caffeine_lang/frontend/tokenizer
Values
pub fn tokenize(
source: String,
) -> Result(List(token.Token), tokenizer_error.TokenizerError)Tokenizes source text into a list of tokens.
pub fn tokenize(
source: String,
) -> Result(List(token.Token), tokenizer_error.TokenizerError)Tokenizes source text into a list of tokens.