glisp/tokenizer

Values

pub fn tokenize(source: String) -> List(String)

Tokenize a string into a list of tokens.

This function splits the input string into tokens, ensuring that parentheses are treated as separate tokens.

Examples

tokenize("(add 1 2)") // -> ["(", "add", "1", "2", ")"]
Search Document