ForthVM.Tokenizer (ForthVM v0.5.0) View Source

Parse a text source

Link to this section Summary

Functions

parse_line(source)

Parse a single line, possibly containing multiple words

split_line(line)

Split line by space delimited words, respecting double quoted strings

tokenize_list(source)

Tokenizes a list of words, returning a list of tokens

tokenize_word(source)

tokenize_word(input, acc)

tokenize_word_to_float(source)

tokenize_word_to_integer(source)

tokenize_word_to_number(source)

Link to this section Functions

parse(source)

parse_line(source)

Parse a single line, possibly containing multiple words

split_line(line)

Split line by space delimited words, respecting double quoted strings

tokenize_list(source)

Tokenizes a list of words, returning a list of tokens

tokenize_word(source)

tokenize_word(input, acc)

tokenize_word_to_float(source)

tokenize_word_to_integer(source)

tokenize_word_to_number(source)