View Source TFLiteElixir.Tokenizer.BasicTokenizer (tflite_elixir v0.3.7)

Runs basic tokenization such as punctuation spliting, lower casing.

Summary

Functions

tokenize(text, is_case_insensitive)

Tokenizes a piece of text.

Functions

Link to this function

tokenize(text, is_case_insensitive)

View Source 
@spec tokenize(String.t(), boolean()) :: [String.t()]

Tokenizes a piece of text.