tflite_beam v0.2.1

View Source TFLiteBEAM.Tokenizer.BasicTokenizer (tflite_beam v0.2.1)

Runs basic tokenization such as punctuation spliting, lower casing.

Link to this section Summary

Functions

tokenize(text, is_case_insensitive)

Tokenizes a piece of text.

Link to this section Functions

Link to this function

tokenize(text, is_case_insensitive)

View Source 
@spec tokenize(String.t(), boolean()) :: [String.t()]

Tokenizes a piece of text.