tflite_beam v0.2.1

View Source TFLiteBeam.Tokenizer.FullTokenizer (tflite_beam v0.2.1)

Runs end-to-end tokenization.

Related link: FullTokenizer.swift

Link to this section Summary

Functions

convert_to_id(tokens, vocab)

Convert to ID in the vocab

tokenize(text, is_case_insensitive, vocab)

End-to-end tokenization.

Link to this section Functions

Link to this function

convert_to_id(tokens, vocab)

View Source 
@spec convert_to_id([String.t()], map()) :: {:ok, [integer()]} | {:error, String.t()}

Convert to ID in the vocab

Link to this function

tokenize(text, is_case_insensitive, vocab)

View Source 
@spec tokenize(String.t(), boolean(), map()) :: [String.t()]

End-to-end tokenization.