IREE.Tokenizers.Model.Unigram (iree_tokenizers v0.7.0)

Unigram model specification compatible with IREE.Tokenizers.Tokenizer.init/1.

This model shape is also used internally when SentencePiece Unigram tokenizers are translated into the IREE-backed runtime format.

Summary

options()

Options for Unigram model construction.

empty()

Returns an empty Unigram model specification.

Builds a Unigram model specification from an in-memory scored vocabulary.

@type options() :: [byte_fallback: boolean(), unk_id: integer()]

Options for Unigram model construction.

@spec empty() :: {:ok, IREE.Tokenizers.Model.t()}

Returns an empty Unigram model specification.

@spec init([{String.t(), number()}], options()) :: {:ok, IREE.Tokenizers.Model.t()}

Builds a Unigram model specification from an in-memory scored vocabulary.