IREE.Tokenizers.Model.WordPiece (iree_tokenizers v0.7.0)

WordPiece model specification compatible with IREE.Tokenizers.Tokenizer.init/1.

Summary

options()

Options for WordPiece model construction.

empty()

Returns an empty WordPiece model specification.

Builds a WordPiece model specification from a newline-delimited vocabulary file.

Builds a WordPiece model specification from an in-memory vocabulary.

@type options() :: [
  unk_token: String.t(),
  max_input_chars_per_word: number(),
  continuing_subword_prefix: String.t()
]

Options for WordPiece model construction.

@spec empty() :: {:ok, IREE.Tokenizers.Model.t()}

Returns an empty WordPiece model specification.

@spec from_file(String.t(), options()) ::
  {:ok, IREE.Tokenizers.Model.t()} | {:error, term()}

Builds a WordPiece model specification from a newline-delimited vocabulary file.

@spec init(%{required(String.t()) => integer()}, options()) ::
  {:ok, IREE.Tokenizers.Model.t()}

Builds a WordPiece model specification from an in-memory vocabulary.