API Reference iree_tokenizers v#0.7.0

Fast Hugging Face tokenizer.json, OpenAI .tiktoken, and SentencePiece .model bindings for Elixir backed by the IREE tokenizer runtime.

Streaming decoder state.

Streaming encoder state.

Result returned by encoding operations.

Helpers for building transformation lists passed to IREE.Tokenizers.Encoding.transform/2 or the :encoding_transformations option on encode functions.

Minimal HTTP client used by IREE.Tokenizers.Tokenizer.from_pretrained/2.

Pure Elixir model specification used to build IREE-backed tokenizers.

BPE model specification compatible with IREE.Tokenizers.Tokenizer.init/1.

Unigram model specification compatible with IREE.Tokenizers.Tokenizer.init/1.

WordPiece model specification compatible with IREE.Tokenizers.Tokenizer.init/1.

Core tokenizer API.