# iree_tokenizers v0.7.0 - Table of Contents

Fast Hugging Face tokenizer.json, OpenAI tiktoken, and SentencePiece model bindings for Elixir via the IREE tokenizer runtime

## Modules

- [IREE.Tokenizers](IREE.Tokenizers.md): Fast Hugging Face `tokenizer.json`, OpenAI `.tiktoken`, and SentencePiece
`.model` bindings for Elixir backed by the IREE tokenizer runtime.

- Tokenization
  - [IREE.Tokenizers.DecodeStream](IREE.Tokenizers.DecodeStream.md): Streaming decoder state.
  - [IREE.Tokenizers.EncodeStream](IREE.Tokenizers.EncodeStream.md): Streaming encoder state.
  - [IREE.Tokenizers.Encoding](IREE.Tokenizers.Encoding.md): Result returned by encoding operations.
  - [IREE.Tokenizers.Encoding.Transformation](IREE.Tokenizers.Encoding.Transformation.md): Helpers for building transformation lists passed to
`IREE.Tokenizers.Encoding.transform/2` or the `:encoding_transformations`
option on encode functions.

  - [IREE.Tokenizers.Model](IREE.Tokenizers.Model.md): Pure Elixir model specification used to build IREE-backed tokenizers.
  - [IREE.Tokenizers.Model.BPE](IREE.Tokenizers.Model.BPE.md): BPE model specification compatible with `IREE.Tokenizers.Tokenizer.init/1`.
  - [IREE.Tokenizers.Model.Unigram](IREE.Tokenizers.Model.Unigram.md): Unigram model specification compatible with `IREE.Tokenizers.Tokenizer.init/1`.
  - [IREE.Tokenizers.Model.WordPiece](IREE.Tokenizers.Model.WordPiece.md): WordPiece model specification compatible with `IREE.Tokenizers.Tokenizer.init/1`.

  - [IREE.Tokenizers.Tokenizer](IREE.Tokenizers.Tokenizer.md): Core tokenizer API.

- Other
  - [IREE.Tokenizers.HTTPClient](IREE.Tokenizers.HTTPClient.md): Minimal HTTP client used by `IREE.Tokenizers.Tokenizer.from_pretrained/2`.