# `IREE.Tokenizers`
[🔗](https://github.com/goodhamgupta/iree_tokenizers/blob/v0.7.0/lib/iree/tokenizers.ex#L1)

Fast Hugging Face `tokenizer.json`, OpenAI `.tiktoken`, and SentencePiece
`.model` bindings for Elixir backed by the IREE tokenizer runtime.

The main entrypoint is `IREE.Tokenizers.Tokenizer`.

Supported load formats:

- Hugging Face `tokenizer.json`
- OpenAI `.tiktoken`
- SentencePiece `.model`

Supported runtime capabilities:

- one-shot encode/decode
- batched encode/decode
- streaming encode/decode
- token offsets and type IDs
- vocabulary lookup helpers

The library is intentionally inference-focused. Pair-sequence encoding,
tokenizer training, and full mutation parity with `elixir-nx/tokenizers`
are not yet complete.

---

*Consult [api-reference.md](api-reference.md) for complete listing*