# Tiktokenex v0.1.0 - Table of Contents

> Pure Elixir BPE tokenizer compatible with OpenAI's tiktoken. Supports cl100k_base (GPT-4, GPT-3.5) and o200k_base (GPT-4o) encodings. No NIFs, no Python, no external dependencies.

## Pages

- [Tiktokenex](readme.md)
- [LICENSE](license.md)

## Modules

- [Tiktokenex](Tiktokenex.md): Pure Elixir BPE tokenizer compatible with OpenAI's tiktoken.
- [Tiktokenex.BPE](Tiktokenex.BPE.md): Core Byte-Pair Encoding merge algorithm.
- [Tiktokenex.Pretokenizer](Tiktokenex.Pretokenizer.md): Regex-based pre-tokenization that splits text into chunks before BPE.
- [Tiktokenex.Ranks](Tiktokenex.Ranks.md): Loads and caches tiktoken rank files from priv/ranks/.