# Tiktokenex v0.1.0 - API Reference

## Modules

- [Tiktokenex](Tiktokenex.md): Pure Elixir BPE tokenizer compatible with OpenAI's tiktoken.
- [Tiktokenex.BPE](Tiktokenex.BPE.md): Core Byte-Pair Encoding merge algorithm.
- [Tiktokenex.Pretokenizer](Tiktokenex.Pretokenizer.md): Regex-based pre-tokenization that splits text into chunks before BPE.
- [Tiktokenex.Ranks](Tiktokenex.Ranks.md): Loads and caches tiktoken rank files from priv/ranks/.