# `LlamaCppEx.Tokenizer`
[🔗](https://github.com/nyo16/llama_cpp_ex/blob/main/lib/llama_cpp_ex/tokenizer.ex#L1)

Text tokenization and detokenization.

# `bos_token`

```elixir
@spec bos_token(LlamaCppEx.Model.t()) :: integer()
```

Returns the BOS (beginning of sentence) token ID.

# `decode`

```elixir
@spec decode(LlamaCppEx.Model.t(), [integer()]) ::
  {:ok, String.t()} | {:error, String.t()}
```

Decodes a list of token IDs back into text.

# `encode`

```elixir
@spec encode(LlamaCppEx.Model.t(), String.t(), keyword()) ::
  {:ok, [integer()]} | {:error, String.t()}
```

Encodes text into a list of token IDs.

## Options

  * `:add_special` - Add special tokens (BOS/EOS). Defaults to `true`.
  * `:parse_special` - Parse special token text (e.g., `<|im_start|>`). Defaults to `true`.

# `eog?`

```elixir
@spec eog?(LlamaCppEx.Model.t(), integer()) :: boolean()
```

Returns whether a token is an end-of-generation token.

# `eos_token`

```elixir
@spec eos_token(LlamaCppEx.Model.t()) :: integer()
```

Returns the EOS (end of sentence) token ID.

# `token_to_piece`

```elixir
@spec token_to_piece(LlamaCppEx.Model.t(), integer()) :: String.t()
```

Converts a single token ID to its text representation.

# `vocab_size`

```elixir
@spec vocab_size(LlamaCppEx.Model.t()) :: integer()
```

Returns the vocabulary size.

---

*Consult [api-reference.md](api-reference.md) for complete listing*