# `IREE.Tokenizers.Model.Unigram`
[🔗](https://github.com/goodhamgupta/iree_tokenizers/blob/v0.7.0/lib/iree/tokenizers/model/unigram.ex#L1)

Unigram model specification compatible with `IREE.Tokenizers.Tokenizer.init/1`.

This model shape is also used internally when SentencePiece Unigram
tokenizers are translated into the IREE-backed runtime format.

# `options`

```elixir
@type options() :: [byte_fallback: boolean(), unk_id: integer()]
```

Options for Unigram model construction.

# `empty`

```elixir
@spec empty() :: {:ok, IREE.Tokenizers.Model.t()}
```

Returns an empty Unigram model specification.

# `init`

```elixir
@spec init([{String.t(), number()}], options()) :: {:ok, IREE.Tokenizers.Model.t()}
```

Builds a Unigram model specification from an in-memory scored vocabulary.

---

*Consult [api-reference.md](api-reference.md) for complete listing*