# `IREE.Tokenizers.EncodeStream`
[🔗](https://github.com/goodhamgupta/iree_tokenizers/blob/v0.7.0/lib/iree/tokenizers/encode_stream.ex#L1)

Streaming encoder state.

Use this when you want to feed a tokenizer incrementally from multiple
binary chunks while preserving the same output you would get from one-shot
encoding of the full input.

# `t`

```elixir
@type t() :: %IREE.Tokenizers.EncodeStream{resource: reference() | {:buffered, pid()}}
```

Mutable streaming encode state owned by the NIF or a local buffered fallback.

# `feed`

```elixir
@spec feed(t(), binary()) :: {:ok, [integer()]} | {:error, {atom(), binary()}}
```

Feeds a binary chunk into the stream and returns any newly produced token IDs.

# `finalize`

```elixir
@spec finalize(t()) :: {:ok, [integer()]} | {:error, {atom(), binary()}}
```

Flushes any remaining state and returns the final token IDs.

# `new`

```elixir
@spec new(
  IREE.Tokenizers.Tokenizer.t(),
  keyword()
) :: {:ok, t()} | {:error, {atom(), binary()}}
```

Creates a new encode stream for the given tokenizer.

Options:

- `:add_special_tokens` - whether post-processing special tokens should be
  emitted during finalization, defaults to `true`
- `:max_chunk_bytes` - maximum chunk size expected by `feed/2`, defaults to
  `65536`

---

*Consult [api-reference.md](api-reference.md) for complete listing*