IREE.Tokenizers.DecodeStream (iree_tokenizers v0.6.0)

Streaming decoder state.

Use this when token IDs arrive incrementally and you want to decode them into text while preserving the same result as one-shot decode.

Summary

t()

Mutable streaming decode state owned by the NIF.

Feeds a list of token IDs into the stream and returns any newly produced text.

finalize(stream)

Flushes any remaining decode state and returns the final text chunk.

Creates a new decode stream for the given tokenizer.

@type t() :: %IREE.Tokenizers.DecodeStream{resource: reference()}

Mutable streaming decode state owned by the NIF.

@spec feed(t(), [integer()]) :: {:ok, binary()} | {:error, {atom(), binary()}}

Feeds a list of token IDs into the stream and returns any newly produced text.

@spec finalize(t()) :: {:ok, binary()} | {:error, {atom(), binary()}}

Flushes any remaining decode state and returns the final text chunk.

@spec new(
  IREE.Tokenizers.Tokenizer.t(),
  keyword()
) :: {:ok, t()} | {:error, {atom(), binary()}}

Creates a new decode stream for the given tokenizer.

Options:

:skip_special_tokens - whether to suppress special tokens from output, defaults to true