IREE.Tokenizers.Encoding.Transformation (iree_tokenizers v0.7.0)

Copy Markdown View Source

Helpers for building transformation lists passed to IREE.Tokenizers.Encoding.transform/2 or the :encoding_transformations option on encode functions.

Summary

Types

t()

A single encoding transformation.

Functions

pad(target_length, opts \\ [])

Builds a padding transformation.

set_sequence_id(id)

Builds a transformation that replaces all sequence IDs with the given value.

truncate(max_length, opts \\ [])

Builds a truncation transformation.

Types

t()

@type t() ::
  {:pad, {non_neg_integer(), keyword()}}
  | {:truncate, {non_neg_integer(), keyword()}}
  | {:set_sequence_id, non_neg_integer()}

A single encoding transformation.

Functions

pad(target_length, opts \\ [])

@spec pad(
  non_neg_integer(),
  keyword()
) :: t()

Builds a padding transformation.

set_sequence_id(id)

@spec set_sequence_id(non_neg_integer()) :: t()

Builds a transformation that replaces all sequence IDs with the given value.

truncate(max_length, opts \\ [])

@spec truncate(
  non_neg_integer(),
  keyword()
) :: t()

Builds a truncation transformation.