Tokenizers
v0.4.0

View Source Tokenizers.Encoding.Transformation (Tokenizers v0.4.0)

Module containing handy functions to build the transformations list.

This list is aplied to an encoding using Tokenizers.Encoding.transform/2.

Summary

Types

t()

Functions

pad(target_length, opts \\ [])

Generates the padding transformation.

set_sequence_id(id)

Generates the set_sequence_id transformation.

truncate(max_length, opts \\ [])

Generates the truncation transformation.

Types

Link to this type

t()

View Source 
@type t() :: [
  pad: {non_neg_integer(), Tokenizers.Encoding.padding_opts()},
  truncate: {non_neg_integer(), Tokenizers.Encoding.truncation_opts()},
  set_sequence_id: non_neg_integer()
]

Functions

Link to this function

pad(target_length, opts \\ [])

View Source 
@spec pad(non_neg_integer(), Tokenizers.Encoding.padding_opts()) ::
  {:pad, {non_neg_integer(), Tokenizers.Encoding.padding_opts()}}

Generates the padding transformation.

Check Tokenizers.Encoding.pad/3 for more information.

Link to this function

set_sequence_id(id)

View Source 
@spec set_sequence_id(non_neg_integer()) :: {:set_sequence_id, non_neg_integer()}

Generates the set_sequence_id transformation.

Check Tokenizers.Encoding.set_sequence_id/2 for more information.

Link to this function

truncate(max_length, opts \\ [])

View Source 
@spec truncate(non_neg_integer(), Tokenizers.Encoding.truncation_opts()) ::
  {:truncate, {non_neg_integer(), Tokenizers.Encoding.truncation_opts()}}

Generates the truncation transformation.

Check Tokenizers.Encoding.truncate/3 for more information.