View Source Tokenizers.Encoding.Transformation (Tokenizers v0.5.1)

Module containing handy functions to build the transformations list.

This list is applied to an encoding using Tokenizers.Encoding.transform/2.

Summary

Functions

Generates the padding transformation.

Generates the set_sequence_id transformation.

Generates the truncation transformation.

Types

Functions

Link to this function

pad(target_length, opts \\ [])

View Source

Generates the padding transformation.

Check Tokenizers.Encoding.pad/3 for more information.

@spec set_sequence_id(non_neg_integer()) :: {:set_sequence_id, non_neg_integer()}

Generates the set_sequence_id transformation.

Check Tokenizers.Encoding.set_sequence_id/2 for more information.

Link to this function

truncate(max_length, opts \\ [])

View Source

Generates the truncation transformation.

Check Tokenizers.Encoding.truncate/3 for more information.