@spec bpe(keyword()) :: t()

Creates a BPE decoder.

Options

• suffix - the suffix to add to the end of each word. Defaults to </w>

@spec byte_fallback() :: t()

Creates a ByteFallback decoder.

@spec byte_level() :: t()

Creates a ByteLevel decoder.

@spec ctc(keyword()) :: t()

Creates a CTC decoder.

Options

• pad_token - the token used for padding. Defaults to <pad>

• word_delimiter_token - the token used for word delimiter. Defaults to |

• cleanup - whether to cleanup tokenization artifacts, defaults to true

@spec decode(t(), [String.t()]) :: {:ok, String.t()} | {:error, any()}

Decodes tokens into string with provided decoder.

@spec fuse() :: t()

Creates a Fuse decoder.

@spec metaspace(keyword()) :: t()

Creates a Metaspace decoder.

Options

• replacement - the replacement character. Defaults to ▁ (as char)

• :prepend_scheme - whether to add a space to the first word if there isn't already one. This lets us treat "hello" exactly like "say hello". Either of :always, :never, :first. :first means the space is only added on the first token (relevant when special tokens are used or other pre_tokenizer are used). Defaults to :always

@spec replace(String.t(), String.t()) :: t()

Creates a Replace decoder.

@spec sequence(decoders :: [t()]) :: t()

Combines a list of decoders into a single sequential decoder.

@spec strip(char(), non_neg_integer(), non_neg_integer()) :: t()

Creates a Strip decoder.

It expects a character and the number of times to strip the character on left and right sides.

@spec word_piece(keyword()) :: t()

Creates a WordPiece decoder.

Options

• prefix - The prefix to use for subwords. Defaults to ##

• cleanup - Whether to cleanup tokenization artifacts. Defaults to true