@spec bert_pre_tokenizer() :: t()

Creates a BertPreTokenizer pre-tokenizer.

Splits for use in BERT models.

@spec byte_level(keyword()) :: t()

Creates a ByteLevel pre-tokenizer.

Splits on whitespaces while remapping all the bytes to a set of visible characters. This technique has been introduced by OpenAI with GPT-2 and has some more or less nice properties:

• Since it maps on bytes, a tokenizer using this only requires 256 characters as initial alphabet (the number of values a byte can have), as opposed to the 130,000+ Unicode characters.

• A consequence of the previous point is that it is absolutely unnecessary to have an unknown token using this since we can represent anything with 256 tokens (Youhou!! 🎉🎉)

• For non ascii characters, it gets completely unreadable, but it works nonetheless!

Options

• :add_prefix_space - whether to add a space to the first word if there isn’t already one. This lets us treat hello exactly like say hello. Defaults to true

• :use_regex - set this to false to prevent this pre-tokenizer from using the GPT2 specific regexp for splitting on whitespace. Defaults to true

@spec byte_level_alphabet() :: charlist()

Gets ByteLevel pre-tokenizer's alphabet.

@spec char_delimiter_split(char()) :: t()

Creates a CharDelimiterSplit pre-tokenizer.

This pre-tokenizer simply splits on the provided delimiter. Works almost like simple split function, except that it accounts for multiple consecutive spaces.

@spec digits(keyword()) :: t()

Creates a Digits pre-tokenizer.

Splits the numbers from any other characters.

Options

• :individual_digits - whether to split individual digits or not. Defaults to false

@spec metaspace(keyword()) :: t()

Creates Metaspace pre-tokenizer.

Splits on whitespaces and replaces them with a special char “▁” (U+2581).

Options

• :replacement - the replacement character to use. Defaults to "▁"

• :prepend_scheme - whether to add a space to the first word if there isn't already one. This lets us treat "hello" exactly like "say hello". Either of :always, :never, :first. :first means the space is only added on the first token (relevant when special tokens are used or other pre_tokenizer are used). Defaults to :always

@spec pre_tokenize(t(), String.t()) :: {:ok, [{String.t(), {integer(), integer()}}]}

Converts a string into a sequence of pre-tokens.

@spec punctuation(split_delimiter_behaviour()) :: t()

Creates a Punctuation pre-tokenizer.

Will isolate all punctuation characters.

@spec sequence([t()]) :: t()

Creates a Sequence pre-tokenizer.

Lets you compose multiple pre-tokenizers that will be run in the given order.

@spec split(String.t(), split_delimiter_behaviour(), keyword()) :: t()

Creates a Split pre-tokenizer using a string as split pattern.

Versatile pre-tokenizer that splits on provided pattern and according to provided behavior.

Options

• :invert - whether to invert the split or not. Defaults to false

@spec split_regex(String.t(), split_delimiter_behaviour(), keyword()) :: t()

Creates a Split pre-tokenizer using a regular expression as split pattern.

Versatile pre-tokenizer that splits on provided regex pattern and according to provided behavior.

The pattern should be a string representing a regular expression according to the Oniguruma Regex Engine.

Options

• :invert - whether to invert the split or not. Defaults to false

Example

iex> Tokenizers.PreTokenizer.split_regex(~S(\?\d{2}\?), :removed)
#Tokenizers.PreTokenizer<[pre_tokenizer_type: "Split"]>

@spec whitespace() :: t()

Creates a Whitespace pre-tokenizer.

Splits on word boundaries. Uses the following regular expression: w+|[^w ]+.

@spec whitespace_split() :: t()

Creates a WhitespaceSplit pre-tokenizer.

Splits on any whitespace character.