View Source Bumblebee.Tokenizer behaviour (Bumblebee v0.5.3)

An interface for configuring and applying tokenizers.

A tokenizer is used to convert raw text data into model input.

Every module implementing this behaviour is expected to also define a configuration struct.

Summary

Types

input()

special_token_type()

A type corresponding to a special token in the vocabulary.

t()

token()

token_id()

Callbacks

additional_special_tokens(t)

Returns a list with extra special tokens, in addition to the named special_tokens/1.

apply(t, arg2)

Performs tokenization and encoding on the given input.

decode(t, arg2)

Decodes a list of token ids into a sentence.

id_to_token(t, token_id)

Converts the given token id the corresponding token.

special_tokens(t)

Returns a map with special tokens.

token_to_id(t, token)

Converts the given token into the corresponding numeric id.

Functions

all_special_tokens(tokenizer)

Returns all special tokens, including any extra tokens.

decode(tokenizer, ids)

Decodes a list of token ids into a sentence.

id_to_token(tokenizer, id)

Converts the given token id the corresponding token.

special_token(tokenizer, type)

Returns a special token by name.

special_token_id(tokenizer, type)

Returns id of a special token by name.

token_to_id(tokenizer, token)

Converts the given token into the corresponding numeric id.

Types

input()

@type input() :: String.t() | {String.t(), String.t()}

special_token_type()

@type special_token_type() :: atom()

A type corresponding to a special token in the vocabulary.

Common types

:bos - a token representing the beginning of a sentence
:eos - a token representing the end of a sentence
:unk - a token representing an out-of-vocabulary token
:sep - a token separating two different sentences in the same input
:pad - a token added when processing a batch of sequences with different length
:cls - a token representing the class of the input
:mask - a token representing a masked token, used for masked language modeling tasks

t()

@type t() :: struct()

token()

@type token() :: String.t()

token_id()

@type token_id() :: non_neg_integer()

Callbacks

additional_special_tokens(t)

@callback additional_special_tokens(t()) :: MapSet.t(token())

Returns a list with extra special tokens, in addition to the named special_tokens/1.

apply(t, arg2)

@callback apply(t(), input() | [input()]) :: any()

Performs tokenization and encoding on the given input.

decode(t, arg2)

@callback decode(t(), [token_id()] | [[token_id()]]) :: String.t()

Decodes a list of token ids into a sentence.

id_to_token(t, token_id)

@callback id_to_token(t(), token_id()) :: token()

Converts the given token id the corresponding token.

special_tokens(t)

@callback special_tokens(t()) :: %{required(special_token_type()) => token()}

Returns a map with special tokens.

token_to_id(t, token)

@callback token_to_id(t(), token()) :: token_id()

Converts the given token into the corresponding numeric id.

Functions

all_special_tokens(tokenizer)

@spec all_special_tokens(t()) :: [token_id()]

Returns all special tokens, including any extra tokens.

decode(tokenizer, ids)

@spec decode(
  t(),
  token() | [token_id()] | [[token_id()]] | Nx.Tensor.t()
) :: String.t()

Decodes a list of token ids into a sentence.

id_to_token(tokenizer, id)

Converts the given token id the corresponding token.

special_token(tokenizer, type)

@spec special_token(t(), special_token_type()) :: token() | nil

Returns a special token by name.

special_token_id(tokenizer, type)

@spec special_token_id(t(), special_token_type()) :: token_id() | nil

Returns id of a special token by name.

token_to_id(tokenizer, token)

@spec token_to_id(t(), token()) :: token_id()

@spec token_to_id(t(), token_id()) :: token()

Converts the given token into the corresponding numeric id.