Bumblebee
v0.4.2

View Source Bumblebee.Tokenizer behaviour (Bumblebee v0.4.2)

An interface for configuring and applying tokenizers.

A tokenizer is used to convert raw text data into model input.

Every module implementing this behaviour is expected to also define a configuration struct.

Summary

Types

input()
special_token_type()

A type corresponding to a special token in the vocabulary.

t()
token()
token_id()

Callbacks

additional_special_tokens(t)

Returns a list with extra special tokens, in addition to the named special_tokens/1.

apply(t, arg2, keyword)

Performs tokenization and encoding on the given input.

decode(t, arg2)

Decodes a list of token ids into a sentence.

id_to_token(t, token_id)

Converts the given token id the corresponding token.

special_tokens(t)

Returns a map with special tokens.

token_to_id(t, token)

Converts the given token into the corresponding numeric id.

Functions

all_special_tokens(tokenizer)

Returns all special tokens, including any extra tokens.

decode(tokenizer, ids)

Decodes a list of token ids into a sentence.

id_to_token(tokenizer, id)

Converts the given token id the corresponding token.

special_token(tokenizer, type)

Returns a special token by name.

special_token_id(tokenizer, type)

Returns id of a special token by name.

token_to_id(tokenizer, token)

Converts the given token into the corresponding numeric id.

Types

Link to this type

input()

View Source 
@type input() :: String.t() | {String.t(), String.t()}
Link to this type

special_token_type()

View Source 
@type special_token_type() :: atom()

A type corresponding to a special token in the vocabulary.

Common types

  • :bos - a token representing the beginning of a sentence

  • :eos - a token representing the end of a sentence

  • :unk - a token representing an out-of-vocabulary token

  • :sep - a token separating two different sentences in the same input

  • :pad - a token added when processing a batch of sequences with different length

  • :cls - a token representing the class of the input

  • :mask - a token representing a masked token, used for masked language modeling tasks

Link to this type

t()

View Source 
@type t() :: struct()
Link to this type

token()

View Source 
@type token() :: String.t()
Link to this type

token_id()

View Source 
@type token_id() :: non_neg_integer()

Callbacks

Link to this callback

additional_special_tokens(t)

View Source 
@callback additional_special_tokens(t()) :: MapSet.t(token())

Returns a list with extra special tokens, in addition to the named special_tokens/1.

Link to this callback

apply(t, arg2, keyword)

View Source 
@callback apply(t(), input() | [input()], keyword()) :: any()

Performs tokenization and encoding on the given input.

Link to this callback

decode(t, arg2)

View Source 
@callback decode(t(), [token_id()] | [[token_id()]]) :: String.t()

Decodes a list of token ids into a sentence.

Link to this callback

id_to_token(t, token_id)

View Source 
@callback id_to_token(t(), token_id()) :: token()

Converts the given token id the corresponding token.

Link to this callback

special_tokens(t)

View Source 
@callback special_tokens(t()) :: %{required(special_token_type()) => token()}

Returns a map with special tokens.

Link to this callback

token_to_id(t, token)

View Source 
@callback token_to_id(t(), token()) :: token_id()

Converts the given token into the corresponding numeric id.

Functions

Link to this function

all_special_tokens(tokenizer)

View Source 
@spec all_special_tokens(t()) :: [token_id()]

Returns all special tokens, including any extra tokens.

Link to this function

decode(tokenizer, ids)

View Source 
@spec decode(
  t(),
  token() | [token_id()] | [[token_id()]] | Nx.Tensor.t()
) :: String.t()

Decodes a list of token ids into a sentence.

Link to this function

id_to_token(tokenizer, id)

View Source

Converts the given token id the corresponding token.

Link to this function

special_token(tokenizer, type)

View Source 
@spec special_token(t(), special_token_type()) :: token() | nil

Returns a special token by name.

Link to this function

special_token_id(tokenizer, type)

View Source 
@spec special_token_id(t(), special_token_type()) :: token_id() | nil

Returns id of a special token by name.

Link to this function

token_to_id(tokenizer, token)

View Source 
@spec token_to_id(t(), token()) :: token_id()
@spec token_to_id(t(), token_id()) :: token()

Converts the given token into the corresponding numeric id.