View Source Gpt3Tokenizer (gpt3_tokenizer v0.1.1)

GPT-3 Tokenizer

Link to this section Summary

decode(tokens)

Decode a list of tokens into a string.

encode(text)

Encode a string into a list of tokens.

Count the number of tokens in a string.

Decode a list of tokens into a string.

iex> Gpt3Tokenizer.decode([31373, 995])
"hello world"

Encode a string into a list of tokens.

iex> Gpt3Tokenizer.encode("hello world")
[31373, 995]

Count the number of tokens in a string.

iex> Gpt3Tokenizer.token_count("hello world")
2