View Source AI.Tokenizer.Tokens_cl100k_base (fnord v0.5.5)

This module implements the cl100k-base tokenizer model used by OpenAI's text-embedding-3-large model.

Summary

Functions

get_merges()
get_pattern()
get_reverse_vocab()
get_special_tokens()
get_vocab()

Functions

get_merges()

get_pattern()

get_reverse_vocab()

get_special_tokens()

get_vocab()