View Source AI.Tokenizer.Tokens_o200k_base (fnord v0.5.5)

This module implements the o200k-base tokenizer model used by OpenAI's gpt-4o models.

Summary

Functions

get_merges()
get_pattern()
get_reverse_vocab()
get_special_tokens()
get_vocab()

Functions

get_merges()

get_pattern()

get_reverse_vocab()

get_special_tokens()

get_vocab()