View Source Tiktoken.Encoding behaviour (tiktoken_betafi v0.1.0)

Link to this section Summary

Link to this section Callbacks

@callback decode([integer()]) :: {:ok, String.t()} | {:error, String.t()}
@callback encode(String.t(), [binary()]) :: {:ok, [integer()]} | {:error, String.t()}
@callback encode_ordinary(String.t()) :: {:ok, [integer()]} | {:error, String.t()}
Link to this callback

encode_with_special_tokens(t)

View Source
@callback encode_with_special_tokens(String.t()) ::
  {:ok, [integer()]} | {:error, String.t()}