Jido.AI.Tokenizer (Jido AI v0.5.2)

A simple tokenizer for AI models that provides basic encode/decode functionality. This is a placeholder implementation that should be replaced with actual tokenization logic.

Summary

Functions

decode(tokens, model)

Decodes tokens back into a string for the given model.

encode(input, model)

Encodes a string into tokens for the given model.

Functions

decode(tokens, model)

@spec decode([integer()], String.t()) :: String.t()

Decodes tokens back into a string for the given model.

encode(input, model)

@spec encode(String.t(), String.t()) :: [integer()]

Encodes a string into tokens for the given model.