IREE.Tokenizers.Model (iree_tokenizers v0.7.0)

Pure Elixir model specification used to build IREE-backed tokenizers.

These structs are not tokenizers by themselves. They are small declarative descriptions that can be passed to IREE.Tokenizers.Tokenizer.init/1.

Currently supported model constructors live in:

Summary

t()

A model specification that can be turned into an IREE tokenizer with IREE.Tokenizers.Tokenizer.init/1.

info(model)

Returns metadata about the model specification.

@type t() :: %IREE.Tokenizers.Model{info: map(), spec: map(), type: binary()}

A model specification that can be turned into an IREE tokenizer with IREE.Tokenizers.Tokenizer.init/1.

@spec info(t()) :: map()

Returns metadata about the model specification.

The exact keys vary by model type, but always include "model_type".