Tokenizers
v0.5.0

View Source Tokenizers.Model (Tokenizers v0.5.0)

The struct and associated functions for the tokenizer model.

Summary

Types

t()

Represents different kind of models that can be used across the library.

Functions

info(model)

Retrieves information about the model.

save(model, directory, opts \\ [])

Saves the given model in the given directory.

Types

Link to this type

t()

View Source 
@type t() :: %Tokenizers.Model{resource: reference()}

Represents different kind of models that can be used across the library.

Functions

Link to this function

info(model)

View Source 
@spec info(t()) :: map()

Retrieves information about the model.

Information retrieved differs per model but all include model_type.

Link to this function

save(model, directory, opts \\ [])

View Source 
@spec save(t(), String.t(), keyword()) ::
  {:ok, file_paths :: [String.t()]} | {:error, any()}

Saves the given model in the given directory.

This function generates a couple files with predefined names, you can specify :prefix to scope them. Existing files with the same names in this directory will be overridden.

Options

  • :prefix - the prefix to use for all the files that will get created. Defaults to ""