View Source Tokenizers.Model (Tokenizers v0.5.1)

The struct and associated functions for the tokenizer model.

Summary

Types

t()

Represents different kind of models that can be used across the library.

Functions

info(model)

Retrieves information about the model.

save(model, directory, opts \\ [])

Saves the given model in the given directory.

Types

t()

@type t() :: %Tokenizers.Model{resource: reference()}

Represents different kind of models that can be used across the library.

Functions

info(model)

@spec info(t()) :: map()

Retrieves information about the model.

Information retrieved differs per model but all include model_type.

save(model, directory, opts \\ [])

@spec save(t(), String.t(), keyword()) ::
  {:ok, file_paths :: [String.t()]} | {:error, any()}

Saves the given model in the given directory.

This function generates a couple files with predefined names, you can specify :prefix to scope them. Existing files with the same names in this directory will be overridden.

Options

:prefix - the prefix to use for all the files that will get created. Defaults to ""