@spec clear(t()) :: :ok

Clears the KV cache.

@spec create(
  LlamaCppEx.Model.t(),
  keyword()
) :: {:ok, t()} | {:error, String.t()}

Creates a new inference context for the given model.

Options

• :n_ctx - Context size (max tokens). Defaults to 2048.
• :n_batch - Max tokens per decode batch. Defaults to n_ctx.
• :n_ubatch - Max tokens per micro-batch. Defaults to 512.
• :n_threads - Number of threads for generation. Defaults to system CPU count.
• :n_threads_batch - Number of threads for prompt processing. Defaults to :n_threads.
• :embeddings - Enable embedding extraction. Defaults to false.
• :pooling_type - Pooling type for embeddings. Defaults to :unspecified. Values: :unspecified, :none, :mean, :cls, :last.
• :n_seq_max - Max number of concurrent sequences. Defaults to 1.

@spec decode(t(), [integer()]) :: :ok | {:error, String.t()}

Decodes a list of tokens through the model.

@spec generate(t(), LlamaCppEx.Sampler.t(), [integer()], keyword()) ::
  {:ok, String.t()} | {:error, String.t()}

Runs the generation loop: decodes prompt tokens and generates up to max_tokens new tokens.

Returns the generated text (not including the prompt).

Options

• :max_tokens - Maximum tokens to generate. Defaults to 256.

@spec n_ctx(t()) :: integer()

Returns the context size.

@spec n_seq_max(t()) :: integer()

Returns the max number of sequences.