DeepEvalEx.LLM.Adapter behaviour (DeepEvalEx v0.1.0)

Behaviour for LLM adapters.

Implement this behaviour to add support for a new LLM provider. DeepEvalEx uses adapters to abstract away the differences between LLM providers (OpenAI, Anthropic, Ollama, etc.).

Implementing a Custom Adapter

defmodule MyApp.CustomLLMAdapter do
  @behaviour DeepEvalEx.LLM.Adapter

  @impl true
  def generate(prompt, opts) do
    # Call your LLM API
    {:ok, "response text"}
  end

  @impl true
  def generate_with_schema(prompt, schema, opts) do
    # Call your LLM API with structured output
    {:ok, %{key: "value"}}
  end

  @impl true
  def model_name(opts), do: Keyword.get(opts, :model, "custom-model")

  @impl true
  def supports_structured_outputs?, do: true

  @impl true
  def supports_log_probs?, do: false
end

Using a Custom Adapter

DeepEvalEx.evaluate(test_case, [metric],
  adapter: MyApp.CustomLLMAdapter,
  model: "custom-model-v2"
)

Summary

Types

error()

opts()

prompt()

schema()

Callbacks

generate(prompt, opts)

Generates a response from the LLM.

generate_with_schema(prompt, schema, opts)

Generates a structured response matching the given schema.

model_name(opts)

Returns the model name/identifier.

supports_log_probs?()

Returns whether this adapter supports log probabilities.

supports_multimodal?()

Returns whether this adapter supports multimodal inputs (images).

supports_structured_outputs?()

Returns whether this adapter supports structured outputs.

Functions

default_adapter()

Gets the default adapter based on configuration.

generate(prompt, opts \\ [])

Generates a response using the default or specified adapter.

generate_with_schema(prompt, schema, opts \\ [])

Generates a structured response using the default or specified adapter.

get_adapter(module)

Gets the configured adapter module for a provider.

Types

error()

@type error() :: {:error, term()}

opts()

@type opts() :: keyword()

prompt()

@type prompt() :: String.t()

schema()

@type schema() :: module() | map()

Callbacks

generate(prompt, opts)

@callback generate(prompt(), opts()) :: {:ok, String.t()} | error()

Generates a response from the LLM.

Parameters

prompt - The prompt to send to the LLM
opts - Options including:
- :model - Model name/identifier
- :temperature - Sampling temperature (0.0 - 2.0)
- :max_tokens - Maximum tokens in response
- :api_key - API key (if not configured globally)

Returns

{:ok, response} - The generated text response
{:error, reason} - Error tuple

generate_with_schema(prompt, schema, opts)

@callback generate_with_schema(prompt(), schema(), opts()) ::
  {:ok, struct() | map()} | error()

Generates a structured response matching the given schema.

Uses the LLM's native structured output capability (JSON mode, function calling, or tool use) to ensure the response matches the expected schema.

Parameters

prompt - The prompt to send to the LLM
schema - An Ecto schema module or JSON schema map
opts - Same options as generate/2

Returns

{:ok, struct} - Parsed response matching the schema
{:error, reason} - Error tuple

model_name(opts)

@callback model_name(opts()) :: String.t()

Returns the model name/identifier.

supports_log_probs?()

@callback supports_log_probs?() :: boolean()

Returns whether this adapter supports log probabilities.

Log probs can be used for more accurate scoring in some metrics.

supports_multimodal?()

(optional)

@callback supports_multimodal?() :: boolean()

Returns whether this adapter supports multimodal inputs (images).

supports_structured_outputs?()

@callback supports_structured_outputs?() :: boolean()

Returns whether this adapter supports structured outputs.

When true, generate_with_schema/3 uses native structured output features. When false, it falls back to parsing JSON from the response.

Functions

default_adapter()

@spec default_adapter() :: {module(), String.t()}

Gets the default adapter based on configuration.

generate(prompt, opts \\ [])

@spec generate(prompt(), opts()) :: {:ok, String.t()} | error()

Generates a response using the default or specified adapter.

This is a convenience function that resolves the adapter and calls generate.

Options

:adapter - Adapter module or provider atom
:model - Model name
All other options are passed to the adapter

generate_with_schema(prompt, schema, opts \\ [])

@spec generate_with_schema(prompt(), schema(), opts()) :: {:ok, map()} | error()

Generates a structured response using the default or specified adapter.

get_adapter(module)

@spec get_adapter(atom()) :: module()

Gets the configured adapter module for a provider.

Examples

DeepEvalEx.LLM.Adapter.get_adapter(:openai)
#=> DeepEvalEx.LLM.Adapters.OpenAI

DeepEvalEx.LLM.Adapter.get_adapter(:anthropic)
#=> DeepEvalEx.LLM.Adapters.Anthropic