Behaviour for integrating with 3rd party eval services like Braintrust, LangSmith, etc.

Eval integrations observe LLM calls and scorer results via Telemetry events and can combine them into complete eval records for external services.

Example Implementation

defmodule MyApp.Eval.BraintrustIntegration do
  @behaviour Synaptic.Eval.Integration

  @impl Synaptic.Eval.Integration
  def on_llm_call(_event, measurements, metadata, config) do
    # Log LLM call with tokens, input, output
    usage = Map.get(metadata, :usage, %{})

    Braintrust.log({
      run_id: metadata.run_id,
      step: metadata.step_name,
      input: metadata.input,  # You'd need to capture this separately
      output: metadata.output,  # You'd need to capture this separately
      model: metadata.model,
      prompt_tokens: Map.get(usage, :prompt_tokens, 0),
      completion_tokens: Map.get(usage, :completion_tokens, 0),
      total_tokens: Map.get(usage, :total_tokens, 0),
      duration_ms: System.convert_time_unit(measurements.duration, :native, :millisecond)
    })
  end

  @impl Synaptic.Eval.Integration
  def on_scorer_result(_event, measurements, metadata, config) do
    # Log scorer result
    Braintrust.log_score({
      run_id: metadata.run_id,
      step: metadata.step_name,
      scorer: metadata.scorer,
      score: metadata.score,
      reason: metadata.reason
    })
  end
end

Attaching the Integration

Attach your integration in your application startup (e.g., in application.ex):

defmodule MyApp.Application do
  def start(_type, _args) do
    # ... other setup ...

    Synaptic.Eval.Integration.attach(MyApp.Eval.BraintrustIntegration, %{
      # Your config here
    })

    # ... rest of startup ...
  end
end

Combining LLM Metrics with Scorer Results

To combine LLM metrics with scorer results, you can:

1. Store LLM call data in a process dictionary or ETS table keyed by {run_id, step_name}
2. When scorer results arrive, look up the corresponding LLM call
3. Combine both into a single eval record

See the README for more detailed examples.