Evaluation primitives for testing agents built on Puck.

Puck.Eval provides minimal building blocks for evaluating LLM agents. These primitives can be composed however you need - with ExUnit, custom runners, or production monitoring.

Core Primitives

• Puck.Eval.Trajectory - Captures what happened during execution
• Puck.Eval.Step - A single LLM call within a trajectory
• Puck.Eval.Collector - Captures trajectory via telemetry
• Puck.Eval.Grader - Behaviour for scoring
• Puck.Eval.Graders - Built-in graders
• Puck.Eval.Result - Aggregates grader results

Quick Example

alias Puck.Eval.{Collector, Graders, Result}

# Capture trajectory from your agent
{output, trajectory} = Collector.collect(fn ->
  MyAgent.run("Find John's email")
end)

# Apply graders
result = Result.from_graders(output, trajectory, [
  Graders.contains("john@example.com"),
  Graders.max_steps(5),
  Graders.output_produced(LookupContact)
])

# Check result
result.passed?  # => true or false

In ExUnit

test "agent finds contact" do
  {output, trajectory} = Puck.Eval.collect(fn ->
    MyAgent.run("Find John's email")
  end)

  assert trajectory.total_steps <= 3
  assert output =~ "john@example.com"
end

In Production Monitoring

def monitor_agent_call(input) do
  {output, trajectory} = Puck.Eval.collect(fn ->
    MyAgent.run(input)
  end)

  :telemetry.execute([:my_app, :agent, :metrics], %{
    tokens: trajectory.total_tokens,
    steps: trajectory.total_steps,
    duration_ms: trajectory.total_duration_ms
  })

  output
end