Optimization engine for finding optimal agent configurations.

The optimizer runs evaluation suites with different parameter combinations to find configurations that maximize performance metrics.

Supported Strategies

• :grid_search - Exhaustive search over parameter grid
• :bayesian - Bayesian optimization with TPE (Tree-structured Parzen Estimator)
• :random - Random search over parameter space

Example

# Define parameter space
params = [
  Optimizer.Parameter.float(:temperature, 0.0, 1.0, step: 0.1),
  Optimizer.Parameter.integer(:max_tokens, 100, 1000, step: 100),
  Optimizer.Parameter.choice(:model, [
    "lmstudio:ministral-3-14b-reasoning",
    "lmstudio:qwen-7b"
  ])
]

# Run optimization
{:ok, result} = Optimizer.optimize(suite, params,
  strategy: :grid_search,
  metric: :score,
  maximize: true
)

IO.inspect(result.best_config)
IO.inspect(result.best_score)

Bayesian Optimization

For expensive evaluations, use Bayesian optimization which learns from previous trials to focus on promising regions:

{:ok, result} = Optimizer.optimize(suite, params,
  strategy: :bayesian,
  n_trials: 50,
  metric: :score
)

Metrics

Optimization can target different metrics:

• :score - Aggregate evaluation score (default)
• :pass_rate - Percentage of tests passing
• :latency_p50 - Median latency
• :latency_p95 - 95th percentile latency
• :total_tokens - Token efficiency
• :cost - Estimated cost