mix nous.optimize (nous v0.13.3)

View Source

Run parameter optimization for Nous agent configurations.

Usage

# Optimize with default parameters (temperature, max_tokens)
mix nous.optimize --suite test/eval/suites/basic.yaml

# Use different optimization strategy
mix nous.optimize --suite basic.yaml --strategy bayesian
mix nous.optimize --suite basic.yaml --strategy random
mix nous.optimize --suite basic.yaml --strategy grid_search

# Configure trials and timeout
mix nous.optimize --suite basic.yaml --trials 50 --timeout 1800000

# Optimize specific metric
mix nous.optimize --suite basic.yaml --metric pass_rate
mix nous.optimize --suite basic.yaml --metric latency_p50 --minimize

# Custom parameter file
mix nous.optimize --suite basic.yaml --params params.exs

# Early stopping
mix nous.optimize --suite basic.yaml --early-stop 0.95

# Output results
mix nous.optimize --suite basic.yaml --output results.json

Options

  • --suite - Path to evaluation suite file (required)
  • --strategy - Optimization strategy: grid_search, random, bayesian (default: bayesian)
  • --trials - Number of trials to run (default: 20)
  • --metric - Metric to optimize: score, pass_rate, latency_p50, latency_p95, total_tokens, cost (default: score)
  • --minimize - Minimize metric instead of maximize (useful for latency/cost)
  • --timeout - Total timeout in ms (default: 3600000 = 1 hour)
  • --early-stop - Stop if metric reaches threshold
  • --params - Path to parameters file (Elixir script returning list of parameters)
  • --output - Output file path for JSON results
  • --verbose - Show detailed progress
  • --quiet - Suppress progress output

Parameters File

Create a .exs file that returns a list of parameters:

# params.exs
alias Nous.Eval.Optimizer.Parameter

[
  Parameter.float(:temperature, 0.0, 1.0, step: 0.1),
  Parameter.integer(:max_tokens, 100, 1000, step: 100),
  Parameter.choice(:model, ["gpt-4", "gpt-3.5-turbo"])
]

Default Parameters

If no params file is specified, optimizes these parameters:

  • temperature - Float from 0.0 to 1.0 (step: 0.1)
  • max_tokens - Integer from 256 to 2048 (step: 256)

Example

# Quick optimization with Bayesian search
mix nous.optimize --suite test/eval/suites/basic.yaml --trials 30

# Grid search for small parameter space
mix nous.optimize --suite basic.yaml --strategy grid_search --max-trials 50

# Minimize latency
mix nous.optimize --suite basic.yaml --metric latency_p50 --minimize