Nous.Eval.Evaluators.Schema (nous v0.9.0)

View Source

Evaluator that validates structured output against an Ecto schema.

Expected Format

The expected value should be an Ecto schema module:

TestCase.new(
  id: "structured",
  input: "Generate a user with name Alice",
  expected: MyApp.User,
  eval_type: :schema
)

Or a map with the schema and optional field checks:

TestCase.new(
  id: "structured",
  input: "Generate a user",
  expected: %{
    schema: MyApp.User,
    required_fields: [:name, :email],
    field_values: %{name: "Alice"}
  },
  eval_type: :schema
)

Configuration

  • :allow_extra_fields - Allow fields not in schema (default: true)
  • :validate_changeset - Run changeset validations (default: true)