Nous.Eval.Evaluators.Contains (nous v0.9.0)

View Source

Evaluator that checks if output contains expected substrings or patterns.

Expected Format

The expected value can be:

  • A list of strings: ["word1", "word2"]
  • A map with :contains key: %{contains: ["word1", "word2"]}
  • A map with :contains_any key: %{contains_any: ["word1", "word2"]}
  • A map with :regex key: %{regex: ["pattern1", "pattern2"]}

Configuration

  • :case_insensitive - Ignore case (default: true)
  • :match_all - Require all items to match (default: true for :contains)

Examples

# Must contain all words
TestCase.new(
  id: "contains_all",
  input: "What's the weather in Tokyo?",
  expected: %{contains: ["weather", "Tokyo"]},
  eval_type: :contains
)

# Must contain any word
TestCase.new(
  id: "contains_any",
  input: "Say hello",
  expected: %{contains_any: ["hello", "hi", "hey"]},
  eval_type: :contains
)

# Regex patterns
TestCase.new(
  id: "regex",
  input: "What is 25*4?",
  expected: %{regex: ["\\d+"]},
  eval_type: :contains
)