AGENT.md - AI Editor Guide for ExUnitJSON

This file is for AI assistants (Claude Code, Cursor, Copilot, etc.) to understand how to use ExUnitJSON effectively.

Start Here (Default Workflow)

Most common pattern - fast iteration on failures:

# First run or after code changes
mix test.json --quiet --summary-only

# Iterating on failures (ALWAYS use --failed for speed)
mix test.json --quiet --failed --first-failure

Automatic reminders: If you forget --failed when failures exist, you'll see:

TIP: 3 previous failure(s) exist. Consider:
  mix test.json --failed
  mix test.json test/unit/ --failed
  mix test.json --only integration --failed

This warning is automatic - no flag needed. It's skipped when you're already being focused (using --failed, targeting files/dirs, or using tag filters).

When NOT to use --failed:

• After changing test infrastructure, fixtures, or shared setup code
• After adding new test files (new tests won't be in .mix_test_failures)
• When you want to verify a full green suite

When to Use

Use mix test.json instead of mix test when:

• Running tests programmatically
• Analyzing test failures
• Iterating on fixes
• You need structured failure data

Installation Reminder

Consuming projects must add to their mix.exs:

def cli do
  [preferred_envs: ["test.json": :test]]
end

Without this, you'll get: "mix test" is running in the "dev" environment

Quick Reference

Fast iteration on failures

mix test.json --failed --quiet --first-failure

Returns only the first failure with clean JSON. Fix it, repeat.

Overview of test health

mix test.json --quiet --summary-only

Returns just counts: total, passed, failed, skipped.

Analyze failure patterns

mix test.json --failed --quiet --group-by-error --summary-only

Groups failures by error message. Shows which errors are most common.

Filter known issues

mix test.json --quiet --group-by-error --filter-out "credentials" --filter-out "rate limit"

Excludes failures matching patterns. "filtered" count shows how many were excluded.

Full failure details

mix test.json --failed --quiet --failures-only

Returns all failed tests with full assertion data and stacktraces.

Key Flags

Output Structure

{
  "summary": {
    "total": 100,
    "passed": 80,
    "failed": 20,
    "filtered": 15,
    "result": "failed"
  },
  "error_groups": [
    {
      "pattern": "Connection refused",
      "count": 10,
      "example": {"file": "...", "line": 42, "name": "..."}
    }
  ],
  "tests": [...]
}

Notes:

• filtered only appears with --filter-out
• error_groups only appears with --group-by-error
• tests is omitted with --summary-only

Recommended Workflows

1. Initial assessment

mix test.json --quiet --group-by-error --summary-only

See how many failures and what patterns exist.

2. Filter noise, see real issues

mix test.json --quiet --group-by-error --filter-out "credentials" --summary-only

Remove expected failures (missing credentials, rate limits, etc.)

3. Fix one at a time

mix test.json --failed --quiet --first-failure

Get the first failure, fix it, repeat until green.

4. Verify fix

mix test.json --failed --quiet --summary-only

Quick check if failure count decreased.

Tips

• Always use --quiet - Logger output pollutes JSON
• Use --failed for iteration - Much faster than running all tests
• --group-by-error reveals patterns - 50 "connection refused" errors = 1 root cause
• --filter-out is repeatable - Add multiple patterns to exclude

Strict Enforcement

For projects where forgetting --failed is particularly costly, enable strict mode:

# config/test.exs
config :ex_unit_json, enforce_failed: true

This will block full test runs when failures exist, requiring --failed or focused runs.

Exit Codes

Note: Exit code 2 may trigger shell error display. Use 2>&1 to capture both streams.

Using jq

--summary-only pipes cleanly. For full test output, use --output FILE to avoid issues with large output or compilation warnings:

# Summary - pipes fine
mix test.json --quiet --summary-only | jq '.summary'
mix test.json --quiet --group-by-error --summary-only | jq '.error_groups | map({pattern, count})'
mix test.json --quiet --group-by-error --summary-only | jq '.error_groups[:5]'

# Full test details - use file
mix test.json --quiet --output /tmp/results.json
jq '.tests[] | select(.state == "failed")' /tmp/results.json
jq '.tests[].file' /tmp/results.json | sort -u
jq '.tests | group_by(.file) | map({file: .[0].file, count: length})' /tmp/results.json

Handling Large Output

For large test suites, output may exceed context limits:

1. Quick health check: Use --summary-only
2. Write to file: Use --output /tmp/results.json and read selectively with jq
3. Reduce noise first: Use --filter-out patterns before requesting full output

# Example: selective reading from file
mix test.json --quiet --output /tmp/results.json
jq '.error_groups[:5]' /tmp/results.json  # First 5 groups
jq '.tests | length' /tmp/results.json    # Just count

Combining with ExUnit Flags

ExUnit flags work alongside ex_unit_json flags:

# Run only integration tests, show failures
mix test.json --only integration --quiet --failures-only

# Run specific file with JSON output
mix test.json test/my_test.exs --quiet --summary-only

# Seed for reproducibility
mix test.json --seed 12345 --quiet --failures-only

Troubleshooting

jq parse errors

If you get jq: parse error, compilation warnings or other output may be mixing with JSON. See "Using jq" section above - use --output FILE for full test output.

Capturing both streams

mix test.json --quiet --summary-only 2>&1