mix test.json (ex_unit_json v0.4.2)

Runs tests and outputs results as JSON.

This task wraps mix test and configures ExUnit to use ExUnitJSON.Formatter for JSON output instead of the default CLI formatter.

Setup

Add this to your project's mix.exs to ensure test.json runs in the test environment:

def cli do
  [preferred_envs: ["test.json": :test]]
end

This is required because Mix doesn't inherit preferred_envs from dependencies.

Usage

mix test.json
mix test.json test/my_test.exs
mix test.json test/my_test.exs:42

Options

All standard mix test options are supported, plus:

• --summary-only - Output only the summary, omit individual test results
• --all - Output all tests (default shows only failures)
• --failures-only - Output only failed tests (DEFAULT - AI-optimized)
• --first-failure - Output only the first failed test (quick iteration)
• --filter-out PATTERN - Mark failures matching pattern as filtered (can repeat)
• --output FILE - Write JSON to file instead of stdout
• --compact - JSONL output with minimal fields (one line per test)
• --group-by-error - Group failures by similar error message
• --quiet - Suppress Logger output and TIP warnings for clean JSON piping
• --no-warn - Suppress the "use --failed" warning when previous failures exist
• --cover - Enable code coverage (off by default for faster runs)
• --cover-threshold N - Fail if overall coverage is below N (0-100). Requires --cover

Coverage

Coverage is disabled by default for faster test runs. Use --cover to enable:

mix test.json --cover

The JSON output includes a coverage key with:

"coverage": {
  "total_percentage": 96.96,
  "total_lines": 330,
  "covered_lines": 320,
  "threshold": 80,
  "threshold_met": true,
  "modules": [
    {
      "module": "MyApp.Module",
      "file": "lib/my_app/module.ex",
      "percentage": 92.68,
      "covered_lines": 38,
      "uncovered_lines": [45, 67, 89]
    }
  ]
}

Default Behavior (v0.3.0+)

By default, mix test.json outputs only failed tests (equivalent to --failures-only). This is optimized for AI agents where passing tests are noise.

Use --all to include all tests in output when needed.

Flag Precedence

When multiple filtering flags are combined, they follow this priority:

1. --summary-only - Highest priority, omits tests array entirely
2. --first-failure - Returns only the first failed test
3. --failures-only / --all - Filter to failures or show all tests

For example, --summary-only --all will omit the tests array.

Iteration Workflow

When previous test failures exist, a tip is shown suggesting to use --failed for faster iteration:

TIP: 3 previous failure(s) exist. Consider:
  mix test.json --failed

This warning is skipped when:

• --failed is already used
• A specific file or directory is targeted
• --only or --exclude tag filters are used
• --no-warn flag is passed

Strict Enforcement

To block full test runs when failures exist (useful for AI-assisted workflows):

# config/test.exs
config :ex_unit_json, enforce_failed: true

This will exit with an error instead of just warning.

Examples

# Run all tests with JSON output
mix test.json

# Run specific file
mix test.json test/my_test.exs

# Output only failures to a file
mix test.json --failures-only --output failures.json