mix test.json (ex_unit_json v0.2.5)

View Source

Runs tests and outputs results as JSON.

This task wraps mix test and configures ExUnit to use ExUnitJSON.Formatter for JSON output instead of the default CLI formatter.

Setup

Add this to your project's mix.exs to ensure test.json runs in the test environment:

def cli do
  [preferred_envs: ["test.json": :test]]
end

This is required because Mix doesn't inherit preferred_envs from dependencies.

Usage

mix test.json
mix test.json test/my_test.exs
mix test.json test/my_test.exs:42

Options

All standard mix test options are supported, plus:

  • --summary-only - Output only the summary, omit individual test results
  • --failures-only - Output only failed tests
  • --first-failure - Output only the first failed test (quick iteration)
  • --filter-out PATTERN - Mark failures matching pattern as filtered (can repeat)
  • --output FILE - Write JSON to file instead of stdout
  • --compact - JSONL output with minimal fields (one line per test)
  • --group-by-error - Group failures by similar error message
  • --quiet - Suppress Logger output for cleaner JSON (sets Logger level to :error)
  • --no-warn - Suppress the "use --failed" warning when previous failures exist

Flag Precedence

When multiple filtering flags are combined, they follow this priority:

  1. --summary-only - Highest priority, omits tests array entirely
  2. --first-failure - Returns only the first failed test
  3. --failures-only - Returns all failed tests

For example, --summary-only --failures-only will omit the tests array.

Iteration Workflow

When previous test failures exist, a tip is shown suggesting to use --failed for faster iteration:

TIP: 3 previous failure(s) exist. Consider:
  mix test.json --failed

This warning is skipped when:

  • --failed is already used
  • A specific file or directory is targeted
  • --only or --exclude tag filters are used
  • --no-warn flag is passed

Strict Enforcement

To block full test runs when failures exist (useful for AI-assisted workflows):

# config/test.exs
config :ex_unit_json, enforce_failed: true

This will exit with an error instead of just warning.

Examples

# Run all tests with JSON output
mix test.json

# Run specific file
mix test.json test/my_test.exs

# Output only failures to a file
mix test.json --failures-only --output failures.json