ExUnitJSON

AI-friendly JSON test output for ExUnit.

ExUnitJSON provides structured JSON output from mix test for use with AI editors like Claude Code, Cursor, and other tools that benefit from machine-parseable test results.

Features

• Drop-in replacement for mix test with JSON output
• All test states: passed, failed, skipped, excluded
• Detailed failure information with assertion values and stacktraces
• Filtering options: --summary-only, --failures-only, --first-failure, --filter-out, --group-by-error, --quiet
• File output: --output results.json
• Deterministic test ordering for reproducible output
• No runtime dependencies (uses Elixir 1.18+ built-in :json)

Installation

Add ex_unit_json to your list of dependencies in mix.exs:

def deps do
  [
    {:ex_unit_json, "~> 0.1.0", only: [:dev, :test], runtime: false}
  ]
end

Configure Mix to run test.json in the test environment:

def cli do
  [preferred_envs: ["test.json": :test]]
end

Note: The cli/0 configuration is required because Mix doesn't inherit preferred_envs from dependencies. Without it, you'll get an error prompting you to add this configuration.

Usage

Basic Usage

# Run all tests with JSON output
mix test.json

# Run specific file
mix test.json test/my_test.exs

# Run specific test by line number
mix test.json test/my_test.exs:42

Options

# Output only the summary (no individual test results)
mix test.json --summary-only

# Output only failed tests
mix test.json --failures-only

# Output only the first failed test (quick iteration)
mix test.json --first-failure

# Mark failures matching pattern as filtered (can repeat)
mix test.json --filter-out "credentials" --filter-out "rate limit"

# Group failures by similar error message
mix test.json --group-by-error

# Suppress Logger output for cleaner JSON
mix test.json --quiet

# Write JSON to a file instead of stdout
mix test.json --output results.json

# Suppress the "use --failed" warning
mix test.json --no-warn

# Combine options
mix test.json --failures-only --output failures.json

All standard mix test options are also supported (file paths, line numbers, etc.).

Iteration Workflow

When previous test failures exist (.mix_test_failures), a helpful tip is shown:

TIP: 3 previous failure(s) exist. Consider:
  mix test.json --failed
  mix test.json test/unit/ --failed
  mix test.json --only integration --failed

This warning is automatically skipped when:

• --failed is already used
• A specific file or directory is targeted
• --only or --exclude tag filters are used
• --no-warn flag is passed

Strict Enforcement

For AI-assisted workflows where forgetting --failed wastes time, enable strict enforcement:

# config/test.exs
config :ex_unit_json, enforce_failed: true

This will exit with an error instead of just warning, forcing the use of --failed or focused runs.

Using with jq

--summary-only pipes cleanly to jq. For full test output, use --output FILE to avoid issues with large output or compilation warnings mixing with JSON:

# Summary - pipes fine
mix test.json --quiet --summary-only | jq '.summary'

# Full test details - use file to avoid parse errors
mix test.json --quiet --output /tmp/results.json
jq '.tests[] | select(.state == "failed")' /tmp/results.json

Output Schema v1

Root Object

{
  "version": 1,
  "seed": 12345,
  "summary": { ... },
  "tests": [ ... ],
  "error_groups": [ ... ],
  "module_failures": [ ... ]
}

Summary Object

{
  "total": 10,
  "passed": 8,
  "failed": 1,
  "skipped": 1,
  "excluded": 0,
  "invalid": 0,
  "filtered": 0,
  "duration_us": 123456,
  "result": "failed"
}

Test Object

{
  "name": "test addition works",
  "module": "MyApp.CalculatorTest",
  "file": "test/calculator_test.exs",
  "line": 10,
  "state": "passed",
  "duration_us": 1234,
  "tags": {},
  "failures": []
}

Failure Object

{
  "kind": "assertion",
  "message": "Assertion with == failed",
  "assertion": {
    "expr": "1 == 2",
    "left": "1",
    "right": "2"
  },
  "stacktrace": [
    {
      "file": "test/calculator_test.exs",
      "line": 15,
      "module": "MyApp.CalculatorTest",
      "function": "test addition works",
      "arity": 1
    }
  ]
}

Stacktrace Frame

Error Group Object

When using --group-by-error, failures are grouped by their error message:

{
  "pattern": "Connection refused",
  "count": 47,
  "example": {
    "name": "test API call",
    "module": "MyApp.APITest",
    "file": "test/api_test.exs",
    "line": 25
  }
}

Groups are sorted by count (descending), so the most common errors appear first.

Example Output

Passing Test Suite

{
  "version": 1,
  "seed": 12345,
  "summary": {
    "total": 3,
    "passed": 3,
    "failed": 0,
    "skipped": 0,
    "excluded": 0,
    "invalid": 0,
    "duration_us": 5432,
    "result": "passed"
  },
  "tests": [
    {
      "name": "test addition",
      "module": "MyApp.MathTest",
      "file": "test/math_test.exs",
      "line": 5,
      "state": "passed",
      "duration_us": 1234,
      "tags": {},
      "failures": []
    }
  ]
}

Failed Test Suite

{
  "version": 1,
  "seed": 67890,
  "summary": {
    "total": 2,
    "passed": 1,
    "failed": 1,
    "skipped": 0,
    "excluded": 0,
    "invalid": 0,
    "duration_us": 3456,
    "result": "failed"
  },
  "tests": [
    {
      "name": "test subtraction",
      "module": "MyApp.MathTest",
      "file": "test/math_test.exs",
      "line": 10,
      "state": "failed",
      "duration_us": 2000,
      "tags": {},
      "failures": [
        {
          "kind": "assertion",
          "message": "Assertion with == failed\ncode:  5 - 3 == 3\nleft:  2\nright: 3",
          "assertion": {
            "expr": "5 - 3 == 3",
            "left": "2",
            "right": "3"
          },
          "stacktrace": [
            {
              "file": "test/math_test.exs",
              "line": 12,
              "module": "MyApp.MathTest",
              "function": "test subtraction",
              "arity": 1
            }
          ]
        }
      ]
    }
  ]
}

Programmatic Usage

You can also use the formatter directly in your test configuration:

# In test/test_helper.exs
ExUnit.configure(formatters: [ExUnitJSON.Formatter])
ExUnit.start()

Or with options:

Application.put_env(:ex_unit_json, :opts,
  summary_only: true,
  output: "test_results.json"
)
ExUnit.configure(formatters: [ExUnitJSON.Formatter])
ExUnit.start()

Requirements

• Elixir 1.18+ (uses built-in :json module)

License

MIT License - see LICENSE file for details.