ExUnitJSON
View SourceAI-friendly JSON test output for ExUnit.
ExUnitJSON provides structured JSON output from mix test for use with AI editors like Claude Code, Cursor, and other tools that benefit from machine-parseable test results.
Features
- Drop-in replacement for
mix testwith JSON output - All test states: passed, failed, skipped, excluded
- Detailed failure information with assertion values and stacktraces
- Filtering options:
--summary-only,--failures-only,--first-failure,--filter-out,--group-by-error,--quiet - File output:
--output results.json - Deterministic test ordering for reproducible output
- No runtime dependencies (uses Elixir 1.18+ built-in
:json)
Installation
Add ex_unit_json to your list of dependencies in mix.exs:
def deps do
[
{:ex_unit_json, "~> 0.1.0", only: [:dev, :test], runtime: false}
]
endConfigure Mix to run test.json in the test environment:
def cli do
[preferred_envs: ["test.json": :test]]
endNote: The cli/0 configuration is required because Mix doesn't inherit preferred_envs from dependencies. Without it, you'll get an error prompting you to add this configuration.
Usage
Basic Usage
# Run all tests with JSON output
mix test.json
# Run specific file
mix test.json test/my_test.exs
# Run specific test by line number
mix test.json test/my_test.exs:42
Options
# Output only the summary (no individual test results)
mix test.json --summary-only
# Output only failed tests
mix test.json --failures-only
# Output only the first failed test (quick iteration)
mix test.json --first-failure
# Mark failures matching pattern as filtered (can repeat)
mix test.json --filter-out "credentials" --filter-out "rate limit"
# Group failures by similar error message
mix test.json --group-by-error
# Suppress Logger output for cleaner JSON
mix test.json --quiet
# Write JSON to a file instead of stdout
mix test.json --output results.json
# Suppress the "use --failed" warning
mix test.json --no-warn
# Combine options
mix test.json --failures-only --output failures.json
All standard mix test options are also supported (file paths, line numbers, etc.).
Iteration Workflow
When previous test failures exist (.mix_test_failures), a helpful tip is shown:
TIP: 3 previous failure(s) exist. Consider:
mix test.json --failed
mix test.json test/unit/ --failed
mix test.json --only integration --failedThis warning is automatically skipped when:
--failedis already used- A specific file or directory is targeted
--onlyor--excludetag filters are used--no-warnflag is passed
Strict Enforcement
For AI-assisted workflows where forgetting --failed wastes time, enable strict enforcement:
# config/test.exs
config :ex_unit_json, enforce_failed: trueThis will exit with an error instead of just warning, forcing the use of --failed or focused runs.
Using with jq
--summary-only pipes cleanly to jq. For full test output, use --output FILE to avoid issues with large output or compilation warnings mixing with JSON:
# Summary - pipes fine
mix test.json --quiet --summary-only | jq '.summary'
# Full test details - use file to avoid parse errors
mix test.json --quiet --output /tmp/results.json
jq '.tests[] | select(.state == "failed")' /tmp/results.json
Output Schema v1
Root Object
{
"version": 1,
"seed": 12345,
"summary": { ... },
"tests": [ ... ],
"error_groups": [ ... ],
"module_failures": [ ... ]
}| Field | Type | Description |
|---|---|---|
version | integer | Schema version (currently 1) |
seed | integer | Random seed used for test ordering |
summary | object | Aggregate test statistics |
tests | array | Individual test results (omitted with --summary-only) |
error_groups | array | Failures grouped by message (only with --group-by-error) |
module_failures | array | setup_all failures (only present when failures occur) |
Summary Object
{
"total": 10,
"passed": 8,
"failed": 1,
"skipped": 1,
"excluded": 0,
"invalid": 0,
"filtered": 0,
"duration_us": 123456,
"result": "failed"
}| Field | Type | Description |
|---|---|---|
total | integer | Total number of tests |
passed | integer | Tests that passed |
failed | integer | Tests that failed |
skipped | integer | Tests skipped with @tag :skip |
excluded | integer | Tests excluded by tag filters |
invalid | integer | Tests with invalid state |
filtered | integer | Failed tests matching --filter-out patterns (only present when non-zero) |
duration_us | integer | Total duration in microseconds |
result | string | "passed" or "failed" |
Test Object
{
"name": "test addition works",
"module": "MyApp.CalculatorTest",
"file": "test/calculator_test.exs",
"line": 10,
"state": "passed",
"duration_us": 1234,
"tags": {},
"failures": []
}| Field | Type | Description |
|---|---|---|
name | string | Test name |
module | string | Test module name |
file | string | Source file path |
line | integer | Line number |
state | string | "passed", "failed", "skipped", or "excluded" |
duration_us | integer | Test duration in microseconds |
tags | object | Test tags (filtered, no internal ExUnit keys) |
failures | array | Failure details (empty for passing tests) |
Failure Object
{
"kind": "assertion",
"message": "Assertion with == failed",
"assertion": {
"expr": "1 == 2",
"left": "1",
"right": "2"
},
"stacktrace": [
{
"file": "test/calculator_test.exs",
"line": 15,
"module": "MyApp.CalculatorTest",
"function": "test addition works",
"arity": 1
}
]
}| Field | Type | Description |
|---|---|---|
kind | string | "assertion", "error", "exit", or "throw" |
message | string | Error message |
assertion | object | Assertion details (only for assertion failures) |
stacktrace | array | Stack frames |
Stacktrace Frame
| Field | Type | Description |
|---|---|---|
file | string | Source file |
line | integer | Line number |
module | string | Module name (optional) |
function | string | Function name (optional) |
arity | integer | Function arity (optional) |
app | string | Application name (optional) |
Error Group Object
When using --group-by-error, failures are grouped by their error message:
{
"pattern": "Connection refused",
"count": 47,
"example": {
"name": "test API call",
"module": "MyApp.APITest",
"file": "test/api_test.exs",
"line": 25
}
}| Field | Type | Description |
|---|---|---|
pattern | string | First line of the error message (truncated at 200 chars) |
count | integer | Number of failures with this error |
example | object | One example test with this failure |
Groups are sorted by count (descending), so the most common errors appear first.
Example Output
Passing Test Suite
{
"version": 1,
"seed": 12345,
"summary": {
"total": 3,
"passed": 3,
"failed": 0,
"skipped": 0,
"excluded": 0,
"invalid": 0,
"duration_us": 5432,
"result": "passed"
},
"tests": [
{
"name": "test addition",
"module": "MyApp.MathTest",
"file": "test/math_test.exs",
"line": 5,
"state": "passed",
"duration_us": 1234,
"tags": {},
"failures": []
}
]
}Failed Test Suite
{
"version": 1,
"seed": 67890,
"summary": {
"total": 2,
"passed": 1,
"failed": 1,
"skipped": 0,
"excluded": 0,
"invalid": 0,
"duration_us": 3456,
"result": "failed"
},
"tests": [
{
"name": "test subtraction",
"module": "MyApp.MathTest",
"file": "test/math_test.exs",
"line": 10,
"state": "failed",
"duration_us": 2000,
"tags": {},
"failures": [
{
"kind": "assertion",
"message": "Assertion with == failed\ncode: 5 - 3 == 3\nleft: 2\nright: 3",
"assertion": {
"expr": "5 - 3 == 3",
"left": "2",
"right": "3"
},
"stacktrace": [
{
"file": "test/math_test.exs",
"line": 12,
"module": "MyApp.MathTest",
"function": "test subtraction",
"arity": 1
}
]
}
]
}
]
}Programmatic Usage
You can also use the formatter directly in your test configuration:
# In test/test_helper.exs
ExUnit.configure(formatters: [ExUnitJSON.Formatter])
ExUnit.start()Or with options:
Application.put_env(:ex_unit_json, :opts,
summary_only: true,
output: "test_results.json"
)
ExUnit.configure(formatters: [ExUnitJSON.Formatter])
ExUnit.start()Requirements
- Elixir 1.18+ (uses built-in
:jsonmodule)
License
MIT License - see LICENSE file for details.