Runs the tests for a project.

This task starts the current application, loads up test/test_helper.exs and then requires all files matching the test/**/*_test.exs pattern in parallel.

A list of files can be given after the task name in order to select the files to compile:

mix test test/some/particular/file_test.exs

Command line options

• --color - enables color in the output
• --cover - runs coverage tool. See "Coverage" section below
• --exclude - excludes tests that match the filter
• --failed - runs only tests that failed the last time they ran
• --force - forces compilation regardless of modification times
• --formatter - formatter module
• --include - includes tests that match the filter
• --listen-on-stdin - runs tests, and then listens on stdin. Receiving a newline will result in the tests being run again. Very useful when combined with --stale and external commands which produce output on stdout upon file system modification
• --max-cases - sets the maximum number of tests running async. Only tests from different modules run in parallel. Defaults to twice the number of cores
• --max-failures - the suite stops evaluating tests when this number of test failures is reached. When not given, it always runs all tests
• --no-archives-check - does not check archives
• --no-color - disables color in the output
• --no-compile - does not compile, even if files require compilation
• --no-deps-check - does not check dependencies
• --no-elixir-version-check - does not check the Elixir version from mix.exs
• --no-start - does not start applications after compilation
• --only - runs only tests that match the filter
• --preload-modules - preloads all modules defined in applications
• --raise - raises if the test suite failed
• --seed - seeds the random number generator used to randomize the order of tests; --seed 0 disables randomization
• --slowest - prints timing information for the N slowest tests. Automatically sets --trace and --preload-modules
• --stale - runs only tests which reference modules that changed since the last time tests were ran with --stale. You can read more about this option in the "Stale" section below
• --timeout - sets the timeout for the tests
• --trace - runs tests with detailed reporting. Automatically sets --max-cases to 1. Note that in trace mode test timeouts will be ignored

See ExUnit.configure/1 for more information on configuration options.

Filters

ExUnit provides tags and filtering functionality that allow developers to select which tests to run. The most common functionality is to exclude some particular tests from running by default in your test helper file:

# Exclude all external tests from running
ExUnit.configure exclude: [external: true]

Then, whenever desired, those tests could be included in the run via the --include option:

mix test --include external:true

The example above will run all tests that have the external option set to true. It is also possible to include all examples that have a given tag, regardless of its value:

mix test --include external

Note that all tests are included by default, so unless they are excluded first (either in the test helper or via the --exclude option) the --include option has no effect.

For this reason, Mix also provides an --only option that excludes all tests and includes only the given ones:

mix test --only external

Which is similar to:

mix test --include external --exclude test

It differs in that the test suite will fail if no tests are executed when the --only option is used.

In case a single file is being tested, it is possible to pass a specific line number:

mix test test/some/particular/file_test.exs:12

Which is equivalent to:

mix test --only line:12 test/some/particular/file_test.exs

If the given line starts a describe block, the line filter runs all tests in it. Otherwise, it runs the closest test on or before the given line number.

Note that in the case where a single file contains more than one test module (test case), the line filter applies to every test case before the given line number. Thus, more than one test might be executed for the run.

Configuration

• :test_paths - list of paths containing test files. Defaults to ["test"] if the test directory exists; otherwise, it defaults to []. It is expected that all test paths contain a test_helper.exs file

• :test_pattern - a pattern to load test files. Defaults to *_test.exs

• :warn_test_pattern - a pattern to match potentially misnamed test files and display a warning. Defaults to *_test.ex

• :test_coverage - a set of options to be passed down to the coverage mechanism

Coverage

The :test_coverage configuration accepts the following options:

• :output - the output directory for cover results. Defaults to "cover"
• :tool - the coverage tool
• :summary - summary output configuration; can be either a boolean or a keyword list. When a keyword list is passed, it can specify a :threshold, which is a boolean or numeric value that enables coloring of code coverage results in red or green depending on whether the percentage is below or above the specified threshold, respectively. Defaults to [threshold: 90]

By default, a very simple wrapper around OTP's cover is used as a tool, but it can be overridden as follows:

def project() do
  [
    ...
    test_coverage: [tool: CoverModule]
    ...
  ]
end

CoverModule can be any module that exports start/2, receiving the compilation path and the test_coverage options as arguments. It must return either nil or an anonymous function of zero arity that will be run after the test suite is done.

"Stale"

The --stale command line option attempts to run only those test files which reference modules that have changed since the last time you ran this task with --stale.

The first time this task is run with --stale, all tests are run and a manifest is generated. On subsequent runs, a test file is marked "stale" if any modules it references (and any modules those modules reference, recursively) were modified since the last run with --stale. A test file is also marked "stale" if it has been changed since the last run with --stale.