Mix
Mix
v1.11.3

View Source mix test (Mix v1.11.3)

Runs the tests for a project.

This task starts the current application, loads up test/test_helper.exs and then requires all files matching the test/**/*_test.exs pattern in parallel.

A list of files and/or directories can be given after the task name in order to select the files to run:

mix test test/some/particular/file_test.exs
mix test test/some/particular/dir

Tests in umbrella projects can be run from the root by specifying the full suite path, including apps/my_app/test, in which case recursive tests for other child apps will be skipped completely:

# To run all tests for my_app from the umbrella root
mix test apps/my_app/test

# To run a given test file on my_app from the umbrella root
mix test apps/my_app/test/some/particular/file_test.exs

Command line options

  • --color - enables color in the output
  • --cover - runs coverage tool. See "Coverage" section below
  • --exclude - excludes tests that match the filter
  • --export-coverage - the name of the file to export coverage results too. Only has an effect when used with --cover
  • --failed - runs only tests that failed the last time they ran
  • --force - forces compilation regardless of modification times
  • --formatter - sets the formatter module that will print the results. Defaults to ExUnit's built-in CLI formatter
  • --include - includes tests that match the filter
  • --listen-on-stdin - runs tests, and then listens on stdin. Receiving a newline will result in the tests being run again. Very useful when combined with --stale and external commands which produce output on stdout upon file system modifications
  • --max-cases - sets the maximum number of tests running asynchronously. Only tests from different modules run in parallel. Defaults to twice the number of cores
  • --max-failures - the suite stops evaluating tests when this number of test failures is reached. It runs all tests if omitted
  • --no-archives-check - does not check archives
  • --no-color - disables color in the output
  • --no-compile - does not compile, even if files require compilation
  • --no-deps-check - does not check dependencies
  • --no-elixir-version-check - does not check the Elixir version from mix.exs
  • --no-start - does not start applications after compilation
  • --only - runs only tests that match the filter
  • --partitions - sets the amount of partitions to split tests in. This option requires the MIX_TEST_PARTITION environment variable to be set. See the "Operating system process partitioning" section for more information
  • --preload-modules - preloads all modules defined in applications
  • --raise - raises if the test suite failed
  • --seed - seeds the random number generator used to randomize the order of tests; --seed 0 disables randomization
  • --slowest - prints timing information for the N slowest tests. Automatically sets --trace and --preload-modules
  • --stale - runs only tests which reference modules that changed since the last time tests were ran with --stale. You can read more about this option in the "The --stale option" section below
  • --timeout - sets the timeout for the tests
  • --trace - runs tests with detailed reporting. Automatically sets --max-cases to 1. Note that in trace mode test timeouts will be ignored as timeout is set to :infinity

Configuration

These configurations can be set in the def project section of your mix.exs:

  • :test_paths - list of paths containing test files. Defaults to ["test"] if the test directory exists; otherwise, it defaults to []. It is expected that all test paths contain a test_helper.exs file

  • :test_pattern - a pattern to load test files. Defaults to *_test.exs

  • :warn_test_pattern - a pattern to match potentially misnamed test files and display a warning. Defaults to *_test.ex

  • :test_coverage - a set of options to be passed down to the coverage mechanism

Filters

ExUnit provides tags and filtering functionality that allow developers to select which tests to run. The most common functionality is to exclude some particular tests from running by default in your test helper file:

# Exclude all external tests from running
ExUnit.configure(exclude: [external: true])

Then, whenever desired, those tests could be included in the run via the --include option:

mix test --include external:true

The example above will run all tests that have the external option set to true. It is also possible to include all examples that have a given tag, regardless of its value:

mix test --include external

Note that all tests are included by default, so unless they are excluded first (either in the test helper or via the --exclude option) the --include option has no effect.

For this reason, Mix also provides an --only option that excludes all tests and includes only the given ones:

mix test --only external

Which is similar to:

mix test --include external --exclude test

It differs in that the test suite will fail if no tests are executed when the --only option is used.

In case a single file is being tested, it is possible to pass one or more specific line numbers to run only those given tests:

mix test test/some/particular/file_test.exs:12

Which is equivalent to:

mix test --exclude test --include line:12 test/some/particular/file_test.exs

Or:

mix test test/some/particular/file_test.exs:12:24

Which is equivalent to:

mix test --exclude test --include line:12 --include line:24 test/some/particular/file_test.exs

If a given line starts a describe block, that line filter runs all tests in it. Otherwise, it runs the closest test on or before the given line number.

Coverage

The :test_coverage configuration accepts the following options:

  • :output - the output directory for cover results. Defaults to "cover"
  • :tool - the coverage tool
  • :summary - summary output configuration; can be either a boolean or a keyword list. When a keyword list is passed, it can specify a :threshold, which is a boolean or numeric value that enables coloring of code coverage results in red or green depending on whether the percentage is below or above the specified threshold, respectively. Defaults to [threshold: 90]
  • :export - a file name to export results to instead of generating the result on the fly. The .coverdata extension is automatically added to the given file. This option is automatically set via the --export-coverage option or when using process partitioning. See mix test.coverage to compile a report from multiple exports.
  • :ignore_modules - modules to ignore from generating reports and in summaries

By default, a very simple wrapper around OTP's cover is used as a tool, but it can be overridden as follows:

def project() do
  [
    ...
    test_coverage: [tool: CoverModule]
    ...
  ]
end

CoverModule can be any module that exports start/2, receiving the compilation path and the test_coverage options as arguments. It must return either nil or an anonymous function of zero arity that will be run after the test suite is done.

Operating system process partitioning

While ExUnit supports the ability to run tests concurrently within the same Elixir instance, it is not always possible to run all tests concurrently. For example, some tests may rely on global resources.

For this reason, mix test supports partitioning the test files across different Elixir instances. This is done by setting the --partitions option to an integer, with the number of partitions, and setting the MIX_TEST_PARTITION environment variable to control which test partition that particular instance is running. This can also be useful if you want to distribute testing across multiple machines.

For example, to split a test suite into 4 partitions and run them, you would use the following commands:

MIX_TEST_PARTITION=1 mix test --partitions 4
MIX_TEST_PARTITION=2 mix test --partitions 4
MIX_TEST_PARTITION=3 mix test --partitions 4
MIX_TEST_PARTITION=4 mix test --partitions 4

The test files are sorted upfront in a round-robin fashion. Note the partition itself is given as an environment variable so it can be accessed in config files and test scripts. For example, it can be used to setup a different database instance per partition in config/test.exs.

If partitioning is enabled and --cover is used, no cover reports are generated, as they only contain a subset of the coverage data. Instead, the coverage data is exported to files such as cover/MIX_TEST_PARTITION.coverdata. Once you have the results of all partitions inside cover/, you can run mix test.coverage to get the unified report.

The --stale option

The --stale command line option attempts to run only the test files which reference modules that have changed since the last time you ran this task with --stale.

The first time this task is run with --stale, all tests are run and a manifest is generated. On subsequent runs, a test file is marked "stale" if any modules it references (and any modules those modules reference, recursively) were modified since the last run with --stale. A test file is also marked "stale" if it has been changed since the last run with --stale.

The --stale option is extremely useful for software iteration, allowing you to run only the relevant tests as you perform changes to the codebase.