View Source mix test.coverage (Mix v1.15.4)

Build reports from exported test coverage.

In this moduledoc, we will describe how the default test coverage works in Elixir and also explore how it is capable of exporting coverage results to group reports from multiple test runs.

Line coverage

Elixir uses Erlang's :cover for its default test coverage. Erlang coverage is done by tracking executable lines of code. This implies blank lines, code comments, function signatures, and patterns are not necessarily executable and therefore won't be tracked in coverage reports. Code in macros are also often executed at compilation time, and therefore may not be covered. Similarly, Elixir AST literals, such as atoms, are not executable either.

Let's see an example:

if some_condition? do
  do_this()
else
  do_that()
end

In the example above, if your tests exercise both some_condition? == true and some_condition? == false, all branches will be covered, as they all have executable code. However, the following code

if some_condition? do
  do_this()
else
  :default
end

won't ever mark the :default branch as covered, as there is no executable code in the else branch. Note, however, this issue does not happen on case or cond, as Elixir is able to mark the clause operator -> as executable in such corner cases:

case some_condition? do
  true ->
    do_this()

  false ->
    :default
end

If the code above is tested with both conditions, you should see entries in both branches marked as covered.

Finally, it is worth discussing that line coverage by itself has its own limitations. For example, take the following code:

do_this() || do_that()

Line coverage is not capable of expressing that both do_this() and do_that() have been executed, since as soon as do_this() is executed, the whole line is covered. Other techniques, such as branch coverage, can help spot those cases, but they are not currently supported by the default coverage tool.

Overall, code coverage can be a great tool for finding flaws in our code (such as functions that haven't been covered) but it can also lead teams into a false sense of security since 100% coverage never means all different executions flows have been asserted, even with the most advanced coverage techniques. It is up to you and your team to specify how much emphasis you want to place on it.

Exporting coverage

This task can be used when you need to group the coverage across multiple test runs. Let's see some examples.

Example: aggregating partitioned runs

If you partition your tests across multiple runs, you can unify the report as shown below:

$ MIX_TEST_PARTITION=1 mix test --partitions 2 --cover
$ MIX_TEST_PARTITION=2 mix test --partitions 2 --cover
$ mix test.coverage

This works because the --partitions option automatically exports the coverage results.

Example: aggregating coverage reports from all umbrella children

If you run mix test.coverage inside an umbrella, it will automatically gather exported cover results from all umbrella children - as long as the coverage results have been exported, like this:

# from the umbrella root
$ mix test --cover --export-coverage default
$ mix test.coverage

Of course, if you want to actually partition the tests, you can also do:

# from the umbrella root
$ MIX_TEST_PARTITION=1 mix test --partitions 2 --cover
$ MIX_TEST_PARTITION=2 mix test --partitions 2 --cover
$ mix test.coverage

On the other hand, if you want partitioned tests but per-app reports, you can do:

# from the umbrella root
$ MIX_TEST_PARTITION=1 mix test --partitions 2 --cover
$ MIX_TEST_PARTITION=2 mix test --partitions 2 --cover
$ mix cmd mix test.coverage

When running test.coverage from the umbrella root, it will use the :test_coverage configuration from the umbrella root.

Finally, note the coverage itself is not measured across the projects themselves. For example, if project B depends on A, and if there is code in A that is only executed from project B, those lines will not be marked as covered, which is important, as those projects should be developed and tested in isolation.

Other scenarios