View Source mix test.coverage (Mix v1.13.4)
Build reports from exported test coverage.
In this moduledoc, we will describe how the default test coverage works in Elixir and also explore how it is capable of export coverage results to group reports from multiple test runs.
Elixir uses Erlang's
for its default test coverage. Erlang coverage is done by tracking
executable lines of code. This implies blank lines, code comments,
function signatures, and patterns are not necessarily executable and
therefore won't be tracked ín coverage reports. Code in macros are
also often executed at compilation time, and therefore may not covered.
Similarly, Elixir AST literals, such as atoms, are not executable either.
Let's see an example:
if some_condition? do do_this() else do_that() end
In the example above, if your tests exercise both
some_condition? == true
some_condition? == false, all branches will be covered, as they all
have executable code. However, the following code
if some_condition? do do_this() else :default end
won't ever mark the
:default branch as covered, as there is no executable
code in the
else branch. Note, however, this issue does not happen on
cond, as Elixir is able to mark the clause operator
-> as executable in
such corner cases:
case some_condition? do true -> do_this() false -> :default end
If the code above is tested with both conditions, you should see entries in both branches marked as covered.
Finally, it is worth discussing that line coverage by itself has its own limitations. For example, take the following code:
do_this() || do_that()
Line coverage is not capable of expressing that both
do_that() have been executed, since as soon as
do_this() is executed,
the whole line is covered. Other techniques, such as branch coverage,
can help spot those cases, but they are not currently supported by the
default coverage tool.
Overall, code coverage can be a great tool for finding out flaws in our code (such as functions that haven't been covered) but it can also lead teams into a false sense of security since 100% coverage never means all different executions flows have been asserted, even with the most advanced coverage techniques. It is up to you and your team to specify how much emphasis you want to place on it.
This task can be used when you need to group the coverage across multiple test runs. Let's see some examples.
Example: aggregating partitioned runs
If you partition your tests across multiple runs, you can unify the report as shown below:
MIX_TEST_PARTITION=1 mix test --partitions 2 --cover MIX_TEST_PARTITION=2 mix test --partitions 2 --cover mix test.coverage
This works because the
automatically exports the coverage results.
Example: aggregating coverage reports from all umbrella children
If you run
mix test.coverage inside an umbrella,
it will automatically gather exported cover results
from all umbrella children - as long as the coverage
results have been exported, like this:
# from the umbrella root mix test --cover --export-coverage default mix test.coverage
Of course, if you want to actually partition the tests, you can also do:
# from the umbrella root MIX_TEST_PARTITION=1 mix test --partitions 2 --cover MIX_TEST_PARTITION=2 mix test --partitions 2 --cover mix test.coverage
On the other hand, if you want partitioned tests but per-app reports, you can do:
# from the umbrella root MIX_TEST_PARTITION=1 mix test --partitions 2 --cover MIX_TEST_PARTITION=2 mix test --partitions 2 --cover mix cmd mix test.coverage
test.coverage from the umbrella root, it
will use the
:test_coverage configuration from the umbrella
Finally, note the coverage itself is not measured across the projects themselves. For example, if project B depends on A, and if there is code in A that is only executed from project B, those lines will not be marked as covered, which is important, as those projects should be developed and tested in isolation.
There may be other scenarios where you want to export coverage.
For example, you may have broken your test suite into two, one
for unit tests and another for integration tests. In such scenarios,
you can explicitly use the
--export-coverage command line option,
:export option under
:test_coverage in your