mix workspace.test.coverage (Workspace v0.2.1)

View Source

Analyze the test coverage of the workspace.

You can use it to generate test coverage for one or more projects of the workspace. It will take care of finding the coverdata files, fixing the absolute paths with respect to the workspace root and formatting the results

$ mix workspace.test.coverage

Invocation requirements

This task assumes that mix test has been executed with --cover and the :export option under :test_coverage set. It is advised to configure a workspace check that ensures that all projects have these options properly set.

[
  module: Workspace.Checks.ValidateProject,
  description: "all projects must have test coverage export option set",
  opts: [
    validate: fn config ->
      coverage_opts = project.config[:test_coverage] || []
      case coverage_opts[:export] do
        nil -> {:error, "export option not defined under :test_coverage settings"}
        _value -> {:ok, ""}
      end
    end
  ]
]

Test coverage configuration best practices

It is advised to set the test coverage :output for all workspace projects pointing to the same directory with the application name set as prefix. This way you can easily cache it in CI pipelines.

A custom check can be used to ensure it:

[
  module: Workspace.Checks.ValidateProject,
  description: "all projects must have test_coverage[:output] properly set",
  opts: [
    validate: fn project ->
      config = project.config
      coverage_opts = config[:test_coverage] || []
      output = coverage_opts[:output]

      cond do
        is_nil(output) ->
          {:error, ":output option not defined under :test_coverage settings"}

        not String.ends_with?(output, Atom.to_string(config[:app])) ->
          {:error, ":output must point to a folder with the same name as the app name"}

        true ->
          {:ok, ""}
      end
    end
  ]
]

And in your projects mix.exs:

test_coverage: [
  output: "path/to/common/app_name"
]

In order to run the tests with --cover enabled for all workspace projects you should run:

mix workspace.run -t test -- --cover

Coverage thresholds

The task supports two coverage thresholds, the error threshold and the warning threshold.

The error threshold can be configured by setting the :threshold option under :test_coverage on the project's config. If not set it defaults to 90%. If any project has a coverage below the error threshold then the command will fail.

Allowing project failures

When restructuring a large codebase some extracted projects will not have the desired coverage. Instead of setting a very low threshold or adding tests directly, you can explicitely allow these projects to fail. In order to do this you have to set the :allow_failure flag of the :test_coverage workspace settings. For example:

test_coverage: [
  allow_failure: [:project_a, :project_b]
  ...
]

In this case the error messages will be normally printed, but the exit code of the command will not be affected.

Notice that this does not affect the output and exit code mix test --cover if executed directly in the project.

Warning threshold can be configured by setting the :warning_threshold option. If not set it defaults to error_threshold + (100 - error_threshold)/2. All project and module coverages that are below the warning threshold are logged as warnings.

Notice that by default modules that have coverage above the warning threshold are not logged. You can force the logging of all modules by setting the --verbose flag.

Workspace overall threshold

Similar to the individual project's coverage the overall test coverage is also calculated on the workspace level.

In order to specify error and warning thresholds you need to set the corresponding options under the :test_coverage key of the workspace settings, for example:

test_coverage: [
  threhsold: 70,
  warning_threshold: 95
]

Notice that the overall coverage will be reported only for the projects which are enabled on each invocation of the mix workspace.test.coverage task

Exporting coverage

By default the coverage results are not exported but only logged. You can however specify one or more exporters in your workspace :test_coverage config. Each exporter is expected to be a function that accepts as input a workspace and a list of tuples of the form:

{module :: module(), path :: binary(), function_data, line_data}

It is responsible for processing the coverage data and generating a report in any format. Officially we support the following exporters

lcov exporter

Generates an lcov file with both line and function coverage stats

Sample config

test_coverage: [
  exporters: [
    lcov: fn workspace, coverage_stats -> 
      Workspace.Coverage.export_lcov(
        workspace,
        coverage_stats,
        [output_path: "artifacts/coverage"]
      )
    end
  ]
]

Command line options

Workspace status options

Status is retrieved from the diff between the given --base and --head. Knowing the changed files we can limit the execution of workspace commands only to relevant projects.

  • -a, --affected (boolean) - Run only on affected projects [default: false]
  • --base (string) - The base git reference to compare the head to. Applied only when --affected or --modified are set.
  • --head (string) - A reference to the git head. Applied only if --base is set for getting the changed files [default: HEAD]
  • -m, --modified (boolean) - Run only on modified projects [default: false]

Filtering options

  • -e, --exclude... (string) - Ignore the given projects [values can be grouped with the , separator]
  • --exclude-tag... (string) - If set, any projects with any of the given tag(s) will be excluded. For scoped tags you should provide a colon separated string (examples: shared, scope:api, type:utils). For selecting a specific tag use --tag [values can be grouped with the , separator]
  • -p, --project... (string) - The project name, can be defined multiple times. If not set all projects are considered [values can be grouped with the , separator]
  • --tag... (string) - If set, only projects with the given tag(s) will be considered. For scoped tags you should provide a colon separated string (examples: shared, scope:api, type:utils). For excluding a specific tag use --exclude-tag [values can be grouped with the , separator]

Display options

  • --silent (boolean) - If set to true only the package coverage is reported and not the individual modules coverages. This has higher priority than the :verbose option. [default: false]
  • --verbose (boolean) - If set enables verbose logging [default: false]

Global workspace options

  • --config-path (string) - The path to the workspace config to be used, relative to the workspace path [default: .workspace.exs]
  • --workspace-path (string) - If set it specifies the root workspace path, defaults to current directory