View Source Benchee.Formatter behaviour (Benchee v1.3.0)

Defines a behaviour for formatters in Benchee, and functions to work with these.

When implementing a Benchee formatter as a behaviour please adopt this behaviour, as it helps with uniformity and also allows at least the .format function of formatters to be run in parallel.

The module itself then has functions to deal with formatters defined in this way allowing for parallel output through output/1 or just output a single formatter through output/3.

Summary

Types

options()

Options given to formatters, entirely defined by formatter authors.

scrubbed_suite()

A suite scrubbed of heavy data.

Callbacks

format(t, options)

Takes a suite and returns a representation write/2 can use.

write(any, options)

Takes the return value of format/1 and then performs some I/O for the user to actually see the formatted data (UI, File IO, HTTP, ...)

Functions

output(suite)

Format and output all configured formatters and formatting functions.

output(suite, formatter, options \\ %{})

Output a suite with a given formatter and options.

Types

options()

@type options() :: any()

Options given to formatters, entirely defined by formatter authors.

scrubbed_suite()

@type scrubbed_suite() :: Benchee.Suite.t()

A suite scrubbed of heavy data.

Type to bring awareness to the fact that format/2 doesn't have access to all data in Benchee.Suite - please read the docs for format/2 to learn more.

Callbacks

format(t, options)

@callback format(Benchee.Suite.t(), options()) :: any()

Takes a suite and returns a representation write/2 can use.

Takes the suite and returns whatever representation the formatter wants to use to output that information. It is important that this function needs to be pure (aka have no side effects) as Benchee will run format/1 functions of multiple formatters in parallel. The result will then be passed to write/2.

Note: Due to memory consumption issues in benchmarks with big inputs, the suite passed to the formatters is missing anything referencing big input data to avoid huge memory consumption and run time. Namely this constitutes:

Benchee.Scenario will have function and input set to nil
Benchee.Configuration will have inputs, but it won't have values only the names, it may be removed in the future please use input_names instead if needed (or input_name of Benchee.Scenario)

Technically speaking this "scrubbing" of Benchee.Suite only occurs when formatters are run in parallel, you still shouldn't rely on those values (and they should not be needed). If you do need them for some reason, please get in touch/open an issue.

write(any, options)

@callback write(any(), options()) :: :ok | {:error, String.t()}

Takes the return value of format/1 and then performs some I/O for the user to actually see the formatted data (UI, File IO, HTTP, ...)

Functions

output(suite)

@spec output(Benchee.Suite.t()) :: Benchee.Suite.t()

Format and output all configured formatters and formatting functions.

Expects a suite that already has been run through all previous functions so has the aggregated statistics etc. that the formatters rely on.

Works by invoking the format/2 and write/2 functions defined in this module. The format/2 functions are actually called in parallel (as they should be pure) - due to potential interference the write/2 functions are called serial.

Also handles pure functions that will then be called with the suite.

You can't rely on the formatters being called in pre determined order.

output(suite, formatter, options \\ %{})

@spec output(Benchee.Suite.t(), module(), options()) :: Benchee.Suite.t()

Output a suite with a given formatter and options.

Replacement for the old MyFormatter.output/1 - calls format/2 and write/2 one after another to create the output defined by the given formatter module. For the given options please refer to the documentation of the formatters you use.