View Source Benchee.Statistics (Benchee v1.3.0)
Statistics related functionality that is meant to take the raw benchmark data and then compute statistics like the average and the standard deviation etc.
See statistics/1 for a breakdown of the included statistics.
Summary
Types
Careful with the mode, might be multiple values, one value or nothing.😱
The samples a Benchee.Collect collected to compute statistics from.
All the statistics statistics/1 computes from the samples.
Types
Careful with the mode, might be multiple values, one value or nothing.😱
@type samples() :: [number()]
The samples a Benchee.Collect collected to compute statistics from.
@type t() :: %Benchee.Statistics{ absolute_difference: float() | nil, average: float(), ips: float() | nil, maximum: number(), median: number(), minimum: number(), mode: mode(), percentiles: %{required(number()) => float()}, relative_less: float() | nil | :infinity, relative_more: float() | nil | :infinity, sample_size: integer(), std_dev: float(), std_dev_ips: float() | nil, std_dev_ratio: float() }
All the statistics statistics/1 computes from the samples.
Overview of all the statistics Benchee currently provides:
- average - average run time of the job in μs (the lower the better)
- ips - iterations per second, how often can the given function be executed within one second (the higher the better)
- std_dev - standard deviation, a measurement how much results vary (the higher the more the results vary)
- std_dev_ratio - standard deviation expressed as how much it is relative to the average
- std_dev_ips - the absolute standard deviation of iterations per second (= ips * std_dev_ratio)
- median - when all measured times are sorted, this is the middle value (or average of the two middle values when the number of times is even). More stable than the average and somewhat more likely to be a typical value you see.
- percentiles - a map of percentile ranks. These are the values below which x% of the run times lie. For example, 99% of run times are shorter than the 99th percentile (99th %) rank. is a value for which 99% of the run times are shorter.
- mode - the run time(s) that occur the most. Often one value, but can be multiple values if they occur the same amount of times. If no value occurs at least twice, this value will be nil.
- minimum - the smallest sample measured for the scenario
- maximum - the biggest sample measured for the scenario
- relative_more - relative to the reference (usually the fastest scenario) how much more was the average of this scenario. E.g. for reference at 100, this scenario 200 then it is 2.0.
- relative_less - relative to the reference (usually the fastest scenario) how much less was the average of this scenario. E.g. for reference at 100, this scenario 200 then it is 0.5.
- absolute_difference - relative to the reference (usually the fastest scenario) what is the difference of the averages of the scenarios. e.g. for reference at 100, this scenario 200 then it is 100.
- sample_size - the number of run time measurements taken
Functions
Takes a suite with scenarios and their data samples, adds the statistics to the
scenarios. For an overview of what the statistics mean see t/0.
Note that this will also sort the scenarios fastest to slowest to ensure a consistent order of scenarios in all used formatters.
Examples
iex> scenarios = [
...> %Benchee.Scenario{
...> job_name: "My Job",
...> run_time_data: %Benchee.CollectionData{
...> samples: [200, 400, 400, 400, 500, 500, 500, 700, 900]
...> },
...> memory_usage_data: %Benchee.CollectionData{
...> samples: [200, 400, 400, 400, 500, 500, 500, 700, 900]
...> },
...> input_name: "Input",
...> input: "Input"
...> }
...> ]
iex> suite = %Benchee.Suite{scenarios: scenarios}
iex> statistics(suite, Benchee.Test.FakeProgressPrinter)
%Benchee.Suite{
scenarios: [
%Benchee.Scenario{
job_name: "My Job",
input_name: "Input",
input: "Input",
run_time_data: %Benchee.CollectionData{
samples: [200, 400, 400, 400, 500, 500, 500, 700, 900],
statistics: %Benchee.Statistics{
average: 500.0,
ips: 2000_000.0,
std_dev: 200.0,
std_dev_ratio: 0.4,
std_dev_ips: 800_000.0,
median: 500.0,
percentiles: %{50 => 500.0, 99 => 900.0},
mode: [500, 400],
minimum: 200,
maximum: 900,
sample_size: 9
}
},
memory_usage_data: %Benchee.CollectionData{
samples: [200, 400, 400, 400, 500, 500, 500, 700, 900],
statistics: %Benchee.Statistics{
average: 500.0,
ips: nil,
std_dev: 200.0,
std_dev_ratio: 0.4,
std_dev_ips: nil,
median: 500.0,
percentiles: %{50 => 500.0, 99 => 900.0},
mode: [500, 400],
minimum: 200,
maximum: 900,
sample_size: 9
}
}
}
],
system: nil
}