Benchee v0.13.1 Benchee.Statistics View Source

Statistics related functionality that is meant to take the raw benchmark run times and then compute statistics like the average and the standard devaition.

Link to this section Summary

Functions

Calculate additional percentiles and add them to the run_time_statistics. Should only be used after statistics/1, to calculate extra values that may be needed for reporting

Calculates statistical data based on a series of run times for a job in microseconds

Sorts the given scenarios fastest to slowest by run_time average

Takes a job suite with job run times, returns a map representing the statistics of the job suite as follows

Link to this section Types

Link to this type mode() View Source
mode() :: [number()] | number() | nil
Link to this type samples() View Source
samples() :: [number()]
Link to this type t() View Source
t() :: %Benchee.Statistics{
  average: float(),
  ips: float() | nil,
  maximum: number(),
  median: number(),
  minimum: number(),
  mode: mode(),
  percentiles: %{optional(number()) => float()},
  sample_size: integer(),
  std_dev: float(),
  std_dev_ips: float() | nil,
  std_dev_ratio: float()
}

Link to this section Functions

Link to this function add_percentiles(suite, percentile_ranks) View Source

Calculate additional percentiles and add them to the run_time_statistics. Should only be used after statistics/1, to calculate extra values that may be needed for reporting.

Examples

iex> scenarios = [ …> %Benchee.Benchmark.Scenario{ …> job_name: “My Job”, …> run_times: [200, 400, 400, 400, 500, 500, 500, 700, 900], …> memory_usages: [200, 400, 400, 400, 500, 500, 500, 700, 900], …> input_name: “Input”, …> input: “Input” …> } …> ] iex> %Benchee.Suite{scenarios: scenarios} …> |> Benchee.Statistics.statistics …> |> Benchee.Statistics.add_percentiles([25, 75]) %Benchee.Suite{ scenarios: [

%Benchee.Benchmark.Scenario{
  job_name: "My Job",
  run_times: [200, 400, 400, 400, 500, 500, 500, 700, 900],
  memory_usages: [200, 400, 400, 400, 500, 500, 500, 700, 900],
  input_name: "Input",
  input: "Input",
  run_time_statistics: %Benchee.Statistics{
    average:       500.0,
    ips:           2000.0,
    std_dev:       200.0,
    std_dev_ratio: 0.4,
    std_dev_ips:   800.0,
    median:        500.0,
    percentiles:   %{25 => 400.0, 50 => 500.0, 75 => 600.0, 99 => 900.0},
    mode:          [500, 400],
    minimum:       200,
    maximum:       900,
    sample_size:   9
  },
  memory_usage_statistics: %Benchee.Statistics{
    average:       500.0,
    ips:           nil,
    std_dev:       200.0,
    std_dev_ratio: 0.4,
    std_dev_ips:   nil,
    median:        500.0,
    percentiles:   %{50 => 500.0, 99 => 900.0},
    mode:          [500, 400],
    minimum:       200,
    maximum:       900,
    sample_size:   9
  }
}

] }

Link to this function job_statistics(measurements) View Source
job_statistics(samples()) :: Benchee.Statistics.t()

Calculates statistical data based on a series of run times for a job in microseconds.

Examples

iex> run_times = [200, 400, 400, 400, 500, 500, 500, 700, 900]
iex> Benchee.Statistics.job_statistics(run_times)
%Benchee.Statistics{
  average:       500.0,
  ips:           nil,
  std_dev:       200.0,
  std_dev_ratio: 0.4,
  std_dev_ips:   nil,
  median:        500.0,
  percentiles:   %{50 => 500.0, 99 => 900.0},
  mode:          [500, 400],
  minimum:       200,
  maximum:       900,
  sample_size:   9
}

iex> Benchee.Statistics.job_statistics([100])
%Benchee.Statistics{
  average:       100.0,
  ips:           nil,
  std_dev:       0,
  std_dev_ratio: 0.0,
  std_dev_ips:   nil,
  median:        100.0,
  percentiles:   %{50 => 100.0, 99 => 100.0},
  mode:          nil,
  minimum:       100,
  maximum:       100,
  sample_size:   1
}

iex> Benchee.Statistics.job_statistics([])
%Benchee.Statistics{
  average:       nil,
  ips:           nil,
  std_dev:       nil,
  std_dev_ratio: nil,
  std_dev_ips:   nil,
  median:        nil,
  percentiles:   nil,
  mode:          nil,
  minimum:       nil,
  maximum:       nil,
  sample_size:   0
}
Link to this function sort(scenarios) View Source
sort([
  %Benchee.Benchmark.Scenario{
    after_each: term(),
    after_scenario: term(),
    before_each: term(),
    before_scenario: term(),
    function: term(),
    input: term(),
    input_name: term(),
    job_name: term(),
    memory_usage_statistics: term(),
    memory_usages: term(),
    name: term(),
    run_time_statistics: term(),
    run_times: term(),
    tag: term()
  }
]) :: [
  %Benchee.Benchmark.Scenario{
    after_each: term(),
    after_scenario: term(),
    before_each: term(),
    before_scenario: term(),
    function: term(),
    input: term(),
    input_name: term(),
    job_name: term(),
    memory_usage_statistics: term(),
    memory_usages: term(),
    name: term(),
    run_time_statistics: term(),
    run_times: term(),
    tag: term()
  }
]

Sorts the given scenarios fastest to slowest by run_time average.

Examples

iex> scenario_1 = %Benchee.Benchmark.Scenario{run_time_statistics: %Statistics{average: 100.0}}
iex> scenario_2 = %Benchee.Benchmark.Scenario{run_time_statistics: %Statistics{average: 200.0}}
iex> scenario_3 = %Benchee.Benchmark.Scenario{run_time_statistics: %Statistics{average: 400.0}}
iex> scenarios = [scenario_2, scenario_3, scenario_1]
iex> Benchee.Statistics.sort(scenarios)
[%Benchee.Benchmark.Scenario{run_time_statistics: %Statistics{average: 100.0}},
 %Benchee.Benchmark.Scenario{run_time_statistics: %Statistics{average: 200.0}},
 %Benchee.Benchmark.Scenario{run_time_statistics: %Statistics{average: 400.0}}]

Takes a job suite with job run times, returns a map representing the statistics of the job suite as follows:

  • average - average run time of the job in μs (the lower the better)
  • ips - iterations per second, how often can the given function be executed within one second (the higher the better)
  • std_dev - standard deviation, a measurement how much results vary (the higher the more the results vary)
  • std_dev_ratio - standard deviation expressed as how much it is relative to the average
  • std_dev_ips - the absolute standard deviation of iterations per second (= ips * std_dev_ratio)
  • median - when all measured times are sorted, this is the middle value (or average of the two middle values when the number of times is even). More stable than the average and somewhat more likely to be a typical value you see.
  • percentiles - a map of percentile ranks. These are the values below which x% of the run times lie. For example, 99% of run times are shorter than the 99th percentile (99th %) rank. is a value for which 99% of the run times are shorter.
  • mode - the run time(s) that occur the most. Often one value, but can be multiple values if they occur the same amount of times. If no value occures at least twice, this value will be nil.
  • minimum - the smallest (fastest) run time measured for the job
  • maximum - the biggest (slowest) run time measured for the job
  • sample_size - the number of run time measurements taken

Parameters

  • suite - the job suite represented as a map after running the measurements, required to have the run_times available under the run_times key

Examples

iex> scenarios = [
...>   %Benchee.Benchmark.Scenario{
...>     job_name: "My Job",
...>     run_times: [200, 400, 400, 400, 500, 500, 500, 700, 900],
...>     memory_usages: [200, 400, 400, 400, 500, 500, 500, 700, 900],
...>     input_name: "Input",
...>     input: "Input"
...>   }
...> ]
iex> suite = %Benchee.Suite{scenarios: scenarios}
iex> Benchee.Statistics.statistics(suite)
%Benchee.Suite{
  scenarios: [
    %Benchee.Benchmark.Scenario{
      job_name: "My Job",
      run_times: [200, 400, 400, 400, 500, 500, 500, 700, 900],
      memory_usages: [200, 400, 400, 400, 500, 500, 500, 700, 900],
      input_name: "Input",
      input: "Input",
      run_time_statistics: %Benchee.Statistics{
        average:       500.0,
        ips:           2000.0,
        std_dev:       200.0,
        std_dev_ratio: 0.4,
        std_dev_ips:   800.0,
        median:        500.0,
        percentiles:   %{50 => 500.0, 99 => 900.0},
        mode:          [500, 400],
        minimum:       200,
        maximum:       900,
        sample_size:   9
      },
      memory_usage_statistics: %Benchee.Statistics{
        average:       500.0,
        ips:           nil,
        std_dev:       200.0,
        std_dev_ratio: 0.4,
        std_dev_ips:   nil,
        median:        500.0,
        percentiles:   %{50 => 500.0, 99 => 900.0},
        mode:          [500, 400],
        minimum:       200,
        maximum:       900,
        sample_size:   9
      }
    }
  ],
  system: nil
}