View Source erlperf_job (erlperf v2.2.0)

Job is an instance of a benchmark.

Every job has a corresponding temporary Erlang module generated. Use source/1 to get the source code of the generated module. The structure of this code is an implementation detail and may change between releases.

Job controls how many workers are executing runner code in a tight loop. It does not restart a failing worker, user must ensure proper error handing and reporting. If a worker process crashes, standard CRASH REPORT message is printed to the log (console).

Job accepts a code_map() containing at least a runner function definition.

See callable() for accepted function definitions.

Different callable forms have different performance overhead. Overhead can be measured with erlperf:compare/2:
      erlang
   erlperf:compare([
      #{runner => fun (V) -> rand:mwc59(V) end, init_runner => {rand, mwc59_seed, []}},
      #{runner => "run(V) -> rand:mwc59(V).", init_runner => {rand, mwc59_seed, []}}
   ], #{}).
   [4371541,131460130]

In the example above, callable defined as fun is 30 times slower than the code compiled from the source. The difference is caused by the Erlang Runtime implementation, where indirect calls via fun are considerably more expensive. As a rule of thumb, source code provides the smallest overhead, followed by MFA tuples.

You can mix & match various definition styles. In the example below, init/0 starts an extra pg scope, done/0 stops it, and init_runner/1 takes the total heap size of pg scope controller to pass it to the runner/1.
      erlang
     erlperf_job:start_link(
         #{
             runner => "run(Max) -> rand:uniform(Max).",
             init => {pg, start_link, [scope]},
             init_runner =>
                 fun ({ok, Pid}) ->
                     {total_heap_size, THS} = erlang:process_info(Pid, total_heap_size),
                     THS
                 end,
             done => fun ({ok, Pid}) -> gen_server:stop(Pid) end
         }
     ).
Same example defined with just the source code:
      erlang
   erlperf_job:start_link(
       #{
           runner => "runner(Max) -> rand:uniform(Max).",
           init => "init() -> pg:start_link().",
           init_runner => "init_runner({ok, Pid}) ->
               {total_heap_size, THS} = erlang:process_info(Pid, total_heap_size),
               THS.",
           done => "done({ok, Pid}) -> gen_server:stop(Pid)."
       }
   ).

runner-function

Runner function

Runner function represents code that is run in the tight loop, counting iterations aggregated between all workers. To give an example, benchmarking a function that takes exactly a millisecond to execute, with 2 workers, for 2 seconds, will result in 4000 iterations in total. This would be the value returned by sample/1.

Runner definition can accept zero, one or two arguments.

runner/0 ignores the value returned by init_runner/0,1.

runner/1 accepts the value returned by init_runner/0,1. It is an error to define runner/1 without init_runner/0,1 defined. This example prints "0" in a tight loop, measuring io:format/2 performance:
      erlang
   #{
      runner => "run(Init) -> io:format(\"~b~n\", [Init]).",
      init_runner => "0."
   }
runner/2 adds second argument, accumulator, initially set to the value returned by init_runner/0,1. Subsequent invocations receive value returned by the previous runner invocation. Example:
      erlang
   #{
      runner => "run(Init, Acc) -> io:format(\"~b~n\", [Init + Acc]), Acc + 1.",
      init_runner => "0."
   }
Running this benchmark prints monotonically increasing numbers. This may be useful to test stateful functions, for example, fast Random Number Generators introduced in OTP 25:
      bash
   ./erlperf --init_runner 'rand:mwc59_seed().' 'run(_, Cur) -> rand:mwc59(Cur).'
   Code                                    ||        QPS       Time
   run(_, Cur) -> rand:mwc59(Cur).          1     123 Mi       8 ns

common-test-usage

Common Test usage

Example using erlperf_job directly, as a part of Common Test test case:
      erlang
   benchmark_rand(Config) when is_list(Config) ->
       %% run timer:sleep(1000) for 5 second, 4 runners
       {ok, Job} = erlperf_job:start_link(#{runner => {timer, sleep, [1000]}}),
       Handle = erlperf_job:handle(Job),
       ok = erlperf_job:set_concurrency(Job, 4), %% 4 runner instances
       InitialIterations = erlperf_job:sample(Handle),
       timer:sleep(5000),
       IterationsIn5Sec = erlperf_job:sample(Handle) - InitialIterations,
       erlperf_job:request_stop(Job), %% use gen:stop(Job) for synchronous call
       %% expect at least 16 iterations (and up to 20)
       ?assert(IterationsIn5Sec >= 16, {too_slow, IterationsIn5Sec}),
       ?assert(IterationsIn5Sec =< 20, {too_fast, IterationsIn5Sec}).

Link to this section Summary

Types

Function definition to use as a runner, init, done or init_runner.

Code map contains definitions for

Module, Function, Args accepted by erlang:apply/3.

Functions

Returns the number of concurrently running workers for this job.

Returns the sampling handle for the job.

Run the timed mode benchmark for a job, similar to timer:tc/3.

Requests this job to stop.

Returns the current iteration counter.

Sets the number of concurrently running workers for this job.

Sets job process priority when there are workers running.

Returns the source code that was generated for this job.

Starts the benchmark job.

Starts the job and links it to caller.

Link to this section Types

-type callable() ::
    string() |
    fun() |
    fun((term()) -> term()) |
    fun((term(), term()) -> term()) |
    mfargs() |
    [mfargs()].

Function definition to use as a runner, init, done or init_runner.

  • string(). Erlang code ending with . (period). Example, zero arity: "runner() -> timer:sleep(1).", arity one: "runner(T) -> timer:sleep(T).", arity two: "runner(Init, Acc) -> Acc + Init.". It is allowed to omit the header for zero arity function, so it becomes "timer:sleep(1)."
  • fun() function accepting no arguments, example: fun() -> timer:sleep(1000) end
  • fun(term()) -> term() function accepting one argument, example: fun(Time) -> timer:sleep(Time) end
  • fun(term(), term()) -> term() function accepting two arguments, example: fun() -> timer:sleep(1000) end
  • mfargs() tuple accepted by erlang:apply/3. Example: {rand, uniform, [10]}
  • [mfargs()] list of MFA tuples, example: [{rand, uniform, [10]}]. This functionality is experimental, and only used to replay a recorded calls list. May not be supported in future releases.
-type code_map() ::
    #{runner := callable(), init => callable(), init_runner => callable(), done => callable()}.

Code map contains definitions for:

  • init/0 - called once when starting the job for the first time. The call is made in the context of the job controller. It is guaranteed to run through the entire benchmark job. So if your benchmark needs to create additional resources - ETS tables, or linked processes, like extra pg scopes, - init/0 is a good choice. If init/0 fails, the entire job startup fails
  • init_runner/0,1 - called when the job starts a new worker. init_runner/1 accepts the value returned by init/0. It is an error to omit init/0 if init_runner/1 is defined. It is allowed to have init_runner/0 when init/0 exists. The call to init_runner is made in the context of the worker process, so you can initialise process-local values (e.g. process dictionary)
  • runner/0,1,2 defines the function that will be called in a tight loop. See Runner Function for overview of a runner function variants.
  • done/0,1 - called when the job terminates, to clean up any resources that are not destroyed automatically. done/0 accepts the return of init/0. Call is made in the context of the job controller
-type exec() :: #exec{}.
-opaque handle()
-type mfargs() :: {Module :: module(), Function :: atom(), Args :: [term()]}.
Module, Function, Args accepted by erlang:apply/3.
-type server_ref() :: gen_server:server_ref().
-type state() :: #erlperf_job_state{}.

Link to this section Functions

-spec concurrency(server_ref()) -> Concurrency :: non_neg_integer().

Returns the number of concurrently running workers for this job.

This number may be lower than the amount requested by set_concurrency/2 if workers crash.
-spec handle(server_ref()) -> handle().

Returns the sampling handle for the job.

The returned value is opaque, and is an implementation detail, do not use it in any quality other than passing to sample/1.
Link to this function

measure(JobId, SampleCount)

View Source
-spec measure(server_ref(), SampleCount :: non_neg_integer()) ->
           TimeUs :: non_neg_integer() | already_started.

Run the timed mode benchmark for a job, similar to timer:tc/3.

Executes the runner SampleCount times. Returns time in microseconds. Has less overhead compared to continuous benchmarking, therefore can be used even for very fast functions.
-spec request_stop(server_ref()) -> ok.

Requests this job to stop.

Job is stopped asynchronously. Caller should monitor the job process to find out when the job actually stopped.
-spec sample(Handle :: handle()) -> non_neg_integer() | undefined.

Returns the current iteration counter.

The iteration counter (sample) monotonically grows by 1 every time the runner function is called (without waiting for it to return, so a function that unconditionally crashes still generates a counter of 1).
Link to this function

set_concurrency(JobId, Concurrency)

View Source
-spec set_concurrency(server_ref(), non_neg_integer()) -> ok.

Sets the number of concurrently running workers for this job.

Does not reset counting. May never return if init_runner hangs and does not return control the the job. Concurrency: number of processes to run. It can be higher than the current count (making the job to start more workers), or lower, making the job to stop some.

Workers that crashes are not restarted automatically.
Link to this function

set_priority(JobId, Priority)

View Source
-spec set_priority(server_ref(), erlang:priority_level()) -> erlang:priority_level().

Sets job process priority when there are workers running.

Worker processes may utilise all schedulers, making job process to lose control over starting and stopping workers. By default, job process sets 'high' priority when there are any workers running. Returns the previous setting.

This function must be called before set_concurrency/2, otherwise it has no effect until all workers are stopped, and then restarted.
-spec source(server_ref()) -> [string()].
Returns the source code that was generated for this job.
-spec start(code_map()) -> {ok, pid()} | {error, term()}.

Starts the benchmark job.

Job starts with no workers, use set_concurrency/2 to start workers.
-spec start_link(code_map()) -> {ok, pid()} | {error, term()}.

Starts the job and links it to caller.

Job starts with no workers, use set_concurrency/2 to start workers.