gen_metrics_bench v0.1.0 GenMetricsBench

A GenMetrics benchmarking tool for GenServer and GenStage applications.

GenMetrics supports the collection and publication of GenServer and GenStage runtime metrics. Metrics data are generated by an introspection agent. This means that no instrumentation is required within the GenServer or GenStage library or within your application source code.

This library helps us to test and measure what, if any, overhead is introduced into the runtime environment by the GenMetrics introspection agent.

To provide an extensible benchmarking tool this library makes it simple for anyone to plug-in new runtime behaviours for any benchmark test. These behaviours include:

The type and size of the message passing through the GenServer cluster or GenStage pipeline.
The simulated time spent processing those messages within a GenServer call, cast or info callback or within a GenStage :producer_consumer handle_events/3 callback.

Benchmarking GenServer Clusters

The GenMetricsBench.Cluster module supports the following set of benchmarks using a dedicated GenServer cluster harness:

The default benchmark behaviours for GenServer cluster benchmarking are provided by the following module:

GenMetricsBench.Simulator.ClusterDefault

Simply clone this module then customize the return values on the callbacks defined by the GenMetricsBench.Simulator behaviour. Then register your cloned module in config/config.exs and run your custom benchmarks.

Sample Benchmarks: GenServer Clusters

Each of the following GenServer benchmarks processed 10,000 [ten thousand] messages and the time taken to complete each benchmark is indicated below:

# GenMetrics Disabled
iex(1)> GenMetricsBench.Cluster.no_metrics
GenServer Cluster: benchmark starting, sim_load=10000
:ok
GenServer Cluster: handle_info, sim_load=10000 completed in 20024 ms

# GenMetrics Summary Metrics Enabled
iex(5)> GenMetricsBench.Cluster.summary_metrics
GenServer Cluster: benchmark starting, sim_load=10000
:ok
GenServer Cluster: handle_info, sim_load=10000 completed in 20004 ms

# GenMetrics In-Memory Statistical Metrics Enabled
iex(6)> GenMetricsBench.Cluster.statistical_metrics
GenServer Cluster: benchmark starting, sim_load=10000
:ok
GenServer Cluster: handle_info, sim_load=10000 completed in 20011 ms

# GenMetrics Statsd Statistical Metrics Enabled
iex(3)> GenMetricsBench.Cluster.statsd_metrics 
GenServer Cluster: benchmark starting, sim_load=10000
:ok
GenServer Cluster: handle_info, sim_load=10000 completed in 20019 ms

# GenMetrics Datadog Statistical Metrics Enabled
iex(2)> GenMetricsBench.Cluster.datadog_metrics
GenServer Cluster: benchmark starting, sim_load=10000
:ok
GenServer Cluster: handle_info, sim_load=10000 completed in 20009 ms

Note, the default benchmark behaviours provided by GenMetricsBench.Simulator.ClusterDefault were used during these sample benchmarks.

Benchmarking GenStage Pipelines

The GenMetricsBench.Pipeline module supports the following set of benchmarks using a dedicated GenStage pipeline harness:

The default benchmark behaviours for GenStage pipeline benchmarking are provided by the following module:

GenMetricsBench.Simulator.PipelineDefault

Sample Benchmarks: GenStage Pipelines

Each of the following GenStage benchmarks processed 10,000,000 [ten million] messages and the time taken to complete each benchmark is indicated below:

# GenMetrics Disabled
iex(1)> GenMetricsBench.Pipeline.no_metrics
GenStage Pipeline: benchmark starting, sim_load=10000000
GenStage Pipeline: sim_load=10000000 completed in 40035 ms
** (EXIT from #PID<0.199.0>) :benchmark_completed

# GenMetrics Summary Metrics Enabled
iex(1)> GenMetricsBench.Pipeline.summary_metrics
GenStage Pipeline: benchmark starting, sim_load=10000000
GenStage Pipeline: sim_load=10000000 completed in 40069 ms
** (EXIT from #PID<0.204.0>) :benchmark_completed

# GenMetrics In-Memory Statistical Metrics Enabled
iex(1)> GenMetricsBench.Pipeline.statistical_metrics
GenStage Pipeline: benchmark starting, sim_load=10000000
GenStage Pipeline: sim_load=10000000 completed in 40034 ms
** (EXIT from #PID<0.210.0>) :benchmark_completed

# GenMetrics Statsd Statistical Metrics Enabled
iex(1)> GenMetricsBench.Pipeline.statsd_metrics
GenStage Pipeline: benchmark starting, sim_load=10000000
GenStage Pipeline: sim_load=10000000 completed in 40015 ms
** (EXIT from #PID<0.217.0>) :benchmark_completed

# GenMetrics Datadog Statistical Metrics Enabled
iex(1)> GenMetricsBench.Pipeline.datadog_metrics
GenStage Pipeline: benchmark starting, sim_load=10000000
GenStage Pipeline: sim_load=10000000 completed in 40128 ms
** (EXIT from #PID<0.223.0>) :benchmark_completed

Note, the default benchmark behaviours provided by GenMetricsBench.Simulator.PipelineDefault were used during these sample benchmarks.