Benchmarks

Performance benchmarks for the instrument library, measuring throughput of core operations.

Running Benchmarks

# Run core benchmarks
./bench/run_bench.escript

# Run real-world scenario benchmarks
rebar3 compile
erl -pa _build/default/lib/*/ebin -noshell -eval \
  "c:c(\"bench/instrument_realworld_bench.erl\"), instrument_realworld_bench:run(), halt()."

# Run client tracing strategy benchmarks
erl -pa _build/default/lib/*/ebin -noshell -eval \
  "c:c(\"bench/instrument_client_bench.erl\"), instrument_client_bench:run(), halt()."

Results

Benchmarks run on Apple M1 Pro, OTP 28, 100,000 iterations per test (April 2026).

Counter Operations

Gauge Operations

Histogram Operations

OpenTelemetry Meter API

OpenTelemetry Tracer API

OpenTelemetry Logger Integration

Real-World Scenarios

Simulated production workloads to measure end-to-end performance.

DB Query Tracing

Tracing overhead: ~4.5 us per operation.

HTTP Request Pipeline

Concurrent Load

• 100 workers x 1,000 requests = 100,000 total
• Result: 27,465 req/sec sustained throughput

Memory Impact

• 100,000 spans with attributes and events
• Throughput: 245,700 spans/sec
• Memory overhead: negligible (GC-friendly design)

Client Tracing Strategies

Comparing different approaches to instrument client operations (DB, HTTP, etc).

Strategy Comparison

Sanitization Performance

Sampling Strategies

Trace Context Injection

Pool Span Helpers

Analysis

Metrics Performance

Counter and gauge operations achieve 38-48 million operations per second thanks to NIF-based atomic operations. The OpenTelemetry Meter API adds minimal overhead for operations without attributes (65-75M ops/sec).

Operations with attributes are slower (2.8-3.7M ops/sec) due to attribute map handling and vec metric creation, but still very fast for most use cases.

Tracing Performance

Span creation costs ~3.2-3.5 us per span due to:

• Context management (process dictionary operations)
• Span ID generation
• Timestamp collection
• Exporter callbacks

The library achieves ~300K spans/second for simple spans. Nested spans scale sub-linearly: 3 nested spans take ~5.4 us (not 3x single span time) due to context reuse.

Sampling Impact

Sampling dramatically reduces overhead:

• always_off: 5x faster than always_on (dropped spans skip most work)
• 1% sampling: Nearly as fast as always_off
• Attribute-based: Scales well, early exit on rule match

Optimization Tips

1. Use sampling in production - even 10% sampling reduces overhead by 3x
2. Prefer counters over histograms when you only need counts
3. Batch operations rather than creating many small spans
4. Pre-create instruments at startup rather than on-demand
5. Sanitization adds ~6-7 us - only use when necessary
6. Attribute operations are slower - minimize attribute cardinality
7. Dropped spans are 5x cheaper - sampling helps significantly