Sampling and Processing Guide

This guide covers how to configure sampling (which spans to record) and processing (how to handle and export spans) in the instrument library.

Table of Contents

1. Overview
2. Samplers
   • AlwaysOn Sampler
   • AlwaysOff Sampler
   • Probability Sampler
   • ParentBased Sampler
3. Span Processors
   • Simple Processor
   • Batch Processor
4. Complete Pipeline
5. Environment Variables
6. Examples

Overview

The tracing pipeline has two key stages:

1. Sampling - Decides whether a span should be recorded and exported. This happens at span creation time.
2. Processing - Handles recorded spans, typically batching and exporting them to backends.

             ┌──────────────┐       ┌─────────────┐       ┌──────────────┐
Span Start ──▶   Sampler    ├──────▶  Processor   ├──────▶   Exporter    │
             │ (drop/record)│       │(batch/export)│       │(OTLP/Console)│
             └──────────────┘       └─────────────┘       └──────────────┘

Samplers

Samplers determine whether spans should be sampled (recorded and exported). The decision is made when a span starts.

Sampling Decisions

• drop - Don't record the span
• record_only - Record but don't export
• record_and_sample - Record and export

AlwaysOn Sampler

Samples every span. Default sampler, suitable for development or low-traffic services.

instrument_sampler:set_sampler(instrument_sampler_always_on).

AlwaysOff Sampler

Drops all spans. Useful for temporarily disabling tracing.

instrument_sampler:set_sampler(instrument_sampler_always_off).

Probability Sampler

Samples a percentage of traces based on trace ID. The decision is deterministic: the same trace ID always produces the same decision, ensuring consistent sampling across services.

%% Sample 10% of traces
instrument_sampler:set_sampler(instrument_sampler_probability, #{ratio => 0.1}).

%% Sample 50% of traces
instrument_sampler:set_sampler(instrument_sampler_probability, #{ratio => 0.5}).

Configuration:

ParentBased Sampler

Respects the parent span's sampling decision. Essential for distributed tracing to maintain consistent sampling across services.

• If parent is sampled, child is sampled
• If parent is not sampled, child is not sampled
• If no parent (root span), delegates to a configurable root sampler

%% ParentBased with probability sampling for root spans
instrument_sampler:set_sampler(instrument_sampler_parent_based, #{
    root => instrument_sampler_probability,
    root_config => #{ratio => 0.1}
}).

%% ParentBased with always-on for root spans (default behavior)
instrument_sampler:set_sampler(instrument_sampler_parent_based, #{
    root => instrument_sampler_always_on
}).

Configuration:

Span Processors

Span processors handle spans after they're created. They receive spans on start and end, and are responsible for batching and exporting.

Simple Processor

Exports each span immediately when it ends. Suitable for development and debugging.

instrument_span_processor:register(instrument_span_processor_simple, #{
    exporter => instrument_exporter_console,
    exporter_config => #{format => text}
}).

Configuration:

When to use:

• Development and debugging
• Low-traffic applications
• When you need immediate visibility

Batch Processor

Queues spans and exports them in batches. Recommended for production use.

instrument_span_processor:register(instrument_span_processor_batch, #{
    exporter => instrument_exporter_otlp,
    exporter_config => #{endpoint => "http://localhost:4318"},
    max_queue_size => 2048,
    max_export_batch_size => 512,
    schedule_delay_millis => 5000
}).

Configuration:

Behavior:

• Spans are exported when the batch reaches max_export_batch_size
• Or after schedule_delay_millis has elapsed
• If queue reaches max_queue_size, new spans are dropped

Managing Processors

%% List registered processors
instrument_span_processor:list().
%% [instrument_span_processor_batch]

%% Force flush all pending spans
instrument_span_processor:force_flush().

%% Unregister a processor
instrument_span_processor:unregister(instrument_span_processor_batch).

%% Shutdown all processors
instrument_span_processor:shutdown().

Complete Pipeline

Here's how to set up a complete tracing pipeline:

%% 1. Configure sampler (10% sampling with parent-based)
instrument_sampler:set_sampler(instrument_sampler_parent_based, #{
    root => instrument_sampler_probability,
    root_config => #{ratio => 0.1}
}).

%% 2. Register processor with exporter
instrument_span_processor:register(instrument_span_processor_batch, #{
    exporter => instrument_exporter_otlp,
    exporter_config => #{
        endpoint => "http://localhost:4318",
        compression => gzip
    },
    max_queue_size => 4096,
    schedule_delay_millis => 2000
}).

%% 3. Create spans - they flow through the pipeline automatically
instrument_tracer:with_span(<<"process_order">>, fun() ->
    instrument_tracer:set_attributes(#{<<"order.id">> => <<"123">>}),
    do_work()
end).

Environment Variables

Configure sampling and processing via environment variables:

Sampler Configuration

Batch Processor Configuration

Using Environment Variables

# Sample 10% of traces with parent-based sampling
export OTEL_TRACES_SAMPLER=parentbased_traceidratio
export OTEL_TRACES_SAMPLER_ARG=0.1

# Configure batch processor
export OTEL_BSP_SCHEDULE_DELAY=2000
export OTEL_BSP_MAX_QUEUE_SIZE=4096

Then initialize configuration in your application:

%% Read and apply environment variables
instrument_config:init().

Examples

Development Setup

Use simple processor with console exporter for immediate feedback:

%% Sample everything
instrument_sampler:set_sampler(instrument_sampler_always_on).

%% Export immediately to console
instrument_span_processor:register(instrument_span_processor_simple, #{
    exporter => instrument_exporter_console,
    exporter_config => #{format => text}
}).

Production Setup with 10% Sampling

%% Parent-based 10% sampling
instrument_sampler:set_sampler(instrument_sampler_parent_based, #{
    root => instrument_sampler_probability,
    root_config => #{ratio => 0.1}
}).

%% Batch export to OTLP collector
instrument_span_processor:register(instrument_span_processor_batch, #{
    exporter => instrument_exporter_otlp,
    exporter_config => #{
        endpoint => "http://otel-collector:4318",
        compression => gzip
    }
}).

High-Volume Setup

For high-throughput services, tune batch processor for efficiency:

%% Low sampling rate
instrument_sampler:set_sampler(instrument_sampler_parent_based, #{
    root => instrument_sampler_probability,
    root_config => #{ratio => 0.01}  %% 1% sampling
}).

%% Optimized batch settings
instrument_span_processor:register(instrument_span_processor_batch, #{
    exporter => instrument_exporter_otlp,
    exporter_config => #{
        endpoint => "http://otel-collector:4318",
        compression => gzip,
        timeout => 10000
    },
    max_queue_size => 8192,
    max_export_batch_size => 1024,
    schedule_delay_millis => 1000  %% Export more frequently
}).

Application Configuration

Configure via sys.config:

[
  {instrument, [
    {sampler, {instrument_sampler_parent_based, #{
        root => instrument_sampler_probability,
        root_config => #{ratio => 0.1}
    }}},
    {span_processor, {instrument_span_processor_batch, #{
        max_queue_size => 4096,
        schedule_delay_millis => 2000,
        max_export_batch_size => 512
    }}},
    {span_exporter, {instrument_exporter_otlp, #{
        endpoint => <<"http://localhost:4318/v1/traces">>
    }}}
  ]}
].

Graceful Shutdown

Ensure spans are exported before shutdown:

stop(_State) ->
    %% Flush pending spans
    instrument_span_processor:force_flush(),
    %% Shutdown processors
    instrument_span_processor:shutdown(),
    ok.