Building Effective Spans

View Source

A span captures more than timing. This chapter covers how to add context that makes debugging easier.

Span Attributes

Attributes are key-value pairs that describe the span. They are indexed by backends, enabling filtering and searching.

Setting Attributes

instrument_tracer:with_span(<<"process_order">>, fun() ->
    %% Set multiple attributes at once
    instrument_tracer:set_attributes(#{
        <<"order.id">> => <<"ORD-12345">>,
        <<"customer.id">> => <<"CUST-789">>,
        <<"order.total">> => 149.99,
        <<"order.item_count">> => 5
    }),

    process(Order)
end).

%% Or set one at a time
instrument_tracer:set_attribute(<<"payment.method">>, <<"credit_card">>).

Attribute Types

Attributes support these value types:

%% Strings
instrument_tracer:set_attribute(<<"user.email">>, <<"alice@example.com">>).

%% Numbers
instrument_tracer:set_attribute(<<"http.status_code">>, 200).
instrument_tracer:set_attribute(<<"order.total">>, 49.99).

%% Booleans
instrument_tracer:set_attribute(<<"user.premium">>, true).

Attribute Best Practices

Useful attributes:

  • Request identifiers (order ID, transaction ID)
  • User context (user ID, tenant ID, role)
  • Request parameters (HTTP method, endpoint)
  • Business data (order total, item count)
  • Error details (error code, message)

Attribute naming:

  • Use dot notation for namespacing: http.method, db.operation
  • Use lowercase with underscores or dots
  • Be consistent across your codebase

Avoid:

  • Sensitive data (passwords, tokens, PII)
  • High-cardinality values in excessive quantities
  • Duplicate information already in the span name

Span Events

Events mark points in time within a span. They're useful for capturing milestones or occurrences.

Adding Events

instrument_tracer:with_span(<<"process_order">>, fun() ->
    instrument_tracer:add_event(<<"order_validated">>),

    Items = fetch_items(Order),
    instrument_tracer:add_event(<<"items_fetched">>, #{
        <<"count">> => length(Items)
    }),

    calculate_shipping(Items),
    instrument_tracer:add_event(<<"shipping_calculated">>),

    complete_order(Order)
end).

Events vs Child Spans

Use events for:

  • Quick checkpoints (validation passed, cache hit)
  • Things that happen but don't have duration
  • Debugging markers

Use child spans for:

  • Operations with meaningful duration
  • Operations you might want to optimize
  • Work that might fail independently
%% Event: quick check, no meaningful duration
instrument_tracer:add_event(<<"input_validated">>).

%% Span: database call with meaningful duration
instrument_tracer:with_span(<<"db_query">>, fun() ->
    run_query(SQL)
end).

Recording Exceptions

The with_span function automatically records exceptions, but you can also record them manually:

try
    risky_operation()
catch
    error:Reason:Stacktrace ->
        instrument_tracer:record_exception(Reason, #{
            stacktrace => Stacktrace
        }),
        handle_error(Reason)
end.

Recorded exceptions appear as span events with:

  • exception.type: The error type
  • exception.message: Formatted error
  • exception.stacktrace: Stack trace if provided

Span Status

Set the span status to indicate success or failure:

%% Successful operation
instrument_tracer:set_status(ok).

%% Failed operation
instrument_tracer:set_status(error).

%% Failed with description
instrument_tracer:set_status(error, <<"Payment declined">>).

Status values:

  • ok: Operation completed successfully
  • error: Operation failed
  • unset: Default, no status set

Status Best Practices

  • Set ok for successful operations
  • Set error for failures that need attention
  • Include a description for errors
  • Don't set status for operations that are "expected" to fail sometimes

Links connect spans that are related but not in a parent-child relationship:

%% Link to a span from another trace
OtherSpanCtx = get_triggering_span_context(),
instrument_tracer:add_link(OtherSpanCtx).

%% Link with attributes
instrument_tracer:add_link(#{
    span_ctx => OtherSpanCtx,
    attributes => #{<<"link.reason">> => <<"retry">>}
}).

Use links for:

  • Batch processing (link to all source items)
  • Fan-out operations (link to triggering span)
  • Retries (link to original attempt)

Updating the Span Name

Sometimes you don't know the final name until later:

instrument_tracer:with_span(<<"http_request">>, fun() ->
    {Method, Path} = parse_request(Req),

    %% Update name with actual details
    instrument_tracer:update_name(<<Method/binary, " ", Path/binary>>),

    handle_request(Method, Path)
end).

Span Options

Create spans with specific options:

instrument_tracer:with_span(<<"operation">>, #{
    kind => server,
    attributes => #{<<"initial">> => <<"value">>},
    links => [OtherSpanCtx],
    start_time => erlang:system_time(nanosecond)
}, fun() ->
    do_work()
end).

Available options:

  • kind: client, server, producer, consumer, internal
  • attributes: Initial attributes map
  • links: List of span contexts to link
  • start_time: Override the start timestamp
  • parent: Override the parent span context

Complete Example: HTTP Handler

-module(http_handler).
-export([handle/2]).

handle(Method, Path) ->
    instrument_tracer:with_span(<<"http_request">>, #{kind => server}, fun() ->
        %% Set HTTP attributes
        instrument_tracer:set_attributes(#{
            <<"http.method">> => Method,
            <<"http.target">> => Path,
            <<"http.scheme">> => <<"https">>
        }),

        %% Validate request
        case validate_request(Method, Path) of
            {error, Reason} ->
                instrument_tracer:set_status(error, Reason),
                {400, Reason};

            ok ->
                instrument_tracer:add_event(<<"request_validated">>),

                %% Process the request
                Result = instrument_tracer:with_span(<<"process_request">>, fun() ->
                    process(Method, Path)
                end),

                %% Set response attributes
                {Status, Body} = Result,
                instrument_tracer:set_attributes(#{
                    <<"http.status_code">> => Status
                }),

                case Status >= 400 of
                    true ->
                        instrument_tracer:set_status(error);
                    false ->
                        instrument_tracer:set_status(ok)
                end,

                Result
        end
    end).

Exercise

Enhance the order processor from the previous chapter:

  1. Add attributes for order ID, customer ID, and total
  2. Add events for validation, payment processing, and completion
  3. Record exceptions properly
  4. Set appropriate status based on outcome

Then introduce a deliberate error and observe how it appears in the span.

Next Steps

Your spans now carry rich context. In the next chapter, you will learn how to connect spans across service boundaries.