Tracing and Instrumentation

Malla includes a zero-overhead observability system called Malla.Tracer. It provides a technology-agnostic API for instrumenting your services with spans, logs, metrics, and events.

The key principle is that you can instrument your code while you write it, without committing to a specific telemetry backend (like OpenTelemetry or Datadog). That decision can be deferred and implemented later by adding a plugin, without changing any of your instrumented code.

Quick Start

To use the tracer, use Malla.Tracer in your service module.

defmodule MyService do
  use Malla.Service
  use Malla.Tracer # Import the tracer macros

  def process_payment(user_id, amount) do
    # 'span' creates a new trace span for this unit of work
    span [:payment, :process] do
      # 'info' creates a log entry associated with the span
      info("Processing payment", user_id: user_id, amount: amount)

      case charge_card(user_id, amount) do
        {:ok, receipt} ->
          # 'metric' records a numerical value
          metric(:payments_successful, 1)
          {:ok, receipt}
        
        {:error, reason} = err ->
          # 'error' logs an error message
          error("Payment failed", reason: reason)
          metric(:payments_failed, 1)
          
          # 'error/1' can also mark the span as having an error
          error(err)
      end
    end
  end
end

How It Works: The Plugin System

• Default Behavior: Without any special plugins, Malla.Tracer provides minimal functionality. Spans emit basic :telemetry events, and logs are sent to the standard Elixir Logger.
• With Plugins: When you add a telemetry plugin (e.g., for OpenTelemetry), that plugin overrides the default tracer callbacks. Your span, info, and metric calls are then routed to the full-featured telemetry backend, enabling distributed tracing, rich context, and custom metrics. We will release soon a powerful plugin capable of sending all traces to Jaeger.

This allows you to add comprehensive observability to your application at any time just by adding a plugin to your service configuration.

Tracer API

Spans

Spans represent a unit of work and can be nested to create a detailed trace.

• span(name, opts, do: block): Executes the block of code within a new span. The name is typically a list of atoms.
• span_update(attributes): Updates the current span with new key-value attributes.

span [:http, :request] do
  span_update(path: "/users", method: "GET")
  
  # Nested span
  span [:database, :query] do
    # ...
  end
end

Logging

Log macros create structured log entries that are automatically associated with the current span. They also support compile-time log level filtering.

• debug(message, metadata \\ [])
• info(message, metadata \\ [])
• notice(message, metadata \\ [])
• warning(message, metadata \\ [])
• error(message, metadata \\ [])
• error(error_term): Marks the current span with an error status.

Compile-Time Filtering

You can set a minimum log level in your configuration to completely remove log calls from your compiled code, ensuring zero performance impact in production.

# In config/prod.exs
config :malla, log_min_level: :info # This will remove all `debug` calls

Metrics and Events

• metric(name, value, metadata \\ []): Records a numerical metric.
• event(name, data \\ [], metadata \\ []): Records a structured event within a span, representing a point-in-time occurrence.

Context Propagation

The tracer system works seamlessly with Malla's remote calls, automatically propagating trace context across nodes to enable distributed tracing.

For custom context propagation (e.g., to manually spawned processes or for integrating with external systems), you can use get_base/0 to retrieve the current span context. A telemetry plugin would then provide a corresponding set_base/1 function to restore that context in another process.

Custom Tracing with Plugins

While Malla may provide official telemetry plugins in the future, you can create your own to integrate with any backend.

A tracing plugin is a standard Malla plugin that implements one or more of the malla_span_* callbacks defined in Malla.Plugins.Tracer.

Here is a simplified example for OpenTelemetry:

defmodule OpenTelemetryPlugin do
  use Malla.Plugin
  
  # Override the 'malla_span' callback
  defcb malla_span(name, opts, fun) do
    # Convert Malla's span name to a string for OpenTelemetry
    span_name = Enum.join(List.wrap(name), ".")
    
    # Use the OpenTelemetry API to start a span
    OpenTelemetry.Tracer.with_span span_name do
      try do
        result = fun.()
        OpenTelemetry.Span.set_status(:ok)
        result
      rescue
        e ->
          OpenTelemetry.Span.record_exception(e)
          OpenTelemetry.Span.set_status(:error, Exception.message(e))
          reraise e, __STACKTRACE__
      end
    end
  end
  
  # Override the 'malla_span_log' callback to add logs as span events
  defcb malla_span_log(level, text_fun, meta) do
    text_str = text_fun.()
    OpenTelemetry.Span.add_event(text_str, Map.new(meta))
    
    # Continue the chain to also send the log to the default Elixir Logger
    :cont
  end
end

By adding this plugin to your service, all span and log calls will be sent to OpenTelemetry without any changes to your business logic.