Tuning and Observability

This guide covers parameter selection, tuning strategies, and observability patterns for every module in the Resiliency library. Each section provides a parameter reference table, explains how parameters interact, and gives concrete recommendations for common workloads.

Table of Contents

1. CircuitBreaker Tuning
2. BackoffRetry Tuning
3. Hedged Requests Tuning
4. SingleFlight Tuning
5. Task Combinator Tuning
6. WeightedSemaphore Tuning
7. RateLimiter Tuning
8. Observability
9. Common Pitfalls

CircuitBreaker Tuning

Parameter Reference

How Parameters Interact

The circuit evaluates after each recorded call:

if window.total >= minimum_calls do
  if failure_rate >= failure_rate_threshold or slow_rate >= slow_call_rate_threshold do
    trip to :open
  end
end

The :minimum_calls parameter acts as a warm-up guard -- the circuit will not trip until enough calls have been observed. This prevents a single failure from tripping a freshly started breaker.

The sliding window is count-based (not time-based). Old outcomes are evicted when new ones push them out of the fixed-size buffer. This means the window naturally adapts to traffic volume without requiring time-based expiry.

Tuning for Common Workloads

Interpreting get_stats

Call Resiliency.CircuitBreaker.get_stats/1 to inspect the breaker at runtime:

%{
  state: :closed,
  total: 87,
  failures: 12,
  slow_calls: 3,
  failure_rate: 0.1379,
  slow_call_rate: 0.0345
}

BackoffRetry Tuning

Parameter Reference

How Parameters Interact

The effective retry sequence is computed as:

delays = backoff_strategy(base_delay)
         |> Backoff.cap(max_delay)
         |> Enum.take(max_attempts - 1)

For exponential backoff with the defaults, the delay sequence before cap is:

attempt 1: immediate (no delay -- first call)
attempt 2: 100 ms
attempt 3: 200 ms

With max_attempts: 5 and base_delay: 100:

attempt 1: immediate
attempt 2: 100 ms
attempt 3: 200 ms
attempt 4: 400 ms
attempt 5: 800 ms

The :budget option acts as a secondary stop condition. Even if max_attempts has not been reached, the retry loop aborts when the next sleep would exceed the total time budget. This makes :budget the right knob for SLA-sensitive callers -- set it to your upstream timeout minus a safety margin.

The Backoff.jitter/2 modifier (usable when passing a custom stream) spreads each delay d uniformly over [d * (1 - proportion), d * (1 + proportion)]. Jitter is critical in multi-instance deployments to avoid thundering herd on the downstream service after a shared failure.

Backoff Strategy Formulas

Tuning for Common Workloads

Custom Backoff Streams

For advanced scenarios, compose your own stream with jitter and cap:

custom_backoff =
  Resiliency.BackoffRetry.Backoff.exponential(base: 200, multiplier: 3)
  |> Resiliency.BackoffRetry.Backoff.jitter(0.25)
  |> Resiliency.BackoffRetry.Backoff.cap(15_000)

Resiliency.BackoffRetry.retry(fn -> call_service() end,
  backoff: custom_backoff,
  max_attempts: 6
)

Or pass a literal list when you want fully deterministic delays:

Resiliency.BackoffRetry.retry(fn -> call_service() end,
  backoff: [100, 500, 2_000, 5_000]
)

When a list is provided, max_attempts is implicitly length(list) + 1 (one initial call plus one retry per delay entry).

Rule of Thumb

Total worst-case latency = sum(delays) + max_attempts * max_call_duration. If this exceeds your caller's timeout, either reduce max_attempts, lower max_delay, or set a :budget.

Hedged Requests Tuning

Stateless Mode Parameter Reference

Adaptive Mode (Tracker) Parameter Reference

When using Resiliency.Hedged.start_link/1, the delay auto-tunes based on observed latency. A token bucket throttles the hedge rate under load.

How Adaptive Delay Works

1. Cold start -- Before min_samples observations, the tracker returns initial_delay as the hedge delay. Pick this conservatively -- too low and you double load during startup.

2. Steady state -- The tracker maintains a circular buffer of the last buffer_size latency samples. On each get_config/1 call, it computes the configured percentile of the buffer and clamps it to [min_delay, max_delay]. This becomes the hedge delay.

3. Token bucket -- Every completed request (success or failure) adds token_success_credit tokens. Every hedge that fires costs token_hedge_cost tokens. When tokens drop below token_threshold, hedging is disabled until enough successful requests replenish the bucket. This creates a natural feedback loop: under sustained load where hedges are not winning, the system backs off automatically.

Effective hedge rate (steady state):

max_hedge_fraction = token_success_credit / token_hedge_cost

With defaults (0.1 / 1.0), at most 10% of requests will be hedged in steady state. To allow up to 20%, set token_success_credit: 0.2. To allow up to 5%, set token_success_credit: 0.05 or token_hedge_cost: 2.0.

Tuning for Different Scenarios

Interpreting Tracker Stats

Call Resiliency.Hedged.Tracker.stats/1 to inspect the tracker at runtime:

%{
  total_requests: 15234,
  hedged_requests: 1412,
  hedge_won: 987,
  p50: 12,
  p95: 45,
  p99: 210,
  current_delay: 45,
  tokens: 7.2
}

SingleFlight Tuning

Resiliency.SingleFlight has no tunable numeric parameters -- its behavior is determined entirely by key design and usage patterns.

When It Helps

• Cache stampede -- Many processes request the same cache key after expiry. Without SingleFlight, all of them hit the database. With it, one process fetches while the rest share the result.

• Expensive computation -- Deduplicating concurrent calls to a heavy aggregation or report-generation function.

• External API rate limits -- Preventing duplicate requests to a rate-limited third-party service.

When It Hurts

• Non-idempotent operations -- If the function has side effects that must execute per-caller (e.g., incrementing a counter, sending a notification), SingleFlight will suppress those side effects for coalesced callers.

• Caller-specific context -- If each caller needs a slightly different variant of the result (different query parameters, different auth tokens), deduplication by a shared key will return the wrong result for some callers.

• Very short functions -- If the function completes in microseconds, the overhead of the GenServer round-trip (message passing, ETS or map lookup) may exceed the savings from deduplication.

Key Design Considerations

Task Combinator Tuning

Choosing the Right Combinator

Timeout Selection

Timeout Rules of Thumb

• For Race.run/1 -- set the timeout to your SLA ceiling. If no backend responds in time, you want a clear timeout rather than an indefinite hang.

• For AllSettled.run/1 -- set the timeout to the slowest acceptable task duration. Tasks that finish within the deadline keep their results; the rest are marked as timed out.

• For Resiliency.Map.run/2 -- multiply your per-item budget by the number of items divided by max_concurrency, then add a margin. Or use :infinity and rely on per-item timeouts inside the function.

• For FirstOk.run/1 -- set the timeout to the total latency budget for the entire fallback chain. Each attempt subtracts from the remaining budget.

Race vs FirstOk Decision

Do you want concurrent execution?
  Yes -> Race.run/1 (all functions run at once, first success wins)
  No  -> FirstOk.run/1 (sequential, stops at first success)

Use Race.run/1 when all backends can handle the load of concurrent requests. Use FirstOk.run/1 when you want to avoid unnecessary calls to slower/more expensive backends.

WeightedSemaphore Tuning

Parameter Reference

max Selection

The :max value represents your concurrency budget. Choose it based on the downstream resource's capacity:

Weight Assignment Strategies

Backpressure Behavior

The semaphore's FIFO queue provides natural backpressure:

• Blocking -- acquire/3 blocks the caller until permits are available. This is the default and simplest mode. Callers queue up and are served in order.

• Non-blocking -- try_acquire/3 returns :rejected immediately if permits are not available or if there are waiters in the queue. Use this for "best effort" work that can be dropped under load.

• Timeout -- acquire/4 accepts a timeout in milliseconds. If permits are not available within the deadline, returns {:error, :timeout}. The caller is removed from the queue.

Fairness guarantee -- Waiters are served in strict FIFO order. A large waiter at the head of the queue blocks smaller waiters behind it, preventing starvation. This means a weight-8 request waiting for permits will not be bypassed by a weight-1 request, even if capacity exists for the smaller request.

Sizing Guidelines

Utilization = avg_concurrent_weight / max

• < 50% -- The semaphore is rarely contended. You may be over-provisioned, or the workload is bursty. Consider lowering max to catch genuine overload earlier.

• 50--80% -- Healthy range. Some queuing occurs during bursts but callers are not waiting long.

• > 90% -- The semaphore is a bottleneck. Callers are frequently blocked. Either increase max (if the downstream can handle it) or reduce the arrival rate.

RateLimiter Tuning

Parameter Reference

How It Works

The rate limiter uses a lazy token-bucket — there is no background timer. Tokens refill on each call based on elapsed time since the last operation:

new_tokens = min(burst_size, old_tokens + elapsed_ms * rate_per_ms)

If new_tokens >= weight, the call is granted and weight tokens are deducted. Otherwise the call is rejected immediately. The hot path (grant and reject) runs entirely in the caller's process via lock-free ETS operations — no GenServer message is sent.

Choosing rate and burst_size

rate vs burst_size Interaction

# Steady-state throughput (tokens/s) = rate
# Burst capacity (tokens) = burst_size
# Time to refill from empty = burst_size / rate seconds

A full empty bucket refills in burst_size / rate seconds. If burst_size equals rate, the bucket refills in exactly one second. If burst_size is much larger than rate, callers can absorb large traffic spikes before the rate limit kicks in.

retry_after_ms Formula

When a call is rejected, the hint is:

retry_after_ms = ceil((weight - current_tokens) / rate * 1000)

This tells the caller exactly how long to wait for enough tokens to refill for their specific weight. Callers should treat this as a minimum — token counts are shared across concurrent callers.

Weighted Calls

Use :weight when different operations have different costs relative to your upstream rate limit. For example, if an API counts bulk requests as equivalent to N single requests, pass weight: N:

# Single item lookup: 1 token
Resiliency.RateLimiter.call(rl, fn -> get_one(id) end)

# Bulk fetch of 50 items: 50 tokens
Resiliency.RateLimiter.call(rl, fn -> get_many(ids) end, weight: 50)

A weight larger than burst_size is always rejected immediately.

get_stats/1 Usage

get_stats/1 computes the projected token count using the current timestamp, without writing to ETS or consuming any tokens. Use it for health checks and dashboards — it does not interfere with the hot path:

%{tokens: tokens, rate: rate, burst_size: burst_size} =
  Resiliency.RateLimiter.get_stats(MyApp.ApiRateLimiter)

utilization = (burst_size - tokens) / burst_size
# 0.0 = full bucket (no recent calls)
# 1.0 = empty bucket (fully rate limited)

Performance Characteristics

The hot path avoids GenServer messages entirely:

• Grant path: persistent_term.get + ETS lookup + float refill math + ETS CAS (select_replace)
• Reject path: persistent_term.get + ETS lookup + float refill math + ETS update_element

Observed on M-series hardware: ~3µs/call for grants, ~2µs/call for rejects. Under 8 concurrent processes, reductions per acquire stay flat (< 100) — no serialisation.

Common Pitfalls

Observability

Emitting Telemetry from CircuitBreaker

Use the :on_state_change callback to emit :telemetry events on every state transition:

{Resiliency.CircuitBreaker,
 name: MyApp.Breaker,
 failure_rate_threshold: 0.5,
 on_state_change: fn name, from, to ->
   :telemetry.execute(
     [:my_app, :circuit_breaker, :state_change],
     %{},
     %{name: name, from: from, to: to}
   )
 end}

Poll Resiliency.CircuitBreaker.get_stats/1 periodically for dashboard metrics:

stats = Resiliency.CircuitBreaker.get_stats(MyApp.Breaker)

:telemetry.execute(
  [:my_app, :circuit_breaker, :stats],
  %{
    failure_rate: stats.failure_rate,
    slow_call_rate: stats.slow_call_rate,
    total: stats.total,
    failures: stats.failures
  },
  %{name: MyApp.Breaker, state: stats.state}
)

Logging Retry Attempts

Use the :on_retry callback to emit structured log lines on every retry:

require Logger

Resiliency.BackoffRetry.retry(
  fn -> MyService.call(params) end,
  max_attempts: 4,
  backoff: :exponential,
  base_delay: 200,
  on_retry: fn attempt, delay, error ->
    Logger.warning(
      "Retry attempt",
      attempt: attempt,
      delay_ms: delay,
      error: inspect(error),
      service: "my_service"
    )
  end
)

Emitting Telemetry from BackoffRetry

Wrap your retry call to emit :telemetry events for each attempt and for the final outcome:

defmodule MyApp.Resilient do
  def call_with_telemetry(fun, opts \\ []) do
    start_time = System.monotonic_time()
    meta = %{service: Keyword.get(opts, :service, :unknown)}

    result =
      Resiliency.BackoffRetry.retry(fun,
        max_attempts: Keyword.get(opts, :max_attempts, 3),
        on_retry: fn attempt, delay, error ->
          :telemetry.execute(
            [:my_app, :retry, :attempt],
            %{delay_ms: delay},
            Map.merge(meta, %{attempt: attempt, error: inspect(error)})
          )
        end,
        retry_if: Keyword.get(opts, :retry_if, fn {:error, _} -> true end)
      )

    duration = System.monotonic_time() - start_time

    case result do
      {:ok, _} ->
        :telemetry.execute(
          [:my_app, :retry, :success],
          %{duration: duration},
          meta
        )

      {:error, _} ->
        :telemetry.execute(
          [:my_app, :retry, :failure],
          %{duration: duration},
          meta
        )
    end

    result
  end
end

Emitting Telemetry from Hedged Requests

Use :on_hedge for per-hedge telemetry, and wrap the call for overall metrics:

defmodule MyApp.HedgedCall do
  def run(fun, opts \\ []) do
    service = Keyword.get(opts, :service, :unknown)
    start_time = System.monotonic_time()

    result =
      Resiliency.Hedged.run(fun,
        delay: Keyword.get(opts, :delay, 100),
        timeout: Keyword.get(opts, :timeout, 5_000),
        on_hedge: fn attempt ->
          :telemetry.execute(
            [:my_app, :hedged, :hedge_fired],
            %{},
            %{service: service, attempt: attempt}
          )
        end
      )

    duration = System.monotonic_time() - start_time

    case result do
      {:ok, _} ->
        :telemetry.execute(
          [:my_app, :hedged, :success],
          %{duration: duration},
          %{service: service}
        )

      {:error, _} ->
        :telemetry.execute(
          [:my_app, :hedged, :failure],
          %{duration: duration},
          %{service: service}
        )
    end

    result
  end
end

Monitoring Adaptive Hedging with Tracker.stats/1

Poll Resiliency.Hedged.Tracker.stats/1 periodically to feed dashboards:

defmodule MyApp.HedgeReporter do
  use GenServer

  def start_link(opts) do
    tracker = Keyword.fetch!(opts, :tracker)
    interval = Keyword.get(opts, :interval, 10_000)
    GenServer.start_link(__MODULE__, %{tracker: tracker, interval: interval})
  end

  @impl true
  def init(state) do
    schedule(state.interval)
    {:ok, state}
  end

  @impl true
  def handle_info(:report, state) do
    stats = Resiliency.Hedged.Tracker.stats(state.tracker)

    :telemetry.execute(
      [:my_app, :hedged, :tracker_stats],
      %{
        p50: stats.p50 || 0,
        p95: stats.p95 || 0,
        p99: stats.p99 || 0,
        current_delay: stats.current_delay,
        tokens: stats.tokens,
        total_requests: stats.total_requests,
        hedged_requests: stats.hedged_requests,
        hedge_won: stats.hedge_won
      },
      %{tracker: state.tracker}
    )

    hedge_rate =
      if stats.total_requests > 0,
        do: stats.hedged_requests / stats.total_requests,
        else: 0.0

    win_rate =
      if stats.hedged_requests > 0,
        do: stats.hedge_won / stats.hedged_requests,
        else: 0.0

    :telemetry.execute(
      [:my_app, :hedged, :tracker_rates],
      %{hedge_rate: hedge_rate, win_rate: win_rate},
      %{tracker: state.tracker}
    )

    schedule(state.interval)
    {:noreply, state}
  end

  defp schedule(interval), do: Process.send_after(self(), :report, interval)
end

Monitoring WeightedSemaphore

The semaphore does not expose internal stats directly. Instrument it by wrapping calls:

defmodule MyApp.InstrumentedSemaphore do
  def acquire(sem, weight, fun) do
    start_time = System.monotonic_time()

    result = Resiliency.WeightedSemaphore.acquire(sem, weight, fn ->
      wait_duration = System.monotonic_time() - start_time

      :telemetry.execute(
        [:my_app, :semaphore, :acquired],
        %{wait_duration: wait_duration, weight: weight},
        %{semaphore: sem}
      )

      fun.()
    end)

    total_duration = System.monotonic_time() - start_time

    case result do
      {:ok, _} ->
        :telemetry.execute(
          [:my_app, :semaphore, :complete],
          %{duration: total_duration, weight: weight},
          %{semaphore: sem, outcome: :ok}
        )

      {:error, _} ->
        :telemetry.execute(
          [:my_app, :semaphore, :complete],
          %{duration: total_duration, weight: weight},
          %{semaphore: sem, outcome: :error}
        )
    end

    result
  end

  def try_acquire(sem, weight, fun) do
    case Resiliency.WeightedSemaphore.try_acquire(sem, weight, fun) do
      :rejected ->
        :telemetry.execute(
          [:my_app, :semaphore, :rejected],
          %{weight: weight},
          %{semaphore: sem}
        )

        :rejected

      other ->
        other
    end
  end
end

Suggested Telemetry Event Names

Common Pitfalls