Simple provider router that implements exponential backoff for failed providers.

This router blocks providers that encounter specific types of errors for a configurable backoff period. The backoff duration increases exponentially with each consecutive failure, helping to avoid overwhelming failing services while allowing for automatic recovery.

Configuration

Configuration is done via application environment:

config :llm_composer, :provider_router,
  min_backoff_ms: 1_000,                    # 1 second minimum backoff (default)
  max_backoff_ms: :timer.minutes(5),        # 5 minutes maximum backoff (default)
  cache_mod: LlmComposer.Cache.Ets,         # Cache module to use (default)
  cache_opts: [                             # Cache options (default shown below)
    name: LlmComposer.ProviderRouter.Simple,
    table_name: :llm_composer_provider_blocks
  ],
  name: LlmComposer.ProviderRouter.Simple   # Router instance name (default)

Backoff Strategy

The router uses exponential backoff with the following formula:

backoff_ms = min(max_backoff_ms, min_backoff_ms * 2^(failure_count - 1))

Examples with default settings:

• 1st failure: 1 second
• 2nd failure: 2 seconds
• 3rd failure: 4 seconds
• 4th failure: 8 seconds
• 5th failure: 16 seconds
• ...continuing until max_backoff_ms (5 minutes)

Behavior

• Success: Provider is unblocked and failure count is reset
• Failure: Provider is blocked for exponential backoff period
• Blocking: Blocked providers are skipped during provider selection
• Recovery: Providers automatically become available after backoff period expires
• Persistence: Blocking state persists across application restarts (stored in ETS with long TTL)