Middleware

Middleware lets you inject logic before and after each provider sample in the agentic loop — without modifying the loop itself. Common uses: logging, token budget enforcement, content filtering, rate limiting, cost tracking.

The interface

Implement the OpenResponses.Middleware behaviour:

defmodule MyApp.Middleware.AuditLog do
  @behaviour OpenResponses.Middleware

  @impl OpenResponses.Middleware
  def before_sample(loop_state) do
    # Called before each provider request.
    # Return {:cont, loop_state} to proceed,
    # or {:halt, reason} to stop the loop.
    Logger.info("Sampling from #{loop_state.response.model}", response_id: loop_state.response.id)
    {:cont, loop_state}
  end

  @impl OpenResponses.Middleware
  def after_sample(loop_state, items) do
    # Called after each provider response is received.
    # Return {:cont, loop_state, items} to proceed with (possibly modified) items,
    # or {:halt, loop_state, reason} to stop.
    {:cont, loop_state, items}
  end
end

Registering middleware

In config/config.exs:

config :open_responses, :middlewares, [
  MyApp.Middleware.AuditLog,
  MyApp.Middleware.TokenBudget,
  MyApp.Middleware.ContentFilter
]

Middleware runs in order. If any before_sample/1 returns {:halt, reason}, the loop stops and the response transitions to failed. Later middleware modules are not called.

Examples

Token budget enforcement

defmodule MyApp.Middleware.TokenBudget do
  @behaviour OpenResponses.Middleware
  @max_tokens 50_000

  @impl OpenResponses.Middleware
  def before_sample(%{response: %{usage: %{"total_tokens" => used}}} = state)
      when used > @max_tokens do
    {:halt, :token_budget_exceeded}
  end

  def before_sample(state), do: {:cont, state}

  @impl OpenResponses.Middleware
  def after_sample(state, items), do: {:cont, state, items}
end

Content filtering

defmodule MyApp.Middleware.ContentFilter do
  @behaviour OpenResponses.Middleware

  @impl OpenResponses.Middleware
  def before_sample(state), do: {:cont, state}

  @impl OpenResponses.Middleware
  def after_sample(state, items) do
    filtered = Enum.reject(items, &contains_pii?/1)
    {:cont, state, filtered}
  end

  defp contains_pii?(item) do
    text = get_text(item)
    String.match?(text, ~r/\b\d{3}-\d{2}-\d{4}\b/)
  end

  defp get_text(%{"content" => [%{"text" => text} | _]}), do: text
  defp get_text(_), do: ""
end

Rate limiting

defmodule MyApp.Middleware.RateLimit do
  @behaviour OpenResponses.Middleware

  @impl OpenResponses.Middleware
  def before_sample(state) do
    user_id = state.response.metadata["user_id"]

    case MyApp.RateLimiter.check(user_id) do
      :ok -> {:cont, state}
      {:error, :rate_limited} -> {:halt, :rate_limited}
    end
  end

  @impl OpenResponses.Middleware
  def after_sample(state, items), do: {:cont, state, items}
end

Execution order

Middleware runs in list order:

before_sample: [AuditLog, TokenBudget, ContentFilter]
  │ → AuditLog.before_sample
  │ → TokenBudget.before_sample  (halts here if over budget)
  │ → ContentFilter.before_sample

[provider samples]

after_sample: [AuditLog, TokenBudget, ContentFilter]
  │ → AuditLog.after_sample
  │ → TokenBudget.after_sample
  │ → ContentFilter.after_sample  (filters items here)

On halt in before_sample, the remaining middleware and the provider call are skipped. On halt in after_sample, the loop terminates after the current items are processed.