The foundation of a Drain app.

This module is meant to be use’d into another module to build a Drain pipeline. It’s an extension of Plug.Builder and it’s based on the same concepts.

When use’d it takes care of the boilerplate configuration to provide the basic behaviour that a Drain app must implement.

More specifically, it defines a default Plug pipeline with these plugs, in order:

• Routemaster.Plugs.RootPostOnly
• Routemaster.Plugs.Auth
• Routemaster.Plugs.Parser
• Routemaster.Plugs.Terminator

It also automatically configures Plug.Logger (only with Mix.env == :dev) and Plug.ErrorHandler to handle some common errors raised by the plugs (e.g. JSON parse errors).

That, by itself, is enough to accept and validate the event-delivery POST requests from the bus server and to respond with the appropriate status codes. To actually process the received events and do something useful with them, this module provides its own drain/2 macro to build a second asynchronous plug pipeline. This second plug pipeline handles the events in the background without blocking the HTTP response and is triggered from the main pipeline before returning a response. All the plugs defined with the drain/2 macro are executed in a supervised Task linked to a Task.Supervisor.

For example:

# Define a Drain app, it will be a valid Plug.
#
defmodule MyApp.MyDrainApp do
  use Routemaster.Drain

  drain Routemaster.Drains.Siphon, topic: "burgers", to: MyApp.BurgerSiphon
  drain Routemaster.Drains.Dedup
  drain Routemaster.Drains.IgnoreStale
  drain :a_function_plug, some: "options"
  drain Routemaster.Drains.FetchAndCache
  drain MyApp.MyCustomDrain, some: "other options"
  drain Routemaster.Drains.Notify, listener: MyApp.EventsSink

  def a_function_plug(conn, opts) do
    {:ok, stuff} = MyApp.Utils.do_something(conn.assigns.events, opts[:some])
    Plug.Conn.assign(conn, :stuff, stuff)
  end
end

# Mount it into a Phoenix or Plug Router.
#
defmodule MyApp.Web.Router do
  use MyApp.Web, :router

  scope path: "/events" do
    forward "/", MyApp.MyDrainApp
  end
end

Just like with the Plug.Builder.plug/2 macro, multiple plugs can be defined with the drain/2 macro, and the plugs in the Drain pipeline will be executed in the order they’ve been added. In the example above, Routemaster.Drains.Siphon will be called first, followed by Routemaster.Drains.Dedup, then Routemaster.Drains.IgnoreStale, then the :a_function_plug function and so on.

Again, the second drain pipeline is anynchronous and independent from the main plug pipeline. The original HTTP POST request that delivers the batch of events is responded to when the main plug pipeline reaches the Terminator plug, and ideally way before the second drain pipeline has completed.

This has been done for two reasons:

• To respond quickly and without errors not related to the HTTP request. An event-delivery request from the bus server only needs to be a POST to the correct path, be authenticated and with a valid JSON body containing events. Once all of this has been verified without errors, there is no reason to delay the response to the bus server, and it would be inappropriate to respond a 500 just because something else later in the pipeline fails to process an event.
• With a supervised root Task that fans out to other independently supervised tasks, errors (e.g. timeouts fetching a resource) can be handled and retried with more flexibility.