Shoehorn.Handler behaviour (shoehorn v0.8.0) View Source

A behaviour module for implementing handling of failing applications

A Shoehorn.Handler is a module that knows how to respond to specific applications going down. There are two types of failing applications. The first is an application that fails to initialize and the second is an application that stops while it is running.

Example

The Shoehorn.Handler behaviour requires developers to implement two callbacks.

The init callback sets up any state the handler needs.

The application_started callback is called when an application starts up.

The application_exited callback processes the incoming failure and replies with the reaction that Shoehorn should take in case of application failure.

    defmodule Example.ShoehornHandler do
      use Shoehorn.Handler

      def init(_opts) do
        {:ok, %{restart_counts: 0}}
      end

      def application_started(app, state) do
        {:continue, state}
      end

      def application_exited(:non_essential_app, _reason,  state) do
        {:continue, state}
      end

      def application_exited(:essential_app, _reason, %{restart_counts: restart_counts} = state) when restart_counts < 2 do
        # repair actions to make before restarting
        Application.ensure_all_started(:essential_app)
        {:continue, %{state | restart_counts: restart_counts + 1}}
      end

      def application_exited(_, state) do
        {:halt, state}
      end
    end

We initialize our Shoehorn.Handler with a restart count for state by calling init with the configuration options from our shoehorn config. The stored state is passed in from the Shoehorn.ApplicationController.

When we have an application startup, we will put a message on the console to notify the developer.

When we have a non-essential application fail we return :continue to inform the system to keep going like nothing happened.

We restart the essential application of our system two times, and then we tell the system to halt if starting over wasn't fixing the system.

Link to this section Summary

Types

cause()

The cause that is firing the handler

opts()

reaction()

The reaction letting Shoehorn.ApplicationController know what to do

t()

Callbacks

application_exited(app, cause, state)

Callback for handling application crashes

application_started(app, state)

Callback for handling application starts

init(opts)

Callback to initialize the handle

Functions

init(opts)

invoke(event, app, handler)

invoke(event, app, cause, handler)

Link to this section Types

cause()

Specs

cause() :: any()

The cause that is firing the handler

opts()

Specs

opts() :: [{:handler, atom()}]

reaction()

Specs

reaction() :: :continue | :halt

The reaction letting Shoehorn.ApplicationController know what to do

:continue - keep the system going like nothing happened
:halt - stop the application and bring the system down

t()

Specs

t() :: %Shoehorn.Handler{module: atom(), state: any()}

Link to this section Callbacks

application_exited(app, cause, state)

Specs

application_exited(app :: atom(), cause(), state :: any()) ::
  {reaction(), state :: any()}

Callback for handling application crashes

Called with the application name, cause, and the handler's state. It must return a tuple containing the reaction that the Shoehorn.ApplicationController should take, and the new state of the handler.

The code that you execute here can be used to notify or capture some information before halting the system. This information can later be used to recreate the issue or debug the problem causing the application to exit.

Use application_exited as a place for a last-ditch effort to fix the issue and restart the application. Ideally, capture some information on the system state, and solve it upstream. Shoehorn restarts should be used as a splint to keep a critical system running.

The default implementation returns the previous state, and a :halt reaction.

Callback Timeout

This callback is limited in the amount of time it has to execute by a shutdown timer. If the callback does not return within the defined shutdown time, the node is instructed to halt. The default shutdown time is 30 seconds but this value can be changed in the application config.

application_started(app, state)

Specs

application_started(app :: atom(), state :: any()) ::
  {reaction(), state :: any()}

Callback for handling application starts

Called with the application name, and the handler's state. It must return a tuple containing the reaction that the Shoehorn.ApplicationController should take, and the new state of the handler.

      def application_exited(:essential_app, _reason, state) do
        # repair actions to make before restarting
        # notify someone of the crash and the details
        # log debug data
        Application.ensure_all_started(:essential_app)
        {:continue, %{state | restart_counts: restart_counts + 1}}
      end

The default implementation returns unchanged state, and a :continue reaction.

init(opts)

Specs

init(opts :: keyword()) :: {:ok, state :: any()}

Callback to initialize the handle

The callback will receive a keyword list of all the configuration for shoehorn.

The callback must return a tuple of {:ok, state}. Where state is the initial state of the handler. The system will halt if the return is anything other than :ok.

Link to this section Functions

init(opts)

Specs

init(opts()) :: t() | no_return()

invoke(event, app, handler)

Specs

invoke(:application_started, app :: atom(), t()) :: {reaction(), t()}

invoke(event, app, cause, handler)

Specs

invoke(:application_exited, app :: atom(), cause(), t()) :: {reaction(), t()}