Horde.DynamicSupervisor behaviour (Horde v0.8.6) View Source

A distributed supervisor.

Horde.DynamicSupervisor implements a distributed DynamicSupervisor backed by a add-wins last-write-wins δ-CRDT (provided by DeltaCrdt.AWLWWMap). This CRDT is used for both tracking membership of the cluster and tracking supervised processes.

Using CRDTs guarantees that the distributed, shared state will eventually converge. It also means that Horde.DynamicSupervisor is eventually-consistent, and is optimized for availability and partition tolerance. This can result in temporary inconsistencies under certain conditions (when cluster membership is changing, for example).

Cluster membership is managed with Horde.Cluster. Joining a cluster can be done with Horde.Cluster.set_members/2. To take a node out of the cluster, call Horde.Cluster.set_members/2 without that node in the list. Alternatively, setting the members startup option to :auto will make Horde auto-manage cluster membership so that all (and only) visible nodes are members of the cluster.

Each Horde.DynamicSupervisor node wraps its own local instance of DynamicSupervisor. Horde.DynamicSupervisor.start_child/2 (for example) delegates to the local instance of DynamicSupervisor to actually start and monitor the child. The child spec is also written into the processes CRDT, along with a reference to the node on which it is running. When there is an update to the processes CRDT, Horde makes a comparison and corrects any inconsistencies (for example, if a conflict has been resolved and there is a process that no longer should be running on its node, it will kill that process and remove it from the local supervisor). So while most functions map 1:1 to the equivalent DynamicSupervisor functions, the eventually consistent nature of Horde requires extra behaviour not present in DynamicSupervisor.

Divergence from standard DynamicSupervisor behaviour

While Horde wraps DynamicSupervisor, it does keep track of processes by the id in the child specification. This is a divergence from the behaviour of DynamicSupervisor, which ignores ids altogether. Using DynamicSupervisor is useful for its shutdown behaviour (it shuts down all child processes simultaneously, unlike Supervisor).

Graceful shutdown

When a node is stopped (either manually or by calling :init.stop), Horde restarts the child processes of the stopped node on another node. The state of child processes is not preserved, they are simply restarted.

To implement graceful shutdown of worker processes, a few extra steps are necessary.

  1. Trap exits. Running Process.flag(:trap_exit) in the init/1 callback of any worker processes will convert exit signals to messages and allow running terminate/2 callbacks. It is also important to include the shutdown option in your child spec (the default is 5000ms).

  2. Use :init.stop() to shut down your node. How you accomplish this is up to you, but by simply calling :init.stop() somewhere, graceful shutdown will be triggered.

Module-based Supervisor

Horde supports module-based supervisors to enable dynamic runtime configuration.

defmodule MySupervisor do
  use Horde.DynamicSupervisor

  def start_link(init_arg, options \ []) do
    Horde.DynamicSupervisor.start_link(__MODULE__, init_arg, options)

  def init(init_arg) do
    [strategy: :one_for_one, members: members()]
    |> Keyword.merge(init_arg)
    |> Horde.DynamicSupervisor.init()

  defp members() do

Then you can use MySupervisor.child_spec/1 and MySupervisor.start_link/1 in the same way as you'd use Horde.DynamicSupervisor.child_spec/1 and Horde.DynamicSupervisor.start_link/1.

Link to this section Summary

Link to this section Types


option() ::
  {:name, name :: atom()}
  | {:strategy, Supervisor.strategy()}
  | {:max_restarts, integer()}
  | {:max_seconds, integer()}
  | {:extra_arguments, [term()]}
  | {:distribution_strategy, Horde.DistributionStrategy.t()}
  | {:shutdown, integer()}
  | {:members, [Horde.Cluster.member()] | :auto}
  | {:delta_crdt_options, [DeltaCrdt.crdt_option()]}
  | {:process_redistribution, :active | :passive}


options() :: [option()]

Link to this section Callbacks


child_spec(options :: options()) :: Supervisor.child_spec()


init(options()) :: {:ok, options()} | :ignore

Link to this section Functions

See start_link/2 for options.

Link to this function


View Source

Works like DynamicSupervisor.count_children/1.

This function delegates to all supervisors in the cluster and returns the aggregated output.

Works like DynamicSupervisor.init/1.

Link to this function

start_child(supervisor, child_spec)

View Source

Works like DynamicSupervisor.start_child/2.

Works like DynamicSupervisor.start_link/1. Extra options are documented here:

Link to this function

start_link(mod, init_arg, opts \\ [])

View Source
Link to this function

stop(supervisor, reason \\ :normal, timeout \\ :infinity)

View Source

Works like DynamicSupervisor.stop/3.

Link to this function

terminate_child(supervisor, child_pid)

View Source


terminate_child(Supervisor.supervisor(), child_pid :: pid()) ::
  | {:error, :not_found}
  | {:error, {:node_dead_or_shutting_down, String.t()}}

Terminate a child process.

Works like DynamicSupervisor.terminate_child/2.

Link to this function

wait_for_quorum(horde, timeout)

View Source


wait_for_quorum(horde :: GenServer.server(), timeout :: timeout()) :: :ok

Waits for Horde.DynamicSupervisor to have quorum.

Link to this function


View Source

Works like DynamicSupervisor.which_children/1.

This function delegates to all supervisors in the cluster and returns the aggregated output. Where memory warnings apply to DynamicSupervisor.which_children, these count double for Horde.DynamicSupervisor.which_children.