GenServer that owns the ETS role/node table and coordinates cluster sync.
Supports overlapping scopes, allowing a node to participate in multiple scopes simultaneously with full isolation between them. A node can have different roles in different scopes.
Responsibilities
- Maintains an ETS
:bagtable keyed on{:scope, scope, :role, role}and{:scope, scope, :node, node}for O(1) lookups in either direction, scoped. - Maintains a separate ETS
:settable for O(1) node enumeration per scope. - Registers this node in a
:pggroup per scope for isolated messaging. - Monitors node connections via
:net_kernel.monitor_nodes/2(shared across scopes). - On startup, pulls the current role state from every already-connected node for each active scope.
- Broadcasts incremental add/remove events scoped to their respective groups
using
:pg.get_members/2+send/2for efficient delivery. - Periodically re-syncs using generation-based change detection: each scope tracks a monotonically increasing generation counter that increments on every local role change. Remote nodes are only fully pulled when their generation has changed, dramatically reducing unnecessary RPC traffic.
- Invokes the optional
ClusterHelper.EventHandlercallbacks when roles or nodes are added or removed from the local ETS table.
Performance characteristics
- Read operations (
get_nodes/2,get_roles/2,all_nodes/1) bypass the GenServer and read directly from ETS tables, which are:protectedwithread_concurrency: true. This provides microsecond-level lookups without GenServer bottleneck. - Batch ETS inserts are used for bulk role additions, significantly reducing the overhead of adding multiple roles at once.
- Task.Supervisor (
ClusterHelper.TaskSupervisor) is used for all async pull tasks, providing better fault tolerance and back-pressure compared to bareTask.start/1. :pg.get_members/2+send/2is used for cluster-wide event broadcasting, leveraging the VM's native process group membership for efficient delivery.
Generation-based sync
Each scope maintains a local generation counter (starting at 0) that is
incremented every time roles are added or removed on this node. During the
periodic pull, the GenServer first checks the remote node's generation via a
lightweight :erpc.call to __get_generation__/1. Only if the generation has
changed (or the node is newly discovered) does a full role pull occur. This
avoids expensive full-sync RPCs when nothing has changed.
Smart node up/down handling
On :nodeup, the GenServer discovers which scopes the new node participates
in by calling __get_scopes__/0 on the remote node. It then only pulls roles
for matching scopes, avoiding unnecessary cross-scope RPCs. The on_node_added
callback fires for each scope where the node is discovered.
On :nodedown, the node is removed from all scopes and the on_node_removed
callback is fired.
Internal message protocol
| Message | Direction | Meaning |
|---|---|---|
{:new_roles, scope, roles, node} | broadcast → handle_info | Remote node added roles in scope |
{:remove_roles, scope, roles, node} | broadcast → handle_info | Remote node removed roles in scope |
{:nodeup, node, info} | net_kernel → handle_info | Remote node came online |
{:nodedown, node, info} | net_kernel → handle_info | Remote node went offline |
{:pull_new_node, scope, node, roles} | Task → handle_info | Pulled roles from a newly discovered node |
{:pull_new_node, scope, node, roles, gen} | Task → handle_info | Same, with generation metadata |
{:pull_update_node, scope, node, roles} | Task → handle_info | Full-sync result for a known node |
{:pull_update_node, scope, node, roles, gen} | Task → handle_info | Same, with generation metadata |
{:stale_node, scope, node} | Task → handle_info | Node left the cluster, clean up |
:pull_roles | Process.send_after → handle_info | Periodic sync tick |
Summary
Functions
Adds role to the current node in the given scope and propagates the change
cluster-wide.
Adds each role in roles to the current node in the given scope and
propagates cluster-wide.
Returns a deduplicated list of every node that has at least one role in the
given scope.
Returns a specification to start this module under a supervisor.
Returns all roles assigned to the local node.
Returns every node in the given scope that has been assigned role.
Returns all roles assigned to node in the given scope.
Joins the current node to an additional scope.
Leaves the given scope, removing all local roles and cleaning up ETS entries
for this scope.
Returns a list of all scopes the local node is currently participating in.
Returns true when node is the local node, false otherwise.
Removes role from the current node in the given scope and propagates the
change cluster-wide.
Removes each role in roles from the current node in the given scope and
propagates cluster-wide.
Functions
@spec add_role(ClusterHelper.role(), scope :: atom() | nil) :: :ok
Adds role to the current node in the given scope and propagates the change
cluster-wide.
If scope is nil, uses the default scope. The scope is automatically
joined if it hasn't been already.
@spec add_roles([ClusterHelper.role()], scope :: atom() | nil) :: :ok
Adds each role in roles to the current node in the given scope and
propagates cluster-wide.
If scope is nil, uses the default scope.
Returns a deduplicated list of every node that has at least one role in the
given scope.
If scope is nil, uses the default scope.
Reads directly from ETS for microsecond-level lookups without GenServer overhead.
Returns a specification to start this module under a supervisor.
See Supervisor.
@spec get_my_roles(scope :: atom() | nil) :: [ClusterHelper.role()]
Returns all roles assigned to the local node.
If scope is nil, returns roles from the default scope.
@spec get_nodes(ClusterHelper.role(), scope :: atom() | nil) :: [node()]
Returns every node in the given scope that has been assigned role.
If scope is nil, uses the default scope.
Reads directly from ETS for microsecond-level lookups without GenServer overhead.
@spec get_roles(node(), scope :: atom() | nil) :: [ClusterHelper.role()]
Returns all roles assigned to node in the given scope.
If scope is nil, uses the default scope.
Reads directly from ETS for microsecond-level lookups without GenServer overhead.
@spec join_scope(atom()) :: :ok | {:error, :already_joined}
Joins the current node to an additional scope.
The scope is started (if not already), the node joins the :all_nodes group,
and an initial pull is performed from all connected nodes for this scope.
@spec leave_scope(atom()) :: :ok | {:error, :not_joined}
Leaves the given scope, removing all local roles and cleaning up ETS entries
for this scope.
@spec list_scopes() :: [atom()]
Returns a list of all scopes the local node is currently participating in.
Returns true when node is the local node, false otherwise.
@spec remove_role(ClusterHelper.role(), scope :: atom() | nil) :: :ok
Removes role from the current node in the given scope and propagates the
change cluster-wide.
If scope is nil, uses the default scope.
@spec remove_roles([ClusterHelper.role()], scope :: atom() | nil) :: :ok
Removes each role in roles from the current node in the given scope and
propagates cluster-wide.
If scope is nil, uses the default scope.
@spec start_link(any()) :: GenServer.on_start()