ExRaft (ExRaft v0.2.1) View Source
ExRaft provides an Elixir implementation and an easy to use API
for the raft consensus protocol.
Internals
Servers
When starting an ExRaft server, there are 2 processes spawned:
- a
Supervisorprocess that ensures that the logs are not lost when the server crashes - the actual server process
Note that the Supervisor will generate an additional atom
to register itself (for more information, check out the
ExRaft.start_server/3 function).
Timeouts
When calling ExRaft.write/3, ExRaft.add_server/3 or
ExRaft.remove_server/3, you may experience timeouts.
This is most likely caused by the majority of the cluster not being available to the leader.
Note that even though the call may time out, once the majority comes back online your command might still be applied, provided the rest of the cluster did not advance their log in the meantime.
Cluster membership changes
Dynamic cluster membership is supported with one-at-a-time changes to the configuration, but as of now new servers are added without any sort of catch-up mechanism or validation (these features will be present in a future release).
Servers can be added to- or removed from the cluster by making ExRaft.add_server/3
or ExRaft.remove_server/3 calls to the leader.
Log compaction
As of now log compaction is not supported but is planned to be implemented and released in the future.
Example
init_arg = [very_cool: true]
initial_config = [raft1: node(), raft2: node()]
# after starting these servers, they will time out and eventually elect a leader
# amongst themselves
{:ok, _} = ExRaft.start_server(YourStateMachine, init_arg, name: :raft1, initial_config: initial_config)
{:ok, _} = ExRaft.start_server(YourStateMachine, init_arg, name: :raft2, initial_config: initial_config)
# this server can never become leader unless its added to an existing cluster
# since it doesn't know of any other server but is required to achieve
# a minimum majority of 2 in elections and log replication
{:ok, _} = ExRaft.start_server(YourStateMachine, init_arg, name: :raft3, min_majority: 2)
# we could pick any server to await the leader
leader = ExRaft.await_leader(:raft1)
:ok = ExRaft.add_server(leader, :raft3)
# the success result of write/3 depends on the state machine
:ok = ExRaft.write(leader, :hello)
# making a write/3 call to a follower results in an error
follower = List.delete(initial_config, leader) |> List.first()
{:error, {:redirect, ^leader}} = ExRaft.write(follower, :hello)
# if we stop the active leader (or it crashes)
# the rest of the cluster will be able to recover
:ok = ExRaft.stop_server(leader)
:ok = ExRaft.trigger_election(:raft3)
new_leader = ExRaft.await_leader(:raft3)
true = Enum.member?([raft2: node(), raft3: node()], new_leader)Link to this section Summary
Types
Option values used by the start_server functions.
Options used by the start_server functions.
Reference to a peer.
Reference to a server.
Functions
Makes a synchronous call to server to add a new server to the configuration.
Makes a synchronous call to server and waits until
server learns of; or becomes the cluster leader.
Sends an asynchronous request to server to inspect and write
its context or log to the device.
Makes a synchronous call to server to retrieve its known leader
and waits for its reply.
Makes a synchronous call to server to read query from the state
and waits for its reply.
Same as read/3, but if :dirty_read is set to true (see start_server/3),
followers are also allowed to execute the query.
Makes a synchronous call to server to remove a server from the configuration.
Same as start_server/3 but sets [] as init_arg.
Starts a server process linked to the current process.
Makes a synchronous call to server to retrieve its state
and waits for its reply.
Synchronously stops the server with the given reason.
Makes a synchronous call to server to start a new election.
Makes a synchronous call to server to apply command to the state
and waits for its reply.
Link to this section Types
Specs
option() ::
{:name, atom()}
| {:debug, boolean()}
| {:dirty_read, boolean()}
| {:initial_config, [peer()]}
| {:min_majority, nil | non_neg_integer()}
| {:min_election_timeout, pos_integer()}
| {:max_election_timeout, pos_integer()}
| {:heartbeat_timeout, pos_integer()}
| {:batch_size, pos_integer()}
Option values used by the start_server functions.
See start_server/3 for more information.
Specs
options() :: [option()]
Options used by the start_server functions.
See start_server/3 for more information.
Specs
Reference to a peer.
Used internally by ExRaft servers to communicate with each other.
Specs
Reference to a server.
Used to make calls to ExRaft servers.
Link to this section Functions
Specs
add_server(server :: server(), new_server :: peer(), timeout :: timeout()) :: :ok | {:error, reason :: atom() | {:redirect, peer()}}
Makes a synchronous call to server to add a new server to the configuration.
Return values
If all checks pass, returns :ok once the majority of the new configuration
(including new_server) knows of new_server being a member of the cluster.
(Returns :ok immediately if new_server is already a member of the cluster.)
Otherwise returns {:error, reason}, where reason is one of the following:
:no_commit-serverhas yet to commit an entry in it's term as leader.:unstable_config- another configuration change is pending.- can also return any of the error reasons specified at
write/3
Specs
Makes a synchronous call to server and waits until
server learns of; or becomes the cluster leader.
Returns the elected cluster leader.
Specs
inspect(server :: server(), flag :: :context | :log) :: :ok
Sends an asynchronous request to server to inspect and write
its context or log to the device.
Specs
Makes a synchronous call to server to retrieve its known leader
and waits for its reply.
Specs
read(server :: server(), query :: any(), timeout :: timeout()) :: StateMachine.reply() | {:error, reason :: atom() | {:redirect, peer()}}
Makes a synchronous call to server to read query from the state
and waits for its reply.
Return values
If server is leader, it returns the result of calling the handle_read/2
function of the underlying state machine (see ExRaft.StateMachine).
Otherwise if server is not leader, it returns the same error as write/3.
Specs
Same as read/3, but if :dirty_read is set to true (see start_server/3),
followers are also allowed to execute the query.
If :dirty_read was set to false, server never replies.
remove_server(server, server_to_remove, timeout \\ 60000)
View Source (since 0.1.0)Specs
remove_server( server :: server(), server_to_remove :: peer(), timeout :: timeout() ) :: :ok | {:error, reason :: atom() | {:redirect, peer()}}
Makes a synchronous call to server to remove a server from the configuration.
Return values
If all checks pass, returns :ok once the majority of the new configuration
(without server_to_remove) knows of server_to_remove not being a member of the cluster.
(Returns :ok immediately if server_to_remove is not a member of the cluster.)
Otherwise returns {:error, reason}, where reason is one of the following:
:last_server-server_to_removeis the last server in the configuration.:no_commit-serverhas yet to commit an entry in it's term as leader.:unstable_config- another configuration change is pending.- can also return any of the error reasons specified at
write/3
Specs
start_server(module :: module(), options :: options()) :: Supervisor.on_start()
Same as start_server/3 but sets [] as init_arg.
Specs
start_server(module :: module(), init_arg :: any(), options :: options()) :: Supervisor.on_start()
Starts a server process linked to the current process.
Once the server is started, the init/1 function (see ExRaft.StateMachine)
of the given module is called with init_arg as its argument to initialize the server.
Options
:name- used for name registration as described in the "Name Registration" section of theGenServerdocumentation. Note that only local names (atoms) are accepted. By designExRaftwill create an additional atom for the server supervisor under:"Elixir.ExRaft.Supervisor.#{name}".This is the only option required to ensure internal server communication.
:debug- can be set totrueto enable verbose server logs. Defaults tofalse.:dirty_read- can be set tofalseto disable reading "dirty" (maybe out of date) state from followers. Defaults totrue.:initial_config- used to start the server with a known set of peers. Defaults to[].:min_majority- used to set a minimum expected majority. When set, servers will decide on achieving majority in elections and log replication based on the active configuration and this value (the greater value will be used). As an example, with a value of 2 in a 2 member cluster, the required majority will be 2, with the same value in a 5 member cluster, the required majority will be 3. Defaults tonil.:min_election_timeout- used to tune the minimum amount of time in milliseconds, that must pass before a follower times out. Note that the actual timeout will be picked randomly between:min_election_timeoutand:max_election_timeout. Defaults to1_000.:max_election_timeout- used to tune the maximum amount of time in milliseconds, that must pass before a follower times out. Note that the actual timeout will be picked randomly between:min_election_timeoutand:max_election_timeout. Defaults to10_000.:heartbeat_timeout- used to set the heartbeat timeout (aka.: the time that will pass between leader heartbeats in log replication) in milliseconds. Defaults to100.:batch_size- used to set the number of entries sent by leaders to followers in heartbeat RPCs. Defaults to100.
Specs
Makes a synchronous call to server to retrieve its state
and waits for its reply.
stop_server(server, reason \\ :normal, timeout \\ :infinity)
View Source (since 0.1.0)Specs
Synchronously stops the server with the given reason.
server will call the terminate/2 function of the underlying state machine
before exiting (see ExRaft.StateMachine).
This function keeps OTP semantics regarding error reporting.
If the reason is any other than :normal, :shutdown or {:shutdown, _},
an error report is logged.
Specs
Makes a synchronous call to server to start a new election.
server replies with :ok immediately and becomes candidate
to start a new election unless server is already a candidate
in which case the reply will still be :ok but nothing happens.
Specs
write(server :: server(), command :: any(), timeout :: timeout()) :: StateMachine.reply() | {:error, reason :: atom() | {:redirect, peer()}}
Makes a synchronous call to server to apply command to the state
and waits for its reply.
Return values
If server is leader, it appends command to its log, and once the
majority of the cluster replicates command, it applies command to the state
by calling the handle_write/2 function of the underlying state machine
(see ExRaft.StateMachine). It then returns reply from the handle_write/2 call.
Otherwise if server is not leader, it returns {:error, reason}
where reason is one of the following:
:no_leader-serverdoes not know the leader of the cluster.{:redirect, leader}- whereleaderis the leader of the cluster.