Barrel P2P

Barrel P2P is an Erlang/OTP library for peer-to-peer clusters.

It keeps the Erlang programming model: Pid ! Msg, gen_server, rpc, links, monitors, and the usual supervision habits. It changes what happens underneath:

distribution runs over QUIC, not TCP;
nodes authenticate with Ed25519;
membership is bounded with HyParView, not full mesh;
services are discovered through a CRDT registry;
EPMD is not required.

The result is a cluster where each node keeps a small number of gossip connections, while any process can still talk to any process in the cluster.

Why Erlang developers may care

Standard Erlang distribution is simple and good. It becomes harder to operate when the cluster grows, when every node must connect to every other node, when EPMD is not welcome, or when nodes move between network paths.

Barrel P2P keeps your application code close to normal OTP code. You still register processes, call servers, monitor pids, and send messages. The library takes care of the cluster shape, peer identity, service discovery, and the QUIC dist carrier.

The cluster shape

Read this graph from node A. The green nodes are the active view: the small set of peers node A uses for gossip and membership maintenance. The grey nodes are known peers kept in the passive view. They are not connected now, but they are warm spares when the topology changes.

HyParView active view: a node connects to a small set of gossip peers, with additional known peers held in a passive cache.

The important point: the active view is not the cluster. It is not the list of nodes your application may talk to. It is only the maintenance topology.

Application traffic is different. If code on node A sends to a pid on node E, OTP can open a dist channel on demand. Barrel P2P authenticates that channel, then the normal Erlang message is delivered.

Sending a message to a pid on a node that is not in the local active view: OTP opens a QUIC dist channel on demand, runs Ed25519 auth, then delivers the message.

Once you hold the pid, Barrel P2P is no longer on the application data path. You use Erlang.

Project status

Barrel P2P is experimental and pre-1.0. APIs may change between minor releases until a 1.0 tag.

The cryptographic and transport layers, Ed25519 dist auth and the QUIC carrier, have unit and multi-node test coverage. They have not been independently audited. Do not use Barrel P2P where a transport compromise would be costly without doing your own review first.

Bug reports and PRs are welcome. Security reports go through SECURITY.md.

Five-minute start

Add Barrel P2P to rebar.config:

{deps, [
    {barrel_p2p, "0.1.0"}
]}.

Use Barrel P2P as the Erlang distribution carrier:

-proto_dist barrel_p2p
-epmd_module barrel_p2p_epmd
-start_epmd false

Start the application and join a seed:

application:ensure_all_started(barrel_p2p).
ok = barrel_p2p:join('seed@192.168.1.10').

ok = barrel_p2p:register_service(worker_pool, #{shard => 1}).

Find it from another node:

{ok, _Node, Worker} = barrel_p2p:whereis_service(worker_pool),
Worker ! {work, <<"payload">>}.

That send is a standard Erlang send. whereis_service/1 returns {ok, Pid} for a local service and {ok, Node, Pid} for a remote one. If the target node is not already connected, the dist channel is opened on demand over QUIC.

Configuration

A small development config:

{barrel_p2p, [
    {active_size, 5},
    {passive_size, 30},
    {listen_port, 9100},
    {contact_nodes, ['seed@192.168.1.10']},
    {auth_enabled, true},
    {auth_trust_mode, tofu}
]}.

The defaults are intentionally narrow:

active_size bounds the number of gossip peers.
passive_size bounds the known-but-disconnected peer cache.
listen_port is the UDP port for QUIC distribution.
contact_nodes gives the node a first place to join.
auth_trust_mode is tofu or strict.

For production notes, see run in production.

What is included

QUIC distribution. -proto_dist barrel_p2p plugs into Erlang's alternative distribution layer. No stock EPMD daemon is required.
HyParView membership. Each node keeps a bounded active view instead of a full mesh.
Service registry. Processes can be registered by name and found from any node through a CRDT-backed registry.
Replicated state. barrel_p2p_map is a gossiped, last-write-wins key-value map for cluster-wide config, flags, and routing tables; the barrel_p2p_replica behaviour underneath is public for custom merge.
Plumtree broadcast. Registry changes and gossip move through an efficient epidemic broadcast tree.
Ed25519 peer identity. Nodes prove their identity after the QUIC TLS handshake and before application traffic flows.
Tagged QUIC streams. Applications can open tagged streams for large or byte-oriented transfers.
Operational tools. Helpers exist for one-shot RPC, TLS material, key rotation, metrics, migration, and test clusters.

Documentation

The docs are organised into five sections. Each section has a hub page that lists its children; the hub pages are linked below.

Overview

Read this first if barrel_p2p is new to you.

What is barrel_p2p? — the project in one page, with the architecture diagram and the load-bearing ideas.
Benefits and trade-offs — why pick barrel_p2p, and what you give up.
Introduction — the long narrative through every layer.
Getting started — boot two nodes and send a real message.

Core concepts

How each subsystem works, explained one page at a time.

Cluster membership — HyParView's bounded active and passive views.
Service registry — OR-Map CRDT, eventual consistency, the registration lifecycle.
Gossip broadcast — Plumtree push-lazy-push trees, self-healing graft/prune.
Dist channel — -proto_dist barrel_p2p over QUIC, the discovery chain, the idle GC.
Authentication — Ed25519 mutual challenge-response, trust modes.
Streams — tagged user-stream multiplex over the same QUIC connection.
Connection migration — RFC 9000 §9 path rebind.
Hybrid logical clocks — the timestamps the CRDT uses.

Tutorials

End-to-end walkthroughs.

Hello, cluster — the smallest two-node walkthrough.
Distributed chat — a small application that uses the service registry and service events.

How-to guides

Task-focused recipes for operating a cluster.

Run in production — sizing, network surface, secrets, shutdown.
Configure authentication — TOFU and strict modes, provisioning, rotation.
Observe a cluster — metrics catalogue and exporter wiring.
Troubleshoot — symptom-cause-fix tables.
Migrate connections — a watchdog recipe for path migration.
Route through a relay — wiring an external tunnel or proxy adapter.
Run the tests — EUnit, Common Test, docker, and soak suites.

Reference

Authoritative material when you know what you are looking for.

API overview — every public function in barrel_p2p.erl, grouped by subsystem.
Configuration — every key under {barrel_p2p, [...]} in sys.config.
Architecture — the full supervision tree and protocol-level details.
Comparison with Partisan — side-by-side, when to pick which library.
Feature stability — public API tiers and coverage.

Example application

A small distributed chat app lives under examples/chat.

cd examples/chat
./scripts/run-demo.sh seed
./scripts/run-demo.sh node 1

The demo starts Erlang nodes with -proto_dist barrel_p2p, generates local TLS and Ed25519 material on first boot, and uses the service registry to find chat rooms across the cluster.

Testing

Run the local suites:

rebar3 eunit
rebar3 ct

The Docker scripts under docker/scripts/ exercise multi-node clusters. See run the tests for the full command list.

API reference

Generate HTML documentation:

rebar3 ex_doc

Versioning

Barrel P2P is still in 0.x.

Minor bumps, 0.x to 0.y, may change documented public APIs.
Patch bumps, 0.x.y to 0.x.y+1, are non-breaking.
1.0 will come after an external audit of the dist auth and transport layers, and after public user feedback on a 0.x release.

Public API stability tiers are tracked in docs/features.md. Anything not listed there is internal.

License

Apache-2.0

Next Page → Overview