ExESDB - A BEAM-native Event Store

ExESDB is a BEAM-native Event Store, built on top of the khepri and ra subsystems.

Motivation

One of the arguments for BEAM development is that it comes "batteries included". Be it caching, storage, pub/sub, observability etc... the Erlang ecosystem always has the option to avoid external dependencies.

For Event Sourcing use cases however, the Event Store is often a separate service.

This project is an attempt at addressing this point, by building further upon the work of the rabbitmq/khepri and rabbitmq/ra subsystems.

Features

ExESDB is a distributed, BEAM-native Event Store that provides high-availability event sourcing capabilities with automatic cluster formation and coordination. Built on top of Khepri and Ra (Raft consensus), it offers enterprise-grade reliability and performance.

Core Event Store Functionality

Event Stream Management

Stream Creation: Automatic stream creation on first event append
Event Appending: Atomic append operations with optimistic concurrency control
Event Retrieval: Query events with forward/backward traversal support
Stream Versioning: Track stream versions for conflict detection and resolution
Stream Listing: Enumerate all streams in the store

Event Storage

Persistent Storage: Durable event storage using Khepri's distributed key-value store
ACID Compliance: Atomic, consistent, isolated, and durable operations
Conflict Resolution: Built-in optimistic concurrency control
Data Integrity: Checksum validation and corruption detection

Subscription System

Multiple Subscription Types:
- :by_stream - Subscribe to specific event streams
- :by_event_type - Subscribe to events by type classification
- :by_event_pattern - Pattern-based event matching
- :by_event_payload - Content-based subscription filtering
Persistent Subscriptions: Durable subscriptions that survive node restarts
Transient Subscriptions: Temporary subscriptions for real-time processing
Event Replay: Start subscriptions from any stream version
Acknowledgment System: Reliable event delivery with ACK/NACK support
"Follow-the-Leader": Subscription processes automatically migrate to cluster leader

Snapshot Management

Aggregate Snapshots: Store and retrieve aggregate state snapshots
Version-based Snapshots: Snapshots tied to specific stream versions
Snapshot Lifecycle: Create, read, update, and delete snapshot operations
Performance Optimization: Reduce replay time for large aggregates
Distributed Storage: Snapshots stored across the cluster for availability

Distributed Architecture & Clustering

LibCluster Integration

ExESDB uses LibCluster for automatic cluster discovery and formation:

Strategy: Gossip-based multicast discovery
Protocol: UDP multicast on configurable port (default: 45892)
Network: Automatic node discovery on shared networks
Security: Shared secret authentication for cluster membership
Broadcast Discovery: Configurable multicast addressing

Cluster Formation Process

Initialization: Node starts and initializes LibCluster topology
Discovery: Uses gossip multicast to discover peer nodes
Authentication: Validates cluster membership using shared secrets
Coordination: ClusterCoordinator manages join/leave operations
Consensus: Khepri cluster formation using Raft consensus
Monitoring: Continuous health monitoring and leader election

High Availability Features

Automatic Clustering: Nodes automatically discover and join clusters
Split-Brain Prevention: ClusterCoordinator prevents network partition issues
Leader Election: Automatic leader election using Raft consensus
Failover: Seamless handling of node failures
Data Replication: Events replicated across cluster nodes
Consensus Protocol: Ra/Raft ensures data consistency

Storage Engine

Khepri Integration

Distributed Tree Store: Hierarchical key-value storage
MVCC: Multi-version concurrency control
Transactions: ACID transaction support
Schema Evolution: Support for data structure changes
Triggers: Event-driven data processing

Ra (Raft) Consensus

Strong Consistency: Linearizable read/write operations
Partition Tolerance: Operates correctly during network partitions
Leader-based Replication: Single leader for write operations
Log Compaction: Automatic cleanup of old log entries
Snapshot Support: Efficient state transfer for new nodes

Configuration & Deployment

Environment Configuration

EX_ESDB_STORE_ID: Unique identifier for the store instance
EX_ESDB_DB_TYPE: Deployment type (:single or :cluster)
EX_ESDB_DATA_DIR: Data directory for persistent storage
EX_ESDB_TIMEOUT: Operation timeout configuration
EX_ESDB_CLUSTER_SECRET: Shared secret for cluster authentication
EX_ESDB_COOKIE: Erlang distribution cookie
EX_ESDB_PUB_SUB: PubSub configuration for event broadcasting

LibCluster Configuration

config :libcluster,
  topologies: [
    ex_esdb_cluster: [
      strategy: Cluster.Strategy.Gossip,
      config: [
        port: 45_892,
        if_addr: "0.0.0.0",
        multicast_addr: "255.255.255.255",
        broadcast_only: true,
        secret: System.get_env("EX_ESDB_CLUSTER_SECRET")
      ]
    ]
  ]

Gateway API Integration

High-Availability Proxy

Load Balancing: Distribute requests across gateway workers
Service Discovery: Automatic discovery of available gateway workers
Fault Tolerance: Handle worker failures gracefully
Request Routing: Smart routing based on operation type

Worker Distribution

Swarm Integration: Distributed worker management
Process Migration: Workers can move between cluster nodes
Resource Management: Efficient resource utilization across cluster
Monitoring: Real-time worker health and performance tracking

Operational Features

Monitoring & Observability

Cluster Status: Real-time cluster membership and health
Leader Tracking: Monitor current cluster leader
Performance Metrics: Operation latency and throughput
Error Tracking: Comprehensive error logging and reporting
Health Checks: Built-in health check endpoints

Development Tools

Cluster Manager: Interactive cluster management script
Docker Compose: Multi-node development environment
Chaos Engineering: Built-in chaos testing capabilities
Validation Scripts: Automated cluster validation tools

Network Topology

┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   ExESDB Node   │    │   ExESDB Node   │    │   ExESDB Node   │
│    (Leader)     │◄──►│   (Follower)    │◄──►│   (Follower)    │
│   Khepri + Ra   │    │   Khepri + Ra   │    │   Khepri + Ra   │
└─────────────────┘    └─────────────────┘    └─────────────────┘
         ▲                       ▲                       ▲
         │                       │                       │
         │     Gossip Multicast Network (UDP:45892)     │
         │                       │                       │
         ▼                       ▼                       ▼
    Raft Consensus           Event Storage           Subscription
     & Replication          & Retrieval             Management

Deployment Scenarios

Single Node Deployment

Development: Local development and testing
Small Applications: Simple event sourcing needs
Embedded Usage: Integration within existing applications

Multi-Node Cluster

Production: High-availability production deployments
Horizontal Scaling: Scale read/write capacity
Geographic Distribution: Multi-region deployments
Fault Tolerance: Survive individual node failures

Container Orchestration

Docker Compose: Development and testing environments
Kubernetes: Production container orchestration
Docker Swarm: Simplified container clustering
Health Checks: Container-level health monitoring

Performance Characteristics

Throughput

Write Performance: Optimized for high-volume event appending
Read Performance: Efficient event retrieval and streaming
Concurrent Operations: Handle multiple simultaneous operations
Batch Processing: Support for batch event operations

Scalability

Horizontal Scaling: Add nodes to increase capacity
Storage Scalability: Distributed storage across cluster
Subscription Scaling: Distribute subscription load
Resource Utilization: Efficient use of available resources

Integration Capabilities

BEAM Ecosystem

Phoenix Integration: Real-time web applications
LiveView Support: Real-time UI updates
GenServer Integration: Native BEAM process integration
OTP Supervision: Fault-tolerant supervision trees

External Systems

REST APIs: HTTP-based integration
Message Queues: Integration with external queuing systems
Databases: Projection and read model support
Monitoring Systems: Metrics and alerting integration

Installation

Docker Installation

ExESDB is available as a Docker image on Docker Hub with automatic versioning based on the mix.exs version.

Available Tags

beamcampus/ex_esdb:latest - Latest build from master branch
beamcampus/ex_esdb:0.0.18 - Specific version (current version)
beamcampus/ex_esdb:0.0.x - Any specific version tag

Quick Start

Single Node:

docker run -d \
  --name ex-esdb \
  -p 4369:4369 \
  -p 9000-9100:9000-9100 \
  -p 45892:45892/udp \
  -e EX_ESDB_STORE_ID="my-store" \
  -e EX_ESDB_DB_TYPE="single" \
  -e EX_ESDB_DATA_DIR="/data" \
  -v ex-esdb-data:/data \
  beamcampus/ex_esdb:latest

Multi-Node Cluster:

# Node 1 (seed node)
docker run -d \
  --name ex-esdb-node1 \
  --network ex-esdb-net \
  -p 4369:4369 \
  -p 9001:9000 \
  -p 45892:45892/udp \
  -e EX_ESDB_STORE_ID="cluster-store" \
  -e EX_ESDB_DB_TYPE="cluster" \
  -e EX_ESDB_DATA_DIR="/data" \
  -e EX_ESDB_CLUSTER_SECRET="your-secret-key" \
  -e EX_ESDB_COOKIE="your-erlang-cookie" \
  -v ex-esdb-node1-data:/data \
  beamcampus/ex_esdb:latest

# Node 2
docker run -d \
  --name ex-esdb-node2 \
  --network ex-esdb-net \
  -p 9002:9000 \
  -e EX_ESDB_STORE_ID="cluster-store" \
  -e EX_ESDB_DB_TYPE="cluster" \
  -e EX_ESDB_DATA_DIR="/data" \
  -e EX_ESDB_CLUSTER_SECRET="your-secret-key" \
  -e EX_ESDB_COOKIE="your-erlang-cookie" \
  -v ex-esdb-node2-data:/data \
  beamcampus/ex_esdb:latest

# Node 3
docker run -d \
  --name ex-esdb-node3 \
  --network ex-esdb-net \
  -p 9003:9000 \
  -e EX_ESDB_STORE_ID="cluster-store" \
  -e EX_ESDB_DB_TYPE="cluster" \
  -e EX_ESDB_DATA_DIR="/data" \
  -e EX_ESDB_CLUSTER_SECRET="your-secret-key" \
  -e EX_ESDB_COOKIE="your-erlang-cookie" \
  -v ex-esdb-node3-data:/data \
  beamcampus/ex_esdb:latest

Docker Compose

For development and testing, use the provided Docker Compose setup:

# Clone the repository
git clone https://github.com/beam-campus/ex-esdb.git
cd ex-esdb/dev-env

# Start a 3-node cluster
./start-core-only.sh

# Or use the interactive cluster manager
./ez-cluster.sh

The Docker Compose setup includes:

Core Cluster: 3-node ExESDB cluster (ex-esdb0, ex-esdb1, ex-esdb2)
Extended Tier: Additional 2 nodes (ex-esdb10, ex-esdb11)
Massive Tier: Additional 8 nodes (ex-esdb20-27)
Automatic Networking: Configured Docker networks for cluster communication
Data Persistence: Named volumes for data persistence
Health Checks: Built-in container health monitoring

Environment Variables

📌 Important: When using the dev-env Docker Compose configurations, you must export the EX_ESDB_COOKIE environment variable on your host machine. This single environment variable is used for all cluster authentication purposes (cookies, secrets, etc.).
export EX_ESDB_COOKIE="your-secure-cookie-value"

Variable	Description	Default	Required
`EX_ESDB_STORE_ID`	Unique store identifier	-	Yes
`EX_ESDB_DB_TYPE`	Deployment type (`single` or `cluster`)	`single`	No
`EX_ESDB_DATA_DIR`	Data directory path	`/data`	No
`EX_ESDB_TIMEOUT`	Operation timeout (ms)	`5000`	No
`EX_ESDB_CLUSTER_SECRET`	Cluster authentication secret	-	Yes (cluster)
`EX_ESDB_COOKIE`	Erlang distribution cookie	-	Yes (cluster)
`EX_ESDB_PUB_SUB`	PubSub process name	`:ex_esdb_pubsub`	No

Ports

Port	Protocol	Description
`4369`	TCP	EPMD (Erlang Port Mapper Daemon)
`9000-9100`	TCP	Erlang distribution ports
`45892`	UDP	LibCluster gossip multicast

Health Checks

The Docker image includes a built-in health check script:

# Check container health
docker exec ex-esdb ./check-ex-esdb.sh

# View health status
docker inspect --format='{{.State.Health.Status}}' ex-esdb

Production Considerations

Security: Use strong, unique values for EX_ESDB_CLUSTER_SECRET and EX_ESDB_COOKIE
Networking: Ensure proper firewall rules for cluster communication
Storage: Use named volumes or bind mounts for data persistence
Monitoring: Implement external monitoring for cluster health
Backups: Regular backup of data volumes
Resource Limits: Set appropriate CPU and memory limits

Hex Installation

ExESDB is also available as a Hex package for direct integration:

def deps do
  [
    {:ex_esdb, "~> 0.0.18"}
  ]
end

Getting Started

Releases

← Previous Page Configuring ExESDB Applications