Snakepit Architecture

Overview

Snakepit is a high-performance Python bridge for Elixir that enables seamless execution of Python code from Elixir applications. It uses a pure gRPC-based architecture with stateless Python workers and centralized session management in Elixir.

High-Level Architecture

┌─────────────────────────────────────────────────────────────────────┐
│                          Elixir Application                         │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│  ┌──────────────┐     ┌──────────────────┐    ┌────────────────┐  │
│  │    Pool      │────▶│ WorkerSupervisor │───▶│ Worker.Starter │  │
│  │  (GenServer) │     │ (DynamicSupervisor)   │  (Supervisor)  │  │
│  └──────────────┘     └──────────────────┘    └────────────────┘  │
│         │                                               │           │
│         │                                               ▼           │
│         │              ┌────────────────┐       ┌──────────────┐  │
│         │              │  SessionStore  │       │  GRPCWorker  │  │
│         └─────────────▶│  (GenServer)   │       │ (GenServer)  │  │
│                        │  + ETS Table   │       └──────────────┘  │
│                        └────────────────┘              │           │
│                                                         │ gRPC      │
└─────────────────────────────────────────────────────────┼───────────┘
                                                          │
┌─────────────────────────────────────────────────────────┼───────────┐
│                        Python Worker                    │           │
│                                                         ▼           │
│  ┌──────────────────┐    ┌──────────────────┐  ┌──────────────┐  │
│  │  grpc_server.py  │────│ SessionContext   │──│    Types &    │  │
│  │ (gRPC Service)   │    │ (Cache + Client) │  │ Serialization │  │
│  └──────────────────┘    └──────────────────┘  └──────────────┘  │
│           │                                                         │
│           ▼                                                         │
│  ┌──────────────────┐    ┌──────────────────┐                    │
│  │  User Adapter    │────│   User Tools     │                    │
│  │  (BaseAdapter)   │    │  (Custom Code)   │                    │
│  └──────────────────┘    └──────────────────┘                    │
└─────────────────────────────────────────────────────────────────────┘

Key Components

Elixir Side

Pool (lib/snakepit/pool/pool.ex)

  • Purpose: Manages a pool of Python workers for concurrent execution
  • Design: GenServer that maintains available/busy worker sets and a request queue
  • Features:
    • Concurrent worker startup
    • Session affinity (routes session requests to same worker when possible)
    • Automatic request queueing when all workers are busy
    • Non-blocking async execution using Task.Supervisor

WorkerSupervisor (lib/snakepit/pool/worker_supervisor.ex)

  • Purpose: DynamicSupervisor for managing worker lifecycle
  • Design: Starts Worker.Starter processes which in turn manage actual workers
  • Features: Provides clean separation between supervision and worker logic

Worker.Starter (lib/snakepit/pool/worker_starter.ex)

  • Purpose: Implements the "Permanent Wrapper" pattern for automatic worker restarts
  • Design: A permanent supervisor that manages a transient worker
  • Features:
    • Automatic restart of crashed workers without Pool intervention
    • Clean shutdown during application termination
    • Decouples Pool from worker replacement logic

GRPCWorker (lib/snakepit/grpc_worker.ex)

  • Purpose: Manages a single Python worker process
  • Design: GenServer that spawns and communicates with Python via gRPC
  • Features:
    • Port-based process management
    • Health checking
    • Automatic reconnection
    • Request timeout handling
    • Statistics tracking

SessionStore (lib/snakepit/bridge/session_store.ex)

  • Purpose: Centralized session and variable management
  • Design: GenServer backed by ETS table for high-performance concurrent access
  • Features:
    • TTL-based session expiration
    • Type-safe variable storage
    • Atomic operations
    • High-performance cleanup using ETS select_delete
    • Read-concurrency optimization

Python Side

grpc_server.py (priv/python/grpc_server.py)

  • Purpose: Main gRPC service implementation
  • Design: Stateless server that delegates to adapters and manages sessions
  • Features:
    • Unified protocol supporting both simple execution and session-based operations
    • Tool registration and execution
    • Streaming support
    • Comprehensive error handling

SessionContext (priv/python/snakepit_bridge/session_context.py)

  • Purpose: Client-side session state with intelligent caching
  • Design: Thread-safe context manager with TTL-based cache
  • Features:
    • Local variable cache to reduce gRPC round-trips
    • Automatic cache invalidation on TTL expiry
    • Lazy loading of variables
    • Batch operations support

Types & Serialization

  • Purpose: Consistent type system across Elixir and Python
  • Design: Centralized modules for type conversion and validation
  • Features:
    • Support for basic types (integer, float, string, boolean)
    • Complex types (list, map, embedding, tensor)
    • Special value handling (NaN, Infinity)
    • Binary data optimization

Design Principles

1. Stateless Python Workers

Python workers are completely stateless. All persistent state is managed by the Elixir SessionStore. This enables:

  • Easy horizontal scaling
  • Crash resilience
  • Simple worker replacement
  • No state synchronization issues

2. Centralized State Management

The SessionStore in Elixir is the single source of truth for all session state:

  • Variables are stored with type information
  • Sessions have TTL-based expiration
  • All operations are atomic
  • High concurrency via ETS

3. Performance Optimization

  • ETS for Storage: Read-concurrency optimized ETS tables for session data
  • Client-side Caching: Python SessionContext caches variables locally
  • Batch Operations: Support for bulk variable operations
  • Binary Protocol: gRPC with protobuf for efficient serialization

4. Fault Tolerance

  • Supervision Tree: Proper OTP supervision at every level
  • Process Monitoring: Multiple layers of process monitoring
  • Automatic Cleanup: ApplicationCleanup prevents orphaned processes
  • Health Checks: Periodic health monitoring of workers

Protocol Specification

The system uses a unified gRPC protocol defined in priv/proto/snakepit_bridge.proto:

Core Services

  • Execute: Simple command execution
  • ExecuteStream: Streaming command execution
  • Tool Operations: RegisterTool, ListTools, UnregisterTool
  • Session Operations: InitializeSession, CleanupSession
  • Variable Operations: RegisterVariable, GetVariable, SetVariable, etc.
  • Health & Info: Health checks and system information

Message Flow

  1. Client calls Pool with request
  2. Pool assigns available worker (preferring session affinity)
  3. Request forwarded to GRPCWorker
  4. GRPCWorker makes gRPC call to Python
  5. Python executes via adapter/tools
  6. For variables: Python may call back to Elixir SessionStore
  7. Response returned through the chain

Architecture Evolution

As of v0.4.0, Snakepit uses a unified gRPC-only architecture that provides:

  • Stateless Python workers with centralized SessionStore for state management
  • Binary gRPC protocol with protobuf for efficient communication
  • Intelligent routing with session affinity and multi-level caching
  • Native streaming support for real-time progress updates
  • Bidirectional tool execution between Elixir and Python

Binary Serialization

Overview

The architecture includes automatic binary serialization for efficient handling of large numerical data:

  • Threshold-based: Automatically switches to binary encoding for data > 10KB
  • Type-aware: Optimized for tensor and embedding types
  • Transparent: No API changes required - works automatically
  • Protocol: Uses Erlang Term Format (ETF) on Elixir side, Python pickle on Python side

Implementation Details

  1. Detection: Serialization.should_use_binary?/2 checks data size
  2. Encoding:
    • Small data: JSON via encode_as_json/2
    • Large data: Binary via encode_with_binary/2
  3. Transport: Binary data travels in separate protobuf fields
  4. Decoding: Automatic detection of binary format via type URL suffix

Performance Impact

  • 10x faster serialization for large tensors
  • 5x reduction in message size
  • Zero overhead for small data (still uses JSON)

Future Enhancements

The architecture is designed to support future features:

  • Distributed Sessions: SessionStore could be backed by distributed ETS/Mnesia
  • Multi-node Support: Workers could run on different nodes
  • Advanced Caching: Redis-backed caching for large datasets
  • Metrics & Tracing: OpenTelemetry integration end-to-end
  • Tool Marketplace: Dynamic tool loading from external sources
  • Compression: Optional compression for binary data
  • Custom Serializers: Pluggable serialization formats