Worker lifecycle manager for automatic recycling and health monitoring.

Manages worker lifecycle events:

• TTL-based recycling: Recycle workers after configured time
• Request-count recycling: Recycle after N requests
• Memory monitoring: Recycle when the BEAM worker process exceeds a configurable threshold (optional)
• Health checks: Monitor worker health and restart if needed

Why Worker Recycling?

Long-running Python processes can accumulate memory due to:

• Memory fragmentation
• Cache growth
• Subtle memory leaks in C libraries
• ML model weight accumulation

Automatic recycling prevents these issues from impacting production. The current implementation samples the BEAM Snakepit.GRPCWorker process memory via :get_memory_usage; Python child process memory is not yet measured directly.

Configuration

config :snakepit,
  pools: [
    %{
      name: :hpc_pool,
      worker_profile: :thread,
      worker_ttl: {3600, :seconds},      # Recycle hourly
      worker_max_requests: 1000,          # Or after 1000 requests
      memory_threshold_mb: 2048           # Or at 2GB (optional)
    }
  ]

Usage

The LifecycleManager runs automatically when started in the supervision tree. It monitors all workers across all pools.

# Manual worker recycling
Snakepit.Worker.LifecycleManager.recycle_worker(pool_name, worker_id)

# Get lifecycle statistics
Snakepit.Worker.LifecycleManager.get_stats()

Implementation

• Runs periodic health checks (every 60 seconds)
• Tracks worker metadata (start time, request count)
• Gracefully replaces workers when recycling
• Emits telemetry events for monitoring