⛵ crew

Assemble a crew of worker processes to tackle your computational tasks.

A lightweight, generic worker/task pool for Gleam that distributes work across multiple managed worker processes. The pool manages a queue of work items and distributes them to idle workers; when no workers are available, work is queued until a worker becomes free.

Worker pools are named and can be used from multiple sources concurrently, making them useful for limiting concurrency globally in a system. crew offers blocking and non-blocking APIs for a variety of use-cases.

Installation

gleam add crew@1

Example

import crew
import gleam/erlang/process
import gleam/int
import gleam/list

pub fn main() {
  // Start a pool with 4 workers
  let pool_name = process.new_name("image_pool")
  let assert Ok(_) =
    crew.new(pool_name)
    |> crew.fixed_size(4)
    |> crew.start

  // Process images concurrently, but keeping the result order in sync
  let images =
    list.range(1, 20)
    |> list.map(fn(i) { "photo" <> int.to_string(i) <> ".jpg" })
    |> crew.parallel_map(pool_name, 6000, process_image)

  echo images
}

fn process_image(filename: String) -> String {
  // Simulate image processing work
  process.sleep(1000)
  "Processed: " <> filename
}

See also

This worker pool is inspired by poolboy and lifeguard, adapted for Gleam’s type system and actor model.