Elastic compute via FLAME for distributed Dux queries.
FLAME boots ephemeral cloud machines with a full copy of your application,
starts Dux.Remote.Worker on each, and auto-terminates when idle. No Docker
builds, no cluster management.
Quick start
# Start a FLAME pool (e.g. in Livebook)
Dux.Flame.start_pool(
backend: {FLAME.FlyBackend,
token: System.fetch_env!("FLY_API_TOKEN"),
cpus: 4,
memory_mb: 16_384},
max: 10
)
# Spin up workers and distribute
Dux.from_parquet("s3://bucket/data/**/*.parquet")
|> Dux.distribute(Dux.Flame.spin_up(5))
|> Dux.filter(amount > 100)
|> Dux.group_by(:region)
|> Dux.summarise(total: sum(amount))
|> Dux.to_rows()Workers read S3 data directly — nothing flows through your machine. After 5 minutes idle (configurable), machines auto-terminate.
Pools
You can run multiple pools for different workloads:
Dux.Flame.start_pool(name: Dux.CpuPool, backend: cpu_backend, max: 10)
Dux.Flame.start_pool(name: Dux.GpuPool, backend: gpu_backend, max: 4)
cpu_workers = Dux.Flame.spin_up(5, pool: Dux.CpuPool)
gpu_workers = Dux.Flame.spin_up(2, pool: Dux.GpuPool)Summary
Functions
Spin up n Dux workers on FLAME runners.
Start a FLAME pool for Dux workers.
Get status of the FLAME-backed Dux cluster.
Functions
Spin up n Dux workers on FLAME runners.
Each worker is placed on a separate FLAME runner via FLAME.place_child/3.
Returns a list of worker PIDs suitable for Dux.distribute/2.
Options
:pool— FLAME pool name (default:Dux.FlamePool)
Start a FLAME pool for Dux workers.
Options
:name— pool name (default:Dux.FlamePool):backend— FLAME backend (required). E.g.{FLAME.FlyBackend, token: ..., cpus: 4}:max— maximum number of runners (default: 10):min— minimum runners to keep warm (default: 0):idle_shutdown_after— ms before idle runner terminates (default: 5 minutes):boot_timeout— ms to wait for runner boot (default: 30 seconds)
Returns {:ok, pid} of the pool supervisor.
Get status of the FLAME-backed Dux cluster.
Returns worker count and PIDs, grouped by node.