Spin up Dux workers on FLAME runners for distributed queries.

FLAME handles pool management and machine lifecycle — see the FLAME docs for pool configuration. This module provides spin_up/2 to place Dux.Remote.Worker processes on FLAME runners, and status/1 to inspect the cluster.

Livebook

# 1. Start a FLAME pool (see FLAME docs for backend options)
Kino.start_child!(
  {FLAME.Pool,
    name: :dux_pool,
    code_sync: [
      start_apps: true,
      sync_beams: [Path.join(System.tmp_dir!(), "livebook_runtime")]
    ],
    min: 0,
    max: 10,
    max_concurrency: 1,
    backend: {FLAME.FlyBackend,
      cpu_kind: "performance", cpus: 4, memory_mb: 8192,
      token: System.fetch_env!("FLY_API_TOKEN"),
      env: %{"LIVEBOOK_COOKIE" => Atom.to_string(Node.get_cookie())}
    },
    boot_timeout: 120_000,
    idle_shutdown_after: :timer.minutes(5)}
)

# 2. Spin up workers and distribute
workers = Dux.Flame.spin_up(5, pool: :dux_pool)

Dux.from_parquet("s3://bucket/data/**/*.parquet")
|> Dux.distribute(workers)
|> Dux.filter(amount > 100)
|> Dux.group_by(:region)
|> Dux.summarise(total: sum(amount))
|> Dux.compute()

Deployed app

# In your application supervisor children:
{FLAME.Pool,
  name: :dux_pool,
  backend: {FLAME.FlyBackend, ...},
  max: 10,
  max_concurrency: 1,
  code_sync: [start_apps: [:dux], copy_apps: true],
  idle_shutdown_after: :timer.minutes(5)}

# Then at runtime:
workers = Dux.Flame.spin_up(5, pool: :dux_pool)

Workers read S3 data directly — nothing flows through your machine. After idle timeout, FLAME auto-terminates the runners.