Dux.Remote.Coordinator (Dux v0.2.0)

Copy Markdown View Source

Orchestrates distributed query execution across a cluster of Dux workers.

The coordinator:

  1. Discovers available workers via :pg
  2. Partitions the data source across workers
  3. Fans out the pipeline to each worker via Worker.execute/2
  4. Merges partial results on the coordinator node

Usage

# Execute a pipeline across all available workers
result = Dux.Remote.Coordinator.execute(pipeline)

# Execute with specific workers
result = Dux.Remote.Coordinator.execute(pipeline, workers: [w1, w2, w3])

The result is a %Dux{} struct with the merged data.

Summary

Functions

execute(pipeline, opts \\ [])

Execute a %Dux{} pipeline across distributed workers.

fan_out_raw(pipeline, opts \\ [])

Execute a pipeline across workers, returning raw Arrow IPC binaries.

Functions

execute(pipeline, opts \\ [])

Execute a %Dux{} pipeline across distributed workers.

Partitions the source, fans out to workers, collects Arrow IPC results, and merges on the coordinator node.

Options

  • :workers — list of worker PIDs (default: all workers from :pg)
  • :timeout — per-worker timeout in ms (default: :infinity)
  • :strategy — partitioning strategy (default: :round_robin)

Returns a %Dux{} struct with the merged results.

fan_out_raw(pipeline, opts \\ [])

Execute a pipeline across workers, returning raw Arrow IPC binaries.

Useful when you want to handle merging yourself or stream results.