ExqBatch (exq_batch v0.1.3) View Source

ExqBatch provides a building block to create complex workflows using Exq jobs. A batch monitors a group of Exq jobs and creates callback job when all the jobs are processed.

Telemetry

The following telemetry events are published. progress event is published on each job completion (either dead or done). done is published when the batch is done (all the jobs are either dead or done).

[
  [:exq_batch, :batch, :new],
  [:exq_batch, :batch, :add],
  [:exq_batch, :batch, :create],
  [:exq_batch, :batch, :progress],
  [:exq_batch, :batch, :done]
]

Measurements

All the events include duration in native time unit. Note that duration here measures the time taken for ExqBatch to complete the operation.

Metadata

All the events include id attribute (batch id). add and progress events include jid attribute as well.

Link to this section Summary

Types

t()

Functions

add(batch, jid)

Add a job to the given batch.

create(batch)

Finalize the batch creation process.

new(options)

Initialize a new batch.

Link to this section Types

t()

Specs

t() :: %ExqBatch{
  id: String.t(),
  on_complete: term(),
  prefix: term(),
  redis: term(),
  ttl: term()
}

Link to this section Functions

add(batch, jid)

Specs

add(t(), String.t() | Keyword.t()) :: {:ok, t(), binary()} | {:error, term()}

Add a job to the given batch.

There are two ways to add a job to batch

Pass the job params and let ExqBatch enqueue the job. ExqBatch will both enqueue the job and add it to the batch using a atomic MULTI EXEC operation. Refer the on_complete option in new/1 for job options.
```
 {:ok, batch, jid} = ExqBatch.add(batch, queue: "default", class: Worker, args: [1])
```
Add a job using jid. Note that, the add/2 should be called before the job is enqueued. Otherwise, there is a potential race condition where the job could finish before add/2 is called and would cause the batch to hang. Exq allows to specify the jid of the job, so generate a jid first, then add it to the batch and after that enqueue the job.
```
 jid = UUID.uuid4()
 {:ok, batch, ^jid} = ExqBatch.add(batch, jid)
 {:ok, ^jid} = Exq.enqueue(Exq, "default", SuccessWorker, [1], jid: jid)
```

create(batch)

Specs

create(t()) :: {:ok, t()} | {:error, term()}

Finalize the batch creation process.

new(options)

Specs

new(Keyword.t()) :: {:ok, t()} | {:error, term()}

Initialize a new batch.

on_complete job will receive an extra arg which includes the list of dead and succeeded jids. Example %{"dead" => [], "succeeded" => [jid1, jid2]}

Options

on_complete (keyword) required - A Keyword list that specifies the details of job that will get enqueued on when all the jobs in a batch get completed
- queue (string) required - exq job queue
- args (array) required - exq job args.
- class (string) required - exq job class.
- retry (integer) - no of times the job should be retried.
- jid (string) - if not present, A UUID is used.
id (string) - A UUID is used by default. If same id is used for two batch jobs, the previous batch jobs will get cleared.