Reprioritization Plugin

🌟 This plugin is available through Oban.Pro

Using mixed priorities in a queue ensures that certain jobs are executed before others. For example, a queue that processes jobs from various customers may prioritize customers that are in a higher tier or plan. All high priority (0) jobs are guaranteed to run before any with lower priority (1..3), which is wonderful for the higher tier customers but can lead to resource starvation. When there is a constant flow of high priority jobs the lower priority jobs will never get the chance to run.

The Reprioritization plugin automatically adjusts lower job's priorities so that all jobs are eventually processed.

Using and Configuring

To use the Reprioritization plugin add the module to your list of Oban plugins in config.exs:

config :my_app, Oban,
  plugins: [Oban.Pro.Plugins.Reprioritization]
  ...

Without any additional options the plugin will automatically increase the priority of any jobs that are available for 5 minutes or more. To reprioritize after less time waiting you can configure the :after time:

plugins: [{Oban.Pro.Plugins.Reprioritization, after: :timer.minutes(2)}]

Now lower job priorities are bumped after 2 minutes of waiting, and every minute after that. To lower the reprioritization frequency you can change the :interval along with the :after time:

plugins: [{
  Oban.Pro.Plugins.Reprioritization,
  after: :timer.minutes(10),
  interval: :timer.minutes(5)
}]

Here we've specified that job priority will increase every 5 minutes after the first 10 minutes of waiting.

Providing Overrides

The after option applies to jobs for any workers across all queues. The Reprioritizer plugin allows you to specify per-queue and per-worker overrides that fine tune reprioritization.

Let's configure reprioritization for the analysis queue so that it nudges jobs after only 1 minute:

plugins: [{
  Oban.Pro.Plugins.Reprioritization,
  queue_overrides: [analysis: :timer.minutes(1)]
}]

We can also effectively disable reprioritization for all other queues by setting the period to :infinity:

plugins: [{
  Oban.Pro.Plugins.Reprioritization,
  after: :infinity,
  queue_overrides: [analysis: :timer.minutes(1)]
}]

If per-queue overrides aren't enough we can override on a per-worker basis instead:

plugins: [{
  Oban.Pro.Plugins.Reprioritization,
  interval: :timer.seconds(15),
  worker_overrides: [
    "MyApp.HighSLAWorker": :timer.seconds(30),
    "MyApp.LowSLAWorker": :timer.minutes(10)
  ]
}]

Naturally you can mix and match overrides to finely control reprioritization:

plugins: [{
  Oban.Pro.Plugins.Reprioritization,
  interval: :timer.minutes(2),
  after: :timer.minutes(5),
  queue_overrides: [media: :timer.minutes(10)],
  worker_overrides: ["MyApp.HighSLAWorker": :timer.seconds(30)]
}]

Instrumenting with Telemetry

The Reprioritization plugin uses Oban.Telemetry.span/3 to emit the following telemetry events:

[:oban, :reprioritizer, :start] — emitted when reprioritization starts for the base case and any overrides
[:oban, :reprioritizer, :stop] — emitted when reprioritization finishes successfully
[:oban, :reprioritizer, :exception] — emitted when an exception occurs while reprioritizing

v2.0.0-rc.1

Reprioritization Plugin

Using and Configuring

Providing Overrides

Instrumenting with Telemetry