Oban
Oban v2.11.0

View Source Reprioritizer Plugin

🌟 This plugin is available through Oban.Pro

Using mixed priorities in a queue ensures that certain jobs are executed before others. For example, a queue that processes jobs from various customers may prioritize customers that are in a higher tier or plan. All high priority (0) jobs are guaranteed to run before any with lower priority (1..3), which is wonderful for the higher tier customers but can lead to resource starvation. When there is a constant flow of high priority jobs the lower priority jobs will never get the chance to run.

The Reprioritizer plugin automatically adjusts lower job's priorities so that all jobs are eventually processed.

Using and Configuring

To use the Reprioritizer plugin add the module to your list of Oban plugins in config.exs:

config :my_app, Oban,
  plugins: [Oban.Pro.Plugins.Reprioritizer]
  ...

Without any additional options the plugin will automatically increase the priority of any jobs that are available for 5 minutes or more. To reprioritize after less time waiting you can configure the :after time:

plugins: [{Oban.Pro.Plugins.Reprioritizer, after: :timer.minutes(2)}]

Now lower job priorities are bumped after 2 minutes of waiting, and every minute after that. To lower the reprioritization frequency you can change the :interval along with the :after time:

plugins: [{
  Oban.Pro.Plugins.Reprioritizer,
  after: :timer.minutes(10),
  interval: :timer.minutes(5)
}]

Here we've specified that job priority will increase every 5 minutes after the first 10 minutes of waiting.

Providing Overrides

The after option applies to jobs for any workers across all queues. The Reprioritizer plugin allows you to specify per-queue and per-worker overrides that fine tune reprioritization.

Let's configure reprioritization for the analysis queue so that it nudges jobs after only 1 minute:

plugins: [{
  Oban.Pro.Plugins.Reprioritizer,
  queue_overrides: [analysis: :timer.minutes(1)]
}]

We can also effectively disable reprioritization for all other queues by setting the period to :infinity:

plugins: [{
  Oban.Pro.Plugins.Reprioritizer,
  after: :infinity,
  queue_overrides: [analysis: :timer.minutes(1)]
}]

If per-queue overrides aren't enough we can override on a per-worker basis instead:

plugins: [{
  Oban.Pro.Plugins.Reprioritizer,
  interval: :timer.seconds(15),
  worker_overrides: [
    "MyApp.HighSLAWorker": :timer.seconds(30),
    "MyApp.LowSLAWorker": :timer.minutes(10)
  ]
}]

Naturally you can mix and match overrides to finely control reprioritization:

plugins: [{
  Oban.Pro.Plugins.Reprioritizer,
  interval: :timer.minutes(2),
  after: :timer.minutes(5),
  queue_overrides: [media: :timer.minutes(10)],
  worker_overrides: ["MyApp.HighSLAWorker": :timer.seconds(30)]
}]

Instrumenting with Telemetry

The Reprioritizer plugin adds the following metadata to the [:oban, :plugin, :stop] event:

  • :reprioritized_count — the number of jobs reprioritized

See the docs on Plugin Events for details.