Oban isn't an application and won't be started automatically. It is started by a supervisor that must be included in your application's supervision tree. All of your configuration is passed into the Oban supervisor, allowing you to configure Oban like the rest of your application.

# confg/config.exs
config :my_app, Oban,
  repo: MyApp.Repo,
  queues: [default: 10, events: 50, media: 20]

# lib/my_app/application.ex
defmodule MyApp.Application do
  @moduledoc false

  use Application

  alias MyApp.{Endpoint, Repo}

  def start(_type, _args) do
    children = [
      Repo,
      Endpoint,
      {Oban, Application.get_env(:my_app, Oban)}
    ]

    Supervisor.start_link(children, strategy: :one_for_one, name: MyApp.Supervisor)
  end
end

Configuring Queues

Queues are specified as a keyword list where the key is the name of the queue and the value is the maximum number of concurrent jobs. The following configuration would start four queues with concurrency ranging from 5 to 50:

queues: [default: 10, mailers: 20, events: 50, media: 5]

There isn't a limit to the number of queues or how many jobs may execute concurrently. Here are a few caveats and guidelines:

• Each queue will run as many jobs as possible concurrently, up to the configured limit. Make sure your system has enough resources (i.e. database connections) to handle the concurrent load.
• Only jobs in the configured queues will execute. Jobs in any other queue will stay in the database untouched.
• Be careful how many concurrent jobs make expensive system calls (i.e. FFMpeg, ImageMagick). The BEAM ensures that the system stays responsive under load, but those guarantees don't apply when using ports or shelling out commands.

Defining Workers

Worker modules do the work of processing a job. At a minimum they must define a perform/2 function, which is called first with the full Oban.Job struct, and subsequently with the args map if no clause matches.

Define a worker to process jobs in the events queue:

defmodule MyApp.Business do
  use Oban.Worker, queue: "events", max_attempts: 10

  @impl Worker
  def perform(%{"id" => id}, _job) do
    model = MyApp.Repo.get(MyApp.Business.Man, id)

    IO.inspect(model)
  end
end

The value returned from perform/2 is ignored, unless it an {:error, reason} tuple. With an error return or when perform has an uncaught exception or throw then the error will be reported and the job will be retried (provided there are attempts remaining).

See Oban.Worker for more details on failure conditions and Oban.Telemetry for details on job reporting.

Enqueueing Jobs

Jobs are simply Ecto structs and are enqueued by inserting them into the database. For convenience and consistency all workers provide a new/2 function that converts an args map into a job changeset suitable for insertion:

%{in_the: "business", of_doing: "business"}
|> MyApp.Business.new()
|> Oban.insert()

The worker's defaults may be overridden by passing options:

%{vote_for: "none of the above"}
|> MyApp.Business.new(queue: "special", max_attempts: 5)
|> Oban.insert()

Jobs may be scheduled at a specific datetime in the future:

%{id: 1}
|> MyApp.Business.new(scheduled_at: ~U[2020-12-25 19:00:56.0Z])
|> Oban.insert()

Jobs may also be scheduled down to the second any time in the future:

%{id: 1}
|> MyApp.Business.new(schedule_in: 5)
|> Oban.insert()

Unique jobs can be configured in the worker, or when the job is built:

%{email: "brewster@example.com"}
|> MyApp.Mailer.new(unique: [period: 300, fields: [:queue, :worker])
|> Oban.insert()

Multiple jobs can be inserted in a single transaction:

Ecto.Multi.new()
|> Oban.insert(:b_job, MyApp.Business.new(%{id: 1}))
|> Oban.insert(:m_job, MyApp.Mailer.new(%{email: "brewser@example.com"}))
|> Repo.transaction()

Occasionally you may need to insert a job for a worker that exists in another application. In that case you can use Oban.Job.new/2 to build the changeset manually:

%{id: 1, user_id: 2}
|> Oban.Job.new(queue: :default, worker: OtherApp.Worker)
|> Oban.insert()

Oban.insert/2,4 is the preferred way of inserting jobs as it provides some of Oban's advanced features (i.e., unique jobs). However, you can use your application's Repo.insert/2 function if necessary.

See Oban.Job.new/2 for a full list of job options.

Unique Jobs

The unique jobs feature lets you specify constraints to prevent enqueuing duplicate jobs. Uniquness is based on a combination of args, queue, worker, state and insertion time. It is configured at the worker or job level using the following options:

• :period — The number of seconds until a job is no longer considered duplicate. You should always specify a period.
• :fields — The fields to compare when evaluating uniqueness. The available fields are :args, :queue and :worker, by default all three are used.
• :states — The job states that will be checked for duplicates. The available states are :available, :scheduled, :executing, :retryable and :completed. By default all states are checked, which prevents any duplicates, even if the previous job has been completed.

For example, configure a worker to be unique across all fields and states for 60 seconds:

use Oban.Worker, unique: [period: 60]

Configure the worker to be unique only by :worker and :queue:

use Oban.Worker, unique: [fields: [:queue, :worker], period: 60]

Or, configure a worker to be unique until it has executed:

use Oban.Worker, unique: [period: 300, states: [:available, :scheduled, :executing]]

Stronger Guarantees

Oban's unique job support is built on a client side read/write cycle. That makes it subject to duplicate writes if two transactions are started simultaneously. If you absolutely must ensure that a duplicate job isn't inserted then you will have to make use of unique constraints within the database. Oban.insert/2,4 will handle unique constraints safely through upsert support.

Performance Note

If your application makes heavy use of unique jobs you may want to add indexes on the args and inserted_at columns of the oban_jobs table. The other columns considered for uniqueness are already covered by indexes.

Testing

As noted in the Usage section above there are some guidelines for running tests:

• Disable all job dispatching by setting queues: false or queues: nil in your test.exs config. Keyword configuration is deep merged, so setting queues: [] won't have any effect.

• Disable pruning via prune: :disabled. Pruning isn't necessary in testing mode because jobs created within the sandbox are rolled back at the end of the test. Additionally, the periodic pruning queries will raise DBConnection.OwnershipError when the application boots.

• Be sure to use the Ecto Sandbox for testing. Oban makes use of database pubsub events to dispatch jobs, but pubsub events never fire within a transaction. Since sandbox tests run within a transaction no events will fire and jobs won't be dispatched.

  config :my_app, MyApp.Repo, pool: Ecto.Adapters.SQL.Sandbox

Oban provides some helpers to facilitate testing. The helpers handle the boilerplate of making assertions on which jobs are enqueued. To use the assert_enqueued/1 and refute_enqueued/1 helpers in your tests you must include them in your testing module and specify your app's Ecto repo:

use Oban.Testing, repo: MyApp.Repo

Now you can assert, refute or list jobs that have been enqueued within your tests:

assert_enqueued worker: MyWorker, args: %{id: 1}

# or

refute_enqueued queue: "special", args: %{id: 2}

# or

assert [%{args: %{"id" => 1}}] = all_enqueued worker: MyWorker

See the Oban.Testing module for more details on making assertions.

Integration Testing

During integration testing it may be necessary to run jobs because they do work essential for the test to complete, i.e. sending an email, processing media, etc. You can execute all available jobs in a particular queue by calling Oban.drain_queue/1 directly from your tests.

For example, to process all pending jobs in the "mailer" queue while testing some business logic:

defmodule MyApp.BusinessTest do
  use MyApp.DataCase, async: true

  alias MyApp.{Business, Worker}

  test "we stay in the business of doing business" do
    :ok = Business.schedule_a_meeting(%{email: "monty@brewster.com"})

    assert %{success: 1, failure: 0} == Oban.drain_queue(:mailer)

    # Make an assertion about the email delivery
  end
end

See Oban.drain_queue/1 for additional details.

Error Handling

When a job returns an error value, raises an error or exits during execution the details are recorded within the errors array on the job. When the number of execution attempts is below the configured max_attempts limit, the job will automatically be retried in the future.

The retry delay has an exponential backoff, meaning the job's second attempt will be after 16s, third after 31s, fourth after 1m 36s, etc.

See the Oban.Worker documentation on "Customizing Backoff" for alternative backoff strategies.

Error Details

Execution errors are stored as a formatted exception along with metadata about when the failure ocurred and which attempt caused it. Each error is stored with the following keys:

• at The utc timestamp when the error occurred at
• attempt The attempt number when the error ocurred
• error A formatted error message and stacktrace

See the Instrumentation docs below for an example of integrating with external error reporting systems.

Limiting Retries

By default jobs will be retried up to 20 times. The number of retries is controlled by the max_attempts value, which can be set at the Worker or Job level. For example, to instruct a worker to discard jobs after three failures:

use Oban.Worker, queue: "limited", max_attempts: 3

Pruning Historic Jobs

Job stats and queue introspection is built on keeping job rows in the database after they have completed. This allows administrators to review completed jobs and build informative aggregates, but at the expense of storage and an unbounded table size. To prevent the oban_jobs table from growing indefinitely, Oban provides active pruning of completed jobs.

By default, pruning is disabled. To enable pruning we configure a supervision tree with the :prune option. There are three distinct modes of pruning:

• :disabled - This is the default, where no pruning happens at all
• {:maxlen, count} - Pruning is based on the number of rows in the table, any rows beyond the configured count will be deleted
• {:maxage, seconds} - Pruning is based on a row's age, any rows older than the configured number of seconds will be deleted. The age unit is always specified in seconds, but values on the scale of days, weeks or months are perfectly acceptable.

Pruning is best-effort and performed out-of-band. This means that all limits are soft; jobs beyond a specified length or age may not be pruned immediately after jobs complete. Prune timing is based on the configured prune_interval, which is one minute by default.

Only jobs in a completed or discarded state will be deleted. Currently executing jobs and older jobs that are still in the available state will be retained.

Instrumentation & Logging

Oban provides integration with Telemetry, a dispatching library for metrics. It is easy to report Oban metrics to any backend by attaching to :oban events.

For instructions on using the default structured logger and information on event metadata see docs for the Oban.Telemetry module.

Prefix Support

Oban supports namespacing through PostgreSQL schemas, also called "prefixes" in Ecto. With prefixes your jobs table can reside outside of your primary schema (usually public) and you can have multiple separate job tables.

To use a prefix you first have to specify it within your migration:

defmodule MyApp.Repo.Migrations.AddPrefixedObanJobsTable do
  use Ecto.Migration

  def up do
    Oban.Migrations.up(prefix: "private")
  end

  def down do
    Oban.Migrations.down(prefix: "private")
  end
end

The migration will create the "private" schema and all tables, functions and triggers within that schema. With the database migrated you'll then specify the prefix in your configuration:

config :my_app, Oban,
  prefix: "private",
  repo: MyApp.Repo,
  queues: [default: 10]

Now all jobs are inserted and executed using the private.oban_jobs table. Note that Oban.insert/2,4 will write jobs in the private.oban_jobs table, you'll need to specify a prefix manually if you insert jobs directly through a repo.

Isolation

Not only is the oban_jobs table isolated within the schema, but all notification events are also isolated. That means that insert/update events will only dispatch new jobs for their prefix. You can run multiple Oban instances with different prefixes on the same system and have them entirely isolated, provided you give each supervisor a distinct id.

Here we configure our application to start three Oban supervisors using the "public", "special" and "private" prefixes, respectively:

def start(_type, _args) do
  children = [
    Repo,
    Endpoint,
    Supervisor.child_spec({Oban, name: ObanA, repo: Repo}, id: ObanA),
    Supervisor.child_spec({Oban, name: ObanB, repo: Repo, prefix: "special"}, id: ObanB),
    Supervisor.child_spec({Oban, name: ObanC, repo: Repo, prefix: "private"}, id: ObanC)
  ]

  Supervisor.start_link(children, strategy: :one_for_one, name: MyApp.Supervisor)
end