Specs

start_link(module(), keyword()) :: on_start()

Starts a Broadway process linked to the current process.

• module is the module implementing the Broadway behaviour.

Options

In order to set up how the pipeline created by Broadway should work, you need to specify the blueprint of the pipeline. You can do this by passing a set of options to start_link/2. Each component of the pipeline has its own set of options.

The broadway options are:

• :name - Required. Used for name registration. All processes/stages created will be named using this value as prefix.

• :shutdown - Optional. The time in milliseconds given for Broadway to gracefully shutdown without discarding events. Defaults to 30_000(ms). The default value is 30000.

• :max_restarts - The default value is 3.

• :max_seconds - The default value is 5.

• :resubscribe_interval - The interval in milliseconds that processors wait until they resubscribe to a failed producers. Defaults to 100(ms). The default value is 100.

• :context - A user defined data structure that will be passed to handle_message/3 and handle_batch/4. The default value is :context_not_set.

• :producer - Required. A keyword list of options. See "Producers options" section below. Only a single producer is allowed.

• :processors - Required. A keyword list of named processors where the key is an atom as identifier and the value is another keyword list of options. See "Processors options" section below. Currently only a single processor is allowed.

• :batchers - A keyword list of named batchers where the key is an atom as identifier and the value is another keyword list of options. See "Batchers options" section below. The default value is [].

• :partition_by - A function that controls how data is partitioned across all processors and batchers. It receives a Broadway.Message and it must return a non-negative integer, starting with zero, that will be mapped to one of the existing processors. See "Ordering and Partitioning" in the module docs for more information.

• :spawn_opt - Low-level options given when starting a process. Applies to producers, processors, and batchers. See erlang:spawn_opt/2 for more information.

• :hibernate_after - If a process does not receive any message within this interval, it will hibernate, compacting memory. Applies to producers, processors, and batchers. Defaults to 15_000(ms). The default value is 15000.

Producers options

The producer options allow users to set up the producer.

The available options are:

• :module - Required. A tuple representing a GenStage producer. The tuple format should be {mod, arg}, where mod is the module that implements the GenStage behaviour and arg the argument that will be passed to the init/1 callback of the producer. See Broadway.Producer for more information.

• :concurrency - The number of concurrent producers that will be started by Broadway. Use this option to control the concurrency level of each set of producers. The default value is 1. The default value is 1.

• :transformer - A tuple representing a transformer that translates a produced GenStage event into a %Broadway.Message{}. The tuple format should be {mod, fun, opts} and the function should have the following spec (event :: term, opts :: term) :: Broadway.Message.t This function must be used sparingly and exclusively to convert regular messages into Broadway.Message. That's because a failure in the :transformer callback will cause the whole producer to terminate, possibly leaving unacknowledged messages along the way. The default value is nil.

• :spawn_opt - Overrides the top-level :spawn_opt.

• :hibernate_after - Overrides the top-level :hibernate_after.

• :rate_limiting - A list of options to enable and configure rate limiting for producing. If this option is present, rate limiting is enabled, otherwise it isn't. Rate limiting refers to the rate at which producers will forward messages to the rest of the pipeline. The rate limiting is applied to and shared by all producers within the time limit. The following options are supported:

  • :allowed_messages - Required. An integer that describes how many messages are allowed in the specified interval.
  • :interval - Required. An integer that describes the interval (in milliseconds)

during which the number of allowed messages is allowed. If the producer produces more than allowed_messages in interval, only allowed_messages will be published until the end of interval, and then more messages will be published.

Processors options

• :concurrency - The number of concurrent process that will be started by Broadway. Use this option to control the concurrency level of the processors. The default value is System.schedulers_online() * 2.

• :min_demand - Set the minimum demand of all processors stages. Default value is 5.

• :max_demand - Set the maximum demand of all processors stages. Default value is 10. The default value is 10.

• :partition_by - Overrides the top-level :partition_by.

• :spawn_opt - Overrides the top-level :spawn_opt.

• :hibernate_after - Overrides the top-level :hibernate_after.

Batchers options

• :concurrency - The number of concurrent batch processors that will be started by Broadway. Use this option to control the concurrency level. Note that this only sets the numbers of batch processors for each batcher group, not the number of batchers. The number of batchers will always be one for each batcher key defined. The default value is 1. The default value is 1.

• :batch_size - The size of the generated batches. Default value is 100. The default value is 100.

• :batch_timeout - The time, in milliseconds, that the batcher waits before flushing the list of messages. When this timeout is reached, a new batch is generated and sent downstream, no matter if the :batch_size has been reached or not. Default value is 1000 (1 second). The default value is 1000.

• :partition_by - Optional. Overrides the top-level :partition_by.

• :spawn_opt - Overrides the top-level :spawn_opt.

• :hibernate_after - Overrides the top-level :hibernate_after.