Broadway.start_link
start_link
, go back to Broadway module for more information.
Specs
Starts a Broadway
process linked to the current process.
module
is the module implementing theBroadway
behaviour.
Options
In order to set up how the pipeline created by Broadway should work,
you need to specify the blueprint of the pipeline. You can
do this by passing a set of options to start_link/2
.
Each component of the pipeline has its own set of options.
The broadway options are:
:name
- Required. Used for name registration. All processes/stages created will be named using this value as prefix.:shutdown
- Optional. The time in milliseconds given for Broadway to gracefully shutdown without discarding events. Defaults to30_000
(ms). The default value is30000
.:max_restarts
- The default value is3
.:max_seconds
- The default value is5
.:resubscribe_interval
- The interval in milliseconds that processors wait until they resubscribe to a failed producers. Defaults to100
(ms). The default value is100
.:context
- A user defined data structure that will be passed to handle_message/3 and handle_batch/4. The default value is:context_not_set
.:producer
- Required. A keyword list of options. See "Producers options" section below. Only a single producer is allowed.:processors
- Required. A keyword list of named processors where the key is an atom as identifier and the value is another keyword list of options. See "Processors options" section below. Currently only a single processor is allowed.:batchers
- A keyword list of named batchers where the key is an atom as identifier and the value is another keyword list of options. See "Batchers options" section below. The default value is[]
.:partition_by
- A function that controls how data is partitioned across all processors and batchers. It receives aBroadway.Message
and it must return a non-negative integer, starting with zero, that will be mapped to one of the existing processors. See "Ordering and Partitioning" in the module docs for more information.:spawn_opt
- Low-level options given when starting a process. Applies to producers, processors, and batchers. Seeerlang:spawn_opt/2
for more information.:hibernate_after
- If a process does not receive any message within this interval, it will hibernate, compacting memory. Applies to producers, processors, and batchers. Defaults to15_000
(ms). The default value is15000
.
Producers options
The producer options allow users to set up the producer.
The available options are:
:module
- Required. A tuple representing a GenStage producer. The tuple format should be{mod, arg}
, wheremod
is the module that implements the GenStage behaviour andarg
the argument that will be passed to theinit/1
callback of the producer. SeeBroadway.Producer
for more information.:concurrency
- The number of concurrent producers that will be started by Broadway. Use this option to control the concurrency level of each set of producers. The default value is1
. The default value is1
.:transformer
- A tuple representing a transformer that translates a produced GenStage event into a%Broadway.Message{}
. The tuple format should be{mod, fun, opts}
and the function should have the following spec(event :: term, opts :: term) :: Broadway.Message.t
This function must be used sparingly and exclusively to convert regular messages intoBroadway.Message
. That's because a failure in the:transformer
callback will cause the whole producer to terminate, possibly leaving unacknowledged messages along the way. The default value isnil
.:spawn_opt
- Overrides the top-level:spawn_opt
.:hibernate_after
- Overrides the top-level:hibernate_after
.:rate_limiting
- A list of options to enable and configure rate limiting for producing. If this option is present, rate limiting is enabled, otherwise it isn't. Rate limiting refers to the rate at which producers will forward messages to the rest of the pipeline. The rate limiting is applied to and shared by all producers within the time limit. The following options are supported::allowed_messages
- Required. An integer that describes how many messages are allowed in the specified interval.:interval
- Required. An integer that describes the interval (in milliseconds)
during which the number of allowed messages is allowed.
If the producer produces more than allowed_messages
in interval
, only allowed_messages
will be published until
the end of interval
, and then more messages will be published.
Processors options
:concurrency
- The number of concurrent process that will be started by Broadway. Use this option to control the concurrency level of the processors. The default value isSystem.schedulers_online() * 2
.:min_demand
- Set the minimum demand of all processors stages. Default value is5
.:max_demand
- Set the maximum demand of all processors stages. Default value is10
. The default value is10
.:partition_by
- Overrides the top-level:partition_by
.:spawn_opt
- Overrides the top-level:spawn_opt
.:hibernate_after
- Overrides the top-level:hibernate_after
.
Batchers options
:concurrency
- The number of concurrent batch processors that will be started by Broadway. Use this option to control the concurrency level. Note that this only sets the numbers of batch processors for each batcher group, not the number of batchers. The number of batchers will always be one for each batcher key defined. The default value is1
. The default value is1
.:batch_size
- The size of the generated batches. Default value is100
. The default value is100
.:batch_timeout
- The time, in milliseconds, that the batcher waits before flushing the list of messages. When this timeout is reached, a new batch is generated and sent downstream, no matter if the:batch_size
has been reached or not. Default value is1000
(1 second). The default value is1000
.:partition_by
- Optional. Overrides the top-level:partition_by
.:spawn_opt
- Overrides the top-level:spawn_opt
.:hibernate_after
- Overrides the top-level:hibernate_after
.