View Source Introduction

Broadway is a library for building concurrent and multi-stage data ingestion and data processing pipelines with Elixir. Broadway pipelines are concurrent and robust, thanks to the Erlang VM and its actors. It features:

  • Back-pressure
  • Automatic acknowledgements at the end of the pipeline
  • Batching
  • Fault tolerance
  • Graceful shutdown
  • Built-in testing
  • Custom failure handling
  • Ordering and partitioning
  • Rate-limiting
  • Metrics

Official Producers

Currently we officially support four Broadway producers:

The guides links above will help you get started with your adapter of choice. For API reference, you can check out the Broadway module.

Non-official (Off-Broadway) Producers

For those interested in rolling their own Broadway Producers (which we actively encourage!), we recommend using the OffBroadway namespace, mirroring the Off-Broadway theaters. For example, if you want to publish your own integration with Amazon SQS, you can package it as off_broadway_sqs, which uses the OffBroadway.SQS namespace.

The following Off-Broadway libraries are available (feel free to send a PR adding your own in alphabetical order):