Introduction
Broadway is a library for building concurrent and multi-stage data ingestion and data processing pipelines with Elixir. Broadway pipelines are concurrent and robust, thanks to the Erlang VM and its actors. It features:
- Back-pressure
- Automatic acknowledgements at the end of the pipeline
- Batching
- Fault tolerance
- Graceful shutdown
- Built-in testing
- Custom failure handling
- Ordering and partitioning
- Rate-limiting
- Metrics
Official Producers
Currently we officially support four Broadway producers:
- Amazon SQS: Source - Guide
- Apache Kafka: Source - Guide
- Google Cloud Pub/Sub: Source - Guide
- RabbitMQ: Source - Guide
The guides links above will help you get started with your adapter of choice. For API reference, you can check out the Broadway module.
Non-official (Off-Broadway) Producers
For those interested in rolling their own Broadway Producers (which we actively encourage!), we recommend using the OffBroadway namespace, mirroring the Off-Broadway theaters. For example, if you want to publish your own integration with Amazon SQS, you can package it as off_broadway_sqs, which uses the OffBroadway.SQS namespace.
The following Off-Broadway libraries are available (feel free to send a PR adding your own in alphabetical order):