View Source kpl_agg (erlmld v1.1.0)

Kinesis record aggregator.

Follows the KPL aggregated record format: https://github.com/awslabs/amazon-kinesis-producer/blob/master/aggregation-format.md

This is an Erlang port of the aggregation functionality from: https://pypi.python.org/pypi/aws_kinesis_agg/1.0.0

Creating a new aggregator:

Agg = kpl_agg:new()

Adding user records to an aggregator (the aggregator will emit an aggregated record when it is full):

case kpl_agg:add(Agg, Record) of {undefined, NewAgg} -> ... {FullAggRecord, NewAgg} -> ... end

You can also use kpl:add_all to add multiple records at once. A
Record

is a {PartitionKey, Data} tuple or a {PartitionKey, Data, ExplicitHashKey} tuple.

Getting the current aggregated record (e.g. to get the last aggregated record when you have no more user records to add):

case kpl_agg:finish(Agg) of {undefined, Agg} -> ... {AggRecord, NewAgg} -> ... end

The result currently uses a non-standard magic prefix to prevent the KCL from deaggregating the record automatically. To use compression, instantiate the aggregator using kpl_agg:new(true), which uses another non-standard magic prefix.

Link to this section Summary

Link to this section Functions