View Source kpl_agg (erlmld v1.1.0)
Kinesis record aggregator.
Follows the KPL aggregated record format: https://github.com/awslabs/amazon-kinesis-producer/blob/master/aggregation-format.md
This is an Erlang port of the aggregation functionality from: https://pypi.python.org/pypi/aws_kinesis_agg/1.0.0
Creating a new aggregator:
Agg = kpl_agg:new()
Adding user records to an aggregator (the aggregator will emit an aggregated record when it is full):
case kpl_agg:add(Agg, Record) of {undefined, NewAgg} -> ... {FullAggRecord, NewAgg} -> ... end
You can also use kpl:add_all to add multiple records at once. ARecordis a {PartitionKey, Data} tuple or a {PartitionKey, Data, ExplicitHashKey} tuple.
Getting the current aggregated record (e.g. to get the last aggregated record when you have no more user records to add):
case kpl_agg:finish(Agg) of {undefined, Agg} -> ... {AggRecord, NewAgg} -> ... end
The result currently uses a non-standard magic prefix to prevent the KCL from deaggregating the record automatically. To use compression, instantiate the aggregator using kpl_agg:new(true), which uses another non-standard magic prefix.