DDSketch implementation in Erlang.

This implements an optimised pre-allocated bounded bucket count, that is, on a degenerate case, memory consumption won't grow but quantiles might lose accuracy as values escape bounds.

In this optimisation, buckets are preallocated using ets tables. Values within (0,1) will be inserted into the underflow bucket; values above the expected limit will be inserted into the overflow bucket.

Calculating the bucket size

If you have an expected highest value M and a given error E, your ideal bucket size would be given by the formula: ceil(math:log2(M) * (1.0 / math:log2((1 + E) / (1 - E)))).

For example, if you measure microseconds and you expect no operation to take more than an hour:

1> F = fun(M, E) -> ceil(math:log2(M) * (1.0 / math:log2((1 + E) / (1 - E)))) end.
2> F(3600000000, 0.01).
1101copy

Note than for an expected error of 1% and 2184 buckets, we can fit values all the way as big to the biggest 64bits signed integer. If we're measuring picoseconds, this would suffice to measure 107 days.

ddskerl_ets:new(#{ets_table => Table, name => Name, ...}).
...
ddskerl_ets:insert(Table, Name, Value).
...
ddskerl_ets:quantile(Table, Name, Q).copy