Implements Gorilla compression for time series data streams.
This module provides compression for streams of {timestamp, number} data using the Gorilla compression algorithm, with optional container compression (zlib or zstd) for additional compression at the final step.
Container Compression Options
The :compression option supports:
:none- No container compression (default):zlib- Use zlib compression (always available, built into Erlang):zstd- Use zstd compression (requires ezstd package):auto- Use zstd if available, fall back to zlib
For backward compatibility, the :zlib boolean option is still supported.
Summary
Functions
Compresses a stream of {timestamp, number} data using the Gorilla algorithm.
Decompresses previously compressed data back into the original stream of {timestamp, float} tuples.
Validates that a stream of data is in the correct format for compression.
Functions
Compresses a stream of {timestamp, number} data using the Gorilla algorithm.
By default, VictoriaMetrics-style preprocessing is ENABLED (victoria_metrics: true), which applies value scaling and optional counter handling to improve compression. You can pass keyword options to customize or disable this behavior.
Parameters
stream: An enumerable of {timestamp, number} tuples- Second argument may be either:
zlib_compression?(boolean) to toggle zlib container compression (default: false), OR- keyword options, supporting:
:victoria_metrics(boolean, default: true):is_counter(boolean, default: false):scale_decimals(:auto | integer, default: :auto):compression(:none|:zlib|:zstd|:auto, default: :none):zlib(boolean, default: false) - legacy option, use:compressioninstead
Returns
{:ok, compressed_data}: When compression is successful{:error, reason}: When compression fails
Examples
iex> stream = [{1609459200, 1.23}, {1609459201, 1.24}, {1609459202, 1.25}]
iex> {:ok, compressed} = GorillaStream.Compression.Gorilla.compress(stream, false)
iex> is_binary(compressed)
true
iex> # With zlib
iex> stream = [{1609459200, 1.23}, {1609459201, 1.24}, {1609459202, 1.25}]
iex> {:ok, compressed} = GorillaStream.Compression.Gorilla.compress(stream, true)
iex> is_binary(compressed)
true
iex> # Disable VictoriaMetrics preprocessing explicitly
iex> stream = [{1609459200, 1.23}, {1609459201, 1.24}, {1609459202, 1.25}]
iex> {:ok, compressed} = GorillaStream.Compression.Gorilla.compress(stream, victoria_metrics: false)
iex> is_binary(compressed)
true
iex> # Enable counters and custom scaling via opts
iex> stream = [{1609459200, 1.23}, {1609459201, 1.24}, {1609459202, 1.25}]
iex> {:ok, compressed} = GorillaStream.Compression.Gorilla.compress(stream, victoria_metrics: true, is_counter: true, scale_decimals: :auto)
iex> is_binary(compressed)
trueDecompresses previously compressed data back into the original stream of {timestamp, float} tuples.
Parameters
compressed_data: The compressed data (binary)- Second argument may be either:
zlib_compression?(boolean) indicating if zlib was used (default: false), OR- keyword options, supporting:
:compression(:none|:zlib|:zstd|:auto, default: :none):zlib(boolean, default: false) - legacy option, use:compressioninstead- Other VM options are read from the header automatically by the decoder.
Returns
{:ok, original_stream}: When decompression is successful{:error, reason}: When decompression fails
Examples
iex> stream = [{1609459200, 1.23}, {1609459201, 1.24}, {1609459202, 1.25}]
iex> {:ok, compressed} = GorillaStream.Compression.Gorilla.compress(stream, false)
iex> GorillaStream.Compression.Gorilla.decompress(compressed, false)
{:ok, [{1609459200, 1.23}, {1609459201, 1.24}, {1609459202, 1.25}]}Validates that a stream of data is in the correct format for compression.
Parameters
stream: The stream to validate
Returns
:okif the stream is valid{:error, reason}if the stream is invalid
Examples
iex> GorillaStream.Compression.Gorilla.validate_stream([{1609459200, 1.23}, {1609459201, 1.24}])
:ok
iex> GorillaStream.Compression.Gorilla.validate_stream([{1609459200, "invalid"}])
{:error, "Invalid data format: expected {timestamp, number} tuple"}