GorillaStream.Compression.Container (gorilla_stream v3.0.1)

Copy Markdown View Source

Container compression utilities for GorillaStream.

Provides a unified interface for applying secondary compression (zlib, zstd, or openzl) on top of Gorilla-compressed data. Zstd is preferred when available as it typically achieves better compression ratios and faster speeds than zlib.

Compression Options

The :compression option accepts the following values:

  • :none - No container compression (default)
  • :zlib - Use zlib compression (always available, built into Erlang)
  • :zstd - Use zstd compression (requires ezstd package)
  • :openzl - Use OpenZL compression (requires ex_openzl package)
  • :auto - Use zstd if available, fall back to zlib

Examples

# No compression (default)
{:ok, data} = Container.compress(binary, compression: :none)

# Use zlib
{:ok, data} = Container.compress(binary, compression: :zlib)

# Use zstd (requires ezstd)
{:ok, data} = Container.compress(binary, compression: :zstd)

# Use OpenZL (requires ex_openzl)
{:ok, data} = Container.compress(binary, compression: :openzl)

# Auto-select best available
{:ok, data} = Container.compress(binary, compression: :auto)

Streaming Compression

For continuous streaming with minimal memory overhead, use the streaming API:

# Create a streaming context
{:ok, ctx} = Container.create_stream_context(:zstd, :compress)

# Compress chunks as they arrive
{:ok, compressed1} = Container.stream_compress(ctx, chunk1)
{:ok, compressed2} = Container.stream_compress(ctx, chunk2)

# Finalize the stream
{:ok, final} = Container.stream_finish(ctx)

Legacy Support

For backward compatibility, the :zlib boolean option is still supported:

{:ok, data} = Container.compress(binary, zlib: true)  # Same as compression: :zlib

Summary

Types

compression_type()

Functions

compress(data, opts \\ [])

Compresses data using the specified compression algorithm.

compress_with_dict(data, cdict)

Compresses data using a pre-trained zstd dictionary.

create_stream_context(type, mode, opts \\ [])

Creates a streaming compression or decompression context.

decompress(data, opts \\ [])

Decompresses data using the specified compression algorithm.

decompress_with_dict(data, ddict)

Decompresses data using a pre-trained zstd dictionary.

effective_compression(opts)

Returns the actual compression algorithm that will be used for the given options.

openzl_available?()

Checks if the ex_openzl library is available at runtime.

stream_compress(arg, data)

Compresses data using a streaming context.

stream_decompress(arg, data)

Decompresses data using a streaming context.

stream_finish(arg)

Finishes a streaming compression context and returns any remaining data.

zstd_available?()

Checks if the ezstd library is available at runtime.

Types

compression_type()

@type compression_type() :: :none | :zlib | :zstd | :openzl | :auto

Functions

compress(data, opts \\ [])

@spec compress(
  binary(),
  keyword()
) :: {:ok, binary()} | {:error, String.t()}

Compresses data using the specified compression algorithm.

Parameters

  • data - Binary data to compress
  • opts - Keyword list of options:
    • :compression - Compression type (:none, :zlib, :zstd, :auto)
    • :compression_level - Zstd compression level 1-22 (default: ezstd default)
    • :zlib - Legacy boolean option for zlib compression

Returns

  • {:ok, compressed_data} on success
  • {:error, reason} on failure

compress_with_dict(data, cdict)

@spec compress_with_dict(binary(), reference()) ::
  {:ok, binary()} | {:error, String.t()}

Compresses data using a pre-trained zstd dictionary.

The cdict must be a reference created via :ezstd.create_cdict/2.

Returns

  • {:ok, compressed_data} on success
  • {:error, reason} on failure

create_stream_context(type, mode, opts \\ [])

@spec create_stream_context(:zstd | :zlib, :compress | :decompress, keyword()) ::
  {:ok, term()} | {:error, String.t()}

Creates a streaming compression or decompression context.

Streaming contexts allow compressing/decompressing data incrementally with minimal memory overhead - ideal for continuous data streams.

Parameters

  • type - :zstd or :zlib
  • mode - :compress or :decompress
  • opts - Options:
    • :buffer_size - Buffer size in bytes (default: 64KB)

Returns

  • {:ok, context} - A streaming context
  • {:error, reason} - If the compression type is not available

Examples

{:ok, ctx} = Container.create_stream_context(:zstd, :compress)

decompress(data, opts \\ [])

@spec decompress(
  binary(),
  keyword()
) :: {:ok, binary()} | {:error, String.t()}

Decompresses data using the specified compression algorithm.

Parameters

  • data - Binary data to decompress
  • opts - Keyword list of options:
    • :compression - Compression type (:none, :zlib, :zstd, :auto)
    • :zlib - Legacy boolean option for zlib compression

Returns

  • {:ok, decompressed_data} on success
  • {:error, reason} on failure

decompress_with_dict(data, ddict)

@spec decompress_with_dict(binary(), reference()) ::
  {:ok, binary()} | {:error, String.t()}

Decompresses data using a pre-trained zstd dictionary.

The ddict must be a reference created via :ezstd.create_ddict/1.

Returns

  • {:ok, decompressed_data} on success
  • {:error, reason} on failure

effective_compression(opts)

@spec effective_compression(keyword()) :: compression_type()

Returns the actual compression algorithm that will be used for the given options.

Useful for debugging or understanding what compression will be applied.

Examples

iex> GorillaStream.Compression.Container.effective_compression(compression: :auto)
:zstd  # or :zlib if ezstd not available

openzl_available?()

@spec openzl_available?() :: boolean()

Checks if the ex_openzl library is available at runtime.

Examples

iex> GorillaStream.Compression.Container.openzl_available?()
true  # or false, depending on whether ex_openzl is installed

stream_compress(arg, data)

@spec stream_compress(term(), binary()) :: {:ok, binary()} | {:error, String.t()}

Compresses data using a streaming context.

Data is compressed incrementally. Call stream_finish/1 when done.

Returns

  • {:ok, compressed_data} - Compressed output (may be empty if buffered)
  • {:error, reason} - On failure

stream_decompress(arg, data)

@spec stream_decompress(term(), binary()) :: {:ok, binary()} | {:error, String.t()}

Decompresses data using a streaming context.

Returns

  • {:ok, decompressed_data} - Decompressed output
  • {:error, reason} - On failure

stream_finish(arg)

@spec stream_finish(term()) :: {:ok, binary()} | {:error, String.t()}

Finishes a streaming compression context and returns any remaining data.

For compression, this flushes the final compressed bytes. For decompression, this verifies the stream is complete.

After calling this, the context should not be used again.

Returns

  • {:ok, final_data} - Any remaining compressed/decompressed data
  • {:error, reason} - On failure

zstd_available?()

@spec zstd_available?() :: boolean()

Checks if the ezstd library is available at runtime.

Examples

iex> GorillaStream.Compression.Container.zstd_available?()
true  # or false, depending on whether ezstd is installed