Storage & Compression

This document covers TimelessTraces' block-based storage format, compression options, and the compaction pipeline.

Block-based storage

Spans are stored in blocks -- batches of spans written together and indexed as a unit. Each block contains:

• A batch of serialized spans (typically 500-2000)
• Block metadata: block_id, byte_size, entry_count, timestamp range (ts_min, ts_max), format
• An inverted index of terms for fast querying
• Trace index entries mapping packed trace IDs to the block

Block formats

Raw (.raw)

Uncompressed Erlang binary serialization (:erlang.term_to_binary). This is the initial format when spans are flushed from the buffer. Raw blocks are temporary -- the compactor merges them into compressed blocks.

Zstd (.zst)

Erlang term_to_binary compressed with Zstandard. Simple, fast compression with good decompression speed.

OpenZL (.ozl)

Columnar format with OpenZL compression. Span fields are split into separate typed columns and each column is independently compressed. This exploits per-column redundancy for better ratios and enables selective column decompression at query time.

Columnar layout:

Choosing a format

The default is :openzl (columnar). Set via configuration:

config :timeless_traces,
  compaction_format: :openzl,    # or :zstd
  compression_level: 6           # 1-22

Compaction

New spans are first written as raw (uncompressed) blocks for low-latency ingestion. A background Compactor process periodically merges raw blocks into compressed blocks.

Compaction triggers

Compaction runs when any of these conditions are met:

1. Entry threshold: total raw entries >= compaction_threshold (default: 500)
2. Age threshold: oldest raw block >= compaction_max_raw_age seconds (default: 60)
3. Manual trigger: TimelessTraces.Compactor.compact_now()
4. Periodic check: every compaction_interval ms (default: 30,000)

Compaction process

1. Read all raw block entries from disk
2. Merge entries into a single batch
3. Compress with the configured format (OpenZL or zstd)
4. Write a new compressed block file
5. Update the index (delete old block metadata, add new)
6. Delete old raw block files
7. Update compression statistics

Compaction configuration

config :timeless_traces,
  compaction_threshold: 500,       # Min raw entries to trigger
  compaction_interval: 30_000,     # Check interval (ms)
  compaction_max_raw_age: 60,      # Force compact after this many seconds
  compaction_format: :openzl,      # Output format
  compression_level: 6             # Compression level (1-22)

Manual compaction

TimelessTraces.Compactor.compact_now()
# => :ok or :noop (if nothing to compact)

Disk layout

data_dir/
├── index.snapshot    # Periodic ETS table dump (compressed ETF)
├── index.log         # Write-ahead log (Erlang disk_log)
└── blocks/
    ├── 000000000001.raw   # Raw block (temporary)
    ├── 000000000002.raw   # Raw block (temporary)
    ├── 000000000003.ozl   # OpenZL compressed block
    ├── 000000000004.ozl   # OpenZL compressed block
    └── ...

Block filenames are 12-digit zero-padded block IDs with format-specific extensions.

The separate trace_index persists 32-character hex trace IDs in packed 16-byte binary form to keep the index smaller. Older text rows are still readable during lookup.

Memory storage mode

For testing or ephemeral environments, use in-memory storage:

config :timeless_traces, storage: :memory

In memory mode:

• Block data is stored in ETS tables only
• No block files are written to disk
• The ETS tables still provide lock-free read access
• Data does not survive application restarts

Compression statistics

Track compression efficiency via the stats API:

{:ok, stats} = TimelessTraces.stats()

stats.compression_raw_bytes_in          # Total uncompressed bytes processed
stats.compression_compressed_bytes_out  # Total compressed bytes produced
stats.compaction_count                  # Number of compaction runs

The compression ratio is compression_raw_bytes_in / compression_compressed_bytes_out.

Compression comparison

OpenZL achieves better compression through columnar encoding -- timestamps compress well together, span kinds compress to nearly nothing, and string columns (names, IDs) benefit from prefix deduplication.