Configuration Reference

All configuration is set via config :timeless_logs in your config/config.exs.

Complete options table

Full configuration example

# config/config.exs
config :timeless_logs,
  data_dir: "/var/lib/my_app/logs",
  flush_interval: 2_000,
  max_buffer_size: 2_000,
  zstd_compression_level: 5,
  compaction_format: :openzl,
  openzl_compression_level: 9,
  compaction_threshold: 1_000,
  compaction_interval: 60_000,
  compaction_max_raw_age: 120,
  retention_max_age: 30 * 86_400,
  retention_max_size: 2 * 1024 * 1024 * 1024,
  retention_check_interval: 600_000,
  http: [port: 9428, bearer_token: "my-secret-token"]

HTTP server options

The http option accepts:

HTTP keyword options:

# No auth (trusted network)
config :timeless_logs, http: true

# With auth
config :timeless_logs, http: [port: 9428, bearer_token: "my-secret-token"]

# Custom port only
config :timeless_logs, http: [port: 9500]

Storage backends

Disk storage (default)

config :timeless_logs,
  storage: :disk,
  data_dir: "/var/lib/my_app/logs"

Block files are stored in data_dir/blocks/ as numbered files with format-specific extensions (.raw, .zst, .ozl). The index is persisted as data_dir/index.snapshot (periodic full dump) and data_dir/index.log (write-ahead log).

Memory storage

config :timeless_logs,
  storage: :memory

Block data is stored in ETS tables only. Useful for testing or ephemeral environments. Data does not survive restarts.

Tuning guidance

Buffer settings

flush_interval and max_buffer_size control when buffered log entries are written to disk.

• Lower flush_interval (e.g., 500ms): lower latency for queries to see recent logs, more I/O
• Higher flush_interval (e.g., 5000ms): better batching, higher throughput
• Higher max_buffer_size (e.g., 5000): larger blocks, better compression ratios
• Lower max_buffer_size (e.g., 200): more frequent flushes, lower data loss risk

Compression settings

The compaction format determines how raw blocks are compressed:

Retention settings

Both age-based and size-based retention are enforced independently. Set either to nil to disable that policy:

# Keep 30 days, no size limit
config :timeless_logs,
  retention_max_age: 30 * 86_400,
  retention_max_size: nil

# Keep 1 GB, no age limit
config :timeless_logs,
  retention_max_age: nil,
  retention_max_size: 1_073_741_824

# Disable all retention (keep forever)
config :timeless_logs,
  retention_max_age: nil,
  retention_max_size: nil

Compaction settings

Compaction merges raw (uncompressed) blocks into compressed blocks. Tune these based on your log volume:

• High volume (>1000 entries/sec): increase compaction_threshold to 2000+ and compaction_interval to 60_000 for larger, more efficient batches
• Low volume (<10 entries/sec): decrease compaction_max_raw_age to 30 for faster compaction
• Testing: set compaction_threshold to 10 and compaction_max_raw_age to 5 for immediate compaction

Merge compaction settings

After initial compaction, many small compressed blocks accumulate (one per flush cycle). Merge compaction consolidates them into larger blocks for better compression and fewer blocks to scan during queries.

• High volume: the defaults (target 2000 entries, min 4 blocks) work well
• Low volume (<10 entries/sec): lower merge_compaction_min_blocks to 2 so merges happen with fewer blocks
• Large queries: increase merge_compaction_target_size to 5000+ to produce fewer, larger blocks

Merge compaction can also be triggered manually via TimelessLogs.merge_now().