Operations

This guide covers monitoring, backup, retention, and troubleshooting for TimelessLogs.

Statistics

Get aggregate storage statistics without reading any blocks:

Elixir API

{:ok, stats} = TimelessLogs.stats()

Returns a %TimelessLogs.Stats{} struct:

HTTP API

curl http://localhost:9428/select/logsql/stats

curl http://localhost:9428/health

Flushing

Force flush the buffer to write pending entries to disk immediately:

TimelessLogs.flush()

curl http://localhost:9428/api/v1/flush

Use before backups or graceful shutdowns.

Backup

Create a consistent online backup without stopping the application.

Elixir API

{:ok, result} = TimelessLogs.backup("/tmp/logs_backup")
# => {:ok, %{path: "/tmp/logs_backup", files: ["index.snapshot", "blocks"], total_bytes: 24000000}}

HTTP API

curl -X POST http://localhost:9428/api/v1/backup \
  -H 'Content-Type: application/json' \
  -d '{"path": "/tmp/logs_backup"}'

Backup procedure

1. The buffer is flushed (all pending entries written to disk)
2. ETS index is written as a snapshot file (atomic rename)
3. Block files are copied in parallel to the target directory
4. Returns the backup path, file list, and total bytes

Restore procedure

1. Stop the TimelessLogs application
2. Replace the data directory contents with the backup files
3. Start the application -- it will load from the restored data

Retention

Retention runs automatically to prevent unbounded disk growth. Two independent policies are enforced:

Age-based retention

Delete blocks with ts_max older than the cutoff:

config :timeless_logs,
  retention_max_age: 7 * 86_400  # 7 days (default)

Size-based retention

Delete oldest blocks until total size is under the limit:

config :timeless_logs,
  retention_max_size: 512 * 1024 * 1024  # 512 MB (default)

Disable retention

config :timeless_logs,
  retention_max_age: nil,    # No age limit
  retention_max_size: nil    # No size limit

Manual trigger

TimelessLogs.Retention.run_now()
# => :noop or {:ok, count_deleted}

Check interval

config :timeless_logs,
  retention_check_interval: 300_000  # 5 minutes (default)

Telemetry events

TimelessLogs emits telemetry events for monitoring and observability:

Attaching handlers

:telemetry.attach_many(
  "my-log-metrics",
  [
    [:timeless_logs, :flush, :stop],
    [:timeless_logs, :query, :stop],
    [:timeless_logs, :compaction, :stop]
  ],
  fn event, measurements, metadata, _config ->
    IO.inspect({event, measurements, metadata})
  end,
  nil
)

Key metrics to monitor

Troubleshooting

High memory usage

• Check raw_blocks in stats -- many uncompacted raw blocks use more memory
• Trigger compaction: TimelessLogs.Compactor.compact_now()
• Reduce max_buffer_size to flush smaller batches
• Check for slow subscribers blocking the buffer

Disk space growing

• Verify retention is configured: check retention_max_age and retention_max_size
• Trigger retention manually: TimelessLogs.Retention.run_now()
• Check stats for total_bytes and total_entries trends
• Reduce retention age or size limits

Slow queries

• Use :level and :metadata filters to leverage the term index
• Avoid full scans (no filters) on large datasets
• Reduce the time range with :since and :until
• Check raw_blocks count -- many small raw blocks are slower to query than fewer compressed blocks
• Trigger merge compaction to consolidate small compressed blocks: TimelessLogs.merge_now()

Logs not appearing in queries

• Flush the buffer: TimelessLogs.flush()
• Check that the Logger handler is installed: :logger.get_handler_config(:timeless_logs)
• Verify the data_dir exists and is writable
• Check for block read errors in telemetry events

Compaction not running

• Check raw_blocks and raw_entries in stats
• Verify compaction_threshold isn't set too high for your log volume
• Trigger manually: TimelessLogs.Compactor.compact_now()
• Check that compaction_max_raw_age is reasonable (default: 60 seconds)