Operations

This guide covers monitoring, backup, retention, and troubleshooting for TimelessTraces.

Statistics

Get aggregate storage statistics without reading any blocks:

{:ok, stats} = TimelessTraces.stats()

Returns a %TimelessTraces.Stats{} struct:

HTTP API

curl http://localhost:10428/health

Returns status, blocks, spans, and disk_size.

Flushing

Force flush the buffer to write pending spans to storage immediately:

TimelessTraces.flush()

curl http://localhost:10428/api/v1/flush

Use before backups or graceful shutdowns.

Backup

Create a consistent online backup without stopping the application.

Elixir API

{:ok, result} = TimelessTraces.backup("/tmp/span_backup")
# => {:ok, %{path: "/tmp/span_backup", files: ["index.snapshot", "blocks"], total_bytes: 24000000}}

HTTP API

curl -X POST http://localhost:10428/api/v1/backup \
  -H 'Content-Type: application/json' \
  -d '{"path": "/tmp/span_backup"}'

Backup procedure

1. The buffer is flushed (all pending spans written to storage)
2. ETS index is written as a snapshot file (atomic rename)
3. Block files are copied in parallel to the target directory
4. Returns the backup path, file list, and total bytes

Restore procedure

1. Stop the TimelessTraces application
2. Replace the data directory contents with the backup files
3. Start the application -- it will load from the restored data

Retention

Retention runs automatically to prevent unbounded disk growth. Two independent policies are enforced:

Age-based retention

Delete blocks with ts_max older than the cutoff:

config :timeless_traces,
  retention_max_age: 7 * 86_400  # 7 days (default)

Size-based retention

Delete oldest blocks until total size is under the limit:

config :timeless_traces,
  retention_max_size: 512 * 1024 * 1024  # 512 MB (default)

Disable retention

config :timeless_traces,
  retention_max_age: nil,    # No age limit
  retention_max_size: nil    # No size limit

Manual trigger

TimelessTraces.Retention.run_now()

Check interval

config :timeless_traces,
  retention_check_interval: 300_000  # 5 minutes (default)

Telemetry events

TimelessTraces emits telemetry events for monitoring:

See the Telemetry guide for handler examples.

Key metrics to monitor

Troubleshooting

High memory usage

• Check raw_blocks in stats -- many uncompacted raw blocks use more memory
• Trigger compaction: TimelessTraces.Compactor.compact_now()
• Reduce max_buffer_size to flush smaller batches
• Check for slow subscribers blocking the buffer

Disk space growing

• Verify retention is configured: check retention_max_age and retention_max_size
• Trigger retention manually: TimelessTraces.Retention.run_now()
• Check stats for total_bytes trends
• Reduce retention age or size limits

Slow queries

• Use :service, :kind, and :status filters to leverage the term index
• Avoid full scans (no filters) on large datasets
• Reduce the time range with :since and :until
• Check raw_blocks count -- many small raw blocks are slower to query than fewer compressed blocks
• Trigger merge compaction to consolidate small compressed blocks: TimelessTraces.merge_now()

Spans not appearing in queries

• Flush the buffer: TimelessTraces.flush()
• Check that the OTel exporter is configured: verify traces_exporter: {TimelessTraces.Exporter, []} in config
• Verify the data_dir exists and is writable
• Check for block read errors in telemetry events
• Check the index_publish_interval -- spans become queryable after the index batches them (default 2s)

Compaction not running

• Check raw_blocks and raw_entries in stats
• Verify compaction_threshold isn't set too high for your span volume
• Trigger manually: TimelessTraces.Compactor.compact_now()
• Check that compaction_max_raw_age is reasonable (default: 60 seconds)

OTel exporter not working

• Verify the dependency: {:opentelemetry, "~> 1.5"} must be in your mix.exs
• Check the config: config :opentelemetry, traces_exporter: {TimelessTraces.Exporter, []}
• Ensure spans are being created: use OpenTelemetry.Tracer.with_span/2 in your code
• Check that TimelessTraces.Application has started (it must start before spans are exported)