Scraping

TimelessMetrics includes a built-in Prometheus-compatible scraper that polls /metrics endpoints on a configurable interval. This lets you pull metrics from any application that exposes a Prometheus text exposition endpoint.

How scraping works

Each scrape target is a background worker process that:

1. Sends an HTTP GET to the target's metrics endpoint
2. Parses the Prometheus text exposition response
3. Applies relabel and metric relabel configs (if any)
4. Writes the parsed metrics into the store via write_batch
5. Updates scrape health tracking (up/down, duration, sample count)
6. Auto-registers metric metadata (type and unit inferred from metric name suffixes)

Workers use anchor-based scheduling to avoid clock drift, with initial jitter to spread load across targets.

Elixir API

The scraper process is named :"#{store_name}_scraper". You can interact with it directly or through the HTTP API.

# The scraper name for a store called :metrics
scraper = :metrics_scraper

# Add a target
{:ok, id} = TimelessMetrics.Scraper.add_target(scraper, %{
  "job_name" => "my_app",
  "address" => "localhost:4000",
  "metrics_path" => "/metrics",
  "scrape_interval" => 30
})

# List all targets with health status
{:ok, targets} = TimelessMetrics.Scraper.list_targets(scraper)

# Get a single target
{:ok, target} = TimelessMetrics.Scraper.get_target(scraper, id)

# Update a target
:ok = TimelessMetrics.Scraper.update_target(scraper, id, %{
  "scrape_interval" => 15
})

# Delete a target
:ok = TimelessMetrics.Scraper.delete_target(scraper, id)

HTTP API

Create a scrape target

curl -X POST http://localhost:8428/api/v1/scrape_targets \
  -H 'Content-Type: application/json' \
  -d '{
    "job_name": "my_phoenix_app",
    "address": "localhost:4000",
    "metrics_path": "/metrics",
    "scrape_interval": 30
  }'
# => {"id": 1, "status": "created"}

List all targets

curl http://localhost:8428/api/v1/scrape_targets

Response includes health information for each target:

{
  "data": [
    {
      "id": 1,
      "job_name": "my_phoenix_app",
      "address": "localhost:4000",
      "metrics_path": "/metrics",
      "scrape_interval": 30,
      "enabled": true,
      "health": {
        "health": "up",
        "last_scrape": 1700000060,
        "last_duration_ms": 12,
        "last_error": null,
        "samples_scraped": 42
      }
    }
  ]
}

Get a single target

curl http://localhost:8428/api/v1/scrape_targets/1

Update a target

curl -X PUT http://localhost:8428/api/v1/scrape_targets/1 \
  -H 'Content-Type: application/json' \
  -d '{"job_name": "my_phoenix_app", "address": "localhost:4000", "scrape_interval": 15}'

Delete a target

curl -X DELETE http://localhost:8428/api/v1/scrape_targets/1

Target options

Relabel configs

Relabel configs transform labels using regex-based rules, following the Prometheus relabeling model. Each config is a JSON object with these fields:

Metric relabel configs

Applied after scraping, these let you filter or transform metrics before they're stored. Common uses:

Drop unwanted metrics:

{
  "metric_relabel_configs": [
    {
      "source_labels": ["__name__"],
      "regex": "go_.*",
      "action": "drop"
    }
  ]
}

Keep only specific metrics:

{
  "metric_relabel_configs": [
    {
      "source_labels": ["__name__"],
      "regex": "http_requests_total|http_request_duration_seconds.*",
      "action": "keep"
    }
  ]
}

Rename a metric:

{
  "metric_relabel_configs": [
    {
      "source_labels": ["__name__"],
      "regex": "old_metric_name",
      "target_label": "__name__",
      "replacement": "new_metric_name"
    }
  ]
}

Health monitoring

Each scrape worker tracks health status and writes self-monitoring metrics:

These metrics have job and instance labels matching the target's job_name and address.

Health status is also stored in the scrape_health SQLite table and returned in the target listing API.

Auto-registered metadata

When new metric names are discovered during scraping, the worker automatically registers metadata based on Prometheus naming conventions:

Example: scraping a Phoenix app

A Phoenix app using :telemetry_metrics_prometheus_core exposes metrics at /metrics:

curl -X POST http://localhost:8428/api/v1/scrape_targets \
  -H 'Content-Type: application/json' \
  -d '{
    "job_name": "phoenix_app",
    "address": "localhost:4000",
    "metrics_path": "/metrics",
    "scrape_interval": 15,
    "labels": {"env": "production"},
    "metric_relabel_configs": [
      {"source_labels": ["__name__"], "regex": "go_.*", "action": "drop"}
    ]
  }'

Example: scraping Node Exporter

curl -X POST http://localhost:8428/api/v1/scrape_targets \
  -H 'Content-Type: application/json' \
  -d '{
    "job_name": "node_exporter",
    "address": "localhost:9100",
    "scrape_interval": 30,
    "labels": {"host": "web-1"}
  }'

Disabling scraping

If you don't need scraping, disable it to save resources:

{TimelessMetrics, name: :metrics, data_dir: "/data", scraping: false}