reckon_db_scavenge (reckon_db v1.1.0)

Scavenging and archival for reckon-db

Provides functionality to: - Remove old events beyond a retention period - Optionally archive events before deletion - Maintain stream integrity by requiring snapshots

Use cases: - Reduce storage costs by removing old events - Comply with data retention policies - Archive events to cold storage

Safety guarantees: - By default, requires a snapshot before scavenging - Supports dry-run mode for previewing changes - Telemetry events for monitoring

Summary

Functions

archive_and_scavenge(StoreId, StreamId, _, Opts)

Archive events to a backend, then scavenge.

dry_run(StoreId, StreamId, Opts)

Preview what would be scavenged without making changes.

scavenge(StoreId, StreamId, Opts)

Scavenge a single stream.

scavenge_matching(StoreId, Pattern, Opts)

Scavenge all streams matching a pattern.

Types

event/0

-type event() ::
          #event{event_id :: binary(),
                 event_type :: binary(),
                 stream_id :: binary(),
                 version :: non_neg_integer(),
                 data :: map() | binary(),
                 metadata :: map(),
                 tags :: [binary()] | undefined,
                 timestamp :: integer(),
                 epoch_us :: integer(),
                 data_content_type :: binary(),
                 metadata_content_type :: binary()}.

scavenge_opts/0

-type scavenge_opts() ::
          #{before => integer(),
            before_version => integer(),
            keep_versions => pos_integer(),
            require_snapshot => boolean(),
            dry_run => boolean()}.

scavenge_result/0

-type scavenge_result() ::
          #{stream_id := binary(),
            deleted_count := non_neg_integer(),
            deleted_versions := {non_neg_integer(), non_neg_integer()},
            archived := boolean(),
            archive_key => binary()}.

Functions

archive_and_scavenge(StoreId, StreamId, _, Opts)

-spec archive_and_scavenge(atom(), binary(), {module(), term()}, scavenge_opts()) ->
                              {ok, scavenge_result()} | {error, term()}.

Archive events to a backend, then scavenge.

First archives events to the specified backend, then deletes them. This ensures events are preserved before removal.

dry_run(StoreId, StreamId, Opts)

-spec dry_run(atom(), binary(), scavenge_opts()) -> {ok, scavenge_result()} | {error, term()}.

Preview what would be scavenged without making changes.

scavenge(StoreId, StreamId, Opts)

-spec scavenge(atom(), binary(), scavenge_opts()) -> {ok, scavenge_result()} | {error, term()}.

Scavenge a single stream.

Deletes old events based on the provided options. By default, requires a snapshot to exist (safety measure).

Options: - before`: Delete events with epoch_us before this timestamp - `before_version`: Delete events before this version (alternative to `before`) - `keep_versions`: Always keep at least N latest versions (default: 0) - `require_snapshot`: Require snapshot exists (default: true) - `dry_run`: Preview only, dont delete (default: false)

Example:

  %% Delete events older than 1 year, keep last 10 versions
  OneYearAgo = erlang:system_time(microsecond) - (365 * 24 * 60 * 60 * 1000000),
  {ok, Result} = reckon_db_scavenge:scavenge(my_store, <<"orders-123">>, #{
      before => OneYearAgo,
      keep_versions => 10
  }).

scavenge_matching(StoreId, Pattern, Opts)

-spec scavenge_matching(atom(), binary(), scavenge_opts()) ->
                           {ok, [scavenge_result()]} | {error, term()}.

Scavenge all streams matching a pattern.

Pattern uses shell-like wildcards (e.g., "orders-*"). Applies the same options to all matching streams.