Fact.Seam.Storage.Standard.V2 (Fact v0.3.1)

View Source

Standard V2 implementation of the Fact.Seam.Storage seam.

This module creates 0 to 3 character buckets (i.e. sub-directories) for events. Directories with a large number of files can cause performance and operational issues.

Most filesystems store directory entries in data structures (like B-trees or linear lists) that degrade as the entry count grows. Listing, searching, or opening files requires scanning or
traversing these structures, which becomes slower as directories grow into thousands or millions of files.

Tool Limitations

Many common tools struggle with huge directories.

  • Shell glob expansion can exceed argument length limes or consume excessive memory
  • ls becomes slow and unwieldy
  • File browsers may hang or become unresponsive
  • Backup tools and file synchronization can slow dramatically

Inode and Metadata Overhead

Directory metadata must often be read into memory. A directory with millions of entries can consume significant memory just for the directory itself, separate from the files it contains.

Bucket Configurations

This implementation will create a sub-directory within the base records_path. The default event record encoding is base16, with a default bucket_length of 2, which would result in 256 "buckets" directories for storing events 00 to ff. Using an alternate encoding for record file names or increasing the bucket length will increase this.

encodingbucket_length: 1bucket_length: 2bucket_length: 3
base16162561,024
base32321,02432,768
base64url644,096262,144

Too many buckets

Having too many buckets is also not good, I would recommend not exceeding 4,096. Configure the system accordingly.

Future

A future storage implementation, may add support for nested buckets.

Summary

Types

t()

The configuration options for the Standard v2 storage seam impl.

Functions

Get the default configuration options.

Gets the path to the base directory for all indexes.

Creates the directory structure used for events and indexes.

Gets the path to the directory containing the ledger.

Gets the path to the directory containing the lock file.

Gets the path to the Merkle Mountain Range directory.

Gets the specification for the configuration options.

Gets the configured base path for the database.

Gets the path to the base directory for records, or the path to a specific record.

Gets the path to the directory containing the write ahead log segment files.

Types

t()

(since 0.2.0)
@type t() :: %Fact.Seam.Storage.Standard.V2{
  bucket_length: non_neg_integer(),
  path: Path.t()
}

The configuration options for the Standard v2 storage seam impl.

  • :path - The base path to the database directory.
  • :bucket_length - The length of event bucket directories.

Functions

default_options()

(since 0.2.0)
@spec default_options() :: t()

Get the default configuration options.

indices_path(v2, opts)

(since 0.2.0)
@spec indices_path(
  t(),
  keyword()
) :: Path.t()

Gets the path to the base directory for all indexes.

initialize_storage(this, opts)

(since 0.2.0)
@spec initialize_storage(
  t(),
  keyword()
) :: {:ok, Path.t()} | {:error, term()}

Creates the directory structure used for events and indexes.

ledger_path(v2, opts)

(since 0.2.0)
@spec ledger_path(
  t(),
  keyword()
) :: Path.t()

Gets the path to the directory containing the ledger.

locks_path(v2, opts)

(since 0.2.0)
@spec locks_path(
  t(),
  keyword()
) :: Path.t()

Gets the path to the directory containing the lock file.

merkle_mountain_range_path(v2, opts)

(since 0.3.0)
@spec merkle_mountain_range_path(
  t(),
  keyword()
) :: Path.t()

Gets the path to the Merkle Mountain Range directory.

option_specs()

(since 0.2.0)

Gets the specification for the configuration options.

path(v2, opts)

(since 0.2.0)
@spec path(
  t(),
  keyword()
) :: Path.t()

Gets the configured base path for the database.

records_path(v2, record_id, opts)

(since 0.2.0)
@spec records_path(t(), nil | Fact.record_id(), keyword()) :: Path.t()

Gets the path to the base directory for records, or the path to a specific record.

validate_options(options, specs)

write_ahead_log_path(v2, opts)

(since 0.3.0)
@spec write_ahead_log_path(
  t(),
  keyword()
) :: Path.t()

Gets the path to the directory containing the write ahead log segment files.