Fact.Seam.Storage.Standard.V2 (Fact v0.3.1)
View SourceStandard V2 implementation of the Fact.Seam.Storage seam.
This module creates 0 to 3 character buckets (i.e. sub-directories) for events. Directories with a large number of files can cause performance and operational issues.
Most filesystems store directory entries in data structures (like B-trees or linear lists) that
degrade as the entry count grows. Listing, searching, or opening files requires scanning or
traversing these structures, which becomes slower as directories grow into thousands or millions of files.
Tool Limitations
Many common tools struggle with huge directories.
- Shell glob expansion can exceed argument length limes or consume excessive memory
lsbecomes slow and unwieldy- File browsers may hang or become unresponsive
- Backup tools and file synchronization can slow dramatically
Inode and Metadata Overhead
Directory metadata must often be read into memory. A directory with millions of entries can consume significant memory just for the directory itself, separate from the files it contains.
Bucket Configurations
This implementation will create a sub-directory within the base records_path. The default event
record encoding is base16, with a default bucket_length of 2, which would result in 256 "buckets"
directories for storing events 00 to ff. Using an alternate encoding for record file names or
increasing the bucket length will increase this.
| encoding | bucket_length: 1 | bucket_length: 2 | bucket_length: 3 |
|---|---|---|---|
| base16 | 16 | 256 | 1,024 |
| base32 | 32 | 1,024 | 32,768 |
| base64url | 64 | 4,096 | 262,144 |
Too many buckets
Having too many buckets is also not good, I would recommend not exceeding 4,096. Configure the system accordingly.
Future
A future storage implementation, may add support for nested buckets.
Summary
Functions
Get the default configuration options.
Gets the path to the base directory for all indexes.
Creates the directory structure used for events and indexes.
Gets the path to the directory containing the ledger.
Gets the path to the directory containing the lock file.
Gets the path to the Merkle Mountain Range directory.
Gets the specification for the configuration options.
Gets the configured base path for the database.
Gets the path to the base directory for records, or the path to a specific record.
Gets the path to the directory containing the write ahead log segment files.
Types
@type t() :: %Fact.Seam.Storage.Standard.V2{ bucket_length: non_neg_integer(), path: Path.t() }
The configuration options for the Standard v2 storage seam impl.
:path- The base path to the database directory.:bucket_length- The length of event bucket directories.
Functions
@spec default_options() :: t()
Get the default configuration options.
Gets the path to the base directory for all indexes.
Creates the directory structure used for events and indexes.
Gets the path to the directory containing the ledger.
Gets the path to the directory containing the lock file.
Gets the path to the Merkle Mountain Range directory.
Gets the specification for the configuration options.
Gets the configured base path for the database.
@spec records_path(t(), nil | Fact.record_id(), keyword()) :: Path.t()
Gets the path to the base directory for records, or the path to a specific record.
Gets the path to the directory containing the write ahead log segment files.