A GenServer implementation of a multi-level cache for PMTiles file data.

This GenServer provides efficient concurrent access to PMTiles map data through file-based caching mechanisms optimized for high-concurrency workloads.

Cache Levels

1. Directory Cache (File-based)

• Caches deserialized directory structures from the PMTiles file
• Stored as individual binary files on disk (one file per directory)
• Root directory pre-fetched in background on startup
• Leaf directories fetched on-demand as tiles are requested
• Coordinates concurrent directory fetches to prevent duplicate S3 requests
• No size limits - cache can grow indefinitely
• No corruption issues from concurrent writes

2. Tile Cache (Optional File-based)

• Optional persistent tile caching using individual files (disabled by default)
• When enabled, each tile is stored as a separate file on disk
• No size limits - cache can grow indefinitely
• Maintains hit/miss statistics
• File system handles concurrent access naturally

3. Concurrent Request Handling

• Non-blocking tile fetching using spawned tasks
• Prevents duplicate requests for the same tile
• Coordinates multiple requests through ETS-based pending requests tables
• Broadcasts results to all waiting processes
• Handles errors gracefully with proper cleanup

Configuration

• :name - Optional custom name for the GenServer (defaults to generated name)
• :storage - Storage backend (:s3 or :local, auto-detected from bucket parameter)
• :enable_dir_cache - Enable file-based directory caching (default: false)
• :enable_tile_cache - Enable file-based tile caching (default: false)
• :cache_dir - Directory for cache files (default: System.tmp_dir!() <> "/ex_pmtiles_cache")
• :max_cache_age_ms - Maximum age of cache in milliseconds before automatic clearing (default: nil, disabled). When set, clears cache on GenServer startup and periodically based on age.
• :cleanup_interval_ms - Interval for checking cache age in milliseconds (default: 1 hour)
• :file_check_interval_ms - Interval for checking if source file has changed (default: 5 minutes). When the file changes (detected via ETag for S3 or mtime for local), cache is automatically cleared and repopulated.

Usage

# Start the cache for an S3 PMTiles file
{:ok, pid} = ExPmtiles.Cache.start_link(bucket: "maps", path: "map.pmtiles")

# Start the cache for a local PMTiles file
{:ok, pid} = ExPmtiles.Cache.start_link(bucket: nil, path: "/path/to/map.pmtiles")

# Start with automatic cache clearing after 24 hours
{:ok, pid} = ExPmtiles.Cache.start_link(
  bucket: "maps",
  path: "map.pmtiles",
  enable_tile_cache: true,
  max_cache_age_ms: :timer.hours(24),
  cleanup_interval_ms: :timer.hours(1)
)

# Start with custom file change detection interval
{:ok, pid} = ExPmtiles.Cache.start_link(
  bucket: "maps",
  path: "map.pmtiles",
  enable_dir_cache: true,
  file_check_interval_ms: :timer.minutes(1)  # Check for file changes every minute
)

# Get a tile by coordinates
case ExPmtiles.Cache.get_tile(pid, 10, 512, 256) do
  {:ok, tile_data} ->
    # Handle tile data
    tile_data
  {:error, reason} ->
    # Handle error
    nil
end

# Get cache statistics
stats = ExPmtiles.Cache.get_stats(pid)
# Returns: %{hits: 150, misses: 25}

# Manually clear the cache
ExPmtiles.Cache.clear_cache(pid)

Implementation Details

The GenServer maintains several tables and file structures:

• Directory cache files: Individual binary files in cache_dir/directories/ (when enable_dir_cache: true)
• Tile cache files: Individual binary files in cache_dir/tiles/ (when enable_tile_cache: true)
• Pending tile table: (ETS) Tracks in-progress tile requests to prevent duplicates
• Stats table: (ETS) Maintains hit/miss statistics
• Pending directory table: (ETS) Coordinates concurrent directory fetches

Performance Features

• Non-blocking init: GenServer starts immediately, ready to serve requests
• Concurrent tile fetching: Multiple tiles can be fetched concurrently using tasks
• File-based caching: Persistent storage with no size limits or corruption issues
• Request deduplication: Multiple requests for the same tile/directory are coordinated
• Scalable: Cache grows with your data without size constraints
• Race condition handling: Safe concurrent writes to cache files
• Automatic cache clearing: Optional time-based cache expiration with configurable intervals
• Smart repopulation: Directory cache automatically repopulates after clearing
• File change detection: Automatically detects when the source PMTiles file changes (via ETag for S3 or mtime for local files) and invalidates all caches, preventing stale or corrupted data from being served