PhoenixKit.Modules.Publishing.ListingCache (phoenix_kit v1.7.38)

Copy Markdown View Source

Caches publishing group listing metadata to avoid expensive filesystem scans on every request.

Instead of scanning 50+ files per request, the listing page reads a single .listing_cache.json file containing all post metadata.

How It Works

  1. When a post is created/updated/published, regenerate/1 is called
  2. This scans all posts and writes metadata to .listing_cache.json
  3. render_blog_listing reads from cache instead of scanning filesystem
  4. Cache includes: title, slug, date, status, languages, versions (no content)

Cache File Location

priv/publishing/{group-slug}/.listing_cache.json

(With legacy fallback to priv/blogging/{group-slug}/.listing_cache.json)

Performance

  • Before: ~500ms (50+ file operations)
  • After: ~20ms (1 file read + JSON parse)

Cache Invalidation

Cache is regenerated when:

  • Post is created
  • Post is updated (metadata or content)
  • Post status changes (draft/published/archived)
  • Translation is added
  • Version is created

In-Memory Caching with :persistent_term

For sub-millisecond performance, parsed cache data is stored in :persistent_term.

  • First read after restart: loads from file, parses JSON, stores in :persistent_term (~2ms)
  • Subsequent reads: direct memory access (~0.1μs, no variance)
  • On regenerate: updates both file and :persistent_term
  • On invalidate: clears :persistent_term entry

The JSON file provides persistence across restarts. :persistent_term provides zero-copy, sub-microsecond reads during runtime.

Summary

Functions

Returns the cache file path for a publishing group.

Counts posts by version structure status for a group.

Counts posts by primary_language status in a group.

Checks if a cache exists for a blog (in :persistent_term or file).

Returns whether file caching is enabled. Uses cached settings to avoid database queries on every call. Checks new key first, falls back to legacy key.

Returns the :persistent_term key for tracking the file's generated_at when loaded into memory.

Finds a post by a previous URL slug for 301 redirects.

Finds a post by URL slug for a specific language.

Finds a post by slug in the cache.

Finds a post by path pattern in the cache (for timestamp mode).

Invalidates (deletes) the cache for a blog.

Loads the cache from file into :persistent_term without regenerating the file.

Returns the :persistent_term key for tracking when the memory cache was loaded.

Returns whether memory caching (:persistent_term) is enabled. Uses cached settings to avoid database queries on every call. Checks new key first, falls back to legacy key.

Returns the file's generated_at timestamp that was stored when the memory cache was loaded. This tells us what version of the file data is currently in memory.

Returns when the memory cache was loaded (ISO 8601 string), or nil if not loaded.

Returns the :persistent_term key for a publishing group's cache.

Returns a list of posts that need primary_language migration.

Returns list of posts that need version structure migration.

Reads the cached listing for a publishing group.

Regenerates the listing cache for a blog.

Regenerates only the file cache without loading into memory.

Regenerates the cache if no other process is already regenerating it.

Functions

cache_path(group_slug)

@spec cache_path(String.t()) :: String.t()

Returns the cache file path for a publishing group.

count_legacy_structure_status(blog_slug)

@spec count_legacy_structure_status(String.t()) :: map()

Counts posts by version structure status for a group.

Returns %{versioned: n, legacy: n} where:

  • versioned - posts with v1/, v2/, etc. structure
  • legacy - posts with flat file structure (need migration)

count_primary_language_status(blog_slug)

@spec count_primary_language_status(String.t()) :: map()

Counts posts by primary_language status in a group.

Returns %{current: n, needs_migration: n, needs_backfill: n} where:

  • current - posts with primary_language matching global setting
  • needs_migration - posts with different primary_language (were created under old setting)
  • needs_backfill - posts with no primary_language stored (legacy posts)

exists?(blog_slug)

@spec exists?(String.t()) :: boolean()

Checks if a cache exists for a blog (in :persistent_term or file).

file_cache_enabled?()

@spec file_cache_enabled?() :: boolean()

Returns whether file caching is enabled. Uses cached settings to avoid database queries on every call. Checks new key first, falls back to legacy key.

file_generated_at_key(blog_slug)

@spec file_generated_at_key(String.t()) :: tuple()

Returns the :persistent_term key for tracking the file's generated_at when loaded into memory.

find_by_previous_url_slug(group_slug, language, url_slug)

@spec find_by_previous_url_slug(String.t(), String.t(), String.t()) ::
  {:ok, map()} | {:error, :not_found | :cache_miss}

Finds a post by a previous URL slug for 301 redirects.

When a URL slug changes, the old slug is stored in previous_url_slugs. This function finds posts that previously used the given URL slug.

Returns

  • {:ok, cached_post} - Found post that previously used this slug
  • {:error, :not_found} - No post with this previous slug
  • {:error, :cache_miss} - Cache not available

find_by_url_slug(group_slug, language, url_slug)

@spec find_by_url_slug(String.t(), String.t(), String.t()) ::
  {:ok, map()} | {:error, :not_found | :cache_miss}

Finds a post by URL slug for a specific language.

This enables O(1) lookup from URL slug to internal identifier, supporting per-language URL slugs for SEO-friendly localized URLs.

Parameters

  • group_slug - The publishing group
  • language - The language code to search in
  • url_slug - The URL slug to find

Returns

  • {:ok, cached_post} - Found post (includes internal slug for file lookup)
  • {:error, :not_found} - No post with this URL slug for this language
  • {:error, :cache_miss} - Cache not available

find_post(blog_slug, post_slug)

@spec find_post(String.t(), String.t()) ::
  {:ok, map()} | {:error, :not_found | :cache_miss}

Finds a post by slug in the cache.

This is useful for single post views where we need metadata (language_statuses, version_statuses, allow_version_access) without reading multiple files.

Returns {:ok, cached_post} if found, {:error, :not_found} otherwise.

find_post_by_path(blog_slug, date, time)

@spec find_post_by_path(String.t(), String.t(), String.t()) ::
  {:ok, map()} | {:error, :not_found | :cache_miss}

Finds a post by path pattern in the cache (for timestamp mode).

Matches posts where the path contains the date/time pattern. Returns {:ok, cached_post} if found, {:error, :not_found} otherwise.

invalidate(blog_slug)

@spec invalidate(String.t()) :: :ok

Invalidates (deletes) the cache for a blog.

Clears both the :persistent_term entry and the JSON file. The next read will return :cache_miss, triggering a fallback to the filesystem scan.

load_into_memory(blog_slug)

@spec load_into_memory(String.t()) :: :ok | {:error, any()}

Loads the cache from file into :persistent_term without regenerating the file.

Returns :ok if successful, {:error, :no_file} if file doesn't exist, or {:error, reason} for other failures.

loaded_at_key(blog_slug)

@spec loaded_at_key(String.t()) :: tuple()

Returns the :persistent_term key for tracking when the memory cache was loaded.

memory_cache_enabled?()

@spec memory_cache_enabled?() :: boolean()

Returns whether memory caching (:persistent_term) is enabled. Uses cached settings to avoid database queries on every call. Checks new key first, falls back to legacy key.

memory_file_generated_at(blog_slug)

@spec memory_file_generated_at(String.t()) :: String.t() | nil

Returns the file's generated_at timestamp that was stored when the memory cache was loaded. This tells us what version of the file data is currently in memory.

memory_loaded_at(blog_slug)

@spec memory_loaded_at(String.t()) :: String.t() | nil

Returns when the memory cache was loaded (ISO 8601 string), or nil if not loaded.

persistent_term_key(group_slug)

@spec persistent_term_key(String.t()) :: tuple()

Returns the :persistent_term key for a publishing group's cache.

posts_needing_primary_language_migration(blog_slug)

@spec posts_needing_primary_language_migration(String.t()) :: [map()]

Returns a list of posts that need primary_language migration.

This checks all posts in a group and returns those that either:

  1. Have no primary_language stored (need backfill)
  2. Have primary_language different from global setting (need migration decision)

posts_needing_version_migration(blog_slug)

@spec posts_needing_version_migration(String.t()) :: [map()]

Returns list of posts that need version structure migration.

read(blog_slug)

@spec read(String.t()) :: {:ok, [map()]} | {:error, :cache_miss}

Reads the cached listing for a publishing group.

Returns {:ok, posts} if cache exists and is valid. Returns {:error, :cache_miss} if cache doesn't exist, is corrupt, or caching is disabled.

Respects the publishing_file_cache_enabled and publishing_memory_cache_enabled settings (with fallback to legacy blogging_* keys).

regenerate(blog_slug)

@spec regenerate(String.t()) :: :ok | {:error, any()}

Regenerates the listing cache for a blog.

Scans all posts using the standard list_posts function and writes the metadata to .listing_cache.json.

This should be called after any post operation that changes the listing:

  • create_post
  • update_post
  • add_language_to_post
  • create_new_version

Returns :ok on success or {:error, reason} on failure.

regenerate_file_only(blog_slug)

@spec regenerate_file_only(String.t()) :: :ok | {:error, any()}

Regenerates only the file cache without loading into memory.

This scans all posts and writes to .listing_cache.json but does not update :persistent_term. Use load_into_memory/1 separately if needed.

regenerate_if_not_in_progress(blog_slug)

@spec regenerate_if_not_in_progress(String.t()) ::
  :ok | :already_in_progress | {:error, any()}

Regenerates the cache if no other process is already regenerating it.

This prevents the "thundering herd" problem where multiple concurrent requests all trigger cache regeneration simultaneously after a server restart.

Uses ETS with insert_new/2 for atomic lock acquisition - only one process can acquire the lock at a time. The lock includes a timestamp and will be considered stale after 30000ms to prevent permanent lockout if a process dies mid-regeneration.

Returns:

  • :ok if regeneration was performed successfully
  • :already_in_progress if another process is currently regenerating
  • {:error, reason} if regeneration failed

Usage

On cache miss in read paths, use this instead of regenerate/1:

case ListingCache.regenerate_if_not_in_progress(blog_slug) do
  :ok -> # Cache is ready, read from it
  :already_in_progress -> # Fall back to filesystem scan
  {:error, _} -> # Fall back to filesystem scan
end