LocalFileCacher (Local File Cacher v0.1.2)

View Source

This package manages a local file cache. The local file cache may be used, for example, to ensure that you have backups of files saved when making HTTP requests to API providers. The cached files can be used, for example, to quickly restore backups from the saved responses.

Project overview

The code in this module works on specific "application contexts" and "cache subdirectory paths".

The cache directory path is determined by joining three values together:

  • The base directory path for the entire file cache (e.g. "/tmp").

    • This value can be modified in the runtime application config (i.e. in config/runtime.exs):
      • config :local_file_cacher, base_path: System.tmp_dir()
  • The application context (e.g. YourProject.SomeApi)

  • The cache subdirectory path (e.g. "some_endpoint")

Using the above examples, the cache directory path would be "/tmp/your_project/some_api/some_endpoint".

Using the file cache

This package has functions that can be used to:

  • Save files to the configured directory path
  • Prune old files from the configured directory path

To prevent your local file cache from getting too large, the functions used to prune the file cache could be invoked automatically by any code that saves files to the cache. By pruning the cache at the same time as files are saved to it, this ensures that the cache size will not grow to an unreasonable level over time.

Saving files to the cache

To save a file to the cache, call LocalFileCacher.save_file_to_cache/5 with the relevant parameters for the context in which you are working.

Pruning (deleting old) files from the cache

To prune a part of the file cache, call LocalFileCacher.prune_file_cache!/2 with the relevant parameters for the context in which you are working.

The number of days to keep a cached file/directory can be modified in the runtime application config (i.e. in config/runtime.exs). For example:

  • config :local_file_cacher, days_to_keep_cached_files: 7

Application contexts

The application_context is a module that represents the context of the application you are working with.

Application context examples

  • When working in the YourProject.SomeApi context, the application_context should be YourProject.SomeApi.

  • When working in the YourProject.SomeApi.SomeCategory context, the application_context should be YourProject.SomeApi.SomeCategory.

Note that in the examples, the application context matches the value of the key in the config files that is used to configure that same context.

Cache subdirectory paths

A file_cache_subdirectory_path represents the subdirectory path of the file cache that is being used for the specific type of file being cached. Use a consistent naming strategy to ensure that files for each endpoint end up in their own subdirectory.

Cache subdirectory path examples

  • SomeAPI's someEndpoint endpoint: The endpoint's relative URL path could be used as the file_cache_subdirectory_path: "some_endpoint"

  • SomeAPI's someCategory/someEndpoint endpoint: The endpoint's relative URL path could be used as the file_cache_subdirectory_path, e.g. Path.join("some_category", "some_endpoint").

Tip

For URLs that are more than one layer deep, use Path.join/1 to ensure that the path is rendered correctly on all operating systems.

Cache subdirectory path style guide

  • Use lowercase letters for all file paths. For example, in an endpoint called "Endpoint", prefer a cache subdirectory name of "endpoint" instead of "Endpoint".

  • Separate multiple words with an underscore. For example, in an endpoint called "Some other endpoint", the cache subdirectory path should be called "some_other_endpoint".

  • To ensure that path names are rendered correctly on all operating systems, use Path.join/1 if the cache subdirectory path is more than one layer deep. For example, for an endpoint located at "https://some-api.com/v1/someCategory/someEndpoint", prefer a cache subdirectory path of Path.join(["v1", "some_category", "some_endpoint"]) instead of "v1/someCategory/someEndpoint".

Summary

Functions

Generate a timestamp-esque string that conforms to the Portable Filename Character Set, making it safe and convenient to use as the name of a cached file or directory.

Return a UNIX timestamp that can be used to indicate whether or not a cached file can be pruned (deleted). If the modification timestamp of a file is older than the cutoff timestamp, then it can be pruned.

Return the file cache directory path for a given application_context and optional file_cache_subdirectory_path.

Prune cached files/directories from a given application_context and file_cache_subdirectory_path.

Save file_contents to a given application_context and file_cache_subdirectory_path with a given filename_suffix (e.g. "json", "txt", "xml") and optional filename_prefix (i.e. the part of the filename before the dot).

Functions

generate_filename_friendly_timestamp()

@spec generate_filename_friendly_timestamp() :: String.t()

Generate a timestamp-esque string that conforms to the Portable Filename Character Set, making it safe and convenient to use as the name of a cached file or directory.

Examples

iex> YourProject.Services.FileCache.generate_filename_friendly_timestamp()
"2024-05-17-15-49-14-091430"

get_cutoff_timestamp()

@spec get_cutoff_timestamp() :: non_neg_integer()

Return a UNIX timestamp that can be used to indicate whether or not a cached file can be pruned (deleted). If the modification timestamp of a file is older than the cutoff timestamp, then it can be pruned.

Examples

iex> YourProject.Services.FileCache.get_cutoff_timestamp()
1715978870

get_file_cache_directory_path(application_context, file_cache_subdirectory_path \\ "")

@spec get_file_cache_directory_path(module(), String.t()) :: String.t()

Return the file cache directory path for a given application_context and optional file_cache_subdirectory_path.

Examples

Get the file cache directory path for a given endpoint:

iex> YourProject.Services.FileCache.get_file_cache_directory_path(
...>   YourProject.SomeApi,
...>   "some_endpoint"
...> )
"/tmp/your_project/some_api/some_endpoint"

To get the base file cache directory path for all endpoints in a given category, simply omit the file_cache_subdirectory_path argument:

iex> YourProject.Services.FileCache.get_file_cache_directory_path(YourProject.SomeApi)
"/tmp/your_project/some_api"

prune_file_cache!(application_context, file_cache_subdirectory_path)

Prune cached files/directories from a given application_context and file_cache_subdirectory_path.

Examples

Prune cached files saved from SomeAPI's someEndpoint endpoint:

iex> YourProject.Services.FileCache.prune_file_cache!(YourProject.SomeApi, "some_endpoint")
:ok

Prune cached directories/files saved from SomeAPI's someCategory/someEndpoint endpoint:

iex> YourProject.Services.FileCache.prune_file_cache!(
...>   _application_context = YourProject.SomeApi,
...>   _cache_subdirectory_path = Path.join(["some_category, "some_endpoint"])
...> )
:ok

Prune cached directories/files saved from some API's "someEndpoint" endpoint:

iex> LocalFileCacher.prune_file_cache!(YourProject.SomeApi, "some_endpoint")
:ok

save_file_to_cache(application_context, file_cache_subdirectory_path, file_contents, filename_suffix, filename_prefix \\ nil)

@spec save_file_to_cache(
  application_context :: module(),
  file_cache_subdirectory_path :: String.t(),
  file_contents :: binary(),
  filename_suffix :: String.t(),
  filename_prefix :: String.t() | nil
) :: :ok | {:error, File.posix()}

Save file_contents to a given application_context and file_cache_subdirectory_path with a given filename_suffix (e.g. "json", "txt", "xml") and optional filename_prefix (i.e. the part of the filename before the dot).

If filename_prefix is nil, then a timestamp-esque file prefix will be generated via generate_filename_friendly_timestamp/0.

Examples

Save a SomeAPI someEndpoint response to the "some_endpoint" cache subdirectory:

iex> {:ok, resp_body} = YourProject.SomeApi.get_some_endpoint()

iex> YourProject.Services.FileCache.save_file_to_cache(
...>   YourProject.SomeApi,
...>   "some_endpoint",
...>   resp_body,
...>   "json",
...> )
:ok

Save a SomeAPI someCategory/someEndpoint response to the "some_category/some_endpoint" cache subdirectory with a custom filename_prefix:

iex> {:ok, resp_body} = YourProject.SomeApi.SomeCategory.get_some_endpoint()

iex> YourProject.Services.FileCache.save_file_to_cache(
...>   YourProject.SomeApi,
...>   Path.join(["some_category", "some_endpoint"])
...>   resp_body,
...>   "json",
...>   "offset-000000"
...> )
:ok