HfHub.FS (HfHub v0.2.0)

Copy Markdown View Source

Filesystem utilities for HuggingFace cache management.

Provides low-level functions for managing the local cache directory structure, file paths, and locking mechanisms.

Examples

# Ensure cache directory exists
:ok = HfHub.FS.ensure_cache_dir()

# Get repository path
path = HfHub.FS.repo_path("bert-base-uncased", :model)

# Get file path in repository
path = HfHub.FS.file_path("bert-base-uncased", :model, "config.json")

# Acquire file lock for concurrent downloads
{:ok, lock} = HfHub.FS.lock_file("bert-base-uncased", "pytorch_model.bin")
# ... download file ...
:ok = HfHub.FS.unlock_file(lock)

Summary

Functions

cache_dir()

Gets the configured cache directory.

ensure_cache_dir()

Ensures the cache directory exists.

file_path(repo_id, repo_type, filename, revision \\ "main")

Gets the local path for a file in a repository.

lock_file(repo_id, filename)

Acquires a lock on a file for concurrent download protection.

repo_path(repo_id, repo_type)

Gets the local path for a repository.

unlock_file(lock)

Releases a file lock.

Functions

cache_dir()

@spec cache_dir() :: Path.t()

Gets the configured cache directory.

Examples

dir = HfHub.FS.cache_dir()
# => "/home/user/.cache/huggingface"

ensure_cache_dir()

@spec ensure_cache_dir() :: :ok | {:error, term()}

Ensures the cache directory exists.

Creates the cache directory and any necessary subdirectories if they don't exist.

Examples

:ok = HfHub.FS.ensure_cache_dir()

file_path(repo_id, repo_type, filename, revision \\ "main")

@spec file_path(
  HfHub.repo_id(),
  HfHub.repo_type(),
  HfHub.filename(),
  HfHub.revision()
) :: Path.t()

Gets the local path for a file in a repository.

Arguments

  • repo_id - Repository ID
  • repo_type - Type of repository
  • filename - Name of the file
  • revision - Git revision (defaults to "main")

Examples

path = HfHub.FS.file_path("bert-base-uncased", :model, "config.json")
# => "/home/user/.cache/huggingface/hub/models--bert-base-uncased/snapshots/main/config.json"

lock_file(repo_id, filename)

@spec lock_file(HfHub.repo_id(), HfHub.filename()) ::
  {:ok, reference()} | {:error, term()}

Acquires a lock on a file for concurrent download protection.

Returns {:ok, lock} where lock is a reference that can be used to unlock the file.

Arguments

  • repo_id - Repository ID
  • filename - Name of the file

Examples

{:ok, lock} = HfHub.FS.lock_file("bert-base-uncased", "pytorch_model.bin")
# ... perform download ...
:ok = HfHub.FS.unlock_file(lock)

repo_path(repo_id, repo_type)

@spec repo_path(HfHub.repo_id(), HfHub.repo_type()) :: Path.t()

Gets the local path for a repository.

Arguments

  • repo_id - Repository ID
  • repo_type - Type of repository (:model, :dataset, or :space)

Examples

path = HfHub.FS.repo_path("bert-base-uncased", :model)
# => "/home/user/.cache/huggingface/hub/models--bert-base-uncased"

unlock_file(lock)

@spec unlock_file(reference()) :: :ok | {:error, term()}

Releases a file lock.

Arguments

Examples

{:ok, lock} = HfHub.FS.lock_file("bert-base-uncased", "config.json")
:ok = HfHub.FS.unlock_file(lock)