HfHub.Hub (HfHub v0.2.0)

Bumblebee-compatible Hub interface for HuggingFace.

Provides ETag-based caching compatible with Bumblebee's Bumblebee.HuggingFace.Hub. This module uses URL-based caching with ETag validation, while the existing HfHub.Download module uses revision-based path caching.

Examples

# Download with ETag-based caching
{:ok, path} = HfHub.Hub.cached_download(
  "https://huggingface.co/bert-base-uncased/resolve/main/config.json"
)

# With options
{:ok, path} = HfHub.Hub.cached_download(url,
  cache_dir: "/custom/cache",
  auth_token: "hf_xxx",
  offline: true
)

Summary

Functions

cached_download(url, opts \\ [])

Downloads a file from a URL with ETag-based caching.

file_listing_url(repository_id, subdir, revision)

Returns a URL to list the contents of a HuggingFace repository.

file_url(repository_id, filename, revision)

Returns a URL pointing to a file in a HuggingFace repository.

Functions

cached_download(url, opts \\ [])

@spec cached_download(
  String.t(),
  keyword()
) :: {:ok, String.t()} | {:error, String.t()}

Downloads a file from a URL with ETag-based caching.

The file is cached based on the received ETag. Subsequent requests for the same URL validate the ETag and return a file from the cache if there is a match.

Options

:cache_dir - Override the default cache directory
:offline - If true, only return cached files (no network requests)
:auth_token - Authentication token for private repositories
:etag - If known, skip the HEAD request to fetch ETag
:cache_scope - Namespace for organizing cached files

Examples

{:ok, path} = HfHub.Hub.cached_download(
  "https://huggingface.co/bert-base-uncased/resolve/main/config.json"
)

file_listing_url(repository_id, subdir, revision)

@spec file_listing_url(String.t(), String.t() | nil, String.t() | nil) :: String.t()

Returns a URL to list the contents of a HuggingFace repository.

Examples

iex> HfHub.Hub.file_listing_url("bert-base-uncased", nil, nil)
"https://huggingface.co/api/models/bert-base-uncased/tree/main"

iex> HfHub.Hub.file_listing_url("bert-base-uncased", "tokenizer", "v1.0")
"https://huggingface.co/api/models/bert-base-uncased/tree/v1.0/tokenizer"

file_url(repository_id, filename, revision)

@spec file_url(String.t(), String.t(), String.t() | nil) :: String.t()

Returns a URL pointing to a file in a HuggingFace repository.

Examples

iex> HfHub.Hub.file_url("bert-base-uncased", "config.json", nil)
"https://huggingface.co/bert-base-uncased/resolve/main/config.json"

iex> HfHub.Hub.file_url("bert-base-uncased", "config.json", "v1.0")
"https://huggingface.co/bert-base-uncased/resolve/v1.0/config.json"