HfHub.Repository (HfHub v0.2.0)

Copy Markdown View Source

Repository reference types and helpers for HuggingFace Hub.

Provides Bumblebee-compatible repository tuple types alongside the existing keyword-based API.

Repository Types

A repository can be referenced as:

  • {:hf, repository_id} - HuggingFace Hub repository
  • {:hf, repository_id, opts} - HuggingFace Hub repository with options
  • {:local, directory} - Local directory containing model files

Options for :hf repositories

  • :revision - Git revision (branch, tag, or commit). Defaults to "main".
  • :cache_dir - Override the cache directory
  • :offline - If true, only use cached files
  • :auth_token - Authentication token for private repos
  • :subdir - Subdirectory within the repository

Examples

# Simple HuggingFace repository
{:hf, "bert-base-uncased"}

# With options
{:hf, "bert-base-uncased", revision: "v1.0", auth_token: "hf_xxx"}

# Local directory
{:local, "/path/to/model"}

Summary

Types

opts()
t()

Functions

cache_scope(repository_id)

Converts a repository ID to a cache scope string.

file_listing_url(arg)

Returns a URL to list the contents of a HuggingFace repository.

file_url(arg, filename)

Returns a URL pointing to a file in a HuggingFace repository.

normalize!(other)

Normalizes a repository reference to its canonical form.

Types

opts()

@type opts() :: [
  revision: String.t(),
  cache_dir: Path.t(),
  offline: boolean(),
  auth_token: String.t(),
  subdir: String.t()
]

t()

@type t() :: {:hf, String.t()} | {:hf, String.t(), keyword()} | {:local, Path.t()}

Functions

cache_scope(repository_id)

@spec cache_scope(String.t()) :: String.t()

Converts a repository ID to a cache scope string.

Used for organizing cached files by repository.

Examples

iex> HfHub.Repository.cache_scope("openai/gpt-2")
"openai--gpt-2"

file_listing_url(arg)

@spec file_listing_url({:hf, String.t(), keyword()}) :: String.t()

Returns a URL to list the contents of a HuggingFace repository.

Examples

iex> HfHub.Repository.file_listing_url({:hf, "bert-base-uncased", []})
"https://huggingface.co/api/models/bert-base-uncased/tree/main"

file_url(arg, filename)

@spec file_url({:hf, String.t(), keyword()}, String.t()) :: String.t()

Returns a URL pointing to a file in a HuggingFace repository.

Examples

iex> HfHub.Repository.file_url({:hf, "bert-base-uncased", []}, "config.json")
"https://huggingface.co/bert-base-uncased/resolve/main/config.json"

iex> HfHub.Repository.file_url({:hf, "bert-base-uncased", revision: "v1.0"}, "config.json")
"https://huggingface.co/bert-base-uncased/resolve/v1.0/config.json"

normalize!(other)

@spec normalize!(t()) :: {:hf, String.t(), keyword()} | {:local, Path.t()}

Normalizes a repository reference to its canonical form.

Converts {:hf, id} to {:hf, id, []} for consistent handling.

Examples

iex> HfHub.Repository.normalize!({:hf, "bert-base-uncased"})
{:hf, "bert-base-uncased", []}

iex> HfHub.Repository.normalize!({:hf, "bert-base-uncased", revision: "v1.0"})
{:hf, "bert-base-uncased", [revision: "v1.0"]}

iex> HfHub.Repository.normalize!({:local, "/path/to/model"})
{:local, "/path/to/model"}