HuggingfaceClient.Hub.Buckets (huggingface_client v0.1.0)

Copy Markdown View Source

HuggingFace Storage Buckets — S3-like object storage powered by Xet.

Unlike Git-based repositories, buckets provide mutable, versioned-free object storage: ideal for checkpoints, logs, large artifacts, and any files that don't need Git history.

See: https://huggingface.co/docs/huggingface_hub/guides/buckets

Example

# Create a bucket
{:ok, url} = HuggingfaceClient.create_bucket("my-bucket", access_token: "hf_...")

# Upload a file
:ok = HuggingfaceClient.upload_bucket_file("username/my-bucket",
  source: "./weights.bin",
  destination: "checkpoints/v1.bin",
  access_token: "hf_..."
)

# Browse files
result = HuggingfaceClient.list_bucket_tree("username/my-bucket", access_token: "hf_...")
|> Enum.each(fn item -> IO.puts("#{item["path"]} (#{item["size"]})") end)

# Download
{:ok, content} = HuggingfaceClient.get_bucket_file("username/my-bucket",
  path: "checkpoints/v1.bin",
  access_token: "hf_..."
)

Summary

Functions

Adds and/or deletes files in a bucket in a single operation.

Copies files between Hub locations (repos or buckets) to a bucket destination.

Creates a new storage bucket.

Deletes a bucket and all its contents. This is irreversible.

Downloads a file from a bucket, returning its content as binary.

Gets metadata about a bucket (size, file count, visibility, etc.).

Lists all buckets for a namespace.

Lists files and directories in a bucket, returning a lazy stream.

Moves or renames a bucket (can transfer to a different namespace).

Uploads a single file to a bucket.

Functions

batch_files(bucket_id, opts \\ [])

@spec batch_files(
  String.t(),
  keyword()
) :: :ok | {:error, Exception.t()}

Adds and/or deletes files in a bucket in a single operation.

Options

  • :add — list of {source, destination} tuples (source = file path or binary)
  • :delete — list of file paths to delete from the bucket
  • :access_token

Example

:ok = HuggingfaceClient.batch_bucket_files("username/my-bucket",
  add: [
    {"./model.safetensors", "models/model.safetensors"},
    {Jason.encode!(%{key: "val"}), "config.json"}
  ],
  delete: ["old-weights.bin"],
  access_token: "hf_..."
)

copy_files(opts)

@spec copy_files(keyword()) :: :ok | {:error, Exception.t()}

Copies files between Hub locations (repos or buckets) to a bucket destination.

Options

  • :source — source hf:// path (required)
  • :destination — destination hf://buckets/ path (required)
  • :access_token

Example

:ok = HuggingfaceClient.copy_files(
  source: "hf://datasets/my-user/dataset/data/",
  destination: "hf://buckets/my-user/my-bucket/data/",
  access_token: "hf_..."
)

create(bucket_id, opts \\ [])

@spec create(
  String.t(),
  keyword()
) :: {:ok, map()} | {:error, Exception.t()}

Creates a new storage bucket.

Options

  • :private — if true, bucket is private (default: org/user setting)
  • :exist_ok — don't raise if bucket already exists (default: false)
  • :namespace — namespace (default: authenticated user)
  • :access_token

Returns

{:ok, bucket_url_info} with "bucket_id", "url", and "handle".

Example

{:ok, url} = HuggingfaceClient.create_bucket("checkpoints",
  private: true,
  exist_ok: true,
  access_token: "hf_..."
)
IO.puts("Handle: #{url["handle"]}")
# "hf://buckets/username/checkpoints"

delete(bucket_id, opts \\ [])

@spec delete(
  String.t(),
  keyword()
) :: :ok | {:error, Exception.t()}

Deletes a bucket and all its contents. This is irreversible.

Options

  • :missing_ok — don't raise if bucket doesn't exist (default: false)
  • :access_token

Example

:ok = HuggingfaceClient.delete_bucket("username/old-bucket",
  access_token: "hf_..."
)

get_file(bucket_id, opts \\ [])

@spec get_file(
  String.t(),
  keyword()
) :: {:ok, binary()} | {:error, Exception.t()}

Downloads a file from a bucket, returning its content as binary.

Options

  • :path — file path in the bucket (required)
  • :access_token

Example

{:ok, content} = HuggingfaceClient.get_bucket_file("username/my-bucket",
  path: "config.json",
  access_token: "hf_..."
)
config = Jason.decode!(content)

info(bucket_id, opts \\ [])

@spec info(
  String.t(),
  keyword()
) :: {:ok, map()} | {:error, Exception.t()}

Gets metadata about a bucket (size, file count, visibility, etc.).

Example

{:ok, info} = HuggingfaceClient.bucket_info("username/my-bucket",
  access_token: "hf_..."
)
IO.puts("Size: #{info["size"]} bytes, Files: #{info["total_files"]}")

list(opts \\ [])

@spec list(keyword()) :: {:ok, [map()]} | {:error, Exception.t()}

Lists all buckets for a namespace.

Options

  • :namespace — user or org (default: authenticated user)
  • :access_token

Example

{:ok, buckets} = HuggingfaceClient.list_buckets(access_token: "hf_...")
Enum.each(buckets, fn b ->
  IO.puts("#{b["id"]}: #{b["total_files"]} files, #{b["size"]} bytes")
end)

list_tree(bucket_id, opts \\ [])

@spec list_tree(
  String.t(),
  keyword()
) :: {:ok, [map()]} | {:error, Exception.t()}

Lists files and directories in a bucket, returning a lazy stream.

Each item is a map with "path", "type" (file/directory), "size", and "modified_at".

Options

  • :prefix — only return items under this path prefix
  • :recursive — recurse into subdirectories (default: true)
  • :access_token

Example

HuggingfaceClient.list_bucket_tree("username/my-bucket",
  prefix: "checkpoints/",
  recursive: true,
  access_token: "hf_..."
)
|> Enum.each(fn item ->
  IO.puts("#{item["path"]} - #{item["size"]} bytes")
end)

move(opts)

@spec move(keyword()) :: :ok | {:error, Exception.t()}

Moves or renames a bucket (can transfer to a different namespace).

Options

  • :from_id — source bucket ID (required)
  • :to_id — destination bucket ID (required)
  • :access_token

Example

:ok = HuggingfaceClient.move_bucket(
  from_id: "username/old-name",
  to_id: "my-org/new-name",
  access_token: "hf_..."
)

upload_file(bucket_id, opts \\ [])

@spec upload_file(
  String.t(),
  keyword()
) :: :ok | {:error, Exception.t()}

Uploads a single file to a bucket.

Options

  • :source — local file path (string) or binary content (required)
  • :destination — destination path in the bucket (required)
  • :content_type — MIME type (default: "application/octet-stream")
  • :access_token

Example

:ok = HuggingfaceClient.upload_bucket_file("username/my-bucket",
  source: "./model.safetensors",
  destination: "models/v1/model.safetensors",
  access_token: "hf_..."
)