HuggingFace Storage Buckets — S3-like object storage powered by Xet.
Unlike Git-based repositories, buckets provide mutable, versioned-free object storage: ideal for checkpoints, logs, large artifacts, and any files that don't need Git history.
See: https://huggingface.co/docs/huggingface_hub/guides/buckets
Example
# Create a bucket
{:ok, url} = HuggingfaceClient.create_bucket("my-bucket", access_token: "hf_...")
# Upload a file
:ok = HuggingfaceClient.upload_bucket_file("username/my-bucket",
source: "./weights.bin",
destination: "checkpoints/v1.bin",
access_token: "hf_..."
)
# Browse files
result = HuggingfaceClient.list_bucket_tree("username/my-bucket", access_token: "hf_...")
|> Enum.each(fn item -> IO.puts("#{item["path"]} (#{item["size"]})") end)
# Download
{:ok, content} = HuggingfaceClient.get_bucket_file("username/my-bucket",
path: "checkpoints/v1.bin",
access_token: "hf_..."
)
Summary
Functions
Adds and/or deletes files in a bucket in a single operation.
Copies files between Hub locations (repos or buckets) to a bucket destination.
Creates a new storage bucket.
Deletes a bucket and all its contents. This is irreversible.
Downloads a file from a bucket, returning its content as binary.
Gets metadata about a bucket (size, file count, visibility, etc.).
Lists all buckets for a namespace.
Lists files and directories in a bucket, returning a lazy stream.
Moves or renames a bucket (can transfer to a different namespace).
Uploads a single file to a bucket.
Functions
@spec batch_files( String.t(), keyword() ) :: :ok | {:error, Exception.t()}
Adds and/or deletes files in a bucket in a single operation.
Options
:add— list of{source, destination}tuples (source = file path or binary):delete— list of file paths to delete from the bucket:access_token
Example
:ok = HuggingfaceClient.batch_bucket_files("username/my-bucket",
add: [
{"./model.safetensors", "models/model.safetensors"},
{Jason.encode!(%{key: "val"}), "config.json"}
],
delete: ["old-weights.bin"],
access_token: "hf_..."
)
@spec copy_files(keyword()) :: :ok | {:error, Exception.t()}
Copies files between Hub locations (repos or buckets) to a bucket destination.
Options
:source— sourcehf://path (required):destination— destinationhf://buckets/path (required):access_token
Example
:ok = HuggingfaceClient.copy_files(
source: "hf://datasets/my-user/dataset/data/",
destination: "hf://buckets/my-user/my-bucket/data/",
access_token: "hf_..."
)
@spec create( String.t(), keyword() ) :: {:ok, map()} | {:error, Exception.t()}
Creates a new storage bucket.
Options
:private— iftrue, bucket is private (default: org/user setting):exist_ok— don't raise if bucket already exists (default:false):namespace— namespace (default: authenticated user):access_token
Returns
{:ok, bucket_url_info} with "bucket_id", "url", and "handle".
Example
{:ok, url} = HuggingfaceClient.create_bucket("checkpoints",
private: true,
exist_ok: true,
access_token: "hf_..."
)
IO.puts("Handle: #{url["handle"]}")
# "hf://buckets/username/checkpoints"
@spec delete( String.t(), keyword() ) :: :ok | {:error, Exception.t()}
Deletes a bucket and all its contents. This is irreversible.
Options
:missing_ok— don't raise if bucket doesn't exist (default:false):access_token
Example
:ok = HuggingfaceClient.delete_bucket("username/old-bucket",
access_token: "hf_..."
)
@spec get_file( String.t(), keyword() ) :: {:ok, binary()} | {:error, Exception.t()}
Downloads a file from a bucket, returning its content as binary.
Options
:path— file path in the bucket (required):access_token
Example
{:ok, content} = HuggingfaceClient.get_bucket_file("username/my-bucket",
path: "config.json",
access_token: "hf_..."
)
config = Jason.decode!(content)
@spec info( String.t(), keyword() ) :: {:ok, map()} | {:error, Exception.t()}
Gets metadata about a bucket (size, file count, visibility, etc.).
Example
{:ok, info} = HuggingfaceClient.bucket_info("username/my-bucket",
access_token: "hf_..."
)
IO.puts("Size: #{info["size"]} bytes, Files: #{info["total_files"]}")
@spec list(keyword()) :: {:ok, [map()]} | {:error, Exception.t()}
Lists all buckets for a namespace.
Options
:namespace— user or org (default: authenticated user):access_token
Example
{:ok, buckets} = HuggingfaceClient.list_buckets(access_token: "hf_...")
Enum.each(buckets, fn b ->
IO.puts("#{b["id"]}: #{b["total_files"]} files, #{b["size"]} bytes")
end)
@spec list_tree( String.t(), keyword() ) :: {:ok, [map()]} | {:error, Exception.t()}
Lists files and directories in a bucket, returning a lazy stream.
Each item is a map with "path", "type" (file/directory), "size", and "modified_at".
Options
:prefix— only return items under this path prefix:recursive— recurse into subdirectories (default:true):access_token
Example
HuggingfaceClient.list_bucket_tree("username/my-bucket",
prefix: "checkpoints/",
recursive: true,
access_token: "hf_..."
)
|> Enum.each(fn item ->
IO.puts("#{item["path"]} - #{item["size"]} bytes")
end)
@spec move(keyword()) :: :ok | {:error, Exception.t()}
Moves or renames a bucket (can transfer to a different namespace).
Options
:from_id— source bucket ID (required):to_id— destination bucket ID (required):access_token
Example
:ok = HuggingfaceClient.move_bucket(
from_id: "username/old-name",
to_id: "my-org/new-name",
access_token: "hf_..."
)
@spec upload_file( String.t(), keyword() ) :: :ok | {:error, Exception.t()}
Uploads a single file to a bucket.
Options
:source— local file path (string) or binary content (required):destination— destination path in the bucket (required):content_type— MIME type (default:"application/octet-stream"):access_token
Example
:ok = HuggingfaceClient.upload_bucket_file("username/my-bucket",
source: "./model.safetensors",
destination: "models/v1/model.safetensors",
access_token: "hf_..."
)