HfHub.DatasetFiles (HfHub v0.2.0)

Resolve dataset file paths by config and split.

Summary

tree_entry()

Infers dataset config names from a repo tree.

Resolves dataset file paths for a config and split by listing the repo tree.

Resolves dataset file paths from a pre-fetched repo tree.

Infers available split names from a repo tree for a config.

@type tree_entry() :: %{
  type: :file | :folder,
  path: String.t(),
  size: non_neg_integer() | nil,
  lfs: map() | nil,
  oid: String.t() | nil
}

@spec configs_from_tree([tree_entry()]) :: [String.t()]

Infers dataset config names from a repo tree.

@spec resolve(HfHub.repo_id(), String.t(), String.t(), keyword()) ::
  {:ok, [String.t()]} | {:error, term()}

Resolves dataset file paths for a config and split by listing the repo tree.

@spec resolve_from_tree([tree_entry()], String.t(), String.t()) ::
  {:ok, [String.t()]} | {:error, :no_files_found}

Resolves dataset file paths from a pre-fetched repo tree.

@spec splits_from_tree([tree_entry()], String.t()) :: [String.t()]

Infers available split names from a repo tree for a config.