Tinkex.CheckpointDownload (Tinkex v0.3.4)

View Source

Download and extract checkpoint archives with streaming support.

Provides memory-efficient checkpoint downloads using Finch.stream_while/5. Downloads are streamed directly to disk with O(1) memory usage, making it safe to download large checkpoint files (100MB-GBs) without risk of OOM errors.

Features

  • Streaming downloads - O(1) memory usage regardless of file size
  • Progress callbacks - Track download progress in real-time
  • Automatic extraction - Downloads and extracts tar archives in one operation
  • Force overwrite - Optional overwrite of existing checkpoint directories

Examples

# Basic download with automatic extraction
{:ok, service_pid} = Tinkex.ServiceClient.start_link(config: config)
{:ok, rest_client} = Tinkex.ServiceClient.create_rest_client(service_pid)

{:ok, result} = Tinkex.CheckpointDownload.download(
  rest_client,
  "tinker://run-123/weights/0001",
  output_dir: "./models",
  force: true
)

IO.puts("Downloaded to: #{result.destination}")

# Download with progress tracking
progress_fn = fn downloaded, total ->
  percent = if total > 0, do: Float.round(downloaded / total * 100, 1), else: 0
  IO.write("\rProgress: #{percent}% (#{downloaded} / #{total} bytes)")
end

{:ok, result} = Tinkex.CheckpointDownload.download(
  rest_client,
  "tinker://run-123/weights/0001",
  output_dir: "./models",
  progress: progress_fn
)

Summary

Functions

download(rest_client, checkpoint_path, opts \\ [])

@spec download(Tinkex.RestClient.t(), String.t(), keyword()) ::
  {:ok, map()} | {:error, term()}

Download and extract a checkpoint.

Options

  • :output_dir - Parent directory for extraction (default: current directory)
  • :force - Overwrite existing directory (default: false)
  • :progress - Progress callback function fn(downloaded, total) -> any

Returns

  • {:ok, %{destination: path, checkpoint_path: path}} on success
  • {:error, {:exists, path}} if target exists and force is false
  • {:error, {:invalid_path, message}} if checkpoint path is invalid
  • {:error, reason} for other failures