Nasty.Statistics.ModelDownloader (Nasty v0.3.0)

View Source

Downloads pre-trained statistical models from GitHub releases.

This module provides functionality to download models hosted on GitHub releases, verify their integrity using SHA256 checksums, and install them locally.

Usage

# Download a specific model
ModelDownloader.download("en-pos-v1", output_dir: "priv/models/en")

# Download with automatic prompt
ModelDownloader.download_if_missing(:en, :pos_tagging, "v1")

Model Repository

Models are expected to be hosted at: https://github.com/USER/REPO/releases/download/models-VERSION/MODEL_ID

Each model should have:

  • MODEL_ID.model - The model file
  • MODEL_ID.meta.json - Metadata
  • MODEL_ID.sha256 - SHA256 checksum

Future Implementation

This module is currently a stub for future GitHub integration. To implement:

  1. Add HTTP client dependency (e.g., req or httpoison)
  2. Implement actual download logic
  3. Add progress reporting
  4. Add retry logic and error handling
  5. Configure repository URLs

Summary

Types

model_id()
options()

Functions

download(model_id, opts \\ [])

Downloads a model by ID from GitHub releases.

download_if_missing(language, task, version, opts \\ [])

Downloads a model if it's not already available locally.

list_available()

Lists all available models in the remote repository.

verify_checksum(model_path, expected_hash)

Verifies the SHA256 checksum of a downloaded model.

Types

model_id()

@type model_id() :: String.t()

options()

@type options() :: keyword()

Functions

download(model_id, opts \\ [])

@spec download(model_id(), options()) :: {:ok, String.t()} | {:error, term()}

Downloads a model by ID from GitHub releases.

Options

  • :output_dir - Directory to save the model (default: "priv/models")
  • :repo - GitHub repository (default: from config)
  • :force - Force download even if file exists (default: false)
  • :verify_checksum - Verify SHA256 (default: true)

Returns

  • {:ok, path} - Successfully downloaded to path
  • {:error, reason} - Download failed

Examples

iex> ModelDownloader.download("en-pos-v1")
{:error, :not_implemented}

download_if_missing(language, task, version, opts \\ [])

@spec download_if_missing(atom(), atom(), String.t(), options()) ::
  {:ok, :already_exists | :downloaded} | {:error, term()}

Downloads a model if it's not already available locally.

Checks if the model exists in priv/models/ and downloads it if missing.

Examples

iex> ModelDownloader.download_if_missing(:en, :pos_tagging, "v1")
{:error, :not_implemented}

list_available()

@spec list_available() :: {:ok, [model_id()]} | {:error, term()}

Lists all available models in the remote repository.

Fetches the list of downloadable models from GitHub releases.

Returns

  • {:ok, models} - List of available model IDs
  • {:error, reason} - Failed to fetch list

Examples

iex> ModelDownloader.list_available()
{:error, :not_implemented}

verify_checksum(model_path, expected_hash)

@spec verify_checksum(String.t(), String.t()) :: :ok | {:error, term()}

Verifies the SHA256 checksum of a downloaded model.

Examples

iex> ModelDownloader.verify_checksum("path/to/model.model", "abc123...")
{:error, :not_implemented}