Crucible.Stage behaviour (CrucibleFramework v0.5.2)

View Source

Behaviour for a single experiment pipeline stage.

Stages transform a %Crucible.Context{} and may enrich metrics, outputs, or orchestrate I/O with backends and external systems.

Runner Location

The pipeline runner lives in crucible_framework:

crucible_ir does not execute anything; it only defines specs. All execution logic is owned by crucible_framework.

Required Callback: run/2

Every stage must implement run/2:

@impl true
def run(%Crucible.Context{} = ctx, opts) do
  # Transform context, perform work
  {:ok, updated_ctx}
end
  • Returns {:ok, %Crucible.Context{}} on success
  • Returns {:error, reason} on failure
  • Must not mutate global state or bypass persistence helpers
  • Should be network-mockable and testable in isolation

Required Callback: describe/1

The describe/1 callback is required for all stage implementations. It provides a discoverable schema for stage options:

@impl true
def describe(_opts) do
  %{
    name: :my_stage,
    description: "Human-readable description of what this stage does",
    required: [:model_name, :dataset_path],
    optional: [:batch_size, :seed, :log_level],
    types: %{
      model_name: :string,
      dataset_path: :string,
      batch_size: :integer,
      seed: :integer,
      log_level: {:enum, [:debug, :info, :warn, :error]}
    }
  }
end

Schema Keys

  • :name - Stage identifier (atom)
  • :description - Human-readable description (string)
  • :required - List of required option keys (list of atoms)
  • :optional - List of optional option keys (list of atoms)
  • :types - Map of key to type specification

Type Specifications

  • :string - String value
  • :integer - Integer value
  • :float - Float value
  • :boolean - Boolean value
  • :atom - Atom value
  • :map - Map value
  • :list - List value
  • {:struct, Module} - Struct of the given module
  • {:enum, [values]} - One of the enumerated values
  • {:function, arity} - Function with given arity

Options Handling

  • CrucibleIR.StageDef.options is an opaque map owned by each stage
  • Stages own their own options schema and validation
  • Stages may accept typed configs (e.g., %CrucibleIR.Training.Config{}) but must normalize internally

Summary

Types

opts()

Callbacks

describe(opts)

Returns a schema describing the stage's purpose and options.

run(context, opts)

Executes the stage logic on the given context.

Types

opts()

@type opts() :: map()

Callbacks

describe(opts)

@callback describe(opts :: opts()) :: map()

Returns a schema describing the stage's purpose and options.

This callback is required for all stage implementations. See module documentation for the expected schema format.

Return Value

Must return a map conforming to Crucible.Stage.Schema.t():

%{
  name: :stage_name,
  description: "Human-readable description",
  required: [:key1],
  optional: [:key2, :key3],
  types: %{
    key1: :string,
    key2: :integer,
    key3: {:enum, [:a, :b, :c]}
  }
}

See Crucible.Stage.Schema for the complete schema specification.

run(context, opts)

@callback run(context :: Crucible.Context.t(), opts :: opts()) ::
  {:ok, Crucible.Context.t()} | {:error, term()}

Executes the stage logic on the given context.

Parameters

  • context - The %Crucible.Context{} struct threaded through the pipeline
  • opts - Stage-specific options (from StageDef.options)

Returns

  • {:ok, %Crucible.Context{}} - Updated context on success
  • {:error, reason} - Error tuple on failure