ExFairness.CrucibleStage (ExFairness v0.5.1)

View Source

Crucible.Stage implementation for fairness evaluation.

This stage integrates ExFairness into crucible_framework pipelines, providing fairness metric evaluation on model outputs.

Configuration

The stage uses fairness configuration from the experiment context:

%CrucibleIR.Reliability.Fairness{
  enabled: true,                    # Enable fairness evaluation
  metrics: [:demographic_parity, :equalized_odds, :equal_opportunity, :predictive_parity],
  group_by: :gender,                # Sensitive attribute field name
  threshold: 0.1,                   # Maximum acceptable disparity
  fail_on_violation: false,         # Whether to fail experiment on fairness violation
  options: %{}                      # Additional metric-specific options
}

Data Sources

The stage extracts data from two possible sources (in order of preference):

  1. From assigns (preferred when pre-computed tensors available):

    • context.assigns.fairness_predictions - Binary predictions tensor
    • context.assigns.fairness_labels - Ground truth labels tensor
    • context.assigns.fairness_sensitive - Sensitive attribute tensor
    • context.assigns.fairness_probabilities - (Optional) Probabilities for calibration

  2. From outputs (fallback):

    • context.outputs - List of maps with :prediction, :label, sensitive attribute

Results

Results are stored in context.metrics.fairness:

%{
  metrics: %{
    demographic_parity: %{disparity: 0.05, passes: true, ...},
    equalized_odds: %{tpr_disparity: 0.03, fpr_disparity: 0.04, passes: true, ...},
    ...
  },
  overall_passes: true,
  violations: []
}

Example Usage

config = %CrucibleIR.Reliability.Fairness{
  enabled: true,
  metrics: [:demographic_parity, :equalized_odds],
  group_by: :gender,
  threshold: 0.1
}

context = %Crucible.Context{
  experiment: %{reliability: %{fairness: config}},
  outputs: [
    %{prediction: 1, label: 1, gender: 0},
    %{prediction: 0, label: 0, gender: 1}
  ]
}

{:ok, result} = ExFairness.CrucibleStage.run(context, %{})
result.metrics.fairness
# => %{metrics: %{...}, overall_passes: true, violations: []}

Summary

Types

context()
fairness_result()
opts()

Functions

describe(opts \\ %{})

Returns metadata about the fairness evaluation stage in canonical schema format.

run(context, opts \\ %{})

Runs fairness evaluation on model outputs in the context.

Types

context()

@type context() :: map()

fairness_result()

@type fairness_result() :: %{
  metrics: map(),
  overall_passes: boolean(),
  violations: [map()]
}

opts()

@type opts() :: map()

Functions

describe(opts \\ %{})

@spec describe(opts()) :: map()

Returns metadata about the fairness evaluation stage in canonical schema format.

Parameters

  • opts - Options (currently unused)

Returns

A map containing stage metadata in canonical schema format with:

  • __schema_version__ - Schema version marker
  • name - Stage identifier (atom)
  • description - Human-readable description
  • required - Required option keys
  • optional - Optional option keys
  • types - Type specifications for options
  • defaults - Default values for optional options
  • version - Package version
  • __extensions__ - Fairness-specific metadata

Examples

iex> schema = ExFairness.CrucibleStage.describe(%{})
iex> schema.name
:fairness
iex> schema.__schema_version__
"1.0.0"
iex> is_list(schema.required)
true

run(context, opts \\ %{})

@spec run(context(), opts()) :: {:ok, context()} | {:error, term()}

Runs fairness evaluation on model outputs in the context.

Parameters

  • context - Experiment context containing fairness config and model outputs
  • opts - Additional options (currently unused, reserved for future extensions)

Returns

  • {:ok, updated_context} - Context with fairness results added to context.metrics.fairness
  • {:error, reason} - If configuration is invalid, data missing, or fairness violations detected (when fail_on_violation is true)

Examples

iex> config = %CrucibleIR.Reliability.Fairness{
...>   enabled: true,
...>   metrics: [:demographic_parity],
...>   group_by: :gender,
...>   threshold: 0.1
...> }
iex> context = %{
...>   experiment: %{reliability: %{fairness: config}},
...>   outputs: [
...>     %{prediction: 1, label: 1, gender: 0},
...>     %{prediction: 0, label: 0, gender: 1}
...>   ],
...>   metrics: %{}
...> }
iex> {:ok, result} = ExFairness.CrucibleStage.run(context, %{})
iex> is_map(result.metrics.fairness)
true