Supplemental Modules Guide

This guide explains Metastatic's supplemental module system - a mechanism for extending MetaAST with language-specific library integrations.

Overview
Core Concepts
Architecture
Using Supplementals
Available Supplementals
Creating Supplementals
Best Practices
API Reference

Overview

Supplemental modules bridge the gap between MetaAST's language-agnostic representation and real-world library ecosystems. They enable:

Cross-language transformations - Express actor patterns in Python/Erlang/JavaScript
Library integrations - Map MetaAST constructs to third-party APIs (Pykka, asyncio, etc.)
Semantic preservation - Maintain intent while adapting to language-specific idioms

Example

# MetaAST representing an actor call
ast = {:actor_call, {:variable, "worker"}, "process", [data]}

# Transform using Pykka supplemental for Python
{:ok, python_ast} = Metastatic.Supplemental.Transformer.transform(ast, :python)
# Result: {:function_call, "worker.ask", ["process", data]}

# Same AST, different supplemental for Erlang
{:ok, erlang_ast} = Metastatic.Supplemental.Transformer.transform(ast, :erlang)
# Result: {:function_call, "gen_server:call", [worker, {:process, data}]}

Core Concepts

M2 Layer Extension

Supplementals operate at the M2 meta-model level, adding optional constructs beyond Core/Extended/Native:

M2 Core    - Universals (literals, variables, operators, conditionals)
M2 Extended - Common patterns (loops, lambdas, collections)
M2 Native  - Language escapes
M2 Supplemental - Library-specific extensions (actors, async, etc.)

Supplemental constructs are not part of the base MetaAST grammar - they're opt-in extensions that require explicit transformation.

Semantic Contract

Each supplemental defines:

Constructs - Which MetaAST node types it handles (e.g., :actor_call, :async_await)
Target language - Single language it generates code for (:python, :javascript, etc.)
Dependencies - External libraries required ("pykka >= 3.0", "asyncio")
Transformation - How to convert supplemental constructs to concrete code

Registry System

A centralized GenServer registry maintains all available supplementals, enabling:

Fast lookup by construct type
Language compatibility checking
Conflict detection between competing supplementals
Runtime registration/deregistration

Architecture

Components

flowchart TD
    subgraph System["Supplemental System"]
        Behaviour["Behaviour<br/>(spec)"] --> Info["Info<br/>(metadata)"]
        Behaviour --> Registry
        
        Registry["Registry (GenServer)<br/>- by_construct index<br/>- by_language index<br/>- conflict detection"]
        Registry --> Transformer
        
        Transformer["Transformer<br/>- lookup + invoke supplementals<br/>- error handling"]
        Transformer --> Concrete
        
        Concrete["Concrete Supplementals<br/>- Python.Pykka<br/>- Python.Asyncio<br/>- ... (extensible)"]
    end

Data Flow

flowchart TD
    A["User AST with supplemental constructs"] --> B["Validator"]
    B --> C["Identify required supplementals"]
    C --> D["Registry lookup"]
    D --> E["Transformer"]
    E --> F["Invoke supplemental.transform/3"]
    F --> G["Concrete language AST"]

Using Supplementals

Basic Usage

alias Metastatic.Supplemental.Transformer

# Single construct transformation
ast = {:actor_call, {:variable, "worker"}, "process", [data]}
{:ok, result} = Transformer.transform(ast, :python)

# Check availability before transforming
if Transformer.available?(:actor_call, :python) do
  {:ok, transformed} = Transformer.transform(ast, :python)
end

# Get all supported constructs for a language
constructs = Transformer.supported_constructs(:python)
# => [:actor_call, :actor_cast, :spawn_actor, :async_await, :async_context, :gather]

With Adapters

Supplementals integrate automatically when using the adapter pipeline:

alias Metastatic.Builder

# Source with supplemental constructs
source = """
actor = spawn_actor(Worker, [config])
result = actor_call(actor, "process", [data])
"""

# Adapter automatically uses supplementals during transformation
{:ok, doc} = Builder.from_source(source, :python)

# Round-trip preserves supplemental transformations
{:ok, output} = Builder.to_source(doc)

Configuration

Configure auto-registration in config/config.exs:

config :metastatic, :supplementals,
  auto_register: [
    Metastatic.Supplemental.Python.Pykka,
    Metastatic.Supplemental.Python.Asyncio
  ]

Validation

Use the validator to detect required supplementals in a document:

alias Metastatic.{Document, Supplemental.Validator}

doc = Document.new(ast, :python)
{:ok, analysis} = Validator.validate(doc)

# Check what supplementals are needed
analysis.required_supplementals
# => [:pykka, :asyncio]

# Get warnings about missing supplementals
analysis.warnings
# => ["Document requires supplemental ':pykka' which is not registered"]

Available Supplementals

Python.Pykka

Actor model support via Pykka library

Constructs:

:actor_call - Synchronous actor message (ask pattern)
:actor_cast - Asynchronous actor message (tell pattern)
:spawn_actor - Create new actor instance

Dependencies: pykka >= 3.0

Example:

# Actor call
{:actor_call, {:variable, "worker"}, "process", [data]}
# Transforms to: worker.ask({'process': data})

# Actor cast
{:actor_cast, {:variable, "worker"}, "log", [message]}
# Transforms to: worker.tell({'log': message})

# Spawn actor
{:spawn_actor, "WorkerActor", [config]}
# Transforms to: WorkerActor.start(config)

Python.Asyncio

Async/await patterns via asyncio library

Constructs:

:async_await - Async function execution
:async_context - Async context managers
:gather - Parallel task execution

Dependencies: asyncio (stdlib)

Example:

# Async await
{:async_operation, :async_await, 
  {:function_call, "fetch_data", [url]}}
# Transforms to: asyncio.run(fetch_data(url))

# Gather
{:async_operation, :gather, [
  {:function_call, "fetch_user", [1]},
  {:function_call, "fetch_posts", [1]}
]}
# Transforms to: asyncio.gather(fetch_user(1), fetch_posts(1))

Creating Supplementals

Step 1: Implement the Behaviour

defmodule MyProject.Supplemental.Python.MyLibrary do
  @behaviour Metastatic.Supplemental
  
  alias Metastatic.Supplemental.Info
  
  @impl true
  def info do
    %Info{
      name: :my_library,
      language: :python,
      constructs: [:my_construct],
      requires: ["my-library >= 1.0"],
      description: "My library integration"
    }
  end
  
  @impl true
  def transform(ast, language, opts)
  
  def transform({:my_construct, args}, :python, _opts) do
    # Transform logic here
    result = {:function_call, "my_library.do_thing", args}
    {:ok, result}
  end
  
  def transform(_ast, :python, _opts) do
    {:error, {:unsupported_construct, "..."}}
  end
  
  def transform(_ast, language, _opts) do
    {:error, {:incompatible_language, "..."}}
  end
end

Step 2: Write Tests

defmodule MyProject.Supplemental.Python.MyLibraryTest do
  use ExUnit.Case, async: true
  
  alias MyProject.Supplemental.Python.MyLibrary
  
  describe "info/0" do
    test "returns correct metadata" do
      info = MyLibrary.info()
      assert info.name == :my_library
      assert info.language == :python
    end
  end
  
  describe "transform/3" do
    test "transforms my_construct correctly" do
      ast = {:my_construct, [{:literal, :string, "arg"}]}
      assert {:ok, result} = MyLibrary.transform(ast, :python)
      assert result == {:function_call, "my_library.do_thing", 
                        [{:literal, :string, "arg"}]}
    end
    
    test "returns error for wrong language" do
      ast = {:my_construct, []}
      assert {:error, {:incompatible_language, _}} = 
               MyLibrary.transform(ast, :javascript)
    end
  end
end

Step 3: Register

# Manual registration
alias Metastatic.Supplemental.Registry

{:ok, _} = Registry.register(MyProject.Supplemental.Python.MyLibrary)

# Or via config (auto-registers on startup)
config :metastatic, :supplementals,
  auto_register: [MyProject.Supplemental.Python.MyLibrary]

Step 4: Document

Add module documentation with:

Clear description of what library/pattern it supports
List of all constructs handled
Examples showing MetaAST input and output
Dependency requirements
Any caveats or limitations

Best Practices

Design Guidelines

Single Responsibility - One supplemental per library/pattern
Explicit Constructs - Use distinct construct atoms (:actor_call not :call)
Language Specific - Target exactly one language per supplemental
Idiomatic Output - Generate natural, idiomatic code for target language
Comprehensive Tests - Test all constructs, error cases, edge cases

Naming Conventions

Modules:

Metastatic.Supplemental.<Language>.<Library>
  ├── Python.Pykka       (library name)
  ├── Python.Asyncio     (pattern name)
  └── JavaScript.RxJS    (library name)

Construct atoms:

:actor_call          (prefixed by domain)
:async_await         (verb describing action)
:spawn_actor         (specific, unambiguous)

Info names:

name: :pykka         (short, lowercase atom)
name: :asyncio       (match library name)

Error Handling

Return specific errors for different failure modes:

# Unsupported construct
{:error, {:unsupported_construct, "Pykka does not support: #{inspect(ast)}"}}

# Wrong language
{:error, {:incompatible_language, "Pykka only supports Python, got: #{language}"}}

# Invalid options
{:error, {:invalid_options, "Unknown option: #{inspect(key)}"}}

Version Constraints

Use semantic versioning in requires:

requires: [
  "pykka >= 3.0, < 4.0",  # Major version constraint
  "asyncio",               # Stdlib, no version needed
  "aiohttp >= 3.8"         # Minimum version only
]

Testing Strategy

Cover these scenarios:

Info validation - Correct metadata structure
Happy path - Each construct transforms correctly
Error cases - Wrong language, unsupported constructs
Edge cases - Empty arguments, nested structures, complex types
Options - Handle all option variations
Idempotency - Multiple transformations don't break

Performance Considerations

Keep transformations fast (<1ms per node)
Avoid heavy computation in info/0 (called frequently)
Use pattern matching for dispatch (faster than conditionals)
Don't validate AST structure (adapters already do this)

API Reference

Behaviour Callbacks

info/0

Returns metadata about the supplemental.

@callback info() :: Info.t()

transform/3

Transforms a supplemental construct to target language AST.

@callback transform(
  ast :: AST.meta_ast(),
  language :: atom(),
  opts :: map()
) :: {:ok, AST.meta_ast()} | {:error, term()}

Registry Functions

register/1

Registry.register(MySupplemental)
# => {:ok, :pykka} | {:error, {:already_registered, :pykka}}

lookup/2

Find supplemental for construct and language.

Registry.lookup(:actor_call, :python)
# => {:ok, Metastatic.Supplemental.Python.Pykka} | 
#    {:error, :not_found}

all/0

List all registered supplementals.

Registry.all()
# => [Metastatic.Supplemental.Python.Pykka, ...]

Transformer Functions

transform/3

Transform a construct using registered supplementals.

Transformer.transform(ast, :python)
# => {:ok, transformed_ast} | {:error, reason}

available?/2

Check if a construct is supported for a language.

Transformer.available?(:actor_call, :python)
# => true | false

supported_constructs/1

Get all supported constructs for a language.

Transformer.supported_constructs(:python)
# => [:actor_call, :actor_cast, :spawn_actor, ...]

Validator Functions

validate/1

Analyze a document to detect required supplementals.

Validator.validate(document)
# => {:ok, %{required_supplementals: [...], warnings: [...]}}

Advanced Topics

Conflict Resolution

When multiple supplementals claim the same construct+language:

# Registry detects conflicts
Registry.register(AlternativePykka)
# => {:error, {:conflict, existing: Pykka, new: AlternativePykka}}

# Unregister old, register new
Registry.unregister(Pykka)
Registry.register(AlternativePykka)

Chaining Transformations

Supplementals can compose:

# First supplemental adds intermediate construct
{:ok, ast1} = Supplemental1.transform(input, :python)

# Second supplemental further transforms
{:ok, ast2} = Supplemental2.transform(ast1, :python)

Stateful Transformations

Pass state via options:

opts = %{
  actor_prefix: "my_app",
  async_mode: :gather
}

Transformer.transform(ast, :python, opts)

Testing with Fixtures

Create test fixtures for complex scenarios:

# test/fixtures/python/supplemental/pykka_actors.py
class Worker(pykka.ThreadingActor):
    def process(self, data):
        return data * 2

worker = Worker.start()
result = worker.ask({'process': 42})

Contributing

Want to add a supplemental?

Use the generator to scaffold a new supplemental:

mix metastatic.gen.supplemental Python MyLibrary

Future Work

Planned supplementals:

Python.Celery - Distributed task queue
JavaScript.RxJS - Reactive programming
Ruby.Concurrent - Concurrency primitives
Go.Channels - CSP-style communication
Rust.Tokio - Async runtime

← Previous Page Theoretical Foundations

Next Page → Analyzer Plugin System