Edifice.Recurrent.NativeRecurrence (Edifice v0.2.0)

Copy Markdown View Source

Native Recurrence — unified module for multiple minimal recurrence types.

Provides three lightweight recurrence variants under a single interface, each using a sequential scan over timesteps with different gating mechanics.

Recurrence Types

  • :elu_gru — ELU-gated GRU: z=sigmoid(Wz·x), c=1+elu(Wc·x), h=(1-z)·h+z·c
  • :real_gru — Real-valued MinGRU: z=sigmoid(Wz·x), c=Wc·x, h=(1-z)·h+z·c
  • :diag_linear — Diagonal linear recurrence: h=sigmoid(Wa·x)·h + Wb·x

Architecture

Input [batch, seq_len, embed_dim]
      |
Input projection -> hidden_size
      |
Per layer: pre-norm -> gate+candidate projections -> scan -> residual
      |
Final norm -> last timestep -> [batch, hidden_size]

Usage

model = NativeRecurrence.build(
  embed_dim: 256,
  hidden_size: 256,
  num_layers: 4,
  recurrence_type: :elu_gru
)

References

  • Feng et al., "Were RNNs All We Needed?" (2024) — MinGRU/MinLSTM
  • Orvieto et al., "Resurrecting Recurrent Neural Networks for Long Sequences" (2023)

Summary

Types

build_opt()

Options for build/1.

Functions

build(opts \\ [])

Build a Native Recurrence model.

output_size(opts \\ [])

Get the output size of the model.

recommended_defaults()

Get recommended defaults.

Types

build_opt()

@type build_opt() ::
  {:embed_dim, pos_integer()}
  | {:hidden_size, pos_integer()}
  | {:num_layers, pos_integer()}
  | {:recurrence_type, :elu_gru | :real_gru | :diag_linear}
  | {:dropout, float()}
  | {:window_size, pos_integer()}

Options for build/1.

Functions

build(opts \\ [])

@spec build([build_opt()]) :: Axon.t()

Build a Native Recurrence model.

Options

  • :embed_dim - Input embedding dimension (required)
  • :hidden_size - Internal hidden dimension (default: 256)
  • :num_layers - Number of recurrent layers (default: 4)
  • :recurrence_type - Recurrence variant: :elu_gru, :real_gru, or :diag_linear (default: :elu_gru)
  • :dropout - Dropout rate (default: 0.1)
  • :window_size - Expected sequence length (default: 60)

Returns

An Axon model outputting [batch, hidden_size].

output_size(opts \\ [])

@spec output_size(keyword()) :: pos_integer()

Get the output size of the model.