Feed-Forward Network building blocks for transformer architectures.

Provides standard and gated FFN variants used in the feed-forward sublayer of transformer blocks. This module unifies the duplicated build_ffn/3 pattern found across attention architectures.

Variants

• Standard: dense(hidden * expansion) -> activation -> dropout -> dense(hidden)
• Gated: Delegates to SwiGLU.layer/2 for gated linear unit variants

Usage

# Standard FFN (default in most transformers)
ffn = FFN.layer(input, hidden_size: 256)

# With custom expansion factor and activation
ffn = FFN.layer(input, hidden_size: 256, expansion_factor: 8, activation: :relu)

# Gated variant (SwiGLU/GeGLU/ReGLU)
ffn = FFN.gated_layer(input, hidden_size: 256, activation: :silu)

References

• "Attention Is All You Need" (Vaswani et al., 2017) - original FFN
• "GLU Variants Improve Transformer" (Shazeer, 2020) - gated variants