Hyena: Sub-quadratic attention alternative via long convolutions and gating.

Implements the Hyena Hierarchy from "Hyena Hierarchy: Towards Larger Convolutional Language Models" (Poli et al., ICML 2023). Hyena replaces attention with a hierarchy of long convolutions and element-wise gating, achieving sub-quadratic complexity in sequence length.

Key Innovation: Implicit Long Convolution + Gating

Instead of attention's O(L^2) pairwise interactions, Hyena uses:

1. A learned implicit filter (small MLP) that generates long convolution kernels
2. Element-wise gating for non-linearity
3. Multiple "orders" of this operation for expressivity

Order 2 Hyena:
  v, x1, x2 = linear_projections(input)  # 3 projections
  y = v
  y = long_conv(y, filter_1) * x1        # First order
  y = long_conv(y, filter_2) * x2        # Second order
  output = linear(y)

Architecture

Input [batch, seq_len, embed_dim]
      |
      v
+-----------------------+
| Input Projection      |
+-----------------------+
      |
      v
+-----------------------+
| Hyena Block x N       |
|  ShortConv(input)     |
|  Split: v, x1, x2    |
|  y = v                |
|  y = LongConv(y)*x1   |  <- Implicit filter via MLP
|  y = LongConv(y)*x2   |
|  OutProj + Residual   |
|  FFN                  |
+-----------------------+
      |
      v
[batch, hidden_size]    (last timestep)

Complexity

Usage

model = Hyena.build(
  embed_dim: 287,
  hidden_size: 256,
  order: 2,
  filter_size: 64,
  num_layers: 4
)

Reference

• Paper: "Hyena Hierarchy: Towards Larger Convolutional Language Models"
• arXiv: https://arxiv.org/abs/2302.10866