Nemotron-H: NVIDIA's Hybrid Mamba-Transformer Architecture.

Nemotron-H is a hybrid language model that combines 90% Mamba2 (SSD) layers with 10% full attention layers. This design achieves Transformer-level quality while maintaining linear inference cost from the SSM components.

Key Innovation: Hybrid Layer Mixing

Rather than using all-attention or all-SSM, Nemotron-H interleaves them:

• 90% of layers use Mamba2 (State Space Duality) for efficient linear-time processing
• 10% of layers use full multi-head attention for global reasoning
• Attention blocks placed at regular intervals (every 10th layer by default)

Architecture

Input [batch, seq_len, embed_dim]
      |
      v
[Shared Embedding Projection]
      |
      v
+========================================+
|            Layer 0 (Mamba2)            |
|  RMSNorm -> Mamba2 SSD -> Residual     |
|  RMSNorm -> SwiGLU FFN -> Residual     |
+========================================+
      |
     ... (Mamba2 layers 1-8)
      |
+========================================+
|           Layer 9 (Attention)          |
|  RMSNorm -> MultiHead Attn -> Residual |
|  RMSNorm -> SwiGLU FFN -> Residual     |
+========================================+
      |
     ... (pattern repeats)
      |
      v
[Final RMSNorm]
      |
      v
[Output Projection (tied weights)]
      |
      v
Output [batch, hidden_dim]

Mamba2 (SSD) Blocks

Use State Space Duality from Mamba-2:

• Chunked matmul for tensor core utilization
• Selective state space with input-dependent parameters
• Depthwise convolution + gating

Attention Blocks

Standard multi-head attention with:

• Grouped Query Attention (optional)
• RoPE position embeddings (optional)
• Causal masking

Usage

model = NemotronH.build(
  embed_dim: 287,
  hidden_dim: 2048,
  num_layers: 32,
  attention_every_n: 10,
  num_heads: 16
)

References

• Paper: "Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Language Models" (NVIDIA, 2025)
• Mamba-2: "Transformers are SSMs" (Gu & Dao, 2024)