DoRA: Weight-Decomposed Low-Rank Adaptation.

Implements DoRA from "DoRA: Weight-Decomposed Low-Rank Adaptation of Large Language Models" (Liu et al., 2024). DoRA decomposes pretrained weights into magnitude and direction components, then applies LoRA only to the direction.

Key Innovation: Magnitude-Direction Decomposition

Standard LoRA modifies the full weight: W' = W + BA

DoRA decomposes W into magnitude m and direction V:

W = m * (V / ||V||)

Then applies LoRA only to the direction component:

W' = m * ((V + BA) / ||V + BA||)

Where:

• m is a learnable magnitude vector [output_size]
• V is the original weight direction
• BA is the standard LoRA low-rank update
• ||.|| is column-wise L2 normalization

Why This Works

Separating magnitude from direction gives two benefits:

1. Direction captures "what" features are important (adapted by LoRA)
2. Magnitude captures "how much" each feature matters (learned separately)
3. This mirrors weight normalization, which is known to improve optimization

Architecture

Input x [batch, input_size]
      |
      +---> W * x (frozen base)
      |        |
      +---> A * x -> B * (A * x)     (LoRA delta)
      |        |
      |     V + BA                    (direction update)
      |        |
      |     normalize(V + BA)         (unit direction)
      |        |
      |     m * normalized            (apply magnitude)
      |
      v
Output [batch, output_size]

LoRA+ Note

LoRA+ (Hayou et al., 2024) proposes different learning rates for A vs B matrices. This is a training configuration choice rather than architectural: use a higher learning rate for B (e.g., 5-10x) than for A. We document this recommendation but don't enforce it in the graph structure.

Usage

# Standalone DoRA layer
dora = DoRA.build(input_size: 768, output_size: 768, rank: 8)

# Wrap an existing layer with DoRA
adapted = DoRA.wrap(input, original, rank: 8, name: "dora_attn")

References

• Liu et al., "DoRA: Weight-Decomposed Low-Rank Adaptation" (2024)
• https://arxiv.org/abs/2402.09353
• Hayou et al., "LoRA+: Efficient Low Rank Adaptation of Large Models" (2024)