VAR: Visual Autoregressive Modeling via Next-Scale Prediction.

Implements the VAR architecture from "Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction" (Tian et al., NeurIPS 2024 Best Paper). Instead of generating images token-by-token (like traditional AR), VAR generates images scale-by-scale: 1×1 → 2×2 → 4×4 → ... → N×N.

Key Innovation: Next-Scale Prediction

Traditional autoregressive image generation flattens images to 1D sequences and predicts pixels/tokens one at a time. This is slow and ignores spatial structure. VAR instead:

1. Encodes images at multiple resolutions via a multi-scale VQ tokenizer
2. Autoregressively predicts each scale given all coarser scales
3. Each scale prediction is parallel (all tokens at that scale at once)

Scale 1 (1×1):   [tok]           → Predict via GPT
Scale 2 (2×2):   [tok tok]       → Predict via GPT given Scale 1
                 [tok tok]
Scale 3 (4×4):   [tok tok tok tok]   → Predict via GPT given Scales 1-2
                 [tok tok tok tok]
                 [tok tok tok tok]
                 [tok tok tok tok]
...
Scale K (N×N):   Full resolution

Architecture

Image [batch, H, W, C]
      |
      v
+---------------------------+
| Multi-Scale VQ Tokenizer  |  (encode at scales 1, 2, 4, 8, 16...)
+---------------------------+
      |
      v
[Scale tokens at each resolution]
      |
      v
+---------------------------+
| GPT-2 Backbone            |  (autoregressive over scale sequence)
| (decoder_only pattern)    |
+---------------------------+
      |
      v
| Predict next scale tokens |
      |
      v
+---------------------------+
| Multi-Scale VQ Decoder    |  (decode all scales to image)
+---------------------------+
      |
      v
Output Image [batch, H, W, C]

Usage

# Build tokenizer for encoding/decoding
tokenizer = VAR.build_tokenizer(
  image_size: 256,
  scales: [1, 2, 4, 8, 16],
  codebook_size: 1024,
  embed_dim: 256
)

# Build the full VAR model (GPT backbone for next-scale prediction)
model = VAR.build(
  hidden_size: 512,
  num_layers: 12,
  num_heads: 8,
  scales: [1, 2, 4, 8, 16],
  codebook_size: 1024
)

Reference

• Paper: "Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction"
• Authors: Tian, Yu, et al.
• arXiv: https://arxiv.org/abs/2404.02905
• Award: NeurIPS 2024 Best Paper