GRPO: Group Relative Policy Optimization.

Implements GRPO from DeepSeek's RLHF methodology. GRPO simplifies RLHF by:

1. Sampling G completions per prompt
2. Ranking completions within each group
3. Using group-relative advantages (no critic network needed)

Key Innovation: Group-Relative Advantage

Standard PPO requires a value function (critic) to compute advantages:

A(s,a) = R - V(s)  # Advantage = Return - Value estimate

GRPO eliminates the critic by using group-relative normalization:

For each prompt, sample G responses with rewards r_1, ..., r_G
Normalize: A_i = (r_i - mean(r)) / std(r)

This works because:

• Within a group, the mean reward is a natural baseline
• Standard deviation normalizes the scale
• No learned value function needed

Algorithm

For each prompt x:
    1. Sample G responses: y_1, ..., y_G ~ pi(y|x)
    2. Get rewards: r_1, ..., r_G = Reward(x, y_i)
    3. Compute advantages: A_i = (r_i - mean(r)) / (std(r) + eps)
    4. Policy gradient: L = -sum(A_i * log pi(y_i|x))

Architecture

Prompt x [batch]
      |
      v
+------------------+
| Sample G times   |  -> G responses per prompt
+------------------+
      |
      v
+------------------+
| Reward Model     |  -> Rewards r_1, ..., r_G
+------------------+
      |
      v
+------------------+
| Group Normalize  |  -> Advantages A_1, ..., A_G
+------------------+
      |
      v
+------------------+
| Policy Gradient  |  -> Update policy
+------------------+

Usage

# Build a GRPO policy
policy = GRPO.build(
  hidden_size: 512,
  num_layers: 6,
  vocab_size: 32000
)

# Compute group-relative advantages
advantages = GRPO.compute_advantages(group_rewards)

# Compute policy gradient loss
loss = GRPO.loss(log_probs, advantages, mask)

Reference

• DeepSeek-R1 Technical Report (2024)
• "DeepSeekMath: Pushing the Limits of Mathematical Reasoning" (2024)