Prerequisite — GRPO & PPO (the RL machinery)¶
The Policy Gradient family in this survey transplants GRPO — the algorithm DeepSeek used for LLM reasoning — onto image/video generators. This primer covers the parts that carry over and the vocabulary you need to read the algorithm pages.
What to expect. This page walks you through five short steps:
- The one-line goal of policy-gradient RL — what a "reward" actually is, with concrete examples.
- PPO → GRPO — why PPO needs a separate value network (critic), and how GRPO removes it by grading outputs on a curve against their own group.
- The PPO clip — the leash that keeps each update from straying too far.
- Why none of this is free for diffusion — the deterministic sampler has no per-step probability.
- A vocabulary map from LLM RL to diffusion/flow, plus the two families this survey is built around.
If you only remember one thing: GRPO is PPO with the critic replaced by the average score of a group of samples.
1. Policy-gradient RL in one line¶
You have a policy \(\pi_\theta\) that produces an output, a scalar reward \(r\) that scores it, and you want to raise the expected reward:
The reward is whatever you can score automatically. In the language-model world it is usually verifiable correctness: did the math answer match the gold solution, did the generated code pass its unit tests? These rule-based rewards are exactly what DeepSeek used to train its reasoning models. When there is no ground truth, the reward instead comes from a learned model of human preference — an aesthetic or alignment score — and for the image/video generators in this survey that is the usual case: a reward model (or a VLM acting as judge) rates how well the final image matches the prompt and how good it looks.
Whatever its source, the recipe is the same: push up the probability of outputs that scored well, push down the rest. The difficulty is doing this stably, and that is where PPO and GRPO come in.
2. PPO → GRPO: replacing the critic with a curve¶
Raw reward is a noisy teacher. A single high score might mean the output was genuinely good, or just that the prompt was easy. The standard fix is to subtract a baseline and ask the sharper question — "was this output better than expected?" The reward minus the baseline is the advantage \(\hat A\), and the gradient pushes on the advantage rather than the raw reward.
What PPO needs. Proximal Policy Optimization (the workhorse behind RLHF) estimates that baseline with a second neural network — a value network, or critic — trained alongside the policy to predict the expected reward of a partially generated output. It works, but it is expensive: the critic is typically as large as the policy itself, so you train and hold two big models in memory, and a critic that estimates value badly quietly corrupts every gradient. Getting it to train well is its own headache.
What GRPO does instead: grade on a curve. GRPO — Group Relative Policy Optimization, introduced in the DeepSeekMath paper (Shao et al. 2024) and later used to train DeepSeek-R1 (Guo et al. 2025) — removes the critic entirely. For one prompt it samples a group of \(N\) outputs and uses the group's own average reward as the baseline. An output's advantage is just its z-score within its group:
Like grading a class on a curve, you never need to know the absolute "difficulty" of the prompt — only who did better than the group average. That single substitution is the whole idea: no critic to train, no second model to serve (≈ 50% less memory than PPO), and prompt difficulty cancels out as a confound. It is the single most reused ingredient in this survey.
3. PPO clipping: a leash on each update¶
On-policy data is generated by the previous weights \(\theta_\text{old}\), so we re-weight by the importance ratio
PPO then clips it so no single update strays too far — a leash: don't rewrite the whole recipe because one tasting went well.
One subtlety to file away: the leash only does its job when \(\rho\) sits near 1. If the ratios are systematically off-centre, the clip never engages — exactly the failure GRPO-Guard diagnoses when this is ported to flow models.
4. Why this is non-trivial for diffusion¶
In an LLM each token has a softmax probability, so \(\rho\) is immediate. A flow model's sampler is a deterministic ODE with no per-step probability (see flow_matching_basics.md §5) — so \(\rho\) is undefined out of the box. Bridging that gap is what FlowGRPO does: convert the ODE to an SDE so each denoising step becomes a Gaussian with a real \(\log\pi_\theta\). VeRL-Omni describes the resulting update as a "CLIP-style" loss using trajectory advantages — i.e. the §3 objective, summed over the denoising steps of the rollout.
A second wrinkle is credit assignment. You only get the reward at the very end — you taste the dish only once it is plated (\(x_0\)) — yet the gradient flows through every stirring step (every denoising step). GRPO's simplest answer is to broadcast the single trajectory-level advantage \(\hat A^{(i)}\) to all steps; much of the academia long-tail (DenseGRPO, TreeGRPO, …) is about doing better than "blame every step equally."
5. Vocabulary map: LLM → diffusion / VeRL-Omni¶
| LLM RL term | Diffusion/flow analogue (and VeRL-Omni framing) |
|---|---|
| Token sequence (the "trajectory") | Denoising trajectory \(x_1 \to \cdots \to x_0\) in latent space (a VeRL-Omni rollout) |
| Per-token log-prob | Per-step Gaussian log-prob of the SDE denoising step |
| Reward on the full response | Reward model on the final image/video \(x_0\) — often a VLM-as-judge / OCR scorer |
| Group of \(N\) responses | Group of \(N\) images generated from one prompt |
| Reference policy (KL anchor) | Frozen pre-RL model \(\pi_{\theta_\text{ref}}\) |
6. The two families, in one line¶
- Policy Gradient — keep this PPO-clip machinery and pay the price of making the sampler stochastic (FlowGRPO, DanceGRPO, MixGRPO, GRPO-Guard, …).
- Direct Preference — drop \(\rho\) entirely and learn from a preference/MSE loss on final samples (DPO, AWM, DiffusionNFT, DGPO).
Next: flow_grpo.md — GRPO meets flow matching. For the model side first, see flow_matching_basics.md.