Skip to content

DGPO — Reinforcing Diffusion Models by Direct Group Preference Optimization

Field Value
arXiv 2510.08425
Submitted 2025-10-09
Venue — (preprint)
Authors Yihong Luo, Tianyang Hu, Jing Tang
GitHub https://github.com/Luo-Yihong/DGPO
Paradigm Direct Preference — ELBO-based group preference loss over online ODE rollouts
Cites FlowGRPO (2505.05470), DanceGRPO (2505.07818), DDPO, DPO (Rafailov 2023), Diffusion-DPO (2311.12908)
Cited by —

Context

DGPO bridges two lines of work: (1) GRPO-style online RL — generating fresh image groups each iteration and computing group-relative advantages; and (2) DPO-style preference optimisation — using the diffusion ELBO as a tractable log-probability proxy. The result avoids the SDE requirement of policy-gradient methods while achieving online adaptation (unlike offline Diffusion-DPO). Two ingredients carry it: Diffusion-DPO's ELBO derivation (Diffusion-DPO, 2311.12908; reproduced in Step 1 below) and the GRPO group advantage (defined in Step 2 below).

DGPO is written in DDPM/score-based terms (the ELBO is cleanest there). \(x_0\) is the clean image (or video latent) and \(c\) the text prompt; \(t \in \lbrace{}1, \ldots, T\rbrace\) indexes the \(T\) discrete denoising steps. The forward (noising) process is \(x_t = \sqrt{\bar\alpha_t}x_0 + \sigma_t\epsilon\), where \(\bar\alpha_t \in (0,1]\) is the cumulative signal-retention coefficient (decreasing in \(t\)), \(\sigma_t = \sqrt{1-\bar\alpha_t}\) the corresponding noise scale, and \(\epsilon \sim \mathcal{N}(0,I)\) standard Gaussian noise. \(\epsilon_\theta(x_t,t,c)\) is the noise-prediction network with trainable parameters \(\theta\), and \(\theta_\text{ref}\) are the frozen reference (pre-RL) parameters; \(p_\theta(x_0\mid c)\) and \(p_{\theta_\text{ref}}(x_0\mid c)\) are the marginal likelihoods of the two models. A group is the set of \(N\) images generated from one prompt, indexed by \(i\).

Problem — GRPO requires a stochastic policy for the importance ratio; this blocks fast ODE samplers

Issue: FlowGRPO and DanceGRPO derive a tractable importance ratio \(\rho_t = \pi_\theta / \pi_{\theta_\text{old}}\) by converting the ODE to an SDE. The SDE is mandatory — without it, \(\pi_\theta\) has no density and \(\rho_t\) is undefined. This means policy-gradient methods cannot use DDIM, DPM-Solver++, or any other deterministic fast sampler during training, imposing a \(2\text{–}5\times\) speed penalty compared to ODE-based inference.

Idea: Replace the policy gradient (which needs \(\rho_t\)) with a group-level preference loss derived from the diffusion ELBO. The preference loss does not require per-step densities — it operates on the terminal images \(\lbrace{}x_0^{(i)}\rbrace\) generated by any ODE sampler. GRPO's group advantage is retained as the weighting signal.

Why this works: Diffusion-DPO showed that \(\log p_\theta(x_0|c)\) can be approximated by the ELBO — a sum of per-timestep MSE losses. DGPO extends this from binary preference pairs to group rankings, and from an offline dataset to an online setting where groups are generated fresh each iteration. The ELBO decomposition does not depend on how \(x_0\) was generated (ODE or SDE), so any sampler works.

Result: On SD3.5-M, DGPO reaches GenEval 0.97 vs FlowGRPO 0.95 (base 0.63) and OCR 0.96 vs 0.92, with better human-preference proxies (PickScore 23.89 vs 23.31, ImageReward 1.32 vs 1.28, Aesthetic 6.08 vs 5.92; Tab. 1–2) — while training ~20× faster than SOTA and ~30× faster than FlowGRPO to the same GenEval (Fig. 1), confirming the ODE-only ELBO route loses no quality.

Core Derivation

Step 1 — Log-ratio via ELBO

The marginal log-ratio of the current vs. reference policy is intractable directly:

\[\log\frac{\pi_\theta(x_0|c)}{\pi_{\theta_\text{ref}}(x_0|c)} = \log\frac{p_\theta(x_0|c)}{p_{\theta_\text{ref}}(x_0|c)}\]

Via the diffusion ELBO (Diffusion-DPO, 2311.12908):

\[\log\frac{p_\theta(x_0|c)}{p_{\theta_\text{ref}}(x_0|c)} \approx -\mathbb{E}_t\left[\underbrace{\Vert\epsilon_\theta(x_t,t,c) - \epsilon\Vert^2}_{\mathcal{L}_\theta(x_0)} - \underbrace{\Vert\epsilon_{\theta_\text{ref}}(x_t,t,c) - \epsilon\Vert^2}_{\mathcal{L}_{\theta_\text{ref}}(x_0)}\right]\]

Here \(\mathbb{E}_t\) averages over a timestep \(t\) drawn uniformly (so the intractable marginal log-ratio collapses to a difference of per-step denoising MSE losses), and \(\mathcal{L}_\theta(x_0)\), \(\mathcal{L}_{\theta_\text{ref}}(x_0)\) are the noise-prediction squared errors of the current and reference models on the same noised state \(x_t = \sqrt{\bar\alpha_t}x_0 + \sigma_t\epsilon\) — the forward-noised version of the generated image (independent of the sampling trajectory). Define the per-sample ELBO log-ratio:

\[\Delta\mathcal{L}_i \triangleq \Vert\epsilon_{\theta_\text{ref}}(x_t^{(i)},t,c) - \epsilon^{(i)}\Vert^2 - \Vert\epsilon_\theta(x_t^{(i)},t,c) - \epsilon^{(i)}\Vert^2\]

i.e. how much lower the current model's denoising error is than the reference's on image \(i\) — positive when \(\theta\) improves on \(\theta_\text{ref}\), and a per-sample proxy for \(\log\bigl(p_\theta(x_0^{(i)}|c)/p_{\theta_\text{ref}}(x_0^{(i)}|c)\bigr)\).

Step 2 — Partition images by group advantage

Generate \(N\) images per prompt, compute for each the group-relative advantage \(\hat{A}^{(i)} = (r^{(i)} - \text{mean}(\lbrace{}r^{(j)}\rbrace)) / \text{std}(\lbrace{}r^{(j)}\rbrace)\) — the reward \(r^{(i)} = r(x_0^{(i)}, c)\) standardised within the group (zero-mean, so the sign of \(\hat{A}^{(i)}\) records whether image \(i\) beats the group-mean reward) — and partition into the positive group \(\mathcal{G}^{+}\) (above the mean) and the negative group \(\mathcal{G}^{-}\) (at or below):

\[\mathcal{G}^{+} = \lbrace{}i : \hat{A}^{(i)} > 0\rbrace, \quad \mathcal{G}^{-} = \lbrace{}i : \hat{A}^{(i)} \leq 0\rbrace\]

Assign per-sample advantage-magnitude weights \(w^{+}(i), w^{-}(j) \geq 0\), normalised so the two groups' total weights match (required for the Bradley-Terry partition function \(Z(c)\) to cancel; because \(\hat{A}\) is zero-mean within the group, \(|\hat{A}|\) already sum-matches across the two halves):

\[w^{+}(i) \propto |\hat{A}^{(i)}|,\ i \in \mathcal{G}^{+}; \quad w^{-}(j) \propto |\hat{A}^{(j)}|,\ j \in \mathcal{G}^{-}; \quad \sum_{\mathcal{G}^{+}} w^{+} = \sum_{\mathcal{G}^{-}} w^{-}\]

Step 3 — Group Bradley-Terry objective

Model the preference that the positive group is collectively better than the negative group, with \(\sigma(z) = 1/(1+e^{-z})\) the logistic (sigmoid) function:

\[\max_\theta\ \mathbb{E}\left[\log\sigma\left(R_\theta(\mathcal{G}^{+}|c) - R_\theta(\mathcal{G}^{-}|c)\right)\right]\]

where the group-level reward proxy \(R_\theta(\mathcal{G}|c)\) is the weighted ELBO log-ratio over the group (\(\beta > 0\) is the DPO temperature controlling how sharply the sigmoid responds to the log-ratio margin; the partition constant \(\log Z(c)\) cancels under the sum-matched weights):

\[R_\theta(\mathcal{G}|c) = \beta\sum_{i \in \mathcal{G}} w(i)\cdot\log\frac{p_\theta(x_0^{(i)}|c)}{p_{\theta_\text{ref}}(x_0^{(i)}|c)} \approx \beta\sum_{i \in \mathcal{G}} w(i)\cdot\mathbb{E}_t\left[\Delta\mathcal{L}_i\right]\]

Substituting (folding the positive constant \(\beta\) and the per-step ELBO weight into an overall scale absorbed at training time; \(\mathbb{E}_{t,\epsilon}\) is the Monte-Carlo estimate over the sampled timestep and noise):

\[\boxed{ \mathcal{L}_\text{DGPO}(\theta) = -\mathbb{E}_{t,\epsilon}\left[\log\sigma\left(\beta\left(\sum_{i \in \mathcal{G}^{+}} w^{+}(i)\Delta\mathcal{L}_i - \sum_{j \in \mathcal{G}^{-}} w^{-}(j)\Delta\mathcal{L}_j\right)\right)\right] }\]

Interpretation: the loss pushes \(\Delta\mathcal{L}_i > 0\) for positive-group images (\(\theta\) improves over \(\theta_\text{ref}\)) and \(\Delta\mathcal{L}_j < 0\) for negative-group images (\(\theta\) gets worse than \(\theta_\text{ref}\)), weighted by advantage magnitude. The sigmoid \(\sigma(\cdot)\) provides a soft margin, and modelling preference on the ELBO proxy \(\Delta\mathcal{L}_i\) (rather than per-step importance ratios) means no stochastic policy is needed — any ODE sampler that produces \(x_0^{(i)}\) suffices, giving an online, ODE-compatible alternative to policy-gradient RL.

Why No SDE Is Needed

The key observation: \(\Delta\mathcal{L}_i\) is computed by: 1. Taking \(x_0^{(i)}\) from any ODE sampler. 2. Independently sampling a timestep \(t \sim \mathrm{Uniform}\lbrace{}t_\min, \ldots, T\rbrace\) (with a floor \(t_\min > 0\) that excludes the lowest-noise steps, where the ELBO proxy is unreliable) and fresh noise \(\epsilon^{(i)} \sim \mathcal{N}(0,I)\). 3. Computing the forward-noised state: \(x_t^{(i)} = \sqrt{\bar\alpha_t}x_0^{(i)} + \sigma_t\epsilon^{(i)}\) (forward direction). 4. Evaluating the denoising MSE difference.

Step 3 is the forward process — entirely independent of how \(x_0^{(i)}\) was sampled. No denoising step, no importance ratio, no SDE required.

Algorithm

Input: pretrained θ (= θ_ref initially), ODE sampler, reward r, prompt dist p_c, group size N
Repeat:
  1. Sample prompts {c_j}
  2. For each c_j, generate N images via any ODE sampler (DDIM, DPM-Solver++, etc.):
       x_0^(1),...,x_0^(N) ~ ODE_θ(c_j)       ← any sampler, no SDE needed
  3. Compute rewards: R^(i) = r(x_0^(i), c_j)
  4. Group advantage:  Â^(i) = (R^(i) - mean) / std
  5. Partition and weight:
       G+ = {i: Â^(i) > 0},  G- = {i: Â^(i) ≤ 0}
       w+(i) ∝ |Â^(i)|,  w-(j) ∝ |Â^(j)|  (normalised to sum-match)
  6. For each training batch:
       t ~ Uniform{t_min,...,T};  ε^(i) ~ N(0,I)
       x_t^(i) = sqrt(ᾱ_t)·x_0^(i) + σ_t·ε^(i)    ← forward noising
       ΔL_i = ‖ε_{θ_ref}(x_t^(i),t,c) - ε^(i)‖² - ‖ε_θ(x_t^(i),t,c) - ε^(i)‖²
  7. L = -log σ( Σ_{G+} w+(i)·ΔL_i - Σ_{G-} w-(j)·ΔL_j )
  8. θ ← θ - η ∇_θ L
  9. Periodically soft-update reference: θ_ref ← θ  (or keep fixed)

Comparison

Aspect Diffusion-DPO FlowGRPO (policy-gradient) DGPO (direct-preference)
Data source Offline preference pairs Online SDE rollouts Online ODE rollouts
Preference signal Binary (\(x^w \succ x^l\)) Continuous \(\hat{A}^{(i)}\) (PPO) Group advantage + Bradley-Terry
Sampler during training Offline (no sampling) SDE (mandatory) Any ODE
Log-likelihood ELBO over pairs Per-step IS ratio ELBO over group
Online adaptation No Yes Yes
Training speed vs. FlowGRPO — \(1\times\) ~20× faster

Limitations

  • Requires a reference policy \(\pi_{\theta_\text{ref}}\) — as \(\theta\) drifts far from \(\theta_\text{ref}\), the ELBO approximation of the log-ratio degrades.
  • Group ranking requires \(N\) forward passes per prompt (parallelisable but \(O(N)\) memory for images).
  • The ELBO-based log-ratio inherits the same timestep-weighting approximation as Diffusion-DPO; exact log-likelihood is still intractable.
  • Group partitioning by sign loses the magnitude information within each group (partially recovered by \(w^\pm \propto |\hat{A}|\)).