@svlevine

Diffusion (or flow) makes for excellent policies, but training them with RL is notoriously hard: BPTT is unstable, RL over diffusion blows up the horizon. In our new paper, we show how we can optimize flow matching actors by using "one weird trick" -- "approximate" the Jacobian of the flow denoising process with the identity matrix. 👇 [image] --- ##### Comments > **Siddharth Ancha @siddancha** · [2026-06-10](https://x.com/siddancha/status/2064597954405781732) > > Really cool work! Although I would argue that the "BPTT guidance gradient" you're trying to approximate is not that "principled" to begin with, even if it didn't have its infamous instability issues. That's because the right way to "lookahead" from aₜ to compute the guidance gradient is not the deterministic a₁ = ODE(aₜ), but an expectation over the \*distribution\* p(a₁ | aₜ) induced by the independent coupling used to train the BC flow policy. Unless you're doing variational flow matching (https://arxiv.org/pdf/2406.04843), p(a₁ | aₜ) is not generally accessible. So the 1-step lookahead (your Eq. 7) that gives 𝔼\[a₁ | aₜ\] is arguably a more principled point estimate than ODE(aₜ). > > We used this 1-step lookahead in previous work on test-time diffusion guidance with @du\_yilun (https://siddancha.github.io/anomalies-by-diffusion-synthesis…) with good results. But this still used the Jacobian of the 1-step lookahead; we didn't try to replace it with the identity matrix! That's a neat idea! > > [image] [image] > **Nicholas Boffi @nmboffi** · [2026-06-10](https://x.com/nmboffi/status/2064700605512167813) > > not sure i fully agree with this -- there's several ways to construct principled guidance / finetuning methods. what you are talking about is more like https://arxiv.org/abs/2602.05993 > > what @svlevine is presenting is more like https://arxiv.org/abs/2604.27147 > > [https://t.co/xR5oN5Is6Q](https://t.co/xR5oN5Is6Q) > **Yazid Janati @yjelid** · [2026-06-10](https://x.com/yjelid/status/2064641211764195627) > > what you are presenting here is a well known reward guidance trick introduced here https://arxiv.org/abs/2204.03458 and here https://arxiv.org/abs/2209.14687 in 2022. I don't think this claim in your paper is accurate: > > [image] > **Philip Bachman @philip\_bachman** · [2026-06-10](https://x.com/philip_bachman/status/2064670868744101893) > > Another option is a "Q grad denoiser" that predicts mean of Q grad wrt clean a\_1 given partially denoised a\_t, with a\_1 sampled from the same posterior as for the action denoising. This seems slightly nicer than grad wrt expected a\_1, since expected a\_1 may be OOD for Q.