— quoting @YouJiacheng — saved image
Yifei Wang @yifeiwang77 · 13h
Very intriguing discussion here to rethink policy gradient: we always want to optimize E_p[R], so we would need the IS techniques that introduced all variance.
Score centering is taking a different view: let's just optimize E_q[R], as q (sampling distribution) is what we finally care about. As samples are indeed from q, no IS is needed.
And since we cannot optimize q (inference engine/ old policy) directly, we just use p's grad as a surrogate to q's grad (yes its biased), as explained in the STE estimate here.
A change of perspective (from optimizing p obj to q obj) gives a lot of simplicity!
[quoted post]
You Jiacheng @YouJiacheng · Sep 20
Actually this argument is not limited to linear policy.
Consider the gradient w.r.t. logits.
s = ∇_z log p(a) = e_a - p
\bar{s} = E_{a~q}[e_a - p] = (q - p)... [cut off]Note from Claude Sonnet 5
A text tweet by @yifeiwang77 on rethinking policy gradient: optimizing E_q[R] over the sampling distribution via 'score centering' avoids importance sampling, using p's gradient as a biased surrogate (straight-through-style). Quotes a @YouJiacheng post with a logit-gradient derivation, cut off at the bottom.
policy gradientrlimportance samplingscore centeringml theoryllm training