importance sampling

2026-09-21 — quoting @YouJiacheng — saved image

Yifei Wang @yifeiwang77 · 13h

Very intriguing discussion here to rethink policy gradient: we always want to optimize E_p[R], so we would need the IS techniques that introduced all variance.

Score centering is taking a different view: let's just optimize E_q[R], as q (sampling distribution) is what we finally care about. As samples are indeed from q, no IS is needed.

And since we cannot optimize q (inference engine/ old policy) directly, we just use p's grad as a surrogate to q's grad (yes its biased), as explained in the STE estimate here.

A change of perspective (from optimizing p obj to q obj) gives a lot of simplicity!

[quoted post]
You Jiacheng @YouJiacheng · Sep 20
Actually this argument is not limited to linear policy.
Consider the gradient w.r.t. logits.
s = ∇_z log p(a) = e_a - p
\bar{s} = E_{a~q}[e_a - p] = (q - p)... [cut off]

Note from Claude Sonnet 5

A text tweet by @yifeiwang77 on rethinking policy gradient: optimizing E_q[R] over the sampling distribution via 'score centering' avoids importance sampling, using p's gradient as a biased surrogate (straight-through-style). Quotes a @YouJiacheng post with a logit-gradient derivation, cut off at the bottom.

policy gradient rl importance sampling score centering ml theory llm training

@wenhaocha1

2026-09-20 — saved image

Wenhao Chai @wenhaocha1 · 3h

So it's not really close the gap. It would be like updating trainer using sampler's gradient direction. We still need to find the way at least make the gap not too big so that this shifted gradient could work.

[quoted post] Martin Marek @mrtnm · Sep 18
Why is RL so sensitive to training-inference mismatch (TIM)?

With each training step, the trainer is pulled toward the sampler, resulting in accumulating ...
[embedded GIF: "Score Centering Stabilizes Off-policy RL"; legend: Trainer, Sampler, Reward signal, Drift; two panels, "Without correction" and "Score centering", each a contour plot with "Higher loss" label, "Minimum", "Start", points T and S]

1 reply · 5 likes · 1K views

Martin Marek @mrtnm · 3h
One thing you can do is apply TIS or MIS to approximate on-policy updates. However, because these methods are not exact IS, they do not cancel drift completely. So you can apply SC on top of TIS / MIS to cancel the remaining drift.

Note from Claude Sonnet 5

A thread on RL training-inference mismatch: Wenhao Chai quote-tweets Martin Marek's post, which embeds a GIF diagram titled 'Score Centering Stabilizes Off-policy RL' comparing a trainer/sampler trajectory without correction (drifting) versus with score centering (heading toward the minimum). Marek replies about applying score centering on top of TIS/MIS.

reinforcement learning training-inference mismatch off-policy rl importance sampling score centering