— saved image
Wenhao Chai @wenhaocha1 · 3h So it's not really close the gap. It would be like updating trainer using sampler's gradient direction. We still need to find the way at least make the gap not too big so that this shifted gradient could work. [quoted post] Martin Marek @mrtnm · Sep 18 Why is RL so sensitive to training-inference mismatch (TIM)? With each training step, the trainer is pulled toward the sampler, resulting in accumulating ... [embedded GIF: "Score Centering Stabilizes Off-policy RL"; legend: Trainer, Sampler, Reward signal, Drift; two panels, "Without correction" and "Score centering", each a contour plot with "Higher loss" label, "Minimum", "Start", points T and S] 1 reply · 5 likes · 1K views Martin Marek @mrtnm · 3h One thing you can do is apply TIS or MIS to approximate on-policy updates. However, because these methods are not exact IS, they do not cancel drift completely. So you can apply SC on top of TIS / MIS to cancel the remaining drift.
Note from Claude Sonnet 5
A thread on RL training-inference mismatch: Wenhao Chai quote-tweets Martin Marek's post, which embeds a GIF diagram titled 'Score Centering Stabilizes Off-policy RL' comparing a trainer/sampler trajectory without correction (drifting) versus with score centering (heading toward the minimum). Marek replies about applying score centering on top of TIS/MIS.
reinforcement learningtraining-inference mismatchoff-policy rlimportance samplingscore centering