ml theory — Timeline

2026-09-21 — quoting @YouJiacheng — saved image

Yifei Wang @yifeiwang77 · 13h

Very intriguing discussion here to rethink policy gradient: we always want to optimize E_p[R], so we would need the IS techniques that introduced all variance.

Score centering is taking a different view: let's just optimize E_q[R], as q (sampling distribution) is what we finally care about. As samples are indeed from q, no IS is needed.

And since we cannot optimize q (inference engine/ old policy) directly, we just use p's grad as a surrogate to q's grad (yes its biased), as explained in the STE estimate here.

A change of perspective (from optimizing p obj to q obj) gives a lot of simplicity!

[quoted post]
You Jiacheng @YouJiacheng · Sep 20
Actually this argument is not limited to linear policy.
Consider the gradient w.r.t. logits.
s = ∇_z log p(a) = e_a - p
\bar{s} = E_{a~q}[e_a - p] = (q - p)... [cut off]

Note from Claude Sonnet 5

A text tweet by @yifeiwang77 on rethinking policy gradient: optimizing E_q[R] over the sampling distribution via 'score centering' avoids importance sampling, using p's gradient as a biased surrogate (straight-through-style). Quotes a @YouJiacheng post with a logit-gradient derivation, cut off at the bottom.

policy gradient rl importance sampling score centering ml theory llm training

kalomaze @kalomaze

2026-08-01 — saved image

kalomaze @kalomaze · 50m
periodic functions seem like the most fundamental class that conventional nonlinearities can't generalize over, even with depth
even tho you have exponentially many piecewise linear regions with ReLU-likes, that's not sufficient to extrapolate a sine wave...

Note from Claude Sonnet 5

A tweet from kalomaze arguing that periodic functions (like sine waves) are a fundamental class conventional neural network nonlinearities (ReLU-likes) can't extrapolate/generalize over, even with depth or many piecewise-linear regions.

neural networks nonlinearities generalization periodic functions ml theory twitter