wh @nrehiew_
— quoting Kyle Corbitt (@corbtt), quoting Lucas Beyer (@giffmana)
wh @nrehiew_ · 3h
In domains (eg writing/llm as a judge) with less precise rewards, heavily reinforcing the last trajectory in a [0.99,0.99,0.99,1] group is not ideal. All rollouts are already ~good so reinforcing the last rollout with only marginal "advantage" will likely lead to entropy collapse
> QUOTED: Kyle Corbitt @corbtt · 18h
> GRPO quirk that contradicted my intuition:
>
> If you train on a group with rewards
> [0, 0, 0, 1]
>
> And then you train on another group with rewards
> [0.99, 0.99, 0.99, 1]
>
> Because of how GRPO normalizes within groups, the last trajectory will be equally reinforced in both cases!
> 💬 11 🔁 20 ♥ 426 📊 50K
>
> > QUOTED: Lucas Beyer (bl16) @giffmana · 3h
> > And whether you think this is good or bad entirely depends on what you expect from your data/reward
> > 💬 2 ♥ 7 📊 743
[Below, partially visible: Kyle Corbitt @corbtt · 21h repeating the same GRPO quirk tweet]
Note from Claude Sonnet 5
Technical Twitter/X thread about a GRPO (Group Relative Policy Optimization) reward-normalization quirk — same-magnitude reinforcement regardless of absolute reward values, with discussion of entropy collapse risk in low-precision-reward domains like LLM-as-judge. Relevant to Nathan's RL/training background and AI safety interest in reward hacking and training dynamics.
grporeinforcement learningrlhfllm trainingentropy collapsetwitterml research