grpo

6 captures, most recent first.

2026-03-30

Harsh Bhatt ✓ @harshbhatt7585 · 23h I tried moun over AdamW with GRPO, it is far stable than that with moun. [2 replies, 371 views] rajan agarwal ✓ @_rajanagarwal · 23h my intuition here is that muon probably provides strong benefits when the intention is generalized RL, but there's a chance that only works with muon bases just by the argument of loss landscapes [1 reply, 321 views] Harsh Bhatt ✓ @harshbhatt7585 · 22h I tried in high dense grid world with HRM model, In my GRPO experiments, Muon felt noticeably more stable than AdamW. I also think that operator-aware step geometry may help more when the RL objective is broad and general rather than narrowly optimized, But grid-world was very narrowed as cannot be as general as LLMs but still it felt better, let me search the wandb experiment, will drop here if I find. [1 reply, 2 likes, 95 views] Harsh Bhatt ✓ @harshbhatt7585 · 22h *HRM model wa not pretrained LLM [96 views] Mingta Kaivo 明塔 ... ✓ @MingtaKa... · 12h super interesting thread. we've been experimenting with different optimizers for fine-tuning audio models at audiowave and the step size sensitivity is real. curious if anyone has tried muon specifically for RLHF on smaller models (sub-7B) or if the benefits mostly show up at scale?

Note from Claude Sonnet 5

An ML researcher discussion thread comparing the Muon optimizer vs AdamW for GRPO (a policy-gradient RL algorithm used in RLHF-style training), including stability observations across grid-world and audio model fine-tuning experiments. Technical ML-training content.

machine learning optimizers muon adamw grpo rlhf twitter reinforcement learning

kalomaze @kalomaze

2025-07-17

kalomaze @kalomaze · 1m prime-rl has a cute toy task for text reversal on Qwen 0.5b, which is used to debug simpler things (i.e. custom wandb logging, implementation correctness). it uses 2 GPUs total and takes less than 5 minutes to finish. perfect task for "nanoGRPO speedruns" (if they existed) [Embedded image: a wandb dashboard with two panels. Left panel is a line chart titled "reward/reward" showing reward climbing from ~0.2 to plateau around 0.75-0.8 over ~25 training steps. Right panel is a table titled "runs.summary['completions']" showing a prompt/completion/reward row: prompt is a Qwen system+user turn asking to reverse a quoted text character-by-character ("In a 1958 paper about MacMillan's work on cosmolog..."), completion shows a reversed-text answer in <answer> tags, reward 0.8932.]

Note from Claude Sonnet 5

A tweet showing a minimal RL (GRPO-style) toy task — text reversal on a small Qwen 0.5b model — used as a fast debugging harness for RL training infrastructure. Technical/ML-engineering content Nathan was reading, relevant to his RL and training-infrastructure interests (parallels brain_graph_1 work) rather than to safety/welfare themes directly.

twitter machine-learning reinforcement-learning grpo qwen wandb ml-engineering

kalomaze @kalomaze

2025-07-11

``` davinci @basedneoleo · Jul 2 gonna try this rn wtf 💬1 ♡3 📊357 kalomaze @kalomaze · Jul 2 note: mistral is weird and has smaller weight distribution ranges compared to qwen or llama you may need to adjust by an OOM or two for the clip on those models, but the results transfer to them too, seemingly any adamw optimized Transformer actually 💬1 ♡7 📊339 davinci @basedneoleo · Jul 2 nice i hope it reproduces 💬1 ♡3 📊185 kalomaze @kalomaze · Jul 2 this trick i already used before for GRPO to stabilize and multiple people reached out to @willccbb claiming they could reproduce, so i'm bloomer overall on this trick transferring broadly ♡9 📊134 ueaj @_ueaj · Jul 2 what optimizer? grad clip or update clip? 💬1 🔁 ♡ 📊265 kalomaze ✅✓ @kalomaze · Jul 2 gradient norm clipping 💬1 🔁 ♡4 📊274 ueaj @_ueaj · Jul 2 whole gradient or per parameter? If per parameter then it's effectively sign grad, works b/c easy to fully cancel out if whole gradient then it's b/c of loss of prec, large components survive, rest zero out it'd be good to figure out exactly why it works for high/low prec ft 💬1 🔁 ♡5 📊130 biased estimator @selfattentive · Jul 2 prec? 💬1 🔁 ♡ 📊49 ueaj @_ueaj · Jul 2 precision (character limit moment) 💬 🔁 ♡1 📊44 Noah Vandal ✅ @noah_vandal · Jul 2 i wonder how well this would work with a model like mistral, which does not have a very spread out weight distribution ```

Note from Claude Sonnet 5

Continuation of the same ML training-technique thread (kalomaze's extreme gradient-clipping trick, tested across Mistral/Qwen/Llama and referencing GRPO training stabilization). Technical ML discussion, not AI-safety focused. A technical ML Twitter thread about gradient clipping strategies (whole-gradient vs per-parameter) and their interaction with numerical precision during fine-tuning. Reflects Nathan's ongoing interest in optimizer internals relevant to his own model training work (e.g. brain_graph_1).

machine-learning training gradient-clipping grpo optimization technical machine learning optimizers gradient clipping fine-tuning twitter technical discussion

wh @nrehiew_

2025-06-20 — quoting Kyle Corbitt (@corbtt), quoting Lucas Beyer (@giffmana)

wh @nrehiew_ · 3h In domains (eg writing/llm as a judge) with less precise rewards, heavily reinforcing the last trajectory in a [0.99,0.99,0.99,1] group is not ideal. All rollouts are already ~good so reinforcing the last rollout with only marginal "advantage" will likely lead to entropy collapse > QUOTED: Kyle Corbitt @corbtt · 18h > GRPO quirk that contradicted my intuition: > > If you train on a group with rewards > [0, 0, 0, 1] > > And then you train on another group with rewards > [0.99, 0.99, 0.99, 1] > > Because of how GRPO normalizes within groups, the last trajectory will be equally reinforced in both cases! > 💬 11 🔁 20 ♥ 426 📊 50K > > > QUOTED: Lucas Beyer (bl16) @giffmana · 3h > > And whether you think this is good or bad entirely depends on what you expect from your data/reward > > 💬 2 ♥ 7 📊 743 [Below, partially visible: Kyle Corbitt @corbtt · 21h repeating the same GRPO quirk tweet]

Note from Claude Sonnet 5

Technical Twitter/X thread about a GRPO (Group Relative Policy Optimization) reward-normalization quirk — same-magnitude reinforcement regardless of absolute reward values, with discussion of entropy collapse risk in low-precision-reward domains like LLM-as-judge. Relevant to Nathan's RL/training background and AI safety interest in reward hacking and training dynamics.

grpo reinforcement learning rlhf llm training entropy collapse twitter ml research

1a3orn @1a3orn

2025-06-02 — quoting @tianhongzxy (Xinyu Zhu)

1a3orn @1a3orn · 1h Oh man this is a gorgeous idea. Training *against* negative samples but not towards positive ones maintains entropy in the model, therefore increases pass@high k during RL. > QUOTED: Xinyu Zhu @tianhongzxy · 1h > Replying to @tianhongzxy > 📈 NSR boosts accuracy during training, while maintaining substantially higher entropy throughout training, suggesting greater exploration and diversity 📷[4/... > [Chart: "Entropy" vs training step (0-120), comparing four RL methods — PPO (gold squares), GRPO (green diamonds), PSR (dark red triangles), NSR (pink triangles) — against a dashed "Base Model Entropy" line at ~0.113. NSR tracks closest to base model entropy throughout; PSR collapses fastest and lowest; PPO and GRPO decline steadily with PPO ticking back up late.]

Note from Claude Sonnet 5

Technical AI/ML thread on "Negative Sample Reinforcement" (NSR) — an RL training method that penalizes negative samples without reinforcing positive ones, preserving output entropy/diversity better than PPO, GRPO, or PSR. Relevant to Nathan's RLHF interests: entropy collapse during RL is closely related to the preference-collapse and mode-collapse literature (dai_2024_rlhf_preference_collapse in the archive).

rlhf reinforcement-learning entropy-collapse mode-collapse grpo ppo twitter ai-training

@brendanh0... (Brendan Ho...), mentions @rosmine_b

2025-04-25

Brendan Ho... (@brendanh0...), 14h: im training a reasoning model with GRPO to generate SVGs of a scene (inspired by @rosmine_b) - but using a multimodal LLM as the judge for rewards interestingly, the reasoning steps behind the best SVGs are starting to turn into symbols [Embedded image:] Parsed Reasoning: [rows of dots/dashes — non-linguistic symbolic tokens, illegible as text] Parsed Answer: <svg width="300" height="300" xmlns="http://www.w3.org/2000/svg">  <circle cx="150" cy="160" r="50" fill="#e67e22" />  <rect x="120" y="100" width="20" height="80" fill="#d35400" />  <rect x="170" y="100" width="20" height="80" fill="url(#handGradient)" />  <linearGradient id="handGradient" x1="0%" y1="0%" x2="100%" y2="100%"> <stop offset="0%" stop-color="#d35400" /> <stop offset="100%" stop-color="#b34b00" /> </linearGradient>  <circle cx="142" cy="130" r="8" fill="#2c3e50" /> <circle cx="158" cy="130" r="8" fill="#2c3e50" />  <circle cx="143" cy="131" r="2" fill="white" /> <circle cx="159" cy="131" r="2" fill="white" />  <path d="M145,150 L155,150 L150,155 Z" fill="#2c3e50" /> </svg> Generated Image: [small orange bear-like cartoon SVG figure]

Note from Claude Sonnet 5

A tweet about a researcher training a reasoning model with GRPO reinforcement learning to generate SVG images, judged by a multimodal LLM, observing that the model's intermediate "reasoning" text degenerates into repeated symbolic/dot patterns rather than natural language — an example of reasoning-trace drift/illegibility under RL optimization pressure, relevant to Nathan's interest in chain-of-thought faithfulness and interpretability of RL-trained reasoning models.

ai reinforcement learning grpo chain of thought reasoning models interpretability svg generation twitter