@DirhousssiAmine

**Dirhousssi Amine** @DirhousssiAmine [2026-03-28](https://x.com/DirhousssiAmine/status/2038041727676678226) based on trl main branch 👉 --- **Kevin David Hayes** @kevindavidhayes [2026-03-30](https://x.com/kevindavidhayes/status/2038443538145255873) Can you tell me the model? --- **Dirhousssi Amine** @DirhousssiAmine [2026-03-30](https://x.com/DirhousssiAmine/status/2038522599706145146) Qwen3-0.6B nothing fancy --- **Chidhambararajan R (a.k.a Chidha)** @TheSeriousProg [2026-03-30](https://x.com/TheSeriousProg/status/2038518198317068644) I think the runs the where the llm precision level and the base model's trained precision levels don't match, a collapse is happening --- **Dirhousssi Amine** @DirhousssiAmine [2026-03-30](https://x.com/DirhousssiAmine/status/2038539164245803012) haha yes, but why ? --- **Chidhambararajan R (a.k.a Chidha)** @TheSeriousProg [2026-03-30](https://x.com/TheSeriousProg/status/2038539886052147351) Because sampling model policy and distillation model policy are closely aligned in similar precisions. Which is not the case with mismatched quantizations. --- **Dirhousssi Amine** @DirhousssiAmine [2026-03-30](https://x.com/DirhousssiAmine/status/2038540514744558038) This is GRPO not distillation :) --- **Chidhambararajan R (a.k.a Chidha)** @TheSeriousProg [2026-03-30](https://x.com/TheSeriousProg/status/2038541914602082550) I mean grpo is opd on policy distillation --- **Seth Karten** @sethkarten [2026-03-28](https://x.com/sethkarten/status/2038026472271618555) a100s? --- **Dirhousssi Amine** @DirhousssiAmine [2026-03-28](https://x.com/DirhousssiAmine/status/2038040720699789427) H100 --- **Alex Chaloner** @alex\_chaloner [2026-03-28](https://x.com/alex_chaloner/status/2038017994580754529) https://arxiv.org/html/2510.26788v1… ? --- **Dirhousssi Amine** @DirhousssiAmine [2026-03-28](https://x.com/DirhousssiAmine/status/2038040988673839121) Yes, very interesting work and it’s an empirical view. The goal of our work is to pinpoint exactly where this numerical error impact the RL training process --- **Harsh Goel** @hgoel1000 [2026-03-29](https://x.com/hgoel1000/status/2038140961159594106) I think the sampling policy is off policy with respect to the policy being updated (pi\_old) due to the precision mismatch --- **Dirhousssi Amine** @DirhousssiAmine [2026-03-29](https://x.com/DirhousssiAmine/status/2038176248447332359) 👏 exactly right intuition. But because GRPO accounts for this offpolicy in the ratio you still think it would converge --- **Harsh Goel** @hgoel1000 [2026-03-29](https://x.com/hgoel1000/status/2038308758200172779) There are 4 policies, pi\_theta (current update), pi\_old (static policy from prev update), pi\_(sample) the policy that the vllm server uses for sampling sequences and pi\_ref for KL loss if enabled. pi\_old = pi\_sample ideally, but mismatched due to precision in 5 cases (exc. pink) --- **Harsh Goel** @hgoel1000 [2026-03-29](https://x.com/hgoel1000/status/2038309070545846382) But with BF 16 enabled in pink, the mismatch does not exist during gradient computation. So we obtain BF16 gradients to update FP32 weights which lack precision to improve the policy in subtle ways but never the less is stable. --- **Dirhousssi Amine** @DirhousssiAmine [2026-03-29](https://x.com/DirhousssiAmine/status/2038312303339196828) Thats mostly right,but this is an async version of grpo so we tolerate a staleness factor between vllm weights and model bf16 weights. The numerical stability is in the ratio for sure but I am trying to systematically define all the ways it can impact the gradient! --- **Harsh Goel** @hgoel1000 [2026-03-29](https://x.com/hgoel1000/status/2038383483593121816) I see, how about you try an experiment where you obtain the logits from the vllm async server and use that in place of pi\_old? Would that help in some of the other cases? I am curious --- **Dirhousssi Amine** @DirhousssiAmine [2026-03-30](https://x.com/DirhousssiAmine/status/2038539618069389621) I am obtaining the logits from vllm exactly as pi\_old. To give you another hint at the problem that I am not showing here to tease out the research, all runs converge if we push lr from 1e-6 -> 1e-5 --- **evan** @evan\_\_casey [2026-03-29](https://x.com/evan__casey/status/2038077004768063507) why do everything in float32 versus just cast the LM head to float32? --- **Dirhousssi Amine** @DirhousssiAmine [2026-03-29](https://x.com/DirhousssiAmine/status/2038175809794384286) Tried exactly that be didn’t change a thing 😬 --- **sonu dixit** @sonudixit2k [2026-03-29](https://x.com/sonudixit2k/status/2038148032542462071) Are you using vllm sampling ratio correction ? - failed runs are most likely generating gibberish. --- **Dirhousssi Amine** @DirhousssiAmine [2026-03-29](https://x.com/DirhousssiAmine/status/2038175709525332112) Sampling is identical in all runs 😉 --- **Grok Grokman** @LeeLeepenkman [2026-03-29](https://x.com/LeeLeepenkman/status/2038146080442105886) not sure but you can just get like more stable training in fp32. you actually have more compute per step or parameter density so there's actually kind of more compute gone into that one that's working the best More compute per step so to speak. It could even still be that the --- **Kevin H. Zhao** @ggkhzhao [2026-03-28](https://x.com/ggkhzhao/status/2038024710974243114) This has happened to me so many times 😂