muon

Ji-Ha @Ji_Ha_Kim · 9h Very cool! I worked on this recently, and I actually used an identical approach early on. But I believe there is a significantly better approach – a **single** minimax rational iteration can beat 5 polynomial steps! [Embedded image: math graphing tool (Desmos-like) showing equations φ(x)=φ2(φ1(x)), Z_7(x)=Z_1(x)/1.4093601263725774, p(x)=p5(p4(p3(p2(p1(x))))), and various polynomial definitions φ1(x), φ2(x), p1(x)-p5(x), plotted as oscillating curves converging near y=1] > QUOTED: Jack Zhang ✓ @jcz42 · Mar 30 > We made Muon run up to 2x faster for free! > Introducing Gram Newton-Schulz: a mathematically equivalent but ... > [thumbnail bar chart comparing "1 Dense MLP Weight" pipeline stages, showing speedup]

Harsh Bhatt ✓ @harshbhatt7585 · 23h I tried moun over AdamW with GRPO, it is far stable than that with moun. [2 replies, 371 views] rajan agarwal ✓ @_rajanagarwal · 23h my intuition here is that muon probably provides strong benefits when the intention is generalized RL, but there's a chance that only works with muon bases just by the argument of loss landscapes [1 reply, 321 views] Harsh Bhatt ✓ @harshbhatt7585 · 22h I tried in high dense grid world with HRM model, In my GRPO experiments, Muon felt noticeably more stable than AdamW. I also think that operator-aware step geometry may help more when the RL objective is broad and general rather than narrowly optimized, But grid-world was very narrowed as cannot be as general as LLMs but still it felt better, let me search the wandb experiment, will drop here if I find. [1 reply, 2 likes, 95 views] Harsh Bhatt ✓ @harshbhatt7585 · 22h *HRM model wa not pretrained LLM [96 views] Mingta Kaivo 明塔 ... ✓ @MingtaKa... · 12h super interesting thread. we've been experimenting with different optimizers for fine-tuning audio models at audiowave and the step size sensitivity is real. curious if anyone has tried muon specifically for RLHF on smaller models (sub-7B) or if the benefits mostly show up at scale?