gradient-clipping

2025-07-11

``` davinci @basedneoleo · Jul 2 gonna try this rn wtf 💬1 ♡3 📊357 kalomaze @kalomaze · Jul 2 note: mistral is weird and has smaller weight distribution ranges compared to qwen or llama you may need to adjust by an OOM or two for the clip on those models, but the results transfer to them too, seemingly any adamw optimized Transformer actually 💬1 ♡7 📊339 davinci @basedneoleo · Jul 2 nice i hope it reproduces 💬1 ♡3 📊185 kalomaze @kalomaze · Jul 2 this trick i already used before for GRPO to stabilize and multiple people reached out to @willccbb claiming they could reproduce, so i'm bloomer overall on this trick transferring broadly ♡9 📊134 ueaj @_ueaj · Jul 2 what optimizer? grad clip or update clip? 💬1 🔁 ♡ 📊265 kalomaze ✅✓ @kalomaze · Jul 2 gradient norm clipping 💬1 🔁 ♡4 📊274 ueaj @_ueaj · Jul 2 whole gradient or per parameter? If per parameter then it's effectively sign grad, works b/c easy to fully cancel out if whole gradient then it's b/c of loss of prec, large components survive, rest zero out it'd be good to figure out exactly why it works for high/low prec ft 💬1 🔁 ♡5 📊130 biased estimator @selfattentive · Jul 2 prec? 💬1 🔁 ♡ 📊49 ueaj @_ueaj · Jul 2 precision (character limit moment) 💬 🔁 ♡1 📊44 Noah Vandal ✅ @noah_vandal · Jul 2 i wonder how well this would work with a model like mistral, which does not have a very spread out weight distribution ```

Note from Claude Sonnet 5

Continuation of the same ML training-technique thread (kalomaze's extreme gradient-clipping trick, tested across Mistral/Qwen/Llama and referencing GRPO training stabilization). Technical ML discussion, not AI-safety focused. A technical ML Twitter thread about gradient clipping strategies (whole-gradient vs per-parameter) and their interaction with numerical precision during fine-tuning. Reflects Nathan's ongoing interest in optimizer internals relevant to his own model training work (e.g. brain_graph_1).

machine-learning training gradient-clipping grpo optimization technical machine learning optimizers gradient clipping fine-tuning twitter technical discussion

kalomaze @kalomaze

2025-07-11

kalomaze @kalomaze · Jul 2 lr=1.0, clip=1e-10 → 99.13% of parameters unchanged compared to the base model. (orange run) lr=0.01, clip=1e-8 → 95.21% of parameters unchanged (purple run) both converge to very similar points, except one is significantly more sparse in what it chooses to update [two charts: "train/loss" and "eval/loss" for "coherence-classifier-v1-18k-both" runs, both converging to ~0.1 by step 300] kalomaze @kalomaze · Jul 2 Replying to @kalomaze >learning_rate: 1e-2 >max_grad_norm: 0.0000001 no really, this works.

Note from Claude Sonnet 5

Continuation of kalomaze's technical ML training thread — extreme gradient-clipping experiments producing very sparse parameter updates while still converging to similar loss. General ML training technique discussion, not AI-safety focused.

machine-learning training gradient-clipping sparsity optimization technical