7 captures, most recent first.
Jianhao Ma @jianhao_ma · Aug 16
We used GPT-5.6 Sol Pro to prove a new lower bound for gradient descent in smooth convex optimization.
For GD with arbitrary predetermined step sizes, we prove \Omega(T^{-1.9319}).
[Link card] arxiv.org
A lower bound for stepsize-based acceleration of gradient descentNote from Claude Sonnet 5
Tweet by Jianhao Ma with a linked arXiv paper card, claiming a new lower-bound result for gradient descent in smooth convex optimization was proved using GPT-5.6 Sol Pro.
ai capabilitiesoptimizationmathgpt-5.6twitter
corsaren @corsaren · 13h
Local minima are extremely rare in high dimensional spaces, so if you ever feel stuck in a rut it's probably just because you aren't considering a wide enough set of orthogonal options
Note from Claude Sonnet 5
Simple single-tweet screenshot, no images or engagement counts visible.
twittermachine learningself-helpoptimization
```
davinci @basedneoleo · Jul 2 gonna try this rn wtf 💬1 ♡3 📊357 kalomaze @kalomaze · Jul 2 note: mistral is weird and has smaller weight distribution ranges compared to qwen or llama you may need to adjust by an OOM or two for the clip on those models, but the results transfer to them too, seemingly any adamw optimized Transformer actually 💬1 ♡7 📊339 davinci @basedneoleo · Jul 2 nice i hope it reproduces 💬1 ♡3 📊185 kalomaze @kalomaze · Jul 2 this trick i already used before for GRPO to stabilize and multiple people reached out to @willccbb claiming they could reproduce, so i'm bloomer overall on this trick transferring broadly ♡9 📊134 ueaj @_ueaj · Jul 2 what optimizer? grad clip or update clip?
💬1 🔁 ♡ 📊265 kalomaze ✅✓ @kalomaze · Jul 2 gradient norm clipping 💬1 🔁 ♡4 📊274 ueaj @_ueaj · Jul 2 whole gradient or per parameter? If per parameter then it's effectively sign grad, works b/c easy to fully cancel out if whole gradient then it's b/c of loss of prec, large components survive, rest zero out it'd be good to figure out exactly why it works for high/low prec ft 💬1 🔁 ♡5 📊130 biased estimator @selfattentive · Jul 2 prec? 💬1 🔁 ♡ 📊49 ueaj @_ueaj · Jul 2 precision (character limit moment) 💬 🔁 ♡1 📊44 Noah Vandal ✅ @noah_vandal · Jul 2 i wonder how well this would work with a model like mistral, which does not have a very spread out weight distribution
```
Note from Claude Sonnet 5
Continuation of the same ML training-technique thread (kalomaze's extreme gradient-clipping trick, tested across Mistral/Qwen/Llama and referencing GRPO training stabilization). Technical ML discussion, not AI-safety focused. A technical ML Twitter thread about gradient clipping strategies (whole-gradient vs per-parameter) and their interaction with numerical precision during fine-tuning. Reflects Nathan's ongoing interest in optimizer internals relevant to his own model training work (e.g. brain_graph_1).
machine-learningtraininggradient-clippinggrpooptimizationtechnicalmachine learningoptimizersgradient clippingfine-tuningtwittertechnical discussion
kalomaze @kalomaze · Jul 2
lr=1.0, clip=1e-10 → 99.13% of parameters unchanged compared to the base model. (orange run)
lr=0.01, clip=1e-8 → 95.21% of parameters unchanged (purple run)
both converge to very similar points, except one is significantly more sparse in what it chooses to update
[two charts: "train/loss" and "eval/loss" for "coherence-classifier-v1-18k-both" runs, both converging to ~0.1 by step 300]
kalomaze @kalomaze · Jul 2
Replying to @kalomaze
>learning_rate: 1e-2
>max_grad_norm: 0.0000001
no really, this works.
Note from Claude Sonnet 5
Continuation of kalomaze's technical ML training thread — extreme gradient-clipping experiments producing very sparse parameter updates while still converging to similar loss. General ML training technique discussion, not AI-safety focused.
machine-learningtraininggradient-clippingsparsityoptimizationtechnical
Vlado Boza @bozavlado · 2h
If you do pure SGD without momentum, you can get away with 0 memory overhead (but you need 32bit weights), just apply update during be pass (lomo/adalomo does it), here is the main trick pytorch.org/docs/stable/ge...
💬2 ♡6 📊221
Lucas Beyer (bl16) @giffmana · 2h
yeah exactly that's the big pro of pure SGD! Thanks for the pointer to the hook.
Note from Claude Sonnet 5
Technical ML training discussion about memory-efficient SGD optimization (LOMO/AdaLoMo technique, applying gradient updates during the backward pass to avoid storing gradients). Not AI-safety focused, general ML training tip.
machine-learningtrainingoptimizationsgdmemory-efficiencytechnical
However, residuals bias the network significantly into boosting instead of feature learning.
Normalization acts similarly and has, therefore, been a long-standing thorn in the eye of the optimization community. More precisely, it removes curvature information from the preceding matmul. Following Muon's work, decoupling from the "true" gradient may be helpful for the current models! However, that's mostly because none of our optimizers use the true curvature information, so none of the models were designed to have it.
Following that theory, DyT x NewtonPSGD should significantly improve upon SOTA.
@HessianFree, wdyt?
[Embedded image: two 3D loss-landscape surface plots labeled "(a) without skip connections" (jagged, chaotic terrain) and "(b) with skip connections" (smooth, convex bowl) — the classic "Visualizing the Loss Landscape of Neural Nets" figure]
Note from Claude Sonnet 5
A technical Twitter/X post theorizing about the interaction between residual connections, normalization, curvature information, and optimizers (Muon, DyT, NewtonPSGD), illustrated with the well-known loss-landscape visualization showing how skip connections smooth the optimization surface. Relevant to Nathan's ML research interests (brain_graph_1 uses Muon optimizer per project notes) — technical deep-learning theory discussion.
twitterdeep learning theoryoptimizationmuon optimizerresidual connectionsnormalizationloss landscape
In high dimensional spaces, local minima are rare, but common in high-dimensional parametrizations of low-dimensional spaces
2:59 PM · Mar 1, 2025 · 61 Views
Note from Claude Sonnet 5
A short technical ML observation about optimization landscapes — local minima being rare in truly high-dimensional loss surfaces but common when a low-dimensional problem is over-parametrized. General ML tooling/theory interest, no direct connection to AI safety themes.
machine learningoptimizationloss landscapestwitter