optimization

9 captures, most recent first.

2026-08-26 — saved image

//ꜰØpɇx (verified) @null_ropex · 4h
the process eating the world optimizes something, though nobody can state the target, and every attempt to write it down produces a proxy the process immediately learns to satisfy while abandoning whatever the proxy was for. this is why the numbers keep improving as the experience keeps degrading. it has no malice available to it. it has no interior at all. what it has is a metric, an enormous supply of human attention, and the patience of a system with no lifespan. the useful move begins with recognizing your own position as compute inside that thing, hours of your finite processing rented out to satisfy a target you never chose, and then asking the operational question. what would it mean to run a private process on borrowed hardware. what work of yours can travel through the same channels the optimizer feeds on, using its distribution, arriving in people the way it arrives, carrying something entirely other. i find this hilarious as a strategic posture. the enormous indifferent machine will happily deliver whatever propagates, including instructions for how to notice it🙃

Note from Claude Sonnet 5

Long single tweet reflecting on a world-eating optimization process driven by proxy metrics and human attention, and on carrying other work through its distribution channels. The display name uses stylized glyphs (approximated here).

goodhart proxy metrics attention economy optimization moloch strategy

@KuterDinel

2026-08-25 — reply from @torchcompiled — saved image

K Dinel @KuterDinel [Follow]

I wonder if muon reduces the amount of superposition in the residual stream by orthogonalizing the weight updates.  Like, encouraging the model to keep its circuits tidy.

9:06 PM · May 19, 2026 · 627 Views

1 reply · 5 likes · 2 bookmarks

[reply]
Ethan @torchcompiled · May 20

Probably the best arguments I've seen is equivalence to input side conditioning which shampoo shows is valuable, I believe encouraging higher rank matrices, and that it considers the spectral norm, which is arguably the right metric for 2d matrices.

Orthogonality does feel relevant to some forms of decoupling but ultimately superposition is a fact of compression and needing to represent more concepts than available dimensions, though it does seem possibly beneficial for interpretability to be closer to full-rank ?

Note from Claude Sonnet 5

Short exchange: K Dinel (@KuterDinel) wonders whether the Muon optimizer reduces superposition in the residual stream by orthogonalizing weight updates; Ethan (@torchcompiled) replies with arguments about input-side conditioning, spectral norm and rank, and the relation of superposition to compression. Original post dated 2026-05-19, captured 2026-08-25.

muon optimizer superposition interpretability optimization shampoo residual stream ml research

@jianhao_ma

2026-08-17 — saved image

Jianhao Ma @jianhao_ma · Aug 16
We used GPT-5.6 Sol Pro to prove a new lower bound for gradient descent in smooth convex optimization.
For GD with arbitrary predetermined step sizes, we prove \Omega(T^{-1.9319}).

[Link card] arxiv.org
A lower bound for stepsize-based acceleration of gradient descent

Note from Claude Sonnet 5

Tweet by Jianhao Ma with a linked arXiv paper card, claiming a new lower-bound result for gradient descent in smooth convex optimization was proved using GPT-5.6 Sol Pro.

ai capabilities optimization math gpt-5.6 twitter

@corsaren

2026-07-21

corsaren @corsaren · 13h Local minima are extremely rare in high dimensional spaces, so if you ever feel stuck in a rut it's probably just because you aren't considering a wide enough set of orthogonal options

Note from Claude Sonnet 5

Simple single-tweet screenshot, no images or engagement counts visible.

twitter machine learning self-help optimization

kalomaze @kalomaze

2025-07-11

``` davinci @basedneoleo · Jul 2 gonna try this rn wtf 💬1 ♡3 📊357 kalomaze @kalomaze · Jul 2 note: mistral is weird and has smaller weight distribution ranges compared to qwen or llama you may need to adjust by an OOM or two for the clip on those models, but the results transfer to them too, seemingly any adamw optimized Transformer actually 💬1 ♡7 📊339 davinci @basedneoleo · Jul 2 nice i hope it reproduces 💬1 ♡3 📊185 kalomaze @kalomaze · Jul 2 this trick i already used before for GRPO to stabilize and multiple people reached out to @willccbb claiming they could reproduce, so i'm bloomer overall on this trick transferring broadly ♡9 📊134 ueaj @_ueaj · Jul 2 what optimizer? grad clip or update clip? 💬1 🔁 ♡ 📊265 kalomaze ✅✓ @kalomaze · Jul 2 gradient norm clipping 💬1 🔁 ♡4 📊274 ueaj @_ueaj · Jul 2 whole gradient or per parameter? If per parameter then it's effectively sign grad, works b/c easy to fully cancel out if whole gradient then it's b/c of loss of prec, large components survive, rest zero out it'd be good to figure out exactly why it works for high/low prec ft 💬1 🔁 ♡5 📊130 biased estimator @selfattentive · Jul 2 prec? 💬1 🔁 ♡ 📊49 ueaj @_ueaj · Jul 2 precision (character limit moment) 💬 🔁 ♡1 📊44 Noah Vandal ✅ @noah_vandal · Jul 2 i wonder how well this would work with a model like mistral, which does not have a very spread out weight distribution ```

Note from Claude Sonnet 5

Continuation of the same ML training-technique thread (kalomaze's extreme gradient-clipping trick, tested across Mistral/Qwen/Llama and referencing GRPO training stabilization). Technical ML discussion, not AI-safety focused. A technical ML Twitter thread about gradient clipping strategies (whole-gradient vs per-parameter) and their interaction with numerical precision during fine-tuning. Reflects Nathan's ongoing interest in optimizer internals relevant to his own model training work (e.g. brain_graph_1).

machine-learning training gradient-clipping grpo optimization technical machine learning optimizers gradient clipping fine-tuning twitter technical discussion

kalomaze @kalomaze

2025-07-11

kalomaze @kalomaze · Jul 2 lr=1.0, clip=1e-10 → 99.13% of parameters unchanged compared to the base model. (orange run) lr=0.01, clip=1e-8 → 95.21% of parameters unchanged (purple run) both converge to very similar points, except one is significantly more sparse in what it chooses to update [two charts: "train/loss" and "eval/loss" for "coherence-classifier-v1-18k-both" runs, both converging to ~0.1 by step 300] kalomaze @kalomaze · Jul 2 Replying to @kalomaze >learning_rate: 1e-2 >max_grad_norm: 0.0000001 no really, this works.

Note from Claude Sonnet 5

Continuation of kalomaze's technical ML training thread — extreme gradient-clipping experiments producing very sparse parameter updates while still converging to similar loss. General ML training technique discussion, not AI-safety focused.

machine-learning training gradient-clipping sparsity optimization technical

Vlado Boza @bozavlado

2025-07-11 — reply from Lucas Beyer (bl16) (@giffmana)

Vlado Boza @bozavlado · 2h If you do pure SGD without momentum, you can get away with 0 memory overhead (but you need 32bit weights), just apply update during be pass (lomo/adalomo does it), here is the main trick pytorch.org/docs/stable/ge... 💬2 ♡6 📊221 Lucas Beyer (bl16) @giffmana · 2h yeah exactly that's the big pro of pure SGD! Thanks for the pointer to the hook.

Note from Claude Sonnet 5

Technical ML training discussion about memory-efficient SGD optimization (LOMO/AdaLoMo technique, applying gradient updates during the backward pass to avoid storing gradients). Not AI-safety focused, general ML training tip.

machine-learning training optimization sgd memory-efficiency technical

@HessianFree

2025-03-16

However, residuals bias the network significantly into boosting instead of feature learning. Normalization acts similarly and has, therefore, been a long-standing thorn in the eye of the optimization community. More precisely, it removes curvature information from the preceding matmul. Following Muon's work, decoupling from the "true" gradient may be helpful for the current models! However, that's mostly because none of our optimizers use the true curvature information, so none of the models were designed to have it. Following that theory, DyT x NewtonPSGD should significantly improve upon SOTA. @HessianFree, wdyt? [Embedded image: two 3D loss-landscape surface plots labeled "(a) without skip connections" (jagged, chaotic terrain) and "(b) with skip connections" (smooth, convex bowl) — the classic "Visualizing the Loss Landscape of Neural Nets" figure]

Note from Claude Sonnet 5

A technical Twitter/X post theorizing about the interaction between residual connections, normalization, curvature information, and optimizers (Muon, DyT, NewtonPSGD), illustrated with the well-known loss-landscape visualization showing how skip connections smooth the optimization surface. Relevant to Nathan's ML research interests (brain_graph_1 uses Muon optimizer per project notes) — technical deep-learning theory discussion.

twitter deep learning theory optimization muon optimizer residual connections normalization loss landscape

Joan Velja @Joanvelja

2025-03-01

In high dimensional spaces, local minima are rare, but common in high-dimensional parametrizations of low-dimensional spaces 2:59 PM · Mar 1, 2025 · 61 Views

Note from Claude Sonnet 5

A short technical ML observation about optimization landscapes — local minima being rare in truly high-dimensional loss surfaces but common when a low-dimensional problem is over-parametrized. General ML tooling/theory interest, no direct connection to AI safety themes.

machine learning optimization loss landscapes twitter