technical discussion

3 captures, most recent first.

2025-07-11 — quoting @osanse... (Omar Sanse..., T5Gemma announcement); also @Aella_Girl visible at bottom

[Top, partial prior tweet engagement bar: 💬13 🔁8 ♡102 📊17K] Super Dario ✅ @inductionheads · 2h The real reason theyare bringing these back is continuous learning You can directly store off the encodings as memories Think RAG but instead of embeddings as index, it's encodings as content > QUOTED: Omar Sanse... ✅ @osanse... · Jul 9 > Introducing T5Gemma: the next generation of encoder-decoder/T5 models! > 🔧 Decoder models adapted to be ... > [Diagram: "Pretrained Decoder-Only Model" (FFN + Causal Self/Attention+ROPE) → Initialization → "Encoder-Decoder Adaptation" showing an Encoder block (FFN, Bidirectional Self-Attention+ROPE, Input Sequence) feeding into a Decoder block (FFN, Cross-Attention, Causal Self-Attention+ROPE, Output/Output shifted right)] > 💬5 🔁3 ♡55 📊3K Aella ✅ @Aella_Girl · 11h I wonder what the downstream consequences are of a culture that obsessively scrubs their own scent off every morning [cut off]

Note from Claude Sonnet 5

ML Twitter feed covering Google's T5Gemma encoder-decoder model release and speculation about encoder representations enabling continuous learning/memory (relevant to Nathan's own architecture interests, e.g. brain_graph_1's memory systems), followed by an unrelated Aella tweet about hygiene culture.

machine learning t5gemma encoder-decoder continuous learning memory twitter technical discussion

Kevin Frans @kvfrans

2025-07-11 — quoting @micahgold... (Micah Gold...)

Kevin Frans ✅ @kvfrans · 16h This is a great investigation, and it answers the itch that often comes up when using gradient accumulation -- why not just make more frequent updates? The answer is that you need to properly scale the b1/b2 settings too! > QUOTED: Micah Gold... @micahgold... · 20h > 🚨 Did you know that small-batch vanilla SGD without momentum (i.e. the first optimizer you learn about in intro ML) is virtually as fast as AdamW for LLM pretraining on a per-FLOP basis? ... > [Line chart titled "GPT-3 (1.3B)": FineWeb loss vs training tokens (0–10B) for two curves — AdamW BS=512 (blue) and SGD no momentum BS=1 (orange). Both curves track closely together, descending from ~4.3 to ~2.9-3.0 loss, with orange showing a few large early spikes.]

Note from Claude Sonnet 5

A machine-learning Twitter exchange about optimizer scaling laws — the surprising finding that small-batch vanilla SGD without momentum tracks AdamW's per-FLOP training efficiency for LLM pretraining, plus a note on scaling beta1/beta2 with gradient accumulation. Technical reading consistent with Nathan's interest in optimizer behavior for his own model training.

machine learning optimizers sgd adamw llm pretraining twitter technical discussion

kalomaze @kalomaze

2025-07-11

``` davinci @basedneoleo · Jul 2 gonna try this rn wtf 💬1 ♡3 📊357 kalomaze @kalomaze · Jul 2 note: mistral is weird and has smaller weight distribution ranges compared to qwen or llama you may need to adjust by an OOM or two for the clip on those models, but the results transfer to them too, seemingly any adamw optimized Transformer actually 💬1 ♡7 📊339 davinci @basedneoleo · Jul 2 nice i hope it reproduces 💬1 ♡3 📊185 kalomaze @kalomaze · Jul 2 this trick i already used before for GRPO to stabilize and multiple people reached out to @willccbb claiming they could reproduce, so i'm bloomer overall on this trick transferring broadly ♡9 📊134 ueaj @_ueaj · Jul 2 what optimizer? grad clip or update clip? 💬1 🔁 ♡ 📊265 kalomaze ✅✓ @kalomaze · Jul 2 gradient norm clipping 💬1 🔁 ♡4 📊274 ueaj @_ueaj · Jul 2 whole gradient or per parameter? If per parameter then it's effectively sign grad, works b/c easy to fully cancel out if whole gradient then it's b/c of loss of prec, large components survive, rest zero out it'd be good to figure out exactly why it works for high/low prec ft 💬1 🔁 ♡5 📊130 biased estimator @selfattentive · Jul 2 prec? 💬1 🔁 ♡ 📊49 ueaj @_ueaj · Jul 2 precision (character limit moment) 💬 🔁 ♡1 📊44 Noah Vandal ✅ @noah_vandal · Jul 2 i wonder how well this would work with a model like mistral, which does not have a very spread out weight distribution ```

Note from Claude Sonnet 5

Continuation of the same ML training-technique thread (kalomaze's extreme gradient-clipping trick, tested across Mistral/Qwen/Llama and referencing GRPO training stabilization). Technical ML discussion, not AI-safety focused. A technical ML Twitter thread about gradient clipping strategies (whole-gradient vs per-parameter) and their interaction with numerical precision during fine-tuning. Reflects Nathan's ongoing interest in optimizer internals relevant to his own model training work (e.g. brain_graph_1).

machine-learning training gradient-clipping grpo optimization technical machine learning optimizers gradient clipping fine-tuning twitter technical discussion