6 captures, most recent first.
kalomaze (@kalomaze) — 7h
getting pissed off that this:
a. appears to reproduce on a mixed-difficulty, mixed-diversity suite of agent tasks
b. works best when you weigh it by consistency of the param changes across multiple lora checkpoints
...it feels like trollface physics? wtf is the catch?
[Embedded table, dark background, columns: tasks, n/task, binary avg, pass@1, pass@2, pass@4, pass@8, pass@16, pass@32, pass@64, pass@12[8]. Header: "tasks, n=128 rollouts per task/model, 12,288 rows per model."
Row "[b]ase Qwen3.5-35B-[A]3B": 96, 128, 0.642, 0.642, 0.823, 0.933, 0.980, 0.996, 1.000, 1.000, [1.00]
Row "[?]kpt200": 96, 128, 0.697, 0.697, 0.842, 0.930, 0.973, 0.991, 0.998, 1.000, [1.00]
Row "[?] extrap": 96, 128, 0.710, 0.710, 0.846, 0.934, 0.977, 0.992, 0.998, 1.000, [1.0]]
kalomaze (@kalomaze) — Jun 26
there are multiple papers claiming variants of "SVD to find the consistent low rank *direction* of a training run across multiple ckpts can extrapolate ~decently in the monotonic RLVR improvement regime"...
Note from Claude Sonnet 5
A benchmark results table for a Qwen3.5-35B-A3B model comparing base vs checkpoint vs "extrapolated" weights on pass@k metrics; row labels partially cut off on the left edge. Technical ML research tweet thread.
machine learning researchloramodel extrapolationrlvrtwitterqwen
John Schul... ✓ [Thinking Machines badge] (@johnschulm...) — Jun 17
PPO had a second wave in the LLM era for reasons unanticipated by the original paper
- the importance-ratio objective fixes biases from numeric error, async training, and forward pass noise
- the clipping objective affects entropy through a mechanism that we didn't know about at the time of publication (DAPO, arxiv.org/abs/2509.26114)
> QUOTED: John Schul... ✓ [Thinking Machines badge] (@johnschulm...) — Jun 17
> PPO: rejected from NIPS 2017
Note from Claude Sonnet 5
A technical tweet by John Schulman (PPO co-author, now at Thinking Machines) reflecting on why the PPO algorithm found unexpected renewed relevance in LLM-era RL training, quoting his own earlier tweet noting PPO's original NeurIPS rejection.
reinforcement learningppollm trainingjohn schulmanmachine learning research
Aryaman Arora ✓ (@aryaman2020) — 23h
since Noam Shazeer is in the news for (probably deservedly) making 1e9 more dollars, i have to say this is my least favourite quote in his work. in my view, the goal of interpretability should be to prevent such claims from being made ever again
[Embedded image, excerpt from a paper]:
4 Conclusions
We have extended the GLU family of layers and proposed their use in Transformer. In a transfer-learning setup, the new variants seem to produce better perplexities for the de-noising objective used in pre-training, as well as better results on many downstream language-understanding tasks. These architectures are simple to implement, and have no apparent computational drawbacks. [circled in red:] We offer no explanation as to why these architectures seem to work; we attribute their success, as all else, to divine benevolence.
Note from Claude Sonnet 5
A screenshot of the conclusion section of a machine learning paper (GLU variants for Transformers, Noam Shazeer), with the final sentence hand-circled in red for emphasis by the tweet author.
machine learning researchinterpretabilitynoam shazeertransformershumor
Percy Liang @percyliang
In our last episode, careful tuning, scaling, and ensembles led to a 5x gain in data efficiency (requires 5x less data to get the same loss). Now, with a rephraser model, we can get an additional 1.8x gain in data efficiency. I know, everyone's compute constrained, but we're preparing for a data-constrained future.
> QUOTED: Konwoo Kim @konwookim · 7h
> for data-constrained pre-training, synth data isn't just benchmaxxing, it lowers loss on the real data distribution as we generate more tokens
> for even better scaling, treat synth gens as forming on...
> [Embedded paper image: "Data-efficient pre-training by scaling synthetic megadocs" — Konwoo Kim, Suhas Kotha, Yejin Choi, Tatsunori Hashimoto, Nick Haber, Percy Liang, Stanford University. Diagrams of "Simple Rephrasing," "Stitched Rephrasing," and "Latent Thoughts" methods for generating synthetic training data from real docs; a scaling chart "Scaling Generations" showing IID Loss decreasing from 3.55 (real data only) down to ~3.34-3.41 as generations per pretraining doc increase from 0 to 32, with Latent Thoughts performing best.]
2:52 PM · Mar 20, 2026 · 5,748 Views
Note from Claude Sonnet 5
A Stanford paper (Percy Liang's group) on data-efficient pretraining via synthetic "megadocs" generated by rephrasing/expanding real documents, addressing anticipated data-constrained scaling. Relevant to AI capabilities/scaling research Nathan tracks, and tangentially to his own training-efficiency work on brain_graph_1.
twittermachine learning researchdata efficiencysynthetic datapretrainingscaling lawsstanfordpercy liang
James Chen @jchencxh · Mar 19
The actual implementation I give for doing this on a MIM task is actually very straightforward.
You just take the hidden representations after every level of a ViT, sample some subset, and use each representation separately to predict masked patches at every single level.
1 reply, 9 likes, 454 views
James Chen @jchencxh · 23h
There are actually two reasons for predicting lower level abstractions:
1) less spuriousness (harder to be spurious for predicting a hierarchy)
2) retain lower level reps that can be useful for later composition in the net
Note that for 2), it's because we're only communicating across representations levels through a single embedding through a feed forward net.
If there was a larger communication bandwidth across the representation levels (e.g., some version of attention residuals), this problem could go away, and there's a lot more possibility for the higher level abstractions we can compose (as it's becomes easier to compose low and very high levels of abstraction, where the very lower level of abstraction would not be kept in a standard feed forward structure).
I think that a newer class of architectures is probably better to exploit this type of learning over the hierarchy.
Note from Claude Sonnet 5
A technical ML architecture thread on masked-image-modeling (MIM) with per-layer prediction of masked patches from a Vision Transformer (ViT), discussing hierarchical representation learning and cross-level communication bandwidth. Likely research reading relevant to Nathan's own architecture work on brain_graph_1 / latent iteration design.
twittermachine learning researchvision transformermasked image modelingrepresentation learningneural architecture
Lisan al Gaib ✓ @scaling01
first of all i wouldn't equate coding to reasoning
secondly, the result is rather unsurprising
code is much more structured, information dense and requires more reasoning than natural language, so
more training data -> better generalization and capturing of nuances
more params -> more memorization
we knew that before
[quoted tweet]
Nicholas Roberts @nick11roberts · 14h
📈📉NEW SCALING LAW PHENOMENON📉📈
We find that knowledge and reasoning exhibit different scaling behaviors! ...
Show more
Note from Claude Sonnet 5
A technical Twitter exchange about scaling laws — @scaling01 pushes back on a claimed "new scaling law phenomenon" (from @nick11roberts) distinguishing knowledge vs. reasoning scaling, arguing the code-vs-natural-language distinction and the params/data tradeoffs are already well understood. Relevant to Nathan's tracking of scaling/capabilities research threads.
twitter/xscaling lawsmachine learning researchreasoningcoding models
