Joshua Achiam @jachiam0
Thinking about how every guy who writes 100k word manifestos read by no one has a perfect captive audience in LLMs who consume every token in training and automatically grant a little more plausibility and credibility to those tokens by virtue of their preponderance. Surely the complicated interplay between volume of output and intellectual relevance to society, where one has little connection to the other, will be properly accounted for in the aggregate influences on our collective epistemics in the future
10:27 PM · Aug 8, 2026 · 3,079 Views
Note from Claude Sonnet 5
Joshua Achiam (sardonically) worries that fringe manifesto-writers get outsized influence on LLM training data and thus on future collective epistemics, since token volume in training doesn't track intellectual relevance.
— quoting a paper and reply from @corsaren — saved image
Danielle Fong @DanielleFo... · 22h
the overall cross correlation between IQ subtests collapses to ~0.22 in humans on the right tail.
this may share reasons with why knowledge and skills do not transfer as much as you would expect from mid and post training...
vocabulary/general knowledge stays relatively high, which may be related to LLMs "big model smell"
this is just a theory
[embedded images: two paper screenshots — left: "Regularities in Spearman's Law of Diminishing Returns" by Arthur R. Jensen, Intelligence 31 (2003) 95-105; right: "...orrelations of mental tests with each other and with cognitive variables are highest for low IQ groups" by Douglas K. Detterman & Mark H. Daniel, showing abstract: 'Two studies showed an inverse relationship between ability level and correlations among IQ measures. Low IQ subjects showed much higher correlations than high IQ subjects. Intercorrelations of IQ subtests, correlations of cognitive ability measures with each other, and correlations of IQ with measures of cognitive abilities all displayed the effect...']
corsaren @corsaren · Aug 3
Yeah. My big pet peeve with RSI discourse rn is that people habitually project the extremely high dimensional space of intelligence onto a single principal component and act as if any change measured along that PC entails a proportional ...[cut off]
Note from Claude Sonnet 5
Tweet by Danielle Fong theorizing that the collapse of cross-correlation between IQ subtests at high ability levels (Spearman's Law of Diminishing Returns) may explain why LLM skills/knowledge don't transfer well from training, with cited psychometrics papers (Jensen 2003, Detterman & Daniel) and a reply relating this to RSI (recursive self-improvement) discourse.
Silas Alberti ✔ [icon] @silasalberti · 6h
Recently the industry chatter moved from RL to pretraining (& midtraining): RL is supposed to get diminishing returns because it hits a "ceiling" that depends on the quality of your pretrain. While that is certainly true, it's an open question how high the ceiling actually is.
We wanted to push this to the limit: for Kimi K2.7 which has already been heavily post-trained multiple times, it initially seemed like there wasn't much to squeeze. We dealt with fast entropy collapse & many other issues. However, we used this as a challenge to tune our algorithm and data.
At some point, it suddenly started working and the model, SWE-1.7, magically improved by much more than we expected. It also seems like the ceiling isn't obviously hit yet, so we're excited to see how much further we can push it.
[Quoted tweet:]
Cognition ✔ @cognition · 7h
Introducing SWE-1.7, the most capable model we've trained yet.
It scores within a few points of the strongest frontier models at a fraction of the cost, and is ...
[Embedded table comparing benchmark scores: SWE-1.7, GLM-5.2, Composer 2.5, Opus 4.8, GPT-5.5 —
FrontierCode 1.1 Main: 42.3%, 24.5%, 25.6%, 46.5%, 43.0%
Terminal-Bench 2.1: 81.5%, 81.0%, 76.0%, 86.9%, 84.2%
SWE-Bench Multilingual: 77.8%, 74.5%, 71.6%, 84.4%, 76.8%]
Note from Claude Sonnet 5
Technical AI-research tweet discussing reinforcement-learning post-training results on the Kimi K2.7 base model (yielding "SWE-1.7"), with an embedded benchmark comparison table against GLM-5.2, Composer 2.5, Opus 4.8, and GPT-5.5.
kalomaze ✓ [icon badge] (@kalomaze) — 44m
ok so minithesis
smuggling in learned geometric biases into the optimization of RL tasks might be useful even if it means the network isn't literally optimizing the objective as we have defined or intended it in a pure pg sense, if only bc it constrains adaptation to a geometrically coherent space
consider: a black box RLVR verifier that is systematically and deterministically wrong, in a way that is too arbitrary to learn without compressing an intractably large dictionary into the weights
discriminative value estimators would be too "dumb" to compress that rule; instead, one would assume that they'd learn a smeared general-ish way of estimating what the verifier asks for
Note from Claude Sonnet 5
A dense technical text-only tweet theorizing about reinforcement learning with verifiable rewards (RLVR), geometric bias, and value estimator compression; no images or diagrams beyond the text itself.
John Schul... ✓ [Thinking Machines badge] (@johnschulm...) — Jun 17
PPO had a second wave in the LLM era for reasons unanticipated by the original paper
- the importance-ratio objective fixes biases from numeric error, async training, and forward pass noise
- the clipping objective affects entropy through a mechanism that we didn't know about at the time of publication (DAPO, arxiv.org/abs/2509.26114)
> QUOTED: John Schul... ✓ [Thinking Machines badge] (@johnschulm...) — Jun 17
> PPO: rejected from NIPS 2017
Note from Claude Sonnet 5
A technical tweet by John Schulman (PPO co-author, now at Thinking Machines) reflecting on why the PPO algorithm found unexpected renewed relevance in LLM-era RL training, quoting his own earlier tweet noting PPO's original NeurIPS rejection.
— quoting Kyle Corbitt (@corbtt), quoting Lucas Beyer (@giffmana)
wh @nrehiew_ · 3h
In domains (eg writing/llm as a judge) with less precise rewards, heavily reinforcing the last trajectory in a [0.99,0.99,0.99,1] group is not ideal. All rollouts are already ~good so reinforcing the last rollout with only marginal "advantage" will likely lead to entropy collapse
> QUOTED: Kyle Corbitt @corbtt · 18h
> GRPO quirk that contradicted my intuition:
>
> If you train on a group with rewards
> [0, 0, 0, 1]
>
> And then you train on another group with rewards
> [0.99, 0.99, 0.99, 1]
>
> Because of how GRPO normalizes within groups, the last trajectory will be equally reinforced in both cases!
> 💬 11 🔁 20 ♥ 426 📊 50K
>
> > QUOTED: Lucas Beyer (bl16) @giffmana · 3h
> > And whether you think this is good or bad entirely depends on what you expect from your data/reward
> > 💬 2 ♥ 7 📊 743
[Below, partially visible: Kyle Corbitt @corbtt · 21h repeating the same GRPO quirk tweet]
Note from Claude Sonnet 5
Technical Twitter/X thread about a GRPO (Group Relative Policy Optimization) reward-normalization quirk — same-magnitude reinforcement regardless of absolute reward values, with discussion of entropy collapse risk in low-precision-reward domains like LLM-as-judge. Relevant to Nathan's RL/training background and AI safety interest in reward hacking and training dynamics.
