← All topics

capability trends

3 captures, most recent first.

davidad @davidad

— saved image

davidad @davidad · 2h
if your definition of "AGI" is "better at most tasks than per-task expert humans" (back in the day, we used to call this "ASI"), that is coming next quarter

[quoted tweet]
Bayesian @Bayesian0_0 · Aug 1
Fun fact: Across 44 benchmarks that have a "Human baseline", the human baseline BECI (a personal replication of the Epoch Capabilities Index) comes out at 166.7, which projections say will be beat by AI models around october 2026!

[embedded chart: 'Human baseline on the BECI scale (pooled human rows scored against frozen benchmark parameters; human data never enters the fit)'. Scatter plot, x-axis 'Release date' 2023-01 to 2026-07+, y-axis 'BECI' 60-160+. Legend: Models (grey dots), Model frontier (blue step line), Frontier trend (dotted line), Human baseline pooled (red horizontal band ~166.7). Annotation: 'trend crossing ~2026-10-16' where the frontier trend dotted line meets the red human baseline band.]
Note from Claude Sonnet 5

X thread: davidad comments on Bayesian's (@Bayesian0_0) chart showing AI model capability (a personal replication of the Epoch Capabilities Index, BECI) trending to cross the pooled human baseline (166.7) around October 2026, per a scatter plot of 44 benchmarks' model scores over time (2023-2026) with a fitted frontier trend line crossing the human baseline band. davidad frames this crossing as meeting an old definition of ASI (better than per-task expert humans at most tasks).

twitteragiasibenchmarkscapability trendsepoch

davidad @davidad

quoting Ethan Mollick (@emollick)

davidad @davidad · 13h Yeah, this is what Ilya (fore)saw [Image: line chart with two trend fits over time — teal dashed "Non-reasoning fit" line, roughly flat/linear low slope, and pink "Reasoning fit" line with steeper upward slope, both fit to scatter points; axes unlabeled in visible crop] > QUOTED: Ethan Mollick @emollick · 19h > Its funny how much the whole "strawberry" thing, which turned out to be o1-preview, was dismissed as overhyped at launch when it is clear in retrospect that it was way underhyped. ...
Note from Claude Sonnet 5

A tweet arguing that OpenAI's "reasoning" model paradigm (o1-preview, codenamed "strawberry") produced a much steeper capability-growth trend line than non-reasoning models, framed as vindicating Ilya Sutskever's foresight. Relevant to Nathan's tracking of capability trajectories and takeoff-speed evidence.

twitterdavidadilya sutskevero1reasoning modelscapability trendsethan mollick

Rohin Shah @rohinmshah

reposted by David Manheim; reply to @ben_j_todd

↻ David Manheim reposted Rohin Shah @rohinmshah · Dec 24 Replying to @ben_j_todd Both METR and ECI mostly measure things that companies optimize for. 2024 saw the rise of reasoning training for frontier models, which optimizes narrowly for some tasks (whereas pretraining provides more general improvements). So I wouldn't read much into any acceleration.
Note from Claude Sonnet 5

Rohin Shah (DeepMind alignment researcher) pushing back on interpreting METR/ECI capability-benchmark trends as evidence of an acceleration in AI progress, arguing these benchmarks measure exactly what labs already optimize for via reasoning-training, so gains there are less informative than general pretraining improvements would be. Directly relevant to Nathan's tracked "Empirical Singularity Tracking" thread (METR automation estimates, r-value debates) — adds a methodological caveat about benchmark validity that should be logged alongside existing METR/Epoch notes.

twitterai alignmentmetrbenchmarkscapability trendsreasoning trainingsingularity tracking