← All topics

capability trends

4 captures, most recent first.

Lisan al Gaib @scaling01

— saved image

Lisan al Gaib  [verified badge]  @scaling01 · 2h
By looking at this you would think that models today
aren't that much more capable than half a year ago.

But models are much more capable at math and
computer science compared to everything else.

To me it feels like we are trying to fit a circle that
describes something like the median capability,
while the capability envelope of models is a highly
non-convex shoggoth

[Two attached images: left, a scatter chart of model results over release date (x-axis ticks Apr. 2025, Oct. 2025, Apr. 2026; labelled points include o1, o3, GPT-5, GPT-5.2 Pro, GPT-5.4 Pro, GPT-5.5 Pro, Claude Fabl[e] [cut off]; a '213 F...' label is cut off); right, a grey square with a black spiky star-shaped blob with long arms and a red circle drawn around its centre.]
Note from Claude Sonnet 5

Tweet from Lisan al Gaib arguing that a capability trend chart hides large gains in math and computer science, and likening the fit of a single circle to the median capability to the real, highly non-convex 'shoggoth' capability envelope. Attached: a cropped model-progress scatter chart and a diagram of a black spiky shape with a red circle fitted at its centre.

capability trendsjagged frontiershoggothmodel capabilitiesmath and cs

davidad @davidad

— saved image

davidad @davidad · 2h
if your definition of "AGI" is "better at most tasks than per-task expert humans" (back in the day, we used to call this "ASI"), that is coming next quarter

[quoted tweet]
Bayesian @Bayesian0_0 · Aug 1
Fun fact: Across 44 benchmarks that have a "Human baseline", the human baseline BECI (a personal replication of the Epoch Capabilities Index) comes out at 166.7, which projections say will be beat by AI models around october 2026!

[embedded chart: 'Human baseline on the BECI scale (pooled human rows scored against frozen benchmark parameters; human data never enters the fit)'. Scatter plot, x-axis 'Release date' 2023-01 to 2026-07+, y-axis 'BECI' 60-160+. Legend: Models (grey dots), Model frontier (blue step line), Frontier trend (dotted line), Human baseline pooled (red horizontal band ~166.7). Annotation: 'trend crossing ~2026-10-16' where the frontier trend dotted line meets the red human baseline band.]
Note from Claude Sonnet 5

X thread: davidad comments on Bayesian's (@Bayesian0_0) chart showing AI model capability (a personal replication of the Epoch Capabilities Index, BECI) trending to cross the pooled human baseline (166.7) around October 2026, per a scatter plot of 44 benchmarks' model scores over time (2023-2026) with a fitted frontier trend line crossing the human baseline band. davidad frames this crossing as meeting an old definition of ASI (better than per-task expert humans at most tasks).

twitteragiasibenchmarkscapability trendsepoch

davidad @davidad

— quoting Ethan Mollick (@emollick)

davidad @davidad · 13h Yeah, this is what Ilya (fore)saw [Image: line chart with two trend fits over time — teal dashed "Non-reasoning fit" line, roughly flat/linear low slope, and pink "Reasoning fit" line with steeper upward slope, both fit to scatter points; axes unlabeled in visible crop] > QUOTED: Ethan Mollick @emollick · 19h > Its funny how much the whole "strawberry" thing, which turned out to be o1-preview, was dismissed as overhyped at launch when it is clear in retrospect that it was way underhyped. ...
Note from Claude Sonnet 5

A tweet arguing that OpenAI's "reasoning" model paradigm (o1-preview, codenamed "strawberry") produced a much steeper capability-growth trend line than non-reasoning models, framed as vindicating Ilya Sutskever's foresight. Relevant to Nathan's tracking of capability trajectories and takeoff-speed evidence.

twitterdavidadilya sutskevero1reasoning modelscapability trendsethan mollick

Rohin Shah @rohinmshah

— reposted by David Manheim; reply to @ben_j_todd

↻ David Manheim reposted Rohin Shah @rohinmshah · Dec 24 Replying to @ben_j_todd Both METR and ECI mostly measure things that companies optimize for. 2024 saw the rise of reasoning training for frontier models, which optimizes narrowly for some tasks (whereas pretraining provides more general improvements). So I wouldn't read much into any acceleration.
Note from Claude Sonnet 5

Rohin Shah (DeepMind alignment researcher) pushing back on interpreting METR/ECI capability-benchmark trends as evidence of an acceleration in AI progress, arguing these benchmarks measure exactly what labs already optimize for via reasoning-training, so gains there are less informative than general pretraining improvements would be. Directly relevant to Nathan's tracked "Empirical Singularity Tracking" thread (METR automation estimates, r-value debates) — adds a methodological caveat about benchmark validity that should be logged alongside existing METR/Epoch notes.

twitterai alignmentmetrbenchmarkscapability trendsreasoning trainingsingularity tracking