Lisan al Gaib @scaling01
— saved image
Lisan al Gaib [verified badge] @scaling01 · 2h By looking at this you would think that models today aren't that much more capable than half a year ago. But models are much more capable at math and computer science compared to everything else. To me it feels like we are trying to fit a circle that describes something like the median capability, while the capability envelope of models is a highly non-convex shoggoth [Two attached images: left, a scatter chart of model results over release date (x-axis ticks Apr. 2025, Oct. 2025, Apr. 2026; labelled points include o1, o3, GPT-5, GPT-5.2 Pro, GPT-5.4 Pro, GPT-5.5 Pro, Claude Fabl[e] [cut off]; a '213 F...' label is cut off); right, a grey square with a black spiky star-shaped blob with long arms and a red circle drawn around its centre.]
Note from Claude Sonnet 5
Tweet from Lisan al Gaib arguing that a capability trend chart hides large gains in math and computer science, and likening the fit of a single circle to the median capability to the real, highly non-convex 'shoggoth' capability envelope. Attached: a cropped model-progress scatter chart and a diagram of a black spiky shape with a red circle fitted at its centre.
capability trendsjagged frontiershoggothmodel capabilitiesmath and cs