ai benchmarking

Epoch AI @EpochAIResearch · 53m
We've updated the MirrorCode leaderboard with results for Claude Fable 5 and GPT-5.6 Sol.

Claude Fable 5 leads with a 64% solve rate, followed by GPT-5.6 Sol at 20%.

[Chart: "AI models can autonomously complete some large software projects" — Overall score on MirrorCode (ML, +Private, 2L). Even when they fail to reimplement targets, AIs typically make substantial progress, passing 90% or more of tests. Bar chart, solve@100% rate (per-target mean), whiskers ±1 SE: Claude Fable 5 64%, GPT-5.6 Sol 20%, GPT-5.4 16%, GPT-5.5 10%. EPOCH AI | CC-BY, epoch.ai]

13 replies, 32 reposts, 213 likes, 8.3K views

Epoch AI @EpochAIResearch · 53m
MirrorCode tests whether AI agents can reimplement software projects from scratch. A solve requires passing 100% of visible and hidden tests.

Claude Fable is the first model we evaluated to solve the C preprocessor and Pkl tasks in at least one run.

[Chart: "The hardest MirrorCode targets remain unsolved" — Per-target solve rates for MirrorCode (ML, +Private, 2L), for full reimplementation (100% of tests passed). Table by model x target:
Claude Fable 5 (avg 64% ±10pp): tssql 100, private_M 100, texmacros 100, wren_cli 100, bib2json 100, nonogrid 83, brotild 83, gotree 0, sed 67, mailauth 83, giac_subset 0, cprepro 83, pkl 25, private_L 33, ruff 0.
GPT-5.6 Sol (avg 20% ±9pp): tssql 100, private_M 50, texmacros 0, wren_cli 33, bib2json 100, nonogrid 0, brotild 17, gotree 0, sed 0, mailauth 0, giac_subset 0, cprepro 0, pkl 0, private_L 0, ruff 0.
GPT-5.4 (avg 16% ±8pp): tssql 100, private_M 50, texmacros 0, wren_cli 0, bib2json 50, nonogrid 0, brotild 33, gotree 0, sed 0, mailauth 0, giac_subset 0, cprepro 0, pkl 0, private_L 0, ruff 0.
GPT-5.5 (avg 10% ±6pp): tssql 83, private_M 0, texmacros 0, wren_cli 0, bib2json 33, nonogrid 0, brotild 33, gotree 0, sed 0, mailauth 0, giac_subset 0, cprepro 0, pkl 0, private_L 0, ruff 0.
Footnote: Each model was run three times in each of the two implementation languages for a target (six planned attempts per target). Attempts with infrastructure errors and no valid final evaluation are excluded; scored sample counts are shown in tooltips. Overall scores average the 15 per-target solve rates, weighting each target equally. ± values show 1 SE across target programs. EPOCH AI | CC-BY, epoch.ai]

ai benchmarkingclaude fablesoftware engineeringepoch ai

Morgan @morganlinton · 2h
I am starting to analyze different LLM benchmarks, to see how well they represent real work engineering teams will do with models.

With new models coming out daily at this point, two things have become clear to me:

1. There's a decent amount of benchmaxxing going on. Lots of benchmarks are now in the training data for these models.

2. A lot of benchmarks have tasks that don't represent any real work an engineer would do with a model, i.e. math puzzles, etc.

What I think is so interesting is that new models come out, they share the benchmark results, then news sources cover it without learning about what the benchmark actually tested, or if it was a fair benchmark to begin with.

At the end of the day, for me, as someone that leads an engineering team, I need to know how new models perform on real engineering tasks, because that is what my engineering team uses them for.

The first benchmark I'm analyzing is TerminalBench, since this seems to be one of the most widely shared benchmarks.

More to come.

It's time to start understanding benchmarks vs. just celebrating higher number.

ai benchmarkingllm evaluationsoftware engineering

Our second official benchmarking post is out! The Euler equations may *seem* easy to solve using finite volume methods, but all frontier models (including GPT-5.6 Sol, Fable 5, and Kimi K3) consistently introduce both subtle and unsubtle errors, including numerical oscillations, thermodynamic inconsistencies, and incorrect orders of accuracy. That is, if the code even works at all. Mathematical misformalizations abound, and token costs can easily hit tens of dollars per attempt.

Only Lanyon's neurosymbolic architecture is consistently able to produce robust solvers with end-to-end proofs of correctness, and it does so with costs that are >100x lower. Post below 👇

ai benchmarkingneurosymbolic ainumerical methodsllm evaluation