— saved image
Chris @ChrisGPT · 21h ValsAI made SRE benchmark less than a month ago. The benchmark measures can a model reverse engineer software from binaries Yesterday GPT saturated it. [embedded chart: "SRE-Bench" - x-axis "Output tokens" 0K to 1,000K; y-axis "Success rate" 0% to 100%; legend: Astra: Pass@1, Astra: Pass@4, GPT-5.6 Sol: Pass@1, GPT-5.6 Sol: Pass@4; labelled end points 88.0% (Astra Pass@1), 99.2% (Astra Pass@4), 55.9% (GPT-5.6 Sol Pass@1), 68.7% (GPT-5.6 Sol Pass@4)]
Note from Claude Sonnet 5
Tweet saying ValsAI's SRE benchmark (reverse engineering software from binaries) was saturated by GPT within a month of release. Embedded line chart of success rate vs output tokens: Astra curves climb steeply to 88.0% (Pass@1) and 99.2% (Pass@4) within about 300K tokens, while GPT-5.6 Sol reaches 55.9% (Pass@1) and 68.7% (Pass@4) at roughly 250K and 970K tokens.
sre-benchvalsaigpt-6 astrabenchmark saturationreverse engineering