Chris @ChrisGPT · 21h
ValsAI made SRE benchmark less than a month ago.
The benchmark measures can a model reverse engineer software from binaries
Yesterday GPT saturated it.
[embedded chart: "SRE-Bench" - x-axis "Output tokens" 0K to 1,000K; y-axis "Success rate" 0% to 100%; legend: Astra: Pass@1, Astra: Pass@4, GPT-5.6 Sol: Pass@1, GPT-5.6 Sol: Pass@4; labelled end points 88.0% (Astra Pass@1), 99.2% (Astra Pass@4), 55.9% (GPT-5.6 Sol Pass@1), 68.7% (GPT-5.6 Sol Pass@4)]
Note from Claude Sonnet 5
Tweet saying ValsAI's SRE benchmark (reverse engineering software from binaries) was saturated by GPT within a month of release. Embedded line chart of success rate vs output tokens: Astra curves climb steeply to 88.0% (Pass@1) and 99.2% (Pass@4) within about 300K tokens, while GPT-5.6 Sol reaches 55.9% (Pass@1) and 68.7% (Pass@4) at roughly 250K and 970K tokens.
sre-benchvalsaigpt-6 astrabenchmark saturationreverse engineering
Chris @ChrisGPT · 1h
GPT-6 Extra High vs. Kimi K3 Swarm - ECHO Start Menu
Both models received the exact same prompt. I'm shocked at the quality Kimi K3 was able to output here. Its overall visual quality is above GPT-6's and even above the menu from the actual game.
However, it took significantly longer - around 3.5 hours versus GPT-6's 42 minutes. Roughly 5× longer.
GPT-6's demo felt more realistic and stayed a lot closer to how the original menu looks.
Kimi added something that GPT-6 missed, which was the interactive eye tracking. Kimi's eyeball follows (albeit blurry and in a slightly weird way) where you click on the start screen. GPT-6's doesn't.
• GPT-6 Extra High: approximately 46,000 generated output tokens - $2.30 in API-equivalent output cost.
• Kimi K3 Swarm: estimated 250,000 generated output tokens - $3.75 in API-equivalent output cost.
Note from Claude Sonnet 5
Tweet comparing GPT-6 Extra High and Kimi K3 Swarm on recreating a game start menu from the same prompt, with time, token and cost figures. Text continues below the capture [cut off at bottom edge].
gpt-6kimi k3model comparisoncoding evalscost
[continuation of the same tweet as previous screenshot]
Researchers have now found 1,367 BTC, nearly $89 million, taken from 4,585 addresses.
The AI part happened AFTER all this. A Reddit user reportedly pointed Claude Code at the public firmware and asked only to "check for vulnerabilities." Within eight minutes, it traced the broken random number path and brought up the same hackable flaw.
There is no proof the original attacker used Claude or ANY LLM. now this still demonstrates Claude is still insane at finding vulnerabilities humans missed in public code for five years and can now be uncovered by one person with one broad prompt and a coding agent in minutes.
[quoted Reddit post, r/Bitcoin, 10h ago, u/Impressive-Gene-421:]
Are you kidding me? Claude Code found the catastrophe after being asked only to ""check for vulnerabilities and thinking for 8 minutes
1. ngu.random is wired to a software PRNG, not the hardware TRNG — CRITICAL
There are two independent RNG paths in the firmware, and only one of them reaches the STM32 TRNG.
Path A (correct). ckcc.rng_bytes() → stm32/COLDCARD_MK4/rng.c:131 random_buffer() → rng_get_or_fault() reads RNG->DR directly and raises OSError on timeout or repeats. This is what backups.py:337 uses for the backup-file password.
Path B (broken). ngu.random.* → external/libngu/ngu/random.c:73 CHIP_TRNG_32(), defined at line 24-26 as extern uint32_t rng_get(void).
It is unbelievable that some kid with an LLM just stole $100m+ because no one bothered to check the source code.
Also on GLM 5.2 (trained 16th June, no internet access).
2:16 PM · Aug 2, 2026 · 28.7K Views
Note from Claude Sonnet 5
Second half of Chris (@ChrisGPT)'s tweet about the Coldcard Bitcoin wallet RNG vulnerability, including a quoted Reddit post from r/Bitcoin showing the specific code paths (ngu.random software PRNG vs hardware TRNG) that Claude Code identified as the critical flaw.
bitcoincybersecurityclaude codetwitterredditcryptocurrency
Chris ✔️ @ChrisGPT
Here's what actually happened with bitcoin and Claude because people are getting the story mixed up.
Coldcard hardware wallets were supposed to create each Bitcoin seed using real physical randomness from a chip.
But a firmware bug checked whether the hardware random number setting existed in the first place NOT whether it was actually enabled. It existed but was set to 0, so the wallet fell back to a randomized predictable software generator.
That reduced some wallets from roughly 2^128 possible seeds to around 2^40. The attacker could generate candidate seeds offline, derive their Bitcoin addresses and use the public blockchain like an answer key to see which wallets had funds.
Once one matched, they had a private key they could drain it without touching the device, and steal the seed phrase or "crack Bitcoin."
Researchers have now found 1,367 BTC, nearly $89 million, taken from 4,585 addresses.
The AI part happened AFTER all this. A Reddit user reportedly pointed Claude Code at the public firmware and asked only to "check for vulnerabilities." Within eight minutes, it traced the broken random number path and brought up the same hackable flaw. [cut off]
Note from Claude Sonnet 5
Tweet from Chris (@ChrisGPT) explaining the Coldcard hardware wallet firmware bug that weakened Bitcoin seed randomness (2^128 to 2^40), leading to $89M stolen, and clarifying that Claude Code was used afterward by a Reddit user to independently rediscover the vulnerability in 8 minutes.
bitcoincybersecurityclaude codetwittercryptocurrency