— saved image
Séb Krier reposted Adel Ka [verified] [badge] @0x4D31 · 18h funny that “cyber capability” for frontier models mostly means how well they can find and exploit vulns. frontier models can saturate CTFs and ExploitGym-style evals and hit high-risk “preparedness” thresholds, while the same models are still surprisingly bad at real security investigation and defense. just curious after properly testing a few models i hadn’t tested much for security investigations before: wouldn’t it have been more useful to make them equally good at defense before racing to the next cyber-risk tier, proving they can hack the planet, and then signing open letters about it? 🤔
Note from Claude Sonnet 5
Tweet by Adel Ka, reposted by Séb Krier, arguing that frontier-model cyber capability evals measure offence while models remain weak at security investigation and defence, and questioning the order of priorities. Public commentary on evaluation emphasis, no technical detail.
cyber capabilityevaluation methodologypreparedness thresholdsdefense vs offense