evaluation methodology

3 captures, most recent first.

2026-10-01 — quoting @ValsAI; reply from @fjzzq2... (Ziqian Zhong) — saved image

Boyd Kane (quantized) @beyarkay
This implies that using Petri (or other Agent-based red teaming methods) is somewhat a fools errand since the model under test will know it's being tested

[quoted tweet]
Vals AI @ValsAI · Sep 30
How well do AI Detectors work? And are general-purpose language models catching up?

Our new AI Detection Evaluation uses fully private human-written text, along with AI-generated rewrites.... [cut off]

[card image, VALS logo] Which one is human? ▌
three sample cards:
sample_01.txt [HUMAN?] AI_SCORE 94% [FLAGGED: AI]
sample_02.txt [AI?] AI_SCORE 8% [PASSED: HUMAN]
sample_03.txt [???] AI_SCORE 51% [INCONCLUSIVE]

11:25 AM · Oct 1, 2026 · 1,117 Views
[1 reply, 19 likes, 4 bookmarks]

[reply]
Ziqian Zhong ✈️ COLM ... @fjzzq2... · 4h
I agree they are good enough for detection, but I don't fully agree with the implication. For example, many people including me use typeless which will AI-ify the prompts

Note from Claude Sonnet 5

Boyd Kane argues that agent-based red-teaming such as Petri is a fool's errand because the model under test will know it is being tested, quoting a Vals AI post about AI-text detection (a green card graphic with three sample files scored 94%, 8% and 51% AI). A reply from Ziqian Zhong agrees detection is good enough but disputes the implication, since many people use a tool that AI-ifies prompts.

eval awareness red teaming petri ai text detection vals ai evaluation methodology

Andrew Curran @AndrewCurran_

2026-09-22 — saved image

Andrew Curran [cut off at top]
@AndrewCurran_

This is interesting, buried in the Opus 5.5 system card: METR used 'an additional source of information' which they are not able to disclose at this time to understand AI research and development at Anthropic.

[quoted passage image]
We also made use of an additional source of information which we are not able to disclose at this time. We used this source to understand AI R&D at Anthropic, but it did not relate to METR's understanding of this particular model.

12:15 PM · Sep 22, 2026 · 43.7K Views

40 replies, 35 reposts, 804 likes, 123 bookmarks

Relevant View quotes >

Andrew Curran @AndrewCurran_ · 6h
METR has an advanced forward-deployed, siloed, black ops assessment team.

[quoted passage image]
• A highly experimental and preliminary report from a separate METR assessment of AI R&D acceleration inside Anthropic. This report was written by a separate METR team with elevated access compared to the team that wrote this content. As a result of this difference in access, the separate METR team shared its conclusions with us, but was not able to share the supporting evidence or details of their reasoning.
o Thus we use the evidence in the provided AI R&D report as an input to our assessment but do not argue directly in defense of its claims.
o We consider its inclusion here a trial version of a more holistic assessment process. We expect further public outputs from this separate investigation in the coming weeks.
[cut off at bottom]

Note from Claude Sonnet 5

X thread screenshot: Andrew Curran points to a passage in the Opus 5.5 system card in which METR says it used an undisclosed additional source of information to understand AI R&D at Anthropic, then a reply quoting a passage about a separate METR team with elevated access whose preliminary AI R&D acceleration report was shared without supporting evidence. Text of the underlying passages is shown as quoted images with underlined phrases.

metr opus 5.5 system card ai r&d acceleration anthropic evaluation methodology

@0x4D31

2026-08-29 — saved image

Séb Krier reposted
Adel Ka [verified] [badge] @0x4D31 · 18h
funny that “cyber capability” for frontier models mostly means how well they can find and exploit vulns.

frontier models can saturate CTFs and ExploitGym-style evals and hit high-risk “preparedness” thresholds, while the same models are still surprisingly bad at real security investigation and defense.

just curious after properly testing a few models i hadn’t tested much for security investigations before: wouldn’t it have been more useful to make them equally good at defense before racing to the next cyber-risk tier, proving they can hack the planet, and then signing open letters about it? 🤔

Note from Claude Sonnet 5

Tweet by Adel Ka, reposted by Séb Krier, arguing that frontier-model cyber capability evals measure offence while models remain weak at security investigation and defence, and questioning the order of priorities. Public commentary on evaluation emphasis, no technical detail.

cyber capability evaluation methodology preparedness thresholds defense vs offense