4 captures, most recent first.
Michiel Bakker [verified]
@bakkermichiel
Super important work from @AVERIorg @openminedorg , @GoogleDeepMind and @MLCommons that deserves much more attention, I think! It's the first double-blind eval of a closed-weight model.
A quick post on why I think this is important.
Normally, there are only two options: 1) an evaluator shares their evals with a lab, meaning the lab could game those evals or 2) the lab shares its model weights with the evaluator, creating huge IP and security risks.
In this work, both the evals and model weights remain private inside a "secure enclave": The lab (GDM in this case) can't see the eval prompts, and the evaluator can't see the model weights. Cryptographic attestation verifies that the agreed model and evals were actually used.
Note from Claude Sonnet 5
Michiel Bakker tweet describing a joint AVERI / OpenMined / Google DeepMind / MLCommons double-blind evaluation of a closed-weight model, run inside a secure enclave with cryptographic attestation so the lab cannot see eval prompts and the evaluator cannot see weights. Text is cut at the bottom edge of the screenshot (continues beyond the capture).
ai evaluationsecure enclavedouble-blind evalclosed-weight modelscryptographic attestationgoogle deepmindopenmined
I'm personally very bullish on how important technology like this could become. In the current situation, people inside a frontier lab could encode "secret loyalties" or other hidden behaviors into a model . For example, behaviors that subtly favor its developer or behaving well only when it recognizes that it is being evaluated. This may seem far-fetched today but, as labs automate more of their own workforce, it could be done by a handful of people and become much harder for the outside world to notice. Secret evals like these make such behavior harder to hide.
Over time, this kind of infrastructure could support much more comprehensive oversight across the training and deployment process. Auditing agents with access to all the training and deployment logs, could function as probes for hidden objectives. This could move us from occasional post hoc evaluations like this towards more continuous oversight, without requiring labs to expose their weights, data, or IP.
Congrats @iamtrask , @SolomonMg, @KLdivergence, @Miles_Brundage and everyone involved!
[quoted post]
AVERI [verified] @AVERIorg · 3h
Today we're announcing a historic milestone:
The first ever double-blind evaluation of a proprietary language model. This was made possible by a unique collaboration between AVERI, @GoogleDeepMind, ... [cut off]
Last edited 9:04 AM · Aug 27, 2026 · 1,264 Views
Note from Claude Sonnet 5
Second screenshot of the same Michiel Bakker thread as the previous capture: argues secure-enclave evals could expose secret loyalties or evaluation-aware hidden behaviors inserted by a few lab insiders, and could grow into continuous oversight via auditing agents without exposing weights or IP; quotes AVERI's announcement of the first double-blind evaluation of a proprietary language model. A floating compose button overlaps the quoted post.
ai evaluationsecret loyaltieshidden objectivessecure enclavecontinuous oversightaveriai auditing
@celestepoasts (Celeste) — 48m
I am a bit worried that the world will get so weird, and majority of pretraining data so unrepresentative that models suspect new real world data (through web searches, etc...) is an evaluation
> QUOTED: @robertskmiles (Rob Miles) — 2h
> Whenever I talk to AI recently the chain of thought is like
>
> "Tactfully considering plausibility of user's far fetched scenario"...
Note from Claude Sonnet 5
Quote-tweet exchange about AI models' chain-of-thought increasingly flagging real-world inputs as implausible/eval-like, with Celeste extending Rob Miles's observation into a worry that models may start mistaking genuine novel real-world events for synthetic evaluation scenarios. No engagement counts visible in frame.
twitterchain of thoughtai evaluationai safety discourse
Noam Brown ✓ @polynoamial
There's a lot of talk of LLMs "saturating all the evals" but there's plenty of evals people could make where LLMs would do poorly:
-Beat a Zelda game
-Make a profit in a prediction market
-Write a stand-up set that's original and funny
I'm bullish on AI, but we're far from done.
9:55 AM · Feb 6, 2025 · 2,440 Views
12 replies, 9 reposts, 128 likes, 12 bookmarks
Noam Brown ✓ @polynoamial · 4m
A lot of grad students have asked me how they can best contribute to the field of AI when they are short on GPUs and making better evals is one thing I consistently point to.
[reply, 28 likes]
Sir Mr Meow ... ✓ @SirMrMeow... · 3m
[reply thread continues, cut off]
Note from Claude Sonnet 5
Noam Brown (OpenAI researcher) argues LLM eval saturation claims are overstated, listing tasks LLMs still fail at; follow-up tweet on grad students contributing via better evals. Relevant to Nathan's interest in AI capability evaluation and benchmarking.
ai evaluationbenchmarksllm capabilitiesnoam brownai progress
