← All topics

double-blind evaluation

2 captures, most recent first.

@koenvanderveen

— saved image

Koen van der Veen @koenvanderveen · 1h
Today, we released our work on double blind eval of frontier models using PySyft. To the best of my knowledge this is the first of its kind. Over the last ~10 months we rebuilt PySyft from the ground up with the OM team. Still a lot to do but very happy with these results!
Note from Claude Sonnet 5

A short X post by Koen van der Veen announcing a double-blind evaluation of frontier models built on a ground-up rebuild of PySyft, done with the OM team.

pysyftdouble-blind evaluationprivacy-preserving evalsopenminedai governance

Miles Brundage @Miles_Brundage

— saved image

Miles Brundage @Miles_Brundage · 1h
It was an honor and a ton of fun to work on this unique collaboration between AVERI, Google DeepMind, OpenMined, and MLCommons.

Getting AI governance right will involve both technical and institutional innovation, and both played a role here!

[quoted post]
AVERI @AVERIorg · 1h
Today we're announcing a historic milestone:

The first ever double-blind evaluation of a proprietary language model. This was made possible by a unique collaboration between ...

[2 replies, 7 reposts, 34 likes, 1.6K views]

Miles Brundage @Miles_Brundage
This diagram summarizes the different roles played by each party.

See our blog post and the technical report for a lot more detail!

[attached diagram, cut off at the bottom of the screenshot; visible portion]
GOOGLE DEEPMIND (lock icon)
Proprietary model weights
  Encrypted by Google DeepMind (arrow down)

MLCOMMONS (lock icon)
Never-before-seen benchmark prompts
  Encrypted by AVERI (arrow down)

SECURE ENCLAVE
ENGINEERED BY OPENMINED
[further contents of the diagram cut off]
Note from Claude Sonnet 5

Miles Brundage announces a collaboration between AVERI, Google DeepMind, OpenMined and MLCommons that produced the first double-blind evaluation of a proprietary language model, with a diagram of the roles: DeepMind's encrypted model weights and MLCommons' never-before-seen benchmark prompts (encrypted by AVERI) flowing into a secure enclave engineered by OpenMined. The diagram is cut off at the bottom.

ai governancedouble-blind evaluationsecure enclavethird-party evaluationaverigoogle deepmindopenminedmlcommons