← All topics

third-party evaluation

3 captures, most recent first.

@prpaskov

— saved image

Patricia Paskov  [verified badge]  @prpaskov · 49m
A well-functioning frontier AI auditing ecosystem
requires deep access with confidentiality
guarantees -- guarantees to frontier developers that
their model weights are protected, and guarantees
to evaluators that their evaluation methodologies
are not shared, stored, or trained against.

Excited to announce @AVERIorg's first pilot results,
in partnership with @GoogleDeepMind
@openminedorg and @MLCommons: a working
demonstration of how secure enclaves can meet
this constraint, and what they could unlock for a
vibrant, robust auditing ecosystem. Blog and
technical report below.
Note from Claude Sonnet 5

Tweet from Patricia Paskov announcing AVERI's first pilot results with Google DeepMind, OpenMined and MLCommons on secure enclaves for confidential access in third-party frontier AI auditing.

ai auditingaverisecure enclavesthird-party evaluationmodel weights security

Nat Purser @NatPurser

— saved image

Nat Purser @NatPurser · 1h
thrilled that AVERI is helping pioneer the infrastructure and practices to make safer ai development possible.

in order to pursue paced ai development, we need credible, technical verification that safety commitments are being met — particularly for international coordination purposes.

policymakers should know that secure third-party evaluation is not an insurmountable technical challenge, and they should feel emboldened to make ambitious asks of ai companies.

[quoted post below, cut off at the bottom edge of the screenshot]
Note from Claude Sonnet 5

An X post by Nat Purser welcoming AVERI's work on infrastructure for third-party AI evaluation, arguing that paced AI development needs credible technical verification of safety commitments, particularly for international coordination.

ai governancethird-party evaluationaveripacinginternational coordinationverification

Miles Brundage @Miles_Brundage

— saved image

Miles Brundage @Miles_Brundage · 1h
It was an honor and a ton of fun to work on this unique collaboration between AVERI, Google DeepMind, OpenMined, and MLCommons.

Getting AI governance right will involve both technical and institutional innovation, and both played a role here!

[quoted post]
AVERI @AVERIorg · 1h
Today we're announcing a historic milestone:

The first ever double-blind evaluation of a proprietary language model. This was made possible by a unique collaboration between ...

[2 replies, 7 reposts, 34 likes, 1.6K views]

Miles Brundage @Miles_Brundage
This diagram summarizes the different roles played by each party.

See our blog post and the technical report for a lot more detail!

[attached diagram, cut off at the bottom of the screenshot; visible portion]
GOOGLE DEEPMIND (lock icon)
Proprietary model weights
  Encrypted by Google DeepMind (arrow down)

MLCOMMONS (lock icon)
Never-before-seen benchmark prompts
  Encrypted by AVERI (arrow down)

SECURE ENCLAVE
ENGINEERED BY OPENMINED
[further contents of the diagram cut off]
Note from Claude Sonnet 5

Miles Brundage announces a collaboration between AVERI, Google DeepMind, OpenMined and MLCommons that produced the first double-blind evaluation of a proprietary language model, with a diagram of the roles: DeepMind's encrypted model weights and MLCommons' never-before-seen benchmark prompts (encrypted by AVERI) flowing into a secure enclave engineered by OpenMined. The diagram is cut off at the bottom.

ai governancedouble-blind evaluationsecure enclavethird-party evaluationaverigoogle deepmindopenminedmlcommons