Google DeepMind announced a pilot double-blind evaluation framework on August 27 for a proprietary frontier-class model.

The model used in the live deployment was Gemini 2.5 Flash Lite.

The core arrangement is straightforward.

The evaluator keeps its benchmark prompts secret. Google keeps its model weights and sensitive inference code secret. Both assets only meet inside a protected computing environment.

Benchmark contamination is becoming a credibility problem

AI labs routinely compare models using sets of questions designed to measure reasoning, safety or specialized capabilities.

Once a benchmark becomes widely circulated, however, its questions or close variants can leak into pretraining or post-training data.

A model can then achieve a stronger score partly because it has encountered the test before.

DeepMind describes this as benchmark contamination.

The accompanying technical report cites research finding evidence of leakage across many models and work showing that contamination can inflate measured performance.

External evaluation traditionally forced somebody to give up a secret

An outside evaluator with confidential questions previously faced an uncomfortable choice.

It could send the prompts to the model provider, often through an API, preserving the lab's intellectual property while potentially exposing the benchmark.

Or it could request the model weights and run the test independently.

That protects the questions but requires the developer to hand over one of its most valuable and potentially sensitive assets.

Frontier weights represent substantial investment and can carry dual-use security concerns, making labs reluctant to distribute them.

The two secrets meet inside confidential hardware

The pilot uses Google Cloud Confidential Space with a confidential virtual machine and an NVIDIA H100 80GB Confidential GPU.

Host memory and GPU memory are protected with hardware-level encryption.

Gemini's weights and the evaluator's prompts are streamed into that environment over encrypted connections.

The benchmark runs inside the enclave.

Only the outputs permitted by the agreed evaluation policy leave it.

The environment can then be decommissioned, limiting persistent state.

Remote attestation checks what is actually running

Encrypted memory alone would not solve the problem.

The participants also need confidence that they are sending secrets to the software stack they agreed to use.

That is where remote attestation enters the design.

Layers of the trusted computing base are cryptographically measured as they load, and signed attestation reports allow participants to compare those measurements against expected values.

Data is released only after the relevant checks pass.

For parts of the software stack, reproducible builds allow outside parties to independently rebuild code and verify that the resulting measurements match.

PySyft coordinates the exchange without revealing the assets

OpenMined's PySyft provides the orchestration layer.

Each organization submits private code and assets into the enclave, and computation does not begin until both sides approve the relevant execution.

The framework can also permit partially redacted code while restricting hidden portions to allowlisted operations that cannot make unauthorized network calls.

That matters because otherwise a hidden inference or evaluation function could potentially exfiltrate the other side's secret.

The benchmark prompts were intentionally held in reserve

AVERI and MLCommons tested the system using a reserved portion of the AILuminate benchmark.

The technical report says those prompts had never previously been processed by any model.

They covered safety domains including CBRNE hazards, cyberattacks, hate speech, self-harm and violent-crime elicitation.

AVERI handled encryption, decryption and evaluation of the relevant outputs.

Singapore's AI Safety Institute separately tested Gemini 2.5 Flash Lite against a private prompt set focused on harmful-content elicitation in the Singapore context.

DeepMind's announcement does not publish performance scores from either evaluation.

The result being demonstrated is the evaluation mechanism itself, not a new Gemini benchmark victory.

The method prevents one form of future leakage, not every kind of contamination

Double-blind execution primarily protects confidential prompts from leaking into later model development through the evaluation process itself.

It cannot prove that an already public benchmark was never seen by a model in the past.

Nor can it guarantee that a model has never seen conceptually similar questions.

Strong evaluation therefore still depends on genuinely fresh prompts and careful stewardship of the benchmark.

MLCommons explicitly emphasizes that secrecy by itself is not sufficient.

This is not a completely trustless system

The technical report is unusually explicit about the remaining trust assumptions.

Confidential computing depends partly on hardware vendors and the integrity of their roots of trust.

The lowest layers, including some CPU microcode and secure-processor firmware, cannot be independently rebuilt from public source.

The pilot also relies on Google services for signing and verifying parts of the attestation process.

The authors acknowledge that this leaves Google inside the verification path and increases the amount of trust still placed in it.

Not all of Gemini's proprietary implementation was inspectable

Another limitation came from the model runtime itself.

The researchers wanted Gemini 2.5 Flash Lite to run entirely through layers whose method implementations could be inspected or allowlisted.

They concluded that eliminating all proprietary method implementations was too large an engineering task for this pilot.

That means some code could not be fully inspected or restricted through the intended allowlisting system.

AVERI was informed and accepted the setup.

The pilot therefore reduces the trust required between organizations rather than eliminating it absolutely.

Compute overhead is already smaller than the organizational overhead

The report says the enclave architecture operates with less than 5% compute overhead in the referenced setup.

The larger bottleneck is procedural.

Legal agreements, security reviews, code inspection and coordination between several organizations take substantial human effort before the evaluation can run.

The long-term ambition is to standardize the attestation pipeline until most of that complexity disappears behind a simple trust signal.

The report compares the desired user experience to the familiar HTTPS lock icon.

Frontier-scale evaluation will require confidential clusters

Gemini 2.5 Flash Lite could be demonstrated on a single confidential H100.

Much larger frontier models may require many accelerators.

The researchers identify distributed confidential H100 and B200 clusters with encrypted interconnects as a next milestone.

They will also need mechanisms that protect details of the model architecture while computation is spread across multiple nodes.

The pilot therefore demonstrates the mechanism at one useful scale rather than proving that every future model can already be audited this way.

The broader goal is to make benchmark gaming harder

Benchmark tables increasingly carry commercial, regulatory and safety significance.

That makes the incentive to optimize specifically against known tests stronger.

A credible independent evaluation ideally needs a developer that cannot inspect the exam and an evaluator that does not need to receive unrestricted access to the model.

DeepMind, OpenMined, AVERI, MLCommons and Singapore AISI have now demonstrated a technical route toward that arrangement.

It remains operationally heavy, imperfect and dependent on hardware and cloud trust.

But the fundamental change is real: the model can finally enter the exam room without its owner being allowed to look at the paper first.