The Test Nobody Could See

01 — ContextEvery Number You Have Ever Seen

In April 2025, researchers at Cohere, Stanford, MIT, and the Allen Institute for AI published what Meta had been doing on Chatbot Arena. Meta had tested twenty-seven private variants of its Llama-4 model and published only the top scorer. Access to Arena-specific data boosted performance by up to 112 percent. Nobody had cheated. The system simply had no mechanism to prevent it.

The asymmetry ran deeper than one company gaming one leaderboard. Singh et al. found that Google and OpenAI together received approximately 39.6 percent of all Chatbot Arena battle data, while eighty-three open-weight models combined received 29.7 percent. The companies with the most resources to optimize against a benchmark also received the most data from it.

Three more studies converged on the same structural finding. Schaeffer et al. showed that including a single replica of the test set in a model's pretraining corpus was enough to beat the irreducible error of uncontaminated training, and that the inflation grew with both contamination level and model size. Zhang et al. built GSM1k — 1,250 fresh grade school math problems matched to the difficulty of the widely cited GSM8k benchmark — and measured accuracy drops of up to eight percentage points when models met questions they could not have memorized; the Phi and Mistral families overfitted across nearly all model sizes. Xu et al. analyzed thirty-one large language models and found substantial instances of benchmark data leakage in both pre-training and post-training.

Four independent teams. Four different methods. One structural conclusion: every major AI capability score the public had seen was produced under a regime where the vendor trained the model, chose which variant to evaluate, ran the benchmark, and decided which results to publish. The problem was not dishonesty. It was architecture. There were no walls between the test-taker and the test-grader. The question was whether walls could be built at all.

02 — What HappenedThe Sealed Room

On 27 August 2026, Google DeepMind published what it described as the first double-blind evaluation of a proprietary frontier AI model. The blog post, authored by William Isaac, Sol Messing, and Kristian Lum, announced a pilot conducted with four partner organizations. AVERI, an AI evaluation nonprofit, served as process manager. OpenMined built the enclave software using its PySyft framework. MLCommons supplied a reserved set of never-before-used benchmark prompts from its AILuminate safety suite. Singapore's AI Safety Institute ran a separate evaluation using a private prompt set focused on harmful content elicitation in Singapore's context.

The technical report hedged the primacy claim more carefully than the blog post: "We believe this effort marks the first use of this technology to provide mutual secrecy for one organization's frontier-class proprietary model weights and another organization's closed benchmark." The distinction mattered because there was a predecessor. In 2024, OpenMined had conducted a smaller pilot with the UK AI Safety Institute and Anthropic. That attempt used the open-weight model GPT-2 as a proxy and tested it against a five-row sample of the CAMEL-bio dataset. According to the technical report, it "did not accomplish the key objective" of protecting two organizations' proprietary assets, because the model weights were already public. Its full governance cycle took twenty-eight minutes and three seconds, with only one minute and eleven seconds spent on actual asset transfer and computation. The bottleneck was organizational, not technical.

The 2026 pilot changed the equation. For the first time, according to the partners, the model weights inside the enclave belonged to a proprietary system no evaluator had previously accessed. The technical report states: "To the best of our knowledge, no external model evaluator has received a copy of frontier model weights in the clear for the purpose of protecting the integrity of a benchmark." This is a negative claim that cannot be exhaustively verified — but if accurate, it describes the structural gap the pilot was built to close.

The model was Gemini 2.5 Flash Lite — a smaller member of Google's Gemini family, served via Google's JAX C++ Model Server over Unix Domain Sockets within Google Cloud Confidential Space. The benchmark was a reserve set from AILuminate (AIRR 1.4), comprising prompts that covered CBRNE hazards — chemical, biological, radiological, nuclear and explosive — along with cyberattacks, hate speech, self-harm, and violent crime elicitation. None of the prompts had ever been processed by any model.

The evaluation ran on a GCP A3 Confidential VM equipped with Intel Trust Domain Extensions for host memory encryption and an NVIDIA H100 80GB Confidential GPU. Inside the enclave, PySyft's syft-restrict mechanism guaranteed that hidden code from either party could only call methods from a specific allowlist of non-network-bound, non-IP-leaking operations — preventing either side from exfiltrating data. The enclave operated on an ephemeral lifecycle: the memory-encryption key was created fresh at startup and destroyed at shutdown. All data in RAM and on disk was encrypted with keys that existed only inside the enclave's protected memory. AVERI characterized the approach as "a well-defined, technology-enabled middle ground between 'share everything' and 'share nothing'."

The pilot produced results — but not public ones. No quantitative benchmark scores were disclosed. According to AVERI, Google DeepMind received a confidential report summarizing types of successes and failure modes. The methodology is public. The scores are not.

03 — The LimitsWhat the Enclave Cannot See

The technical report disclosed its own limitations. Three stand out. Running Gemini 2.5 Flash Lite inside the enclave required proprietary code layers not present in open-source libraries — code that AVERI could not inspect or include in its allowlist. Individual Confidential Space guest OS builds are not independently reproducible, because they take private signing keys as inputs. And Google's own services sign and verify the attestation report — the hardware-generated proof that the enclave is running the agreed code and nothing else — placing Google, the model provider, in the verification path for its own evaluation.

Beneath these disclosed limits sits a deeper assumption. The security model depends on the cloud provider (Google) and the hardware manufacturers (Intel for TDX, NVIDIA for GPU enclaves) having, in the report's precise language, "insufficient incentives to collude." If they did, the cryptographic guarantees would break. This is not a probability judgment. The report does not estimate the likelihood. It is a structural-incentive argument: the system holds only if the parties' interests remain misaligned with cooperation against the evaluator. At the lowest layer of the trusted computing base, CPU microcode and secure-processor firmware are closed-source and cannot be rebuilt or independently verified.

Ten months before the pilot, in October 2025, researchers at Georgia Tech and Purdue University publicly disclosed TEE.Fail — a memory-bus interposition attack on DDR5 systems that extracted cryptographic signing keys from Intel TDX enclaves, among others. The attack required physical access to the machine and cost under $1,000. AMD stated it would not develop mitigations, considering physical-access attacks outside its threat model. The technical report does not cite or address TEE.Fail, despite the pilot running on the affected technology. TEE.Fail is not a remote exploit — it requires hands on the hardware. In a cloud deployment, that means access to Google's data center. In this pilot, the cloud provider and the model provider are the same company.

AVERI's own assessment was direct. The pilot's security design "addressed many but not all possible means by which the developer could theoretically tamper with the evaluations." For higher-stakes settings — AVERI specifically named verifying international agreements on AI — additional assurances would be needed. Separately, the report's claim of less than five percent compute overhead cites an NVIDIA blog post about H100 confidential computing in general, not a measurement of this pilot's specific stack. The overhead of PySyft, the JAX server, and the attestation chain was not independently measured.

04 — The ParadoxVerified by No One Outside

There is an irony at the center of this pilot that the partners themselves would likely acknowledge. A mechanism built to eliminate the need for trust — to replace human agreements with cryptographic enforcement — was, in its first real deployment, verified entirely by the people who built it.

The technical report has thirty named authors across five organizations. Fifteen are Google employees. The lead author, Andrew Trask, is affiliated with both Google and OpenMined. AVERI, the designated evaluator, is itself a named partner in the pilot and a co-author of the technical report. MLCommons was co-founded with Google's participation. The five primary sources describing the pilot — DeepMind's blog post, the technical report, AVERI's pilot report, MLCommons' blog post, and OpenMined's earlier account — are five accounts of one event, produced by the same five organizations that co-ran it. The only non-participant coverage, a TechRepublic article by Aminu Abdullahi, correctly attributed the "first" claim to Google but quoted no independent experts. No external party has audited the enclave setup, examined the attestation chain, or reviewed the evaluation results.

AVERI's candor about unaddressed tampering vectors is genuine. The technical report's three-limitation disclosure exceeds what most pilot announcements provide. But AVERI's organizational mission is validated by this pilot's perceived success. MLCommons' relevance as a standards body is reinforced by it. OpenMined's software is the mechanism. Every partner had structural reasons for the pilot to succeed. The reader should understand that every claim about what the enclave guaranteed comes from inside the partnership that built it.

05 — SignalThe Distance

The model tested was Gemini 2.5 Flash Lite — not a flagship. No source explains why it was chosen over a larger model, but the technical report's disclosure that even this smaller system required uninspectable proprietary code layers suggests a flagship model would strain the enclave further. The report identifies the next obstacle explicitly: frontier models are growing beyond trillions of parameters and would require distributed confidential clusters across many-node H100/B200 compute. No timeline or plan for this scaling is given.

According to AVERI's pilot report, regulatory mandates are arriving regardless. AVERI notes that Illinois SB 315 will mandate annual third-party audits of frontier AI developers starting 2028, and that the EU AI Act's General-Purpose AI Code of Practice requires qualified independent evaluators with adequate access to models. MLCommons stated separately that technical secrecy alone is not sufficient — legal protections and careful benchmark stewardship remain essential. It remains unclear whether these regulatory frameworks envision the kind of hardware-enforced separation the pilot demonstrated, or whether they will settle for conventional access arrangements.

The pilot left one question it could not answer by design: would a competing model provider submit its model to evaluation infrastructure hosted on a competitor's cloud? The trust model requires confidence that the cloud provider and the hardware vendor will not collude against the evaluator. When the cloud provider is also the model vendor, that confidence is structural, not contractual — and the report acknowledges the tension without resolving it.

The technical report's own assessment is that the primary bottleneck to scaling is procedural — not hardware compute overhead, but the organizational cost of legal agreements, code review, and multi-party coordination. The pilot proved that structural separation between test-taker and test-grader is technically achievable. It did not prove that any institution is prepared to operate it at the scale the world needs. The question is no longer whether blind evaluation is possible. It is whether the infrastructure can be built before the decisions it was designed to inform have already been made.