LLM Olympiad: Why Model Evaluation Needs a Sealed Exam
Jan Christian Blaise Cruz and Alham Fikri Aji · 2026
Read LLM Olympiad: Why Model Evaluation Needs a Sealed Exam by Jan Christian Blaise Cruz and Alham Fikri Aji (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-23292.txt.
Introduction
Cruz and Aji argue that language model scores need a sealed-exam style of testing. Leaderboard numbers can reflect test chasing, hidden scoring choices, or leaked test items. Closed benchmarks help but hide how results were produced. The pair propose an Olympiad-style event with problems sealed until test day. Entries freeze in advance and all run through one standard harness. After scoring, the full task set and code are released for audit. They say the design makes strong scores harder to fake and easier to trust.
Catalog record
| CREATOR | Jan Christian Blaise Cruz and Alham Fikri Aji |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48612 |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.23292
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LLM Olympiad: Why Model Evaluation Needs a Sealed Exam Jan Christian Blaise Cruz and Alham Fikri Aji Abstract Benchmarks and leaderboards are how NLP most often communicates progress, but in the LLM era they are increasingly easy to misread. Scores can reflect benchmark-chasing, hidden evaluation choices, or accidental exposure to test content—not just broad capability. Closed benchmarks delay some of these issues, but reduce transparency and make it harder for the community to learn from results. We argue for a complementary practice: an Olympiad-style evaluation event where problems are sealed until evaluation, submissions are frozen in advance, and all entries run through one standardized harness. After scoring, the full task set and evaluation code are released so results can be reproduced and audited. This design aims to make strong performance harder to “manufacture” and easier to trust. 1 Introduction Premise. Benchmarks and leaderboards shape how NLP communicates progress. Yet benchmark outcomes can be fragile : small choices in what is measured and how results are aggregated can change which methods look best ( 5 ) . Figure 1 previews the three evaluation formats we contrast in this paper. Two LLM-era forces amplify this fragility. First, evaluation can become “leaky”: models trained on web-scale corpora may have already seen test items (or close variants), complicating generalization claims ( 31 ) . Second, evaluation protocols have many degrees of freedom. In few-shot settings, even demonstration order can swing performance ( 15 ) , and “few-shot ability” can be overstated when prompts and settings are tuned in ways that do not reflect real no-dev evaluation ( 19 ) . Figure 1: Comparison of evaluation formats across three dimensions. Open benchmarks are fully transparent but expose tasks to optimization and leakage. Shared tasks hide test labels but evaluate against a known target under non-standardized protocols. The proposed Olympiad format seals tasks until evaluation, enforces a standardized harness, and releases all artifacts afterward — combining the strengths of existing formats while mitigating their core weaknesses. Intuition. We take inspiration from Olympiad-style competitions: participants prepare broadly, but the exact problems are revealed only at contest time. In the International Olympiad such as IMO 1 1 1 https://www.imo-official.org , IOI 2 2 2 https://ioinformatics.org , and other subjects, organizers prepare the problem set in a confidential manner, keeping it sealed until the Olympiad concludes ( 8 ) . The analogy for LLM evaluation is straightforward: if tasks are known in advance, evaluation naturally becomes a target; if tasks are sealed until scoring, performance is harder to manufacture and easier to interpret. Secrecy here is temporary : tasks and evaluation artifacts are released afterward so the community can audit and reuse them. Our position. We take the position that NLP should complement existing benchmarks with a sealed, Olympiad-style evaluation protocol—an “LLM Olympiad” where (i) evaluation problems are sealed until the event, (ii) submissions are frozen and evaluated centrally under a standardized harness, and (iii) all artifacts are released afterward for community audit. The result is an evaluation where strong performance cannot be manufactured through benchmark-specific optimization or test-set exposure, and where every score comes with a public trail of evidence—tasks, code, and logs—that the community can verify. The key shift is that the benchmark is not only “the dataset,” but the procedure : how tasks are solicited, sealed, executed, scored, and then released. This stance is consistent with prior calls to treat evaluation as infrastructure, including MLPerf’s emphasis on shared rules and compliance ( 22 ) and HELM’s push for transparent, standardized reporting ( 13 ) . Roadmap. We summarize the limits of today’s benchmarking toolbox in Section 2 , define the Olympiad-style protocol in Section 3 , describe operational mechanics in Section 4 , and outline a threat model with mitigations and honest limits in Section 5 . 2 The Limits of Today’s Benchmarking Toolbox Open benchmarks, closed benchmarks, and shared tasks each serve important purposes, and the field is better because they exist. They also optimize different tradeoffs: open benchmarks prioritize transparency and reuse; closed benchmarks prioritize test secrecy; shared tasks prioritize centralized scoring under a common script. Our position is not that these formats are misguided, but that in the LLM era they leave gaps for high-assurance claims—especially around contamination risk, comparability, and incentives. 2.1 Open benchmarks Open benchmarks are transparent and easy to reproduce, which is a major reason suites like GLUE ( 28 ) , MMLU ( 7 ) , GPQA 23 , and other benchmarks became influential. However, once a benchmark becomes a widely watched target, it invites heavy iteration on prompts and evaluation settings, and small measurement choices can affect perceived winners ( 5 ) . For LLMs, this is often amplified by prompt-based evaluation: seemingly minor choices (e.g., demonstration selection or ordering) can shift outcomes, making “the benchmark score” less stable than it appears ( 15 ; 19 ) . Open benchmarks also face rising contamination risk at web scale ( 31 ) , motivating auditing efforts ( 12 ) and deduplication practices ( 11 ) . Finally, benchmarks can saturate, prompting cycles like GLUE \rightarrow SuperGLUE ( 27 ) . 2.2 Closed benchmarks Closed benchmarks can delay direct test-set overfitting, but trade away community auditability and accessibility, which also affect adaptability by the community. Without access to tasks and evaluation details, it is harder to interpret why a score changed, or what it teaches the field. This trust bottleneck motivates calls for transparent, standardized reporting in evaluation ( 13 ) . Closed benchmarks also do not remove incentives to iterate privately and selectively disclose favorable results ( 24 ) . Some benchmarks combine both open and closed data access by utilizing two different test sets: one for public release and one for private use 4 ; 3 . This approach balances accessibility with privacy; developers can easily measure progress on the public set, then later request an evaluation of their model on the private set. However, this approach still falls short, as practitioners may over-optimize for the specific task. 2.3 Shared tasks Shared tasks improve comparability by running centralized evaluation on hidden test labels. However, they typically announce the task and format well in advance; strong results can reflect targeted engineering for a known target rather than general preparedness. This is a feature for shared tasks—they are designed to measure adaptation to a specific target—but it differs from the goal of an “exam-style” evaluation that probes broad preparedness under surprise problem statements. For LLM systems, remaining degrees of freedom (prompting conventions, tool use, and compute budgets) can still make comparisons uneven unless the rulebook and harness are strict ( 22 ; 32 ) . Moreover, many shared-task submissions are highly task-specific. While this is not necessarily wrong, it typically implies that such tasks are not intended for evaluating the capabilities of general-purpose models. Once tasks are released, they can eventually enter training corpora, reintroducing long-term leakage risk ( 31 ) . It is worth noting that some shared tasks already incorporate elements of the protocol we propose. SemEval tasks 20 ; 16 release test data only during the evaluation window; WMT 18 ; 17 has moved toward increasingly controlled evaluation protocols; and the TREC evaluation rounds 1 ; 10