Expanding External Access to Frontier AI Models for Dangerous Capability Evaluations
Jacob Charnock, Alejandro Tlaie, Kyle O’Brien, Stephen Casper, Aidan Homewood · 2026
Read Expanding External Access to Frontier AI Models for Dangerous Capability Evaluations by Jacob Charnock, Alejandro Tlaie, Kyle O’Brien, Stephen Casper, Aidan Homewood (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-11916.txt.
Introduction
Jacob Charnock, Alejandro Tlaie, Kyle O'Brien, Stephen Casper, and Aidan Homewood argue for wider outside access to top AI models for risk tests. Firms rely on outside teams to probe for risky skills before launch. Yet those teams often get weak access, scant facts, and brief time.
Weak access cuts the rigour of the tests and trust in their calls. EU rules for general AI call for apt access but do not spell out what that means. The team set out what apt access should hold in real terms. Clear norms could make risk tests more sound.
Catalog record
| CREATOR | Jacob Charnock, Alejandro Tlaie, Kyle O’Brien, Stephen Casper, Aidan Homewood |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-11916
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Expanding External Access to Frontier AI Models for Dangerous Capability Evaluations Jacob Charnock, Alejandro Tlaie, Kyle O’Brien, Stephen Casper, Aidan Homewood Abstract Frontier AI companies increasingly rely on external evaluations to assess risks from dangerous capabilities before deployment. However, external evaluators often receive limited model access, limited information, and little time, which can reduce evaluation rigour and confidence. The EU General-Purpose AI Code of Practice calls for “appropriate” access, but does not specify what this means in practice. Furthermore, there is no common framework for describing different types and levels of evaluator access. To address this gap, we propose a taxonomy of access methods for dangerous capability evaluations. We disentangle three aspects of access: model access, model information, and evaluation timeframe. For each aspect, we review benefits and risks, including how expanding access can reduce false negatives and improve stakeholder trust, but can also increase security and capacity challenges. We argue that these limitations can likely be mitigated through technical means and safeguards used in other industries. Based on the taxonomy, we propose three descriptive access levels: AL1 (black-box model access and minimal information), AL2 (grey-box model access and substantial information), and AL3 (white-box model access and comprehensive information), to support clearer communication between evaluators, frontier AI companies, and policymakers. We believe these levels correspond to the different standards for appropriate access defined in the EU Code of Practice, though these standards may change over time. Executive Summary This paper provides a taxonomy for describing external access for model evaluations, and offers a preliminary set of Access Levels corresponding to the different standards for appropriate access in the EU Code of Practice (CoP). The current state of external evaluator access ( Section 1 ) Frontier AI companies increasingly rely on external evaluation results to assess whether their models are safe for deployment. However, external evaluators are typically given limited model access, technical information, and time to complete their evaluations. The CoP outlines high-level expectations for “appropriate” access for third-party model evaluations; however, it does not clarify what this means in practice. The relationship between access, rigour, and security ( Section 2 ) Expanding external evaluator access has two key benefits. First, expanding access can make external evaluations more rigorous, helping to reduce false negatives (i.e. cases where dangerous capabilities exist but remain undetected) as well as false positives (i.e. cases where evaluators overestimate dangerous capabilities). Second, expanding external access can increase stakeholder trust (e.g. amongst governments, regulators, and customers) because evaluators can assess systems with greater independence, and in more breadth and depth. Despite this, expanding evaluator access, absent appropriate safeguards, can present challenges. Whilst unlikely, third parties with enough access could leak proprietary information. Furthermore, providing additional external access could strain internal capacity at frontier companies. Nevertheless, we find these risks can likely be mitigated through technical means (e.g. privacy-preserving access), and mitigations used in other industries (e.g. segregating duties, appointing a liaison). A taxonomy of structured access for dangerous capability evaluations ( Section 3 ) Our taxonomy disentangles three distinct aspects of access: Model access can range from black-box querying to full white-box access including activations, gradients, logits, and custom fine-tuning. Deeper model access enables more accurate capability assessments by enabling different evaluation techniques (e.g. using log-probabilities to reduce variance). However, more model access could increase opportunities for model tampering, reverse engineering, and model weight theft unless appropriate safeguards are in place. Model information can include information about training data, deployment configurations, internal evaluation results, and safety mitigation details. Greater information disclosure helps evaluators contextualise their results, identify blind spots, probe potential vulnerabilities, and more accurately verify company safety claims. Despite this, more information disclosure risks exposing proprietary design and training choices that competitors could steal or adversaries could exploit, unless strong information handling protocols are in place. Evaluation time frames can vary significantly for external reviewers, ranging from no access before release, less than a week pre-release, to assessments exceeding 20 business days. Longer evaluation time frames enable more thorough evaluations, as evaluators can design bespoke testing methods, re-run, adapt and debug evaluations. However, longer time frames impose greater operational costs on developers (e.g. allocating internal employees to oversee longer external evaluations) and could interfere with model release schedules. Access Levels ( Section 4 ) Based on the taxonomy described in ( Section 3 ), we outline three descriptive access levels. We relied on existing literature, expert discussions, and definitions from the CoP to categorise the access methods from our taxonomy. We believe these access levels correspond to each definition of “appropriate” access under the EU GPAI Code of Practice: AL1 (i.e. black-box model access and minimal information), AL2 (i.e. grey-box model access and substantial information), and AL3 (i.e. white-box model access and comprehensive information). AL1: Black-box model access and minimal information (“best practice”) This level enables basic external assurance and vulnerability detection. It might be suitable when a developer releases a minor update to an existing model, or when a new model poses low systemic risk. AL2: Grey-box model access and substantial information (“the state of the art”) This level enables more targeted evaluations to improve capability elicitation and vulnerability identification. It might be suitable for model releases with significant capability jumps. AL3: White-box model access and comprehensive information (“more innovative”) This level likely enables the most accurate risk assessments. It might be suitable for models with step-changes in capabilities, when model deployment decisions depend on evaluation results, or if privacy-preserving tools can support white-box access methods. 1 Introduction Frontier AI companies increasingly use results from external model evaluations to demonstrate to stakeholders and governments that their models are sufficiently safe for deployment ( 44 ; 29 ) . External evaluations of the dangerous capabilities of frontier AI models have become part of risk assessments before deployment ( 58 ; 12 ; 31 ) . Results from these evaluations can provide governments and the public with more confidence in the safety of frontier AI models, allowing them to make more informed decisions ( 17 ; 74 ) . Similarly, evaluation results can help companies improve their safeguards ( 60 ; 10 ) , and help governments identify and anticipate emerging risks ( 33 ) . External evaluations are included in the Frontier AI Safety Commitments ( 26 ) , and the EU General-Purpose AI Code of Practice (“CoP”) ( 28 ) . However, external evaluators often receive limited access to the models they are evaluating, which reduces evaluation rigour and confidence 23 ; 74 . External evaluators are often restricted to black-box access, meaning they can query the model and examine outputs (sometimes alongside chains-of-thought), but cannot examine any model internals (e.g. activations, gradients, weights). Furthermore, external evaluators often receive limited technical information, which means they often struggle to fully assess reported properties ( 23 ) and provide stakeholders with high confidence evaluation results ( 61 ) . Moreover, despite an improvement in external access for recent evaluations involving chain-of-thought (CoT) access and longer evaluation time frames ( 45 ; 58 ) , external evaluators continue to receive inconsistent model access, limited (often unverifiable) information about models, and days rather than weeks to evaluate models. 1 1 1 For example, Apollo and UK AISI received less