Position: Ensuring mutual privacy is necessary for effective external evaluation of proprietary AI systems
Ben Bucknall, Robert F. Trager, Michael A. Osborne · 2025
Read Position: Ensuring mutual privacy is necessary for effective external evaluation of proprietary AI systems by Ben Bucknall, Robert F. Trager, Michael A. Osborne (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-01470.txt.
Introduction
Ben Bucknall, Robert Trager, and Michael Osborne argue that outside checks of private AI systems need privacy on both sides, in a 2025 position paper. External review is a key way to learn the risks of new systems. Yet builders fear leaks of their models, and reviewers must guard their held-back test sets, or scores lose meaning. The authors name this two-way need mutual privacy and make it the center of their case.
They set out the problem in formal terms and map what each side must protect. Current review setups, they argue, guard the builder's side far better than the reviewer's. Fixes may come from cryptography or from hardware that keeps data sealed while it is used. The paper is a call to build those tools before outside review becomes the norm, so that access and secrecy can hold at once.
Catalog record
| CREATOR | Ben Bucknall, Robert F. Trager, Michael A. Osborne |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-01470
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Position: Ensuring mutual privacy is necessary for effective external evaluation of proprietary AI systems Ben Bucknall, Robert F. Trager, Michael A. Osborne Abstract The external evaluation of AI systems is increasingly recognised as a crucial approach for understanding their potential risks. However, facilitating external evaluation in practice faces significant challenges in balancing evaluators’ need for system access with AI developers’ privacy and security concerns. Additionally, evaluators have reason to protect their own privacy – for example, in order to maintain the integrity of held-out test sets. We refer to the challenge of ensuring both developers’ and evaluators’ privacy as one of providing mutual privacy . In this position paper, we argue that (i) addressing this mutual privacy challenge is essential for effective external evaluation of AI systems, and (ii) current methods for facilitating external evaluation inadequately address this challenge, particularly when it comes to preserving evaluators’ privacy. In making these arguments, we formalise the mutual privacy problem; examine the privacy and access requirements of both model owners and evaluators; and explore potential solutions to this challenge, including through the application of cryptographic and hardware-based approaches. 1 Introduction External evaluations – including independent audits and red-teaming – are emerging as a cornerstone of AI governance regimes. Their importance is reflected in their inclusion as Measure 16 of the second draft of the AI Act’s General-Purpose AI Code of Practice ( Oliver et al., 2024 ) , and prior research has demonstrated their value in advancing external transparency and scrutiny of advanced AI systems ( Raji et al., 2022 ; Mökander et al., 2023 ; Anderljung et al., 2023 ) . For external evaluations to be effective, evaluators require sufficient access to both the AI system being assessed and relevant contextual information regarding that system ( Bucknall & Trager, 2023 ; Casper et al., 2024 ) . However, in the case of proprietary systems – where model weights are not made openly available – this can create a practical tension between providing external access and maintaining system security, leading many model providers to be hesitant about granting deep access to evaluators. Current research on expanding system access for external evaluators has primarily focused on balancing evaluator access with model providers’ security concerns. However, equally crucial but often overlooked is the privacy and security of the evaluators themselves. When evaluators lack adequate privacy protections, it can be difficult to fully trust either the independence or results of their evaluations. As such, in this paper we argue for the following pair of positions: • Ensuring the mutual privacy of both external auditors and model owners is essential for effective external evaluation of AI systems; and • Current methods for facilitating external evaluations of proprietary systems inadequately address the mutual privacy challenge, particularly when it comes to preserving evaluators’ privacy. We make three key contributions in advocating for these positions. First, we introduce and define the challenge of ensuring mutual privacy during external evaluations of AI systems. Second, we provide an overview of information that both model owners and evaluators may need to keep private or share during evaluations. Third, we discuss current and proposed methods for enabling secure external evaluations, finding that evaluator privacy is rarely considered. 2 Formalising the mutual privacy problem We start by formalising the notion of a mutual privacy problem using set notation. This formalism is given solely for increasing the precision and clarity of the definition. Suppose that we have two actors, a model owner M , and an external evaluator E . Each actor has a set of information known to them, denoted I_{M} and I_{E} , respectively, where the intersection I_{M}\cap I_{E} may be nonempty – that is, there may be common knowledge. For each actor i\in\{M,E\} , there is some subset P_{i}\subset I_{i} , which we call i ’s private set, that they wish to keep private from the other – that is, P_{M}\cap I_{E}=\emptyset , and vice-versa. Furthermore, each actor has a set of required information , R_{i}\subset I_{j} containing information that may be demanded from the other for the purposes of conducting or facilitating an external evaluation. We say that there is a privacy problem if P_{M}\cap R_{E}\neq\emptyset – that is, if the model owner wishes to know something that the evaluator would like to keep private – or vice-versa. A mutual privacy problem occurs if both P_{M}\cap R_{E} and P_{E}\cap R_{M} are nonempty. We take a fairly inclusive conceptualisation for what constitutes ‘information’, including not only factual knowledge (for example, architectural details of a given AI model), but also procedural knowledge (for example, information about what activities are being carried out by the other actor, and how), as well as access to system artefacts including models and datasets. Information could also refer to assurances, for example that the model owner’s private set P_{M} is indeed private and not known to the evaluator, or that the information R_{E} that the evaluator receives from the model owner is accurate and reliable. 3 Information that is private to, and required by, each actor Figure 1: A summary of the relevant information that either the model owner or evaluator may require or want to keep private. Where one party requires information that the other would wish to keep private, we say there is a ‘privacy problem’. Where both parties require information that the other would wish to keep private, we say there is a ‘ mutual privacy problem.’ In this section we analyse a typical external evaluation setting in terms of the information that each of the model owner and evaluator both require and want to keep private. For each set of information we give a brief description, along with an explanation of why this information is may need to be private and/or required. Note that we do not claim that all of the below will hold in all instances of external evaluation. The extent and frequency with which the types of information described below will be required and/or private varies. Rather, we refer to information for which, in some plausible cases, there may be some reason to include in any of P_{M} , P_{E} , R_{M} , or R_{E} for the purpose of conducting an external evaluation. A summary of the information discussed and how conflicting requirements can give rise to a mutual privacy problem is depicted in Figure 1 . 3.1 Information private to the model owner We begin by describing information known to the model owner that they may want to keep private, either from the external evaluator, or the broader public. For each set of information we provide a brief explanation of the potential incentives for this privacy. Model weights. The most advanced contemporary commercial AI models cost on the order of tens of millions of dollars to train, placing considerable value on the resulting model weights ( Cottier et al., 2024 ) . Therefore, unless pursuing an open-weights commercial strategy ( Widder et al., 2024 ) , model owners have large incentives to ensure that model weights are not leaked or stolen, motivating strict controls on who can access the raw weight data. Furthermore, we are seeing an increasing number of statements regarding the importance of ensuring the security of model weights