Audit Cards: Contextualizing AI Evaluations
Leon Staufer, Mick Yang, Anka Reuel, Stephen Casper · 2025
Read Audit Cards: Contextualizing AI Evaluations by Leon Staufer, Mick Yang, Anka Reuel, Stephen Casper (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-13839.txt.
Introduction
Leon Staufer, Mick Yang, Anka Reuel, and Stephen Casper propose audit cards for AI evaluations in a 2025 paper. Governments increasingly lean on audits, yet a technically sound test means little if its context is reported selectively or obscurely. Drawing on literature, stakeholder interviews, and governance frameworks, they define a card format that makes context explicit. Six feature types belong on it, from who audited to how the review was checked.
Their survey of real evaluation reports found wide variation, with most omitting key context like auditor backgrounds, conflicts of interest, and model access levels. Rules and frameworks also rarely demand rigorous reporting. They argue cards would build trust and aid interpretation.
Catalog record
| CREATOR | Leon Staufer, Mick Yang, Anka Reuel, Stephen Casper |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 77 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-13839
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Audit Cards: Contextualizing AI Evaluations Leon Staufer, Mick Yang, Anka Reuel, Stephen Casper Abstract AI governance frameworks increasingly rely on audits, yet the results of their underlying evaluations require interpretation and context to be meaningfully informative. Even technically rigorous evaluations can offer little useful insight if reported selectively or obscurely. Current literature focuses primarily on technical best practices, but evaluations are an inherently sociotechnical process, and there is little guidance on reporting procedures and context. Through literature review, stakeholder interviews, and analysis of governance frameworks, we propose “audit cards” to make this context explicit. We identify six key types of contextual features to report and justify in audit cards: auditor identity, evaluation scope, methodology, resource access, process integrity, and review mechanisms. Through analysis of existing evaluation reports, we find significant variation in reporting practices, with most reports omitting crucial contextual information such as auditors’ backgrounds, conflicts of interest, and the level and type of access to models. We also find that most existing regulations and frameworks lack guidance on rigorous reporting. In response to these shortcomings, we argue that audit cards can provide a structured format for reporting key claims alongside their justifications, enhancing transparency, facilitating proper interpretation, and establishing trust in reporting. 1 ML Alignment & Theory Scholars 2 Technical University of Munich 3 University of Pennsylvania 4 Stanford University 5 Massachusetts Institute of Technology [email protected], [email protected] 1 Introduction AI governance frameworks 1 1 1 This includes frontier AI safety policies from AI companies (e.g. OpenAI Preparedness Framework), regulations (e.g. US AI Action Plan, EU AI Act), and other norms (e.g. NIST AI Risk Management Framework). See Appendix D for a complete list. are being designed to increasingly rely on audits. 2 2 2 We understand audits as formalized evaluation processes of AI models. We refer to the specific evaluation procedure (e.g. benchmarking, red-teaming, uplift studies) underlying the audit as evaluations. See Appendix A for an overview of key terms. Within these frameworks, audits are meant to (1) identify potential risks, (2) incentivize more responsive development practices, and (3) involve more stakeholders in the system deployment process. However, audits can only fulfill this role effectively if the underlying evaluations are both technically rigorous and effectively integrated into decision-making ( Hardy et al. 2024 ) . However, not all evaluations are equally rigorous (e.g., Raji et al. 2022 ; Birhane et al. 2024 ; Mökander 2023 ; Anderljung et al. 2023 ; Kolt et al. 2024 ; Casper et al. 2024 ; Reuel et al. 2024b ; Li and Goel 2024 ). To date, much prior literature has focused on the technical side of audits (see Section 2 ). However, evaluations are never conducted in a vacuum ( Wallach et al. 2025 ) . Their results are intimately shaped, not only by the technical methods employed, but by numerous non-technical details of their context and design (see Section 3 ). For example, while the research field has long understood the importance of conflict of interest disclosures for academic integrity (e.g., Knerr and D’Amelia 2020 ), auditor independence remains a consistent concern in the AI ecosystem ( Costanza-Chock, Raji, and Buolamwini 2022 ; Evans et al. 2023 ) . Even technically rigorous evaluations can be uninformative or actively misleading if reported on selectively or obscurely ( Ananny and Crawford 2018 ) . This risk is magnified given the vested interest of AI developers in obtaining favorable audit results. In AI and other fields, companies undergoing evaluations have strong incentives to game audits and influence reporting in potentially misleading ways (e.g., Krawiec 2003 ; Lu 2006 ; Marquis, Toffel, and Zhou 2016 ). General principles of reporting on audits Justification Limitations Assumptions Specific contextual features to report on in audits Who are the auditors? • Expertise • Background Review & Communication • Review mechanism • Feedback channel • Executive summary • Defintions of key terms • Access to results What is evaluated? • Scope and goal • Type of evaluation • Continuous evaluations • Obsolescence criteria How is it evaluated? • Desc. of evaluation setup • Justification of capability-to-goal translation • Interpretation of scores Integrity • Selection process • Conflicts of interest • Compensation and incentive structures Access and Resources • Level of access • Compute, time, financial resources available Figure 1: Our audit card template requires reporting on three principles and six features . (Left) The three cross-cutting principles (justification, limitations, and assumptions) further transparency about the process behind an audit. (Right) The six features offer key methodological and contextual information. See Section 3 and Appendix C for details. Thus, for AI audits to play a meaningful role in governance, evaluation reporting must be done in a way that minimizes the chance of omitting key contextual information. To address this challenge, we propose “audit cards”: a structured reporting framework to document the context of an AI audit rather than the evaluated system itself ( Figure 1 ). Unlike model and system cards ( Mitchell et al. 2019 ) , which document system characteristics and capabilities, audit cards address the distinct transparency requirements of evaluation processes-specifically the contextual factors that shape how evaluations are conducted and reported. They are designed to offer a standardized approach to reporting that enhances transparency and facilitates informed interpretations of results. This need not necessarily be reported in a separate audit card document, rather the relevant context can be added as sections in existing model and system cards. Overall, we make four key contributions: 1. Audit cards framework addressing evaluation transparency gaps: Based on a survey of 28 prior works on AI evaluations ( Section 3 ), we synthesise existing literature to propose a template and checklist ( Appendix C ) for AI audit cards that addresses transparency needs not covered by existing tools, including three cross-cutting principles (justification, limitations, and assumptions) and six key types of contextual information (auditor identity, evaluation scope, methodology, resource access, process integrity, and review mechanisms). 2. Survey of frontier AI evaluation reports: We analyze the thoroughness of 24 existing evaluation reports for frontier systems ( Section 4 ). 3. Survey of governance frameworks: We analyze gaps between best reporting practices and 21 existing AI governance frameworks ( Section 5 ). 4. Stakeholder insights: We interview and present perspectives from 10 expert stakeholders on AI evaluation reporting practices ( Section 6 ). 2 Related work Transparency and reporting: In the past decade, frontier AI research and development have largely shifted from being predominantly academia-driven to predominantly industry-driven ( Maslej et al. 2024 ) . This shift has been accompanied by a decrease in transparency around proprietary state-of-the-art systems. As such, increasing transparency and awareness has emerged as a key goal of AI governance (e.g., Felzmann et al. 2020 ; Haresamudram, Larsson, and Heintz 2023 ; Winecoff and Bogen 2024 ; Chan et al. 2024 ; Bommasani et al. 2024 ; Kolt et al. 2024 ). One barrier to meaningful transparency is a lack of standards for reporting ( Maslej et al. 2024 ) . In response to this challenge, previous works have proposed data, model, system, agent, and usage “cards” ( Pushkarna, Zaldivar, and Kjartansson 2022 ; Mitchell et al. 2019 ; Gursoy and Kakadiaris 2022 ; Casper et al. 2025 ;