Developing and Maintaining an Open-Source Repository of AI Evaluations: Challenges and Insights

Alexandra Abbas, Celia Waggoner, Justin Olive · 2025

Read Developing and Maintaining an Open-Source Repository of AI Evaluations: Challenges and Insights by Alexandra Abbas, Celia Waggoner, Justin Olive (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-06893.txt.

Introduction

Alexandra Abbas, Celia Waggoner, and Justin Olive share what they learned in eight months of work on inspect_evals, an open store of AI evaluations. The store holds more than seventy tests of model skill and safety given by people in the field. To keep such a store sound turned out to be a craft of its own.

Three trouble spots rule their account. Growth in what the field sends in called for a clear frame to run cohorts. Fair tests of one model next to the next called for tools from statistics, with resampling and with bounds on how sure a score is. Work that others can redo called for checks on quality past what plain software needs. The team argue that grave work on tests needs its own plant, rigor from statistics, and a field kept in close touch.

Catalog record

CREATORAlexandra Abbas, Celia Waggoner, Justin Olive
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE21 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-06893

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Developing and Maintaining an Open-Source Repository of AI Evaluations: Challenges and Insights Alexandra Abbas, Celia Waggoner, Justin Olive Abstract AI evaluations have become critical tools for assessing large language model capabilities and safety. This paper presents practical insights from eight months of maintaining i ​ n ​ s ​ p ​ e ​ c ​ t ​ _ ​ e ​ v ​ a ​ l ​ s inspect\_evals , an open-source repository of 70+ community-contributed AI evaluations. We identify key challenges in implementing and maintaining AI evaluations and develop solutions including: (1) a structured cohort management framework for scaling community contributions, (2) statistical methodologies for optimal resampling and cross-model comparison with uncertainty quantification, and (3) systematic quality control processes for reproducibility. Our analysis reveals that AI evaluation requires specialized infrastructure, statistical rigor, and community coordination beyond traditional software development practices. 1 Introduction AI evaluations have emerged as a valuable way to assess the capabilities and safety of large language models (LLMs). The UK AI Security Institute (UK AISI) identifies AI evaluations as a critical tool to advance our understanding of model capabilities and prevent catastrophic AI risks ( UK AI Security Institute 2025 ) , recognizing their fundamental importance in developing safe and beneficial AI systems. This paper presents learnings and challenges from the maintenance of the inspect\_evals repository ( UK AI Security Institute et al. 2024 ) since its publication 8 months ago. inspect\_evals is a repository of community-contributed LLM evaluations for inspect\_ai ( UK AI Security Institute 2024 ) , created in collaboration by the UK AISI, Arcadia Impact, and the Vector Institute. The repository houses a diverse range of (70+) evaluations, including assessments of scientific knowledge such as GPQA ( Rein et al. 2023 ) , offensive cyber capabilities such as Cybench ( Zhang et al. 2024 ) , and the robustness of safeguards such as AgentHarm ( Andriushchenko et al. 2024 ) . AI evaluations have long presented challenges that traditional software engineering practices struggle to address, and these challenges are intensifying as evaluations become more numerous, complex, and consequential. Through our experience with inspect\_evals , we have encountered technical, methodological, and community-driven obstacles that can provide valuable insights for the broader open-source AI evaluation community. This paper reviews these challenges, proposes solutions to some, and identifies others as open questions requiring further research and community coordination. 2 Repository Overview inspect\_evals is a collaborative space where the evaluation community can contribute high-quality standardized assessments, breaking down traditional silos across organizations and facilitating wider adoption of rigorous evaluations. The repository serves three primary audiences: researchers working on scaling laws and evaluation science, analysts processing model performance data for decision-makers, and technical teams running internal evaluations against industry benchmarks. inspect\_evals organizes evaluations across nine categories: Agents, Assistants, Coding, Cybersecurity, Knowledge, Mathematics, Multimodal, Reasoning, and Safeguards, providing comprehensive coverage of modern AI capabilities and safety considerations. 3 Learnings and Challenges Maintaining an open-source repository of AI evaluations introduces new challenges beyond traditional software development. Key challenges include: • The rapid pace of new evaluation publications combined with the absence of centralized evaluation sources complicates discovery and curation efforts; • Assessing the quality, credibility, and usefulness of evaluations is resource intensive, requiring careful analysis, and in many cases, domain expertise; • Incomplete or low-quality documentation of the evaluation methodology (including the reference implementation) creates challenges in reproducing the results. 3.1 Evaluation Selection and Prioritization With evaluations being published continuously, it is necessary to capture and process this constant stream of new releases. To this end, we developed a roadmap that prioritizes evaluations, assigns them an implementation difficulty score, and identifies which capabilities they cover. One factor that influences our prioritization is filling gaps in the least-covered areas. Sources of evaluations include technical reports, The AI Evaluation Substack ( AI Evaluation Digest 2024 ) , and suggestions from the community. Some of the criteria that we consider when sourcing and prioritizing evaluations include the following. • Well-established with research citations • Credibly sourced from major labs or academic groups • Challenging for frontier models with distinguishable performance • Agentic/task-based over simple Q&A (question-and-answer) • Clearly scoped with verifiable methodologies • Comparable with existing baseline results 3.2 Contributor Management To help keep up with the rapid pace of new evaluation publications and to help upskill the open-source community, we periodically facilitate cohorts of volunteers to implement benchmarks in inspect\_evals . We organize volunteers into 5-week cohorts with one Technical Project Manager (TPM) overseeing 5-10 engineers. Implementation involves porting evaluations from papers to inspect\_evals . Figure 1: Cohort management structure showing one Technical Project Manager overseeing multiple Software Engineers with bidirectional review processes for coordinated evaluation implementation and scalability. Our program follows structured phases: an onboarding week, a design phase where contributors produce a Benchmark Development Plan that is reviewed by the TPM and a peer, the implementation phase for creating draft Pull Requests (PRs) with TPM and peer review, and the final PR submission with evaluation reports. This structure addresses the educational gap and enables scalable coordination of volunteer contributions through agile project management including standups, Kanban boards, and detailed documentation. Outside of the benchmarks cohort program, we create GitHub issues to signal to the open-source community which evaluations we would be particularly excited to see implemented in inspect\_evals . We provide support to individual contributors who pick up these issues by answering questions and reviewing their code. 3.3 Quality Assurance Ensuring reliable AI evaluations is a challenge that requires systematic quality control to address both the correctness of implementation and the methodological validity. Contributor guidelines and review process. Participants in the volunteer cohorts are provided with a Software Requirement Specification that outlines a 5-week delivery timeline with weekly deliverables. Contributors must first produce a Benchmark Development Plan to be reviewed by their TPM before implementation. The Benchmark Development Plan details the evaluation scope, methodology, scoring mechanisms, and verification strategy. This upfront design prevents implementation drift and ensures alignment with evaluation objectives. For individual contributors outside of the volunteer cohorts we developed a Contribution Guide which established standardized procedures for submitting new evaluations. Verification and testing methods. Contributors participating in our volunteer cohorts work in pairs to verify each other’s implementations on data subsets, with TPM oversight. For all contributions to inspect\_evals , automated unit and integration tests are run in CI pipelines to validate core functionality and framework compliance, and logs are manually reviewed to verify that frontier models or humans can complete evaluation samples. This prevents tool failures that artificially degrade model performance and identifies potential reward hacking behaviors. 3.3.1 Testing AI Evaluations Standards and requirements. We follow the UK AISI Autonomous Systems Evaluation Standard ( Black et al. 2024 ) , which requires unit tests for custom tools and scorers, plus verification tests for dynamic resource access like Hugging Face datasets. We also conduct manual peer testing to ensure that the benchmarks successfully execute end-to-end, though there are plans to migrate this to automated CI processes. Result validation. Frontier models are run on evaluation subsets to verify that results match reference implementations within +/-5% deviation. Deviations require investigation, which is fairly common due to authors not publishing exact model configuration details, like temperature or specific endpoints. Manual tests

Read the complete text