An autonomous agent for auditing and improving the reliability of clinical AI models

Lukas Kuhn, Florian Buettner · 2025

Read An autonomous agent for auditing and improving the reliability of clinical AI models by Lukas Kuhn, Florian Buettner (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-05755.txt.

Introduction

AI models that ace medical image benchmarks can fail badly on real hospital scans, where scanners, lighting, or patient mixes differ. Kuhn and Buettner introduce ModelAuditor, an autonomous agent that audits a clinical model before deployment. The agent talks with the user, picks fitting measures, and simulates realistic shifts in the data. It then writes a plain report on likely failure modes, their causes, and fixes.

The team tested it on three real settings: tissue images across hospitals, skin images across demographic groups, and chest X-rays across equipment. The agent correctly exposed hidden weaknesses in strong models, including a well-known melanoma classifier. Following its advice recovered fifteen to twenty-five percent of the accuracy lost under shift, beating standard fixes. An audit runs in under ten minutes on home hardware for less than fifty cents.

Catalog record

CREATORLukas Kuhn, Florian Buettner
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-05755

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

An autonomous agent for auditing and improving the reliability of clinical AI models Lukas Kuhn, Florian Buettner Abstract The deployment of AI models in clinical practice faces a critical challenge: models achieving expert-level performance on benchmarks can fail catastrophically when confronted with real-world variations in medical imaging. Minor shifts in scanner hardware, lighting or demographics can erode accuracy, but currently reliability auditing to identify such catastrophic failure cases before deployment is a bespoke and time-consuming process. Practitioners lack accessible and interpretable tools to expose and repair hidden failure modes. Here we introduce ModelAuditor, a self-reflective agent that converses with users, selects task-specific metrics, and simulates context-dependent, clinically relevant distribution shifts. ModelAuditor then generates interpretable reports explaining how much performance likely degrades during deployment, discussing specific likely failure modes and identifying root causes and mitigation strategies. Our comprehensive evaluation across three real-world clinical scenarios — inter-institutional variation in histopathology, demographic shifts in dermatology, and equipment heterogeneity in chest radiography — demonstrates that ModelAuditor is able correctly identify context-specific failure modes of state-of-the-art models such as the established SIIM-ISIC melanoma classifier. Its targeted recommendations recover 15-25% of performance lost under real-world distribution shift, substantially outperforming both baseline models and state-of-the-art augmentation methods. These improvements are achieved through a multi-agent architecture and execute on consumer hardware in under 10 minutes, costing less than US$0.50 per audit. keywords Out-of-distribution generalization, Robustness Auditing, Computer Vision, Agentic AI 1 Introduction Figure 1: Schematic of the ModelAuditor agent. (1) Agent Instructionset: guideline texts from the MetricsReloaded framework, the complete catalogue of synthetic distribution shifts and a template for the final report are each converted into instructions for the agent (2) Metrics & shifts tools: extensible collection of tools containing a multitude of performance and calibration measures and distribution-shift implementations accessible to the agent (3) ModelAuditor Agent: supplied with a trained clinical AI model and an evaluation dataset, the LLM agent autonomously chooses context-dependent, clinically relevant metrics and relevant distribution shifts and executes the audit, ultimately returning a natural-language report and actionable recommendations. The deployment of artificial intelligence models in clinical practice is hindered by a key challenge: models that achieve expert-level performance on benchmark datasets often fail when confronted with the subtle variations inherent in real-world medical imaging Kelly et al. 2019 ; Zech et al. 2019 ; Koh et al. 2021 . This reliability gap – between laboratory excellence and real-world clinical safety – has emerged as a critical barrier preventing AI from fulfilling its promise to improve healthcare delivery. Consequently, it is also reflected in evolving regulatory frameworks, including the European Union’s AI Act and FDA guidance in the United States, which now mandate systematic auditing of high-risk AI systems in healthcare to ensure reliability and trustworthiness throughout the entire life-cycle of an AI system European Commission 2021 ; U.S. Food and Drug Administration 2021 . Consider a deep learning model trained to detect melanoma in dermoscopic images. In controlled evaluations, it may rival dermatologists in accuracy Esteva et al. 2017 . Yet when deployed in a new clinic, the same model might miss one-third of malignant lesions simply because the lighting differs from its training environment Daneshjou et al. 2022 . Similar failures occur for other imaging modalities: chest X-ray algorithms trained at one hospital misclassify pneumonia at another due to differences in patient positioning protocols Zech et al. 2018 ; MRI classifiers experience accuracy drops exceeding 20% when encountering images from different scanner manufacturers Kushol et al. 2023 ; and histopathology models for tumor detection show marked performance degradation across institutions due to variations in tissue staining procedures Ehteshami Bejnordi et al. 2017 ; Bandi et al. 2019 . Such distribution shifts occur frequently in clinical practice and their exact nature is usually unknown prior to model deployment. Healthcare data varies across institutions, equipment manufacturers, patient populations, imaging protocols, and time. Ensuring that a model is able to generalize across these variations via a comprehensive auditing strategy is therefore both a clinical safety imperative and regulatory requirement. Current approaches to auditing model reliability remain fundamentally limited. Many published models report only aggregate metrics such as area under the receiver operating characteristic curve (AUROC) or accuracy for in-distribution scenarios Maier-Hein et al. 2024 ; U.S. Food and Drug Administration 2019 , which can mask substantial performance drops in real-world deployment scenarios or patient subpopulations. Even well-intentioned practitioners lack standardized tools for systematically probing their models’ failure modes before deployment Salvo et al. 2024 ; Navarro et al. 2021 . A related line of work attempts to harden models during training by synthetically enriching the training data Cubuk et al. 2019 , but suffers from three critical shortcomings. First, they require practitioners to anticipate which distribution shifts will occur – a challenging task requiring expert domain knowledge as well as expert AI expertise given the domain-specific nature of real-world deployment environments. Second, augmentation policies that improve robustness in one imaging modality may harm performance in another Goceri 2023 . Third, they provide no mechanism for understanding when a trained model fails under distribution shift, why it fails and how to mitigate these failures. Taken together, an auditing framework that quantifies the reliability of a trained AI model by comprehensively anticipating potential distribution shifts that may occur throughout its lifecycle and also explains failures in clinically interpretable and actionable terms is needed for closing the reliability-gap. The recent emergence of large language models (LLMs) as autonomous agents offers a transformative opportunity to address these challenges Liu et al. 2024 ; Lu et al. 2024 . When augmented with specialized tools, LLM agents have demonstrated remarkable capabilities in medical contexts: generating radiology reports with iterative self-refinement Zeng et al. 2024 , performing sequential clinical decision support Liu et al. 2024 , and providing treatment recommendations grounded in medical literature Lu et al. 2024 . These examples illustrate that LLMs, when orchestrated as agents, have the potential to also effectively audit the reliability of medical AI systems. Here, we present ModelAuditor, an autonomous agent that conducts a comprehensive reliability evaluation of trained AI models that enables practitioners to understand failure modes and implement corrective measures. Given any trained clinical AI vision model, ModelAuditor conducts a comprehensive audit by: (i) engaging practitioners in natural conversation to understand the clinical context and deployment environment; (ii) automatically selecting statistically sound evaluation metrics that capture clinically meaningful failures; (iii) simulating clinically relevant distribution shifts tailored to the specific clinical context and deployment environment; and (iv) generating interpretable reports that explain how much performance degrades, as well as why it fails and how to fix it. Unlike existing approaches that treat robustness as a monolithic property, ModelAuditor recognizes that clinical relevance depends on context. A dermatology model intended for teledermatology must be robust to smartphone camera variations, while a histopathology model for multi-institutional trials must handle diverse staining protocols. By tailoring its analysis to each specific use case, ModelAuditor provides actionable insights that directly translate to better model reliability and regulatory compliance. We demonstrate ModelAuditor’s capabilities across three complementary clinical scenarios that exemplify common distribution shift challenges: inter-institutional variation in histopathology, demographic shifts in dermatology, and equipment heterogeneity in chest radiography. In each case, ModelAuditor (i) identifies critical failure modes missed by conventional

Read the complete text