Trustworthy Agents for Electronic Health Records through Confidence Estimation

Yongwoo Song, Minbyul Jeong, Mujeen Sung · 2025

Read Trustworthy Agents for Electronic Health Records through Confidence Estimation by Yongwoo Song, Minbyul Jeong, Mujeen Sung (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-19096.txt.

Introduction

Yongwoo Song, Minbyul Jeong, and Mujeen Sung build a health records agent in a 2025 paper. Language models can pull facts from patient records and aid doctors, but they can also make up answers. In care settings, a wrong answer given with full confidence is a real risk.

The team adds step-by-step confidence scoring to their agent and a new metric that trades accuracy against trust. On two hospital data sets, the agent beat rivals by wide margins under strict trust rules where rivals failed outright. It gives an answer when sure and owns up when unsure. Readers get a design for clinical tools that know their limits.

Catalog record

CREATORYongwoo Song, Minbyul Jeong, Mujeen Sung
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-19096

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Trustworthy Agents for Electronic Health Records through Confidence Estimation Yongwoo Song, Minbyul Jeong, Mujeen Sung Abstract Large language models (LLMs) have emerged as a promising approach for extracting and interpreting information from Electronic Health Records (EHR), offering possibilities for clinical decision support and patient care. However, the reliable deployment of LLMs in clinical settings remains challenging due to the risk of hallucinations. Thus, we propose Hallucination-Controlled Accuracy at k% (HCAcc@k%), a novel metric quantifying the accuracy-reliability trade-off at varying confidence thresholds. We also introduce TrustEHRAgent, a confidence-aware agent that incorporates step-wise confidence estimation for clinical question answering. Experiments on MIMIC-III and eICU datasets demonstrate that TrustEHRAgent outperforms baselines under strict reliability constraints, achieving substantial improvements of 44.23%p and 25.34%p at HCAcc@70% while baseline methods fail completely at these stringent thresholds. These performance differentials under reliability constraints underscore the limitations of traditional accuracy metrics in evaluating healthcare AI agents. Our work contributes to the development of trustworthy clinical agents capable of delivering accurate information—or transparently expressing uncertainty—when confidence is low. 1 1 1 ldi-kyunghee/TrustEHRAgent Figure 1: Performance comparison under reliability constraints using the HCAcc@k% metric, where k% represents the minimum required reliability. Traditional accuracy metrics (A) cannot identify the Patient Safety Risk Area, a region where strict reliability is essential for patient safety. In contrast, while (B) PythonAgent completely fails at k\geq 60\% , (C) Our TrustEHRAgent maintains stable performance under high reliability requirements, demonstrating the need for reliability-aware evaluation in medical AI agents. 1 Introduction Large language models (LLMs) have shown remarkable potential in healthcare, particularly in extracting and interpreting information from Electronic Health Records (EHRs) Qiu et al. 2024 . However, deploying these agents in clinical settings presents unique challenges, as hallucinations or factual errors can have serious implications for patient safety and care quality Pal et al. 2023 ; Li et al. 2023 ; Ahmad et al. 2023 . While prior works have largely focused on improving the accuracy of LLM-based clinical assistants Wu et al. 2023 , comparatively little attention has been given to reliability—the model’s ability to recognize its own limitations and abstain from answering when uncertain. This lack of reliability poses a critical barrier to the safe and responsible deployment of LLMs in real-world clinical environments Hager et al. 2024 . The widespread use of traditional evaluation metrics fails to capture the risks posed by hallucinations in high-stakes medical contexts (see Figure 1 A). These metrics typically assume that agents attempt to answer every question, ignoring the practical necessity of abstention in uncertain or ambiguous cases. As shown in Figure 1 B, existing methods suffer substantial performance degradation under strict reliability constraints, underscoring their vulnerability in real clinical scenarios. In clinical applications of AI, the ability to estimate confidence and recognize uncertainty is not merely desirable—it is essential. A reliable LLM-based agent must know when to respond and when to abstain, rather than risk producing harmful outputs in uncertain situations. To address these limitations, we introduce Hallucination-Controlled Accuracy at k% (HCAcc@k%), a novel evaluation metric that quantifies the trade-off between accuracy and reliability across varying confidence thresholds. Unlike traditional metrics, HCAcc@k% explicitly incorporates reliability, making it well-suited for evaluating agents intended for clinical use. Based on this evaluation metric, we propose TrustEHRAgent, a confidence-aware clinical agent that integrates step-wise confidence estimation into its question-answering environments. This enables the agent to make informed decisions about whether to answer or abstain based on its internal uncertainty estimates. As illustrated in Figure 1 C, our method demonstrates improved performance at stringent thresholds compared to existing approaches, providing a foundation for safer clinical decision support. Notably, under stringent reliability constraints (HCAcc@70%), TrustEHRAgent maintains performance with 44.23% and 25.34% accuracy on MIMIC-III and eICU datasets respectively while baseline methods achieve 0% performance. TrustEHRAgent also exhibits superior performance under traditional evaluation, achieving 62.13% and 59.82% accuracy (HCAcc@0%), representing improvements of 3.16%p and 6.72%p over the existing methods. Additionally, the agent shows the ability to appropriately abstain from answering when encountering ambiguous medical abbreviations, preventing potential errors through selective response mechanisms. Our main contributions in this work include: • A novel evaluation metric (HCAcc@k%) that quantifies the accuracy-reliability trade-off at varying confidence thresholds, enabling principled assessment of medical AI agents under patient safety constraints. • A confidence-aware clinical agent framework (TrustEHRAgent) that achieves remarkable performance improvements of 44.23%p and 25.34%p at stringent reliability thresholds where existing methods fail completely. • Evidence that traditional accuracy metrics inadequately reflect agent reliability in clinical environments, with methods showing similar baseline accuracy exhibiting dramatically different performance under safety constraints. Figure 2: Overview of TrustEHRAgent framework. The left panel shows step-wise confidence estimation during clinical query resolution, where the agent assigns low confidence (0.5) when using uncertain database tables. The right panel illustrates the confidence estimator that takes the query, reasoning history, and answer as inputs to compute a final confidence score (C = 0.52) using token probabilities. The computed confidence score is then compared against a predefined threshold to make threshold-based decisions that either provide or reject answers based on reliability requirements. To achieve high HCAcc@k%, the agent must provide accurate answers when confident and appropriately abstain when uncertain. 2 Related Work Reliability in Clinical LLMs Recent advances in LLMs have enabled their application across various medical domains Wu et al. 2023 ; Qiu et al. 2024 . For Electronic Health Record (EHR) systems Wang et al. 2020 ; Yang et al. 2022 introduced Text-to-SQL benchmarks for database querying, while Shi et al. 2024 developed EHRAgent, an agent framework optimized for EHR interaction. Despite these advances, deployment in clinical settings faces significant challenges due to patient safety concerns, as factual errors can lead to harmful clinical decisions. To address these reliability challenges, several approaches have been proposed, including evaluation frameworks for medical question answering Li et al. 2023 ; Agarwal et al. 2024 ; Hager et al. 2024 , hallucination detection methods across NLP tasks Ahmad et al. 2023 ; Ahsan et al. 2024 ; Mehenni and Zouaq 2025 , and medical-specific benchmarks for evaluating reasoning and memory-related hallucinations Pal et al. 2023 . However, these approaches lack integrated evaluation metrics that simultaneously consider both accuracy and reliability, particularly in clinical agent environments that require multiple reasoning steps. Uncertainty and Confidence Estimation Confidence estimation for LLM outputs has become increasingly important for ensuring AI reliability in high-stakes applications Mahaut et al. 2024 . Current approaches can be categorized into white-box methods that utilize internal model representations Jiang et al. 2020 ; Orgad et al. 2024 ; Gekhman et al. 2025 and black-box methods employing sampling-based consistency measures and verbalized confidence techniques Chen and Mueller 2023 ; Xiong et al. 2023 ; Lin et al. 2023 ; Yang et al. 2024 . While recent research has emphasized evaluation methodologies beyond traditional accuracy metrics Wang et al. 2023 ; Lin et al. 2021 , existing confidence estimation approaches primarily focus on single input-output relationships rather than evaluating reliability across multiple reasoning steps Han et al. 2024 . This limitation is particularly problematic in clinical agent environments where errors can cascade through multiple reasoning steps, making it essential to assess confidence at each stage of the decision process. Our approach addresses these limitations by introducing HCAcc@k%, a novel metric

Read the complete text