“Do I Trust the AI?” Towards Trustworthy AI-Assisted Diagnosis: Understanding User Perception in LLM-Supported Clinical Reasoning DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXX
Yuansong Xu, Yichao Zhu, Haokai Wang, Yuchen Wu, Yang Ouyang, Hanlu Li, Wenzhe Zhou, Xinyu Liu, Chang Jiang, Quan Li · 2026
Read “Do I Trust the AI?” Towards Trustworthy AI-Assisted Diagnosis: Understanding User Perception in LLM-Supported Clinical Reasoning DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXX by Yuansong Xu, Yichao Zhu, Haokai Wang, Yuchen Wu, Yang Ouyang, Hanlu Li, Wenzhe Zhou, Xinyu Liu, Chang Jiang, Quan Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-19540.txt.
Introduction
Yuansong Xu and nine colleagues study how doctors judge advice from artificial intelligence. Large language models can suggest diagnoses and support clinical reasoning. Yet doctors often struggle to tell when a model's advice is sound and when it is not. Trust that is too high or too low can both harm care.
The authors look at how users perceive these systems during real diagnostic work. Most past tests used fixed benchmarks and set tasks, which say little about daily practice. This study focuses on the human side of the exchange. Knowing how trust forms helps designers build tools that doctors can calibrate. The goal is AI support that clinicians use with clear eyes, neither blind faith nor blanket doubt.
Catalog record
| CREATOR | Yuansong Xu, Yichao Zhu, Haokai Wang, Yuchen Wu, Yang Ouyang, Hanlu Li, Wenzhe Zhou, Xinyu Liu, Chang Jiang, Quan Li |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 115 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-19540
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
“Do I Trust the AI?” Towards Trustworthy AI-Assisted Diagnosis: Understanding User Perception in LLM-Supported Clinical Reasoning DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 3815 Conference: Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems; April 13–17, 2026; Barcelona, Spain Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems (CHI ’26), April 13–17, 2026, Barcelona, Spain DOI: 10.1145/3772318.3790835 ISBN: 979-8-4007-2278-3/2026/04 CCS: Human-centered computing Empirical studies in interaction design Yuansong Xu, Yichao Zhu, Haokai Wang, Yuchen Wu, Yang Ouyang, Hanlu Li, Wenzhe Zhou, Xinyu Liu, Chang Jiang, Quan Li Abstract . Large language models (LLMs) have shown considerable potential in supporting medical diagnosis. However, their effective integration into clinical workflows is hindered by physicians’ difficulties in perceiving and trusting LLM capabilities, which often results in miscalibrated trust. Existing model evaluations primarily emphasize standardized benchmarks and predefined tasks, offering limited insights into clinical reasoning practices. Moreover, research on human–AI collaboration has rarely examined physicians’ perceptions of LLMs’ clinical reasoning capability. In this work, we investigate how physicians perceive LLMs’ capabilities in the clinical reasoning process. We designed clinical cases, collected the corresponding analyses, and obtained evaluations from physicians (N=37) to quantitatively represent their perceived LLM diagnostic capabilities. By comparing the perceived evaluations with benchmark performance, our study highlights the aspects of clinical reasoning that physicians value and underscores the limitations of benchmark-based evaluation. We further discuss the implications of opportunities for enhancing trustworthy collaboration between physicians and LLMs in LLM-supported clinical reasoning. † † cc-license: by 1. Introduction With the rapid advancement of technology, large language models (LLMs) are increasingly being integrated into medical practice, demonstrating great potential in diagnostic tasks. By leveraging the capabilities of LLMs to retrieve relevant patient data and medical literature ( 97 ) , identify complex symptom patterns ( 61 ) , and generate personalized recommendations ( 8 ) , their integration into diagnostic workflows can help reduce decision-making errors, improve diagnostic efficiency, and support informed clinical decisions. Despite the demonstrated effectiveness of LLMs in medical diagnosis, realizing these benefits and their full potential depends critically on effective collaboration with physicians and the integration of these tools into clinical diagnosis reasoning, which is defined as the iterative process of gathering evidence, forming and evaluating diagnostic hypotheses, and making treatment decisions ( 27 ; 9 ) . A key factor in this collaboration is how physicians perceive and trust the results provided by LLMs ( 112 ; 59 ) . Trust, defined as the willingness to rely on others under uncertainty ( 81 ) , has been shown to depend on users’ perception of LLM capabilities ( 57 ; 89 ) . For example, when physicians perceive LLMs as highly capable, they tend to be more willing to adopt their suggestions, which may lead to increased trust and a higher likelihood of accepting outputs without critical evaluation ( 5 ; 19 ) . Conversely, when LLMs are perceived as less capable, physicians may exhibit lower levels of trust and become less inclined to incorporate their suggestions into the diagnostic process ( 98 ) . Therefore, a central challenge in enabling effective physician-LLM collaboration within clinical reasoning lies in the difficulty physicians face in accurately perceiving the LLMs’ capabilities (e.g., due to model opacity or performance variability), which consequently hinders the development of appropriate trust in the results and ultimately limits the safe and effective integration of LLMs into clinical workflows. While existing studies have demonstrated the performance of models like GPT-4 ( 1 ) , Claude 3 ( 4 ) , and Gemini 1.5 ( 25 ) , these evaluations are often based on standardized benchmarks and predefined tasks such as question-answering or multiple-choice ( 41 ; 39 ; 68 ) . However, real-world clinical diagnosis is more complex and dynamic, involving clinical reasoning-based differential diagnosis that includes forming initial hypotheses, gathering evidence through inquiry, and updating decisions based on new information ( 44 ; 11 ) . In clinical practice, physicians evaluate not only diagnostic outcomes but also consider the reasoning process. The adoption and trust of LLMs follow the same principle—less about absolute correctness, more about clinical validity: alignment with real workflows, responsiveness to patient and clinician needs, and assurance of medical risk mitigation. Therefore, existing metrics fail to capture the depth and dynamic nature of clinical reasoning in practice. In the absence of clinically grounded evaluation frameworks, physicians often resort to subjective judgments to assess LLM capabilities. Yet without standardized criteria and workflow, such assessments are inconsistent and unreliable. This unreliability prevents physicians from forming accurate perceptions of LLMs’ diagnostic competence. Crucially, as established earlier, trust in LLMs fundamentally depends on users’ perception of their capabilities ( 57 ; 89 ) . Therefore, the resulting ambiguity in assessing these capabilities hinders the calibration of appropriate trust. In high-risk clinical reasoning, where informed decision-making is critical, a lack of trust significantly limits the potential for effective physician–LLM collaboration, ultimately undermining models’ capability to enhance diagnostic outcomes. Although prior studies have explored various aspects of human-LLM collaboration, including information presentation, interaction, and explanation integration ( 7 ; 83 ; 53 ; 49 ; 104 ; 86 ) , important gaps remain. First, existing research suggests that users tend to assess an LLM’s capability based on the perceived value of its responses, which in turn influences their trust in the model ( 89 ; 24 ) . LLMs that generate responses perceived as more valuable and actionable are generally considered to be more capable and trustworthy. However, in the specific context of clinical reasoning, the criteria that define the “value” of an LLM response (e.g., diagnostic accuracy, clinical relevance, coherence of reasoning) remain insufficiently defined. Second, while physicians’ subjective perceptions of LLM capabilities offer valuable insights that complement benchmark-based evaluations, the relationship between these two perspectives and potential discrepancies between them has received limited attention. Previous research on AI-assisted decision-making suggests that effective trust calibration requires aligning user trust with specific performance indicators, such as accuracy, confidence, or consistency ( 30 ; 77 ; 109 ) . Such calibration enables users to determine when to rely on AI recommendations and when to exercise caution ( 14 ; 76 ; 58 ) . However, given the unclear relationship between users’ subjective assessments of model capability and objective performance metrics, it remains uncertain how such subjective perceptions can be reliably leveraged to calibrate trust in AI systems. Recognizing potential misalignments between subjective assessment and objective performance of LLM is essential, as such divergences can reveal limitations in current benchmark-based evaluations and inform the development of more robust, clinically grounded frameworks for evaluating and deploying trustworthy AI-assisted diagnostic tools. In this work, we aim to understand physicians’ perceptions of LLMs’ clinical reasoning capabilities and examine their consistency with benchmark evaluations, to advance trustworthy AI-assisted diagnosis. Given that physicians often base their trust in LLMs on subjective perceptions of the model’s capabilities, we aim to develop a structured evaluation approach that integrates these perceptions with objective performance metrics. This approach aims to complement traditional benchmark-based evaluations, which rely solely on objective metrics, and to better support physicians in forming accurate subjective