MedCoT-RAG: Causal Chain-of-Thought RAG for Medical Question Answering
Ziyu Wang, Elahe Khatibi, Amir M. RahmaniUniversity of California, Irvine, USA, {ziyuw31, ekhatibi, amirr1}@uci.edu · 2025
Read MedCoT-RAG: Causal Chain-of-Thought RAG for Medical Question Answering by Ziyu Wang, Elahe Khatibi, Amir M. RahmaniUniversity of California, Irvine, USA, {ziyuw31, ekhatibi, amirr1}@uci.edu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-15849.txt.
Introduction
Ziyu Wang, Elahe Khatibi, and Amir Rahmani build a question-answering system for medicine that reasons in causal steps. Their paper is from two thousand twenty-five. Language models often answer medical questions with shallow guesses or made-up facts. Fetching source text helps, but plain similarity search can pull the wrong evidence for a diagnosis task.
Their system searches for documents that match the causal chain behind a case, not just its keywords. It then prompts the model to reason step by step along that chain, like a clinician would. Tests on three medical question sets beat plain retrieval systems by up to about ten percent. Gains over tuned medical systems were also clear. The design improves accuracy and makes the reasoning easier to inspect.
Catalog record
| CREATOR | Ziyu Wang, Elahe Khatibi, Amir M. RahmaniUniversity of California, Irvine, USA, {ziyuw31, ekhatibi, amirr1}@uci.edu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 18 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-15849
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MedCoT-RAG: Causal Chain-of-Thought RAG for Medical Question Answering Ziyu Wang, Elahe Khatibi, Amir M. RahmaniUniversity of California, Irvine, USA, {ziyuw31, ekhatibi, amirr1}@uci.edu Abstract Large language models (LLMs) have shown promise in medical question answering but often struggle with hallucinations and shallow reasoning, particularly in tasks requiring nuanced clinical understanding. Retrieval-augmented generation (RAG) offers a practical and privacy-preserving way to enhance LLMs with external medical knowledge. However, most existing approaches rely on surface-level semantic retrieval and lack the structured reasoning needed for clinical decision support. We introduce MedCoT-RAG , a domain-specific framework that combines causal-aware document retrieval with structured chain-of-thought prompting tailored to medical workflows. This design enables models to retrieve evidence aligned with diagnostic logic and generate step-by-step causal reasoning reflective of real-world clinical practice. Experiments on three diverse medical QA benchmarks show that MedCoT-RAG outperforms strong baselines by up to 10.3% over vanilla RAG and 6.4% over advanced domain-adapted methods, improving accuracy, interpretability, and consistency in complex medical tasks. I Introduction Large language models (LLMs) have demonstrated strong performance across a wide range of medical NLP tasks, including question answering, clinical summarization, and diagnostic reasoning [ 1 , 2 , 3 ] . Their ability to synthesize biomedical knowledge from large-scale corpora has made them appealing as decision-support tools in healthcare. However, despite this progress, LLMs remain prone to hallucinations, inconsistencies, and reasoning failures—particularly when applied to complex clinical tasks that require causal understanding [ 4 , 5 ] . These challenges are exacerbated in safety-critical domains like medicine, where outputs must be not only accurate but also clinically plausible and interpretable. Retrieval-augmented generation (RAG) has emerged as a promising approach to reduce hallucinations by grounding LLM outputs in relevant external knowledge [ 6 , 7 ] . In medical settings, RAG provides a flexible, privacy-preserving way to enhance LLMs with domain-specific knowledge from biomedical literature, clinical guidelines, or structured EHRs—without the need for costly and often ineffective fine-tuning. However, most medical RAG systems face two core limitations. First, they rely on semantic similarity for retrieval, which often surfaces contextually similar but clinically irrelevant information [ 1 , 8 ] . Second, they lack structured reasoning, producing unstructured answers that fail to reflect how clinicians generate differential diagnoses, assess treatments, or integrate evidence [ 3 , 9 ] . Clinical reasoning in real-world practice is structured and causal: physicians link symptoms to pathophysiology, evaluate differential diagnoses based on key features, and integrate evidence with patient-specific factors. These causal chains underpin real-world diagnostic reasoning but are largely missing from existing RAG pipelines, which focus on shallow context retrieval and unstructured generation. Even advanced medical RAG systems like MedRAG [ 10 ] and those leveraging strong biomedical embeddings such as MedCPT [ 11 ] or LinkBERT [ 12 ] retrieve context without explicitly modeling causal relationships. While chain-of-thought (CoT) prompting has improved reasoning in general LLMs [ 13 ] , its use in medical settings often depends on generic templates that fail to capture the structured, diagnostic reasoning employed by clinicians. Fig. 1 : Comparison of conventional RAG and MedCoT-RAG. Conventional RAG uses static queries and semantic retrieval, often yielding hallucinated, causally incorrect responses. MedCoT-RAG combines causal-aware retrieval and structured reasoning via CoT splitting to produce causally consistent, clinically coherent answers. In this work, we present MedCoT-RAG , a domain-specific RAG framework designed to improve medical question answering through structured causal reasoning and clinically grounded retrieval. MedCoT-RAG introduces a causal CoT prompting strategy that guides LLMs through four stages aligned with clinical workflows: symptom analysis, causal pathophysiology, differential diagnosis, and evidence synthesis. Simultaneously, it performs causal-aware retrieval , prioritizing documents that capture diagnostic reasoning, treatment mechanisms, and clinically relevant cause-effect relationships—beyond surface-level semantic similarity. This dual design enables MedCoT-RAG to generate medically coherent, interpretable, and reliable outputs, addressing key limitations of current biomedical RAG systems. We evaluate MedCoT-RAG on three diverse and challenging medical QA benchmarks spanning various biomedical domains and question formats. Our evaluation includes strong baselines: a plain LLM without retrieval, vanilla RAG, RAG with MedCPT embeddings, and RAG with MedCPT combined with basic CoT prompting. We also compare against recent domain-specific retrieval-augmented methods, including RGAR [ 14 ] and RAG 2 [ 2 ] . Across all benchmarks, MedCoT-RAG consistently improves accuracy, reasoning quality, and clinical alignment. These results demonstrate the effectiveness of our structured causal reasoning framework and reinforce the importance of accuracy-based evaluation for complex medical question answering tasks. • We propose MedCoT-RAG, a domain-specific RAG framework that combines causal-aware retrieval and structured CoT prompting to better capture clinical reasoning. • We design a novel retrieval scoring function that prioritizes documents based on causal medical relevance, including pathophysiological links, treatment rationales, and diagnostic mechanisms. • We demonstrate the effectiveness of MedCoT-RAG on three challenging medical QA benchmarks, showing consistent improvements in accuracy, reasoning depth, and clinical alignment over strong baselines. II Method MedCoT-RAG is a modular RAG framework designed for medical question answering. It is built upon the observation that effective clinical reasoning requires structured causal inference, not merely surface-level fact retrieval. To address this, MedCoT-RAG jointly aligns document retrieval and answer generation under a unified diagnostic reasoning framework. As illustrated in Fig. 2 , MedCoT-RAG consists of two key components: (1) a causal-aware retrieval module that selects clinically salient documents based on both semantic relevance and causal reasoning cues, and (2) a structured prompting scheme that guides the LLM through step-by-step causal reasoning aligned with clinical workflows. II-A Causal-Aware Retrieval Many clinical questions hinge on subtle causal relationships rather than shallow lexical similarities. Traditional semantic retrieval methods often retrieve contextually similar but diagnostically irrelevant documents [ 3 , 5 ] . To overcome this, we design a causal-aware retrieval mechanism that incorporates both semantic similarity and causal relevance. Specifically, we define the retrieval score for a document d given query q as: s(d,q)=\alpha\cdot\text{sim}(q,d)+\beta\cdot\psi(d), where \text{sim}(q,d) denotes the cosine similarity between query and document embeddings, and \psi(d) represents a causal relevance score that estimates the diagnostic utility of the document. The causal score \psi(d) is computed by detecting medically relevant causal patterns in the text, such as causal operators (“leads to”, “causes”, “mediates”), treatment-action-effect relations, and mechanistic disease explanations. We implement this via a weighted keyword matching scheme, normalized by document length to avoid verbosity bias. We initialize our document encoder with the MedCPT biomedical embedding model [ 11 ] and pre-embed a multi-source medical corpus comprising PubMed abstracts, StatPearls clinical reference articles, curated medical textbooks [ 15 ] , and Wikipedia medical pages. During retrieval, the query is optionally enhanced with clinical modifiers before being encoded. We then select the top- k documents based on the composite score s(d,q) , prioritizing both relevance and interpretability for downstream reasoning. Fig. 2 : MedCoT-RAG pipeline combining causal-aware retrieval and four-step causal CoT prompting for clinically coherent, causally consistent answers. II-B Structured Generation via Causal CoT After retrieving the top-ranked documents \mathcal{D}_{q} , we construct a structured reasoning prompt P(q,\mathcal{D}_{q}) that embeds both the retrieved content and a domain-specific diagnostic schema. Unlike generic chain-of-thought (CoT) prompting [ 13 ] , our method explicitly follows a clinically inspired reasoning workflow: (1) identifying clinical features, (2) explaining underlying causal mechanisms, (3) evaluating differential diagnoses, and (4) synthesizing evidence into a