TAGS: A Test-Time Generalist–Specialist Framework with Retrieval-Augmented Reasoning and Verification
Jianghao Wu, Feilong Tang, Yulong Li, Ming Hu, Haochen Xue, Shoaib Jameel, Yutong Xie, Imran Razzak · 2025
Read TAGS: A Test-Time Generalist–Specialist Framework with Retrieval-Augmented Reasoning and Verification by Jianghao Wu, Feilong Tang, Yulong Li, Ming Hu, Haochen Xue, Shoaib Jameel, Yutong Xie, Imran Razzak (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-18283.txt.
Introduction
Jianghao Wu and co-authors introduce TAGS in this 2025 paper, a framework that helps language models answer medical questions. Simple prompting can give shallow or unstable answers, while models tuned only for medicine often fail on new kinds of cases. TAGS needs no retraining of the underlying models.
The framework pairs a broad generalist model with a medical specialist model, so both views inform each answer. A search module fetches helpful past examples at several levels, and a scorer checks how consistent the reasoning is before merging answers. Across nine medical test sets, TAGS raised the scores of several well-known models and beat many tuned medical models.
Catalog record
| CREATOR | Jianghao Wu, Feilong Tang, Yulong Li, Ming Hu, Haochen Xue, Shoaib Jameel, Yutong Xie, Imran Razzak |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-18283
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
TAGS: A Test-Time Generalist–Specialist Framework with Retrieval-Augmented Reasoning and Verification Jianghao Wu, Feilong Tang, Yulong Li, Ming Hu, Haochen Xue, Shoaib Jameel, Yutong Xie, Imran Razzak Abstract Recent advances such as Chain-of-Thought prompting have significantly improved large language models (LLMs) in zero-shot medical reasoning. However, prompting-based methods often remain shallow and unstable, while fine-tuned medical LLMs suffer from poor generalization under distribution shifts and limited adaptability to unseen clinical scenarios. To address these limitations, we present TAGS , a test-time framework that combines a broadly capable generalist with a domain-specific specialist to offer complementary perspectives without any model fine-tuning or parameter updates. To support this generalist–specialist reasoning process, we introduce two auxiliary modules: a hierarchical retrieval mechanism that provides multi-scale exemplars by selecting examples based on both semantic and rationale-level similarity, and a reliability scorer that evaluates reasoning consistency to guide final answer aggregation. TAGS achieves strong performance across nine MedQA benchmarks, boosting GPT-4o accuracy by 13.8%, DeepSeek-R1 by 16.8%, and improving a vanilla 7B model from 14.1% to 23.9%. These results surpass several fine-tuned medical LLMs, without any parameter updates. The code will be available at https://github.com/JianghaoWu/TAGS . † † footnotetext: Corresponding author. 1 Introduction Large Language Models (LLMs) have recently demonstrated promising capabilities in medical question answering (MedQA), achieving strong performance on a range of benchmark datasets Singhal et al. (2025) ; Jin et al. (2022) ; Chen et al. (2023a) ; Chen et al. (2023b) ; Zhou et al. (2023) ; Gao et al. (2024) . However, despite these advances, recent studies have shown that even state-of-the-art models frequently fail on complex cases requiring deep domain knowledge, multi-step reasoning, and generalization to out-of-distribution clinical scenarios Xu et al. (2024) ; Fan et al. (2025) ; Shi et al. (2024) . To improve LLM reasoning in MedQA, two major research directions have emerged. The first involves prompting-based strategies, such as Chain-of-Thought (CoT) Wei et al. (2022) and Multi-Agent Systems (MAS) Chen et al. (2025b) ; Chen et al. (2025a) , which guide LLMs through structured multi-step reasoning or simulate expert collaboration. However, empirical studies Tang et al. (2025) have revealed that interactive multi-agent reasoning is frequently brittle, with redundant outputs, unstable communication, and poor coordination undermining reliability. Recent benchmarks report high failure rates for such systems, with multi-agent discussions yielding limited improvements over single-agent baselines Cemri et al. (2025) . The second direction focuses on fine-tuning LLMs on domain-specific medical data, yielding specialist models like HuatuoGPT Chen et al. (2024c) and MedLLaMA Qiu et al. (2024) . Although these models perform well on in-distribution tasks, they tend to overfit their training domains and exhibit limited adaptability to emerging knowledge or unseen questions Yang et al. (2024b) ; Ye et al. (2024) . While prior work has made progress on reasoning, retrieval, and domain adaptation, these components are typically developed in isolation and lack integration into a unified, inference-time framework. As a result, existing approaches fall short in three critical aspects for robust MedQA: acquiring up-to-date medical knowledge, supporting diverse and complementary reasoning strategies, and ensuring answer reliability under distribution shift, all without relying on any parameter updates. This motivates our work: designing a structured, test-time reasoning framework that unifies generalist and specialist perspectives with retrieval-augmented prompting and verification. Our motivation stems from the observation that single-agent prompting often lacks the depth and diversity required for complex medical reasoning, while multi-agent systems tend to produce redundant or inconsistent outputs due to unclear role assignments and weak complementarity. To address these limitations, we propose TAGS (Test-time Generalist–Specialist framework with retrieval-augmented reasoning and verification), a modular, inference-only framework that enhances medical question answering through structured reasoning collaboration. At its core, TAGS relies on a Generalist–Specialist Reasoning Collaboration ( GSRC ) module, which facilitates complementary reasoning between a generalist agent and a specialist agent. Each agent brings a different perspective to the problem: the generalist offers broad clinical insights, while the specialist contributes precise, domain-specific reasoning. This collaboration is designed to produce diverse and accurate reasoning paths for challenging medical questions. To support GSRC, we introduce two auxiliary modules. First, to promote diverse and contextually grounded reasoning, we propose a Hierarchical Retrieval Augmentation ( HRA ) mechanism that leverages external, high-quality medical Chain-of-Thought (CoT) exemplars. HRA operates in two stages: it first retrieves semantically relevant CoT exemplars based on the input question–answer (QA) pair to serve as initial references for both agents. Then, after the agents generate their initial reasoning steps, HRA performs a second round of retrieval based on the generated rationales to obtain reasoning-level exemplars that further guide and enrich subsequent inference. Second, to ensure the reliability of the generated answers along with their reasoning chains, we introduce an Uncertainty-Aware Answer Aggregation ( UAAA ) module. UAAA evaluates the consistency between the reasoning path and the final answer produced by each agent, and aggregates the outputs based on a confidence score and inter-agent agreement, yielding a robust and interpretable final prediction. This design effectively mitigates issues common in previous methods, such as insufficient reasoning depth and diversity in single-agent prompting and redundant or conflicting outputs arising from ambiguous roles in multi-agent systems, ultimately producing more accurate and robust medical reasoning outcomes. Extensive experiments across nine MedQA benchmarks validate the robustness and adaptability of TAGS under both non-fine-tuned and fine-tuned settings. Our method consistently outperforms strong prompting and agent-based baselines across multiple foundation models, including GPT-4o (18.0% to 31.8%), DeepSeek-R1 (27.2% to 44.0%), and Qwen-2.5-7B (14.1% to 23.9%). Notably, we show that even without retrieving semantically similar exemplars, TAGS maintains strong performance by exposing models to diverse reasoning patterns rather than specific answers. Figure 1: Overview of the proposed TAGS framework. The architecture consists of three modules: (A) HRA (Hierarchical Retrieval Augmentation) , a two-stage retrieval process that selects semantically relevant exemplars ( \mathcal{T}_{1} ) and refines them based on rationale alignment ( \mathcal{T}_{G,2} , \mathcal{T}_{S,2} ). (B) GSRC (Generalist-Specialist Reasoning Collaboration) employs dual-agent reasoning across two rounds, generating four candidate (Rationale, Answer) pairs. (C) UAAA (Uncertainty-Aware Answer Aggregation) assesses rationale consistency using the RCE and aggregates reliability scores ( c ) to determine the final answer. 2 Related Work 2.1 Medical Question Answering Medical question answering (MedQA) aims to predict accurate answers to domain-specific clinical or biomedical questions, often posed in a multiple-choice format. Existing benchmarks span a variety of formats and reasoning challenges, including clinical exam-style questions, evidence-based inference, and multi-subject distractor-rich scenarios Pal et al. (2022) . Recent approaches leverage large language models (LLMs) or chain-of-thought prompting to enhance reasoning Singhal et al. (2025) . MedCoT Liu et al. (2024) explicitly integrates multi-step rationale generation with hierarchical expert feedback. Concurrently, biomedical LLMs such as MedLLaMA † † † https://huggingface.co/johnsnowlabs/ , HuatuoGPT Chen et al. (2024c) , and OpenBioLLM Pal and Sankarasubbu (2024) have achieved strong zero-shot or few-shot performance on MedQA benchmarks. However, these models typically rely on direct answer generation and lack explicit mechanisms for multi-agent reasoning or consistency verification. In contrast, our method introduces a retrieval-augmented multi-agent framework that performs staged reasoning and employs a dedicated verifier to assess the reliability of generated answers, promoting robustness and interpretability. 2.2 Retrieval-Augmented Reasoning Retrieval-augmented reasoning enhances prediction quality, factual consistency, and interpretability by incorporating external knowledge into the