UltrasoundAgents: Hierarchical Multi-Agent Evidence-Chain Reasoning for Breast Ultrasound Diagnosis
Yali Zhu, Kang Zhou ∗, Dingbang Wu, Gaofeng Meng · 2026
Read UltrasoundAgents: Hierarchical Multi-Agent Evidence-Chain Reasoning for Breast Ultrasound Diagnosis by Yali Zhu, Kang Zhou ∗, Dingbang Wu, Gaofeng Meng (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-10852.txt.
Introduction
Yali Zhu and coauthors build a team of AI agents that work together. The team reads breast ultrasound images in the order a clinician uses. It first finds the lesion. It then assesses its signs. It then assigns a category. The main agent joins the four structured signs to output a BI-RADS category. It also gives a benign-or-malignant call, and each step stays reviewable. Training runs in stages, with the sign agent taught first. Corrective trajectories are added later. Tests show gains over strong vision-language baselines.
Catalog record
| CREATOR | Yali Zhu, Kang Zhou ∗, Dingbang Wu, Gaofeng Meng |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 23673 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.10852
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
UltrasoundAgents: Hierarchical Multi-Agent Evidence-Chain Reasoning for Breast Ultrasound Diagnosis Yali Zhu, Kang Zhou ∗, Dingbang Wu, Gaofeng Meng Abstract Breast ultrasound diagnosis typically proceeds from global lesion localization to local sign assessment and then evidence integration to assign a BI-RADS category and determine benignity or malignancy. Many existing methods rely on end-to-end prediction or provide only weakly grounded evidence, which can miss fine-grained lesion cues and limit auditability and clinical review. To align with the clinical workflow and improve evidence traceability, we propose a hierarchical multi-agent framework, termed UltrasoundAgents. A main agent localizes the lesion in the full image and triggers a crop-and-zoom operation. A sub-agent analyzes the local view and predicts four clinically relevant attributes, namely echogenicity pattern, calcification, boundary type, and edge (margin) morphology. The main agent then integrates these structured attributes to perform evidence-based reasoning and output the BI-RADS category and the malignancy prediction, while producing reviewable intermediate evidence. Furthermore, hierarchical multi-agent training often suffers from error propagation, difficult credit assignment, and sparse rewards. To alleviate this and improve training stability, we introduce a decoupled progressive training strategy. We first train the attribute agent, then train the main agent with oracle attributes to learn robust attribute-based reasoning, and finally apply corrective trajectory self-distillation with spatial supervision to build high-quality trajectories for supervised fine-tuning, yielding a deployable end-to-end policy. Experiments show consistent gains over strong vision-language baselines in diagnostic accuracy and attribute agreement, together with structured evidence and traceable reasoning. 1 Introduction Breast ultrasound (BUS) is widely used for breast cancer screening and early diagnosis. Clinicians typically follow a coarse-to-fine reading workflow: localize a suspicious lesion, examine fine-grained signs (e.g., morphology and echo patterns), and integrate evidence to assign a BI-RADS category and decide benignity or malignancy [ 1 , 2 , 3 ] . A practical computer-aided diagnosis system therefore needs not only accurate predictions, but also intermediate evidence that is traceable and consistent with the clinical decision process [ 6 ] . Most BUS methods use end-to-end prediction or tightly coupled multi-stage pipelines [ 7 ] . While interpretable multi-task and multi-stage designs have been explored [ 1 , 6 ] , the produced evidence is often only weakly explicit, lacking a clear evidence-chain to explain how local cues support BI-RADS grading and malignancy decisions. Predicting BI-RADS categories and/or generating BI-RADS descriptor terms as structured outputs has also been studied [ 4 , 5 ] , but these approaches are usually trained with static supervision and do not explicitly model the dynamic, sequential evidence acquisition and integration process inherent in clinical practice. Recent vision-language models (VLMs) enable more explicit reasoning via textual rationales and structured attribute descriptions [ 2 ] . However, supervised fine-tuning (SFT) for reasoning often depends on additional rationale annotations or teacher trajectories and may inherit biases from static supervision. Reinforcement learning (RL) can instead optimize towards outcome-level rewards and encourage the model to discover readable reasoning patterns without reference rationales [ 8 , 10 ] . For medical images, where key evidence can be small and sparse, incorporating ROI-focused observations during reasoning is also effective, and RL can incentivize such behavior in an end-to-end manner [ 9 , 11 , 12 ] . Nevertheless, learning localization, fine-grained perception, and high-level diagnostic reasoning within a single policy remains difficult: localization errors and perceptual noise shift downstream observations, increasing non-stationarity and complicating credit assignment [ 13 ] . Hierarchical multi-agent diagnosis provides a principled alternative by separating evidence acquisition from evidence integration, improving consistency and traceability [ 14 , 15 ] , yet stable training and distillation into deployable policies remain underexplored for unified BUS diagnosis. We propose UltrasoundAgents , a hierarchical multi-agent BUS diagnostic framework aligned with the clinical workflow. We explicitly decouple global localization and high-level evidence integration from fine-grained attribute perception. The main agent localizes lesions on the full image and integrates evidence to output BI-RADS and malignancy conclusions. The sub-agent focuses on the crop-and-zoom lesion view, recognizes clinically relevant attributes, and provides structured evidence to support the main agent. To ensure stability and robustness of hierarchical training, we further adopt a decoupled progressive training scheme. We first train an attribute specialist, then introduce oracle-guided curriculum reinforcement learning for the main agent to mitigate non-stationarity in hierarchical training, and finally construct high-quality supervised trajectories via corrective trajectory self-distillation and distill them into a deployable policy through SFT. At test time, the system relies only on predicted intermediate evidence while maintaining a consistent evidence chain and robust diagnostic decisions. Figure 1 : Hierarchical multi-agent architecture. The main agent analyzes the full image to localize the lesion, triggers a crop-and-zoom operation, and queries the sub-agent on the zoomed view to obtain structured attribute evidence. The main agent then integrates the global context and the attributes to predict malignancy and BI-RADS. This work makes three contributions. (1) We propose a hierarchical multi-agent diagnostic framework that separates full-image lesion localization and high-level evidence-based reasoning in a main agent from local clinical attribute recognition in a sub-agent, using crop-and-zoom to produce an auditable chain of ROI \rightarrow attributes \rightarrow diagnosis. To our knowledge, we present the first agent-based diagnostic framework for ultrasound diagnosis. (2) We introduce an oracle-guided curriculum RL strategy that trains the main agent with ground-truth attributes to learn stable attribute-based reasoning without perceptual noise, alleviating non-stationarity and error propagation. (3) We design a corrective trajectory self-distillation pipeline that refines RL-explored trajectories into supervised data and distills them via SFT into a deployable policy, improving reasoning consistency and localization-aware perception when predicted attributes are used at test time. 2 Method As shown in Fig. 1 , we propose a hierarchical multi-agent framework for breast ultrasound interpretation that follows a coarse-to-fine clinical workflow. The main design decouples global localization and high-level reasoning from fine-grained attribute perception, and uses an explicit crop-and-zoom operation to provide focused lesion observations. The framework contains a main agent A_{M} for full-image analysis, lesion localization, and evidence integration, and a sub-agent A_{S} specialized in attribute recognition. Training proceeds in three stages: RL for A_{S} , oracle-guided curriculum RL for A_{M} , and trajectory refinement followed by supervised fine-tuning (SFT), as illustrated in Fig. 2 . 2.1 Hierarchical multi-agent architecture Figure 2 : Three-stage training. Stage 1: RL trains A_{S} for attribute recognition. Stage 2: oracle-guided RL trains A_{M} with GT attributes for stable reasoning. Stage 3: refine trajectories and distill via SFT for robustness. Fig. 1 summarizes the interaction between the two agents. The input is a breast ultrasound image I\in\mathbb{R}^{H\times W\times C} . The outputs include a malignancy label y_{\text{mal}} and a BI-RADS category y_{\text{bi}} . Each image is associated with a ground-truth lesion box b^{\text{gt}} and four attributes: echogenicity pattern f_{echo} , calcification f_{calc} , boundary type f_{bound} , and edge (margin) morphology f_{edge} . Main agent ( A_{M} ): global localization and reasoning Given I , A_{M} performs global scanning and predicts an ROI box b=[x_{1},y_{1},x_{2},y_{2}] , then triggers crop-and-zoom to obtain a local lesion view I_{\text{crop}}=\mathrm{Crop}(I,b). (1) After receiving the attribute set from A_{S} , A_{M} combines global context from I with local clinical attributes and outputs the diagnosis Y=\{y_{\text{mal}},y_{\text{bi}}\} . Sub-agent ( A_{S} ): fine-grained attribute expert A_{S} operates on I_{\text{crop}} and predicts F=\{f_{echo},f_{calc},f_{bound},f_{edge}\}. (2) We