U2-BENCH: Benchmarking Large Vision-Language Models on Ultrasound Understanding

Anjie Le, Henan Liu, Yue Wang, Zhenyu Liu, Rongkun Zhu, Taohan Weng, Jinze Yu, Boyang Wang, Yalun Wu, Kaiwen Yan, Quanlin Sun, Meirui Jiang, Jialun Pei, Siya Liu, Haoyun Zheng, Zhoujun Li, J. Alison Noble, Jacques Souquet, Xiaoqing Guo, Manxi Lin, Hongcheng Guo · 2025

Read U2-BENCH: Benchmarking Large Vision-Language Models on Ultrasound Understanding by Anjie Le, Henan Liu, Yue Wang, Zhenyu Liu, Rongkun Zhu, Taohan Weng, Jinze Yu, Boyang Wang, Yalun Wu, Kaiwen Yan, Quanlin Sun, Meirui Jiang, Jialun Pei, Siya Liu, Haoyun Zheng, Zhoujun Li, J. Alison Noble, Jacques Souquet, Xiaoqing Guo, Manxi Lin, Hongcheng Guo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-17779.txt.

Introduction

Ultrasound scans are cheap and everywhere, but reading them well takes skill, and modern vision-language models have barely been tested on them. Anjie Le and co-authors build U2-BENCH, the first broad benchmark for such models on ultrasound. It holds 7,241 cases from 15 body regions, arranged into eight clinic-style tasks across 50 use settings, from diagnosis and view naming to finding lesions, estimating measures, and writing reports.

The team runs 23 leading models, both open and commercial, general and medical. Models handled whole-image classification well but stumbled on spatial reasoning and on writing clinical text. The authors offer the benchmark, published in 2025, as a shared yardstick to speed progress in a corner of medical imaging that models have largely skipped.

Catalog record

CREATORAnjie Le, Henan Liu, Yue Wang, Zhenyu Liu, Rongkun Zhu, Taohan Weng, Jinze Yu, Boyang Wang, Yalun Wu, Kaiwen Yan, Quanlin Sun, Meirui Jiang, Jialun Pei, Siya Liu, Haoyun Zheng, Zhoujun Li, J. Alison Noble, Jacques Souquet, Xiaoqing Guo, Manxi Lin, Hongcheng Guo
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE103 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-17779

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

U2-BENCH: Benchmarking Large Vision-Language Models on Ultrasound Understanding Anjie Le, Henan Liu, Yue Wang, Zhenyu Liu, Rongkun Zhu, Taohan Weng, Jinze Yu, Boyang Wang, Yalun Wu, Kaiwen Yan, Quanlin Sun, Meirui Jiang, Jialun Pei, Siya Liu, Haoyun Zheng, Zhoujun Li, J. Alison Noble, Jacques Souquet, Xiaoqing Guo, Manxi Lin, Hongcheng Guo Abstract Ultrasound is a widely-used imaging modality critical to global healthcare, yet its interpretation remains challenging due to variability in image quality caused by operator dependency, noise, and anatomical complexity. Although large vision-language models (LVLMs) have demonstrated impressive multimodal capabilities across natural and medical domains, their performance on ultrasound remains largely unexplored. We introduce U2-BENCH, the first comprehensive benchmark to evaluate LVLMs on ultrasound understanding across classification, detection, regression, and text generation tasks. U2-BENCH aggregates 7,241 cases spanning 15 anatomical regions and defines 8 clinically inspired tasks, such as diagnosis , view recognition , lesion localization , clinical value estimation , and report generation , across 50 ultrasound application scenarios. We evaluate 20 state-of-the-art LVLMs, both open- and closed-source, general-purpose and medical-specific. Our results reveal strong performance on image-level classification, but persistent challenges in spatial reasoning and clinical language generation. U2-BENCH establishes a rigorous and unified testbed to assess and accelerate LVLM research in the uniquely multimodal domain of medical ultrasound imaging. 1 1 1 Codes are available at: https://anonymous.4open.science/r/U2-Bench-F781/VLMEVALKIT/ Data is available at: https://huggingface.co/datasets/DolphinAI/u2-bench/tree/main 1 Introduction “In diagnostics, the eye sees what the mind knows – true understanding requires merging image patterns with clinical wisdom.” – William Osler Figure 1 : Examples of the 8 benchmark tasks in U2-BENCH across diverse anatomical regions. Each callout, consisting of the question prompt, expected output format, and sample output, highlights a representative ultrasound application scenario of the corresponding task. Tasks involve disease diagnosis (DD), view recognition and assessment (VRA), lesion localization (LL), organ detection (OD), keypoint detection (KD), clinical value estimation (CVE), report generation (RG) and caption generation (CG). Ultrasound (US) is one of the most widely used imaging modalities in global healthcare — essential in obstetrics, emergency medicine, cardiology, and low-resource settings — while its interpretation remains notoriously difficult [ 29 ] . Compared to modalities such as computed tomography (CT), magnetic resonance imaging (MRI), positron emission tomography (PET), and whole-slide imaging (WSI), which offer higher spatial resolution, consistent image quality, and standardized anatomical views, ultrasound is real-time and low-cost but highly operator-dependent and frequently affected by imaging artifacts [ 65 ] . In addition, in contrast to these modalities, US is dynamically presenting three-dimensional (3D) anatomies in image sequences. Therefore, accurate interpretation of US demands not only visual pattern recognition in the images, but also an understanding of anatomy and capturing of dynamic spatial-context reasoning, typically requiring extensive prior domain expertise [ 74 ] . These challenges have limited the applicability of earlier artificial intelligence (AI) models. However, recent advances in medical large vision-language models (LVLMs) have shown promise in overcoming these barriers [ 17 , 80 , 31 ] , potentially offering a robust multimodal understanding of complex, noisy, and context-rich ultrasound data. While progress in medical LVLM has been rapid, most previous models and benchmarks focus on those less noisy and static imaging modalities [ 34 , 32 , 69 ] , leaving the complexities of ultrasound largely unaddressed. Prior efforts in ultrasound AI are typically based on small, task-specific datasets [ 81 ] , such as fetal plane identification [ 25 ] or pathology segmentation [ 33 , 59 ] . As model capabilities grow, a public, balanced benchmark for ultrasound understanding is needed to evaluate whether emerging LVLMs can generalize beyond static medical vision tasks, to those requiring spatial reasoning and contextual understanding of anatomical structures. To address these challenges, we introduce U2-Bench , the first benchmark holistically evaluating current LVLMs for ultrasound understanding across diverse tasks and anatomies. The dataset we use comprises 7,241 cases across 15 anatomical regions, involving breast, heart, lung, etc, covering 8 diverse clinical tasks and 50 application scenarios. Each task belongs to one of the four categories: (1) classification (i.e., disease diagnosis, view recognition and assessment), (2) detection (i.e., lesion localization, organ detection, keypoint detection), (3) regression (i.e., clinical value estimation), (4) text generation (i.e., report generation, caption generation). Samples are selected to ensure balance across data sources, anatomies, and task types, to enable robust evaluation and alleviate dataset-specific bias. Several examples in our U2-Bench are shown in Fig. 1 . We benchmark 20 LVLMs, including both open- and closed-source, general-purpose and medical-specialized models, on a diverse set of US tasks. U2-Bench makes the following key contributions: • Comprehensive Dataset: We release the first publicly available benchmark comprising 7,241 ultrasound cases spanning 15 anatomies and 8 clinical tasks, covering 50 application scenarios. Each case is annotated with task-aligned labels in a unified format and paired with carefully designed prompts, enabling standardized and reproducible evaluation. • Task Suite and Evaluation: We define an eight-task taxonomy spanning disease diagnosis , view recognition and assessment , lesion localization , organ detection , keypoint detection , clinical value estimation , report generation , and caption generation . Each task reflects real-world clinical workflows and is paired with standard evaluation metrics. We also introduce an aggregate metric to provide a unified assessment of each model’s overall capability in ultrasound understanding. • Empirical Insights: We conduct the first large-scale evaluation of LVLMs on ultrasound, uncovering consistent trends across model families: models achieve strong performance on image-level disease diagnosis and clinical value estimation tasks, but degrade on spatial reasoning tasks such as view recognition and organ detection. Clinical report generation tasks remain particularly challenging. Performance gains from model scaling can be limited, and compact models occasionally outperform larger ones on certain tasks, suggesting that targeted training may be more impactful than scale alone in ultrasound understanding. 2 Related Work Recent advances in LVLMs have catalyzed new opportunities in medical image understanding. Our work intersects two key directions: (1) the construction of benchmarks for systematically evaluating LVLM capabilities across multimodal medical tasks, and (2) unlocking the potential of LVLMs for ultrasound imaging. Large Vision-Language Models. LVLMs such as GPT-4V [ 54 ] , Claude [ 5 ] , Gemini [ 4 ] , DeepSeek-VL [ 21 ] , LLaVA [ 45 ] , Qwen-VL [ 9 ] , and MiniGPT4 [ 89 ] have emerged as general-purpose multimodal systems capable of handling tasks like image captioning, visual question answering, and multimodal reasoning. These models are trained on large-scale image-text pairs [ 66 , 62 ] , and their performance has been extensively evaluated in domains such as question answering, mathematics, and science [ 15 , 70 , 75 , 30 , 46 ] . However, their clinical reliability remains underexplored. To address this gap, several medical-specialized LVLMs have been proposed. MiniGPT-Med [ 78 ] focuses on X-ray, CT, and MRI for tasks such as medical report generation, VQA, and disease identification. RadFM [ 79 ] further supports both 2D and 3D modalities. MedDr [ 26 ] extends to radiology, pathology, dermatology, retinography, and endoscopy, introducing a retrieval-augmented diagnosis strategy. Yet, these models exclude ultrasound. Med-Gemini [ 71 ] spans numerous modalities including ultrasound, though its capability in this domain is limited to caption generation. Multimodal Benchmarks for

Read the complete text