ReXSonoVQA: A Video QA Benchmark for Procedure-Centric Ultrasound Understanding

Xucheng Wang, BS, Xiaoman Zhang, PhD, Sung Eun Kim, MD, Ankit Pal, BS, Pranav Rajpurkar, PhD · 2026

Read ReXSonoVQA: A Video QA Benchmark for Procedure-Centric Ultrasound Understanding by Xucheng Wang, BS, Xiaoman Zhang, PhD, Sung Eun Kim, MD, Ankit Pal, BS, Pranav Rajpurkar, PhD (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-10916.txt.

Introduction

Wang and four colleagues introduce ReXSonoVQA, a video benchmark for procedural ultrasound understanding. Its 514 clips hold 514 questions across three skills: Action-Goal Reasoning, Artifact Resolution and Optimization, and Procedure Context and Planning. Ultrasound acquisition demands skilled probe handling and live adjustments, yet benchmarks test only still images. Zero-shot tests of Gemini 3 Pro, Qwen3.5-397B, LLaVA-Video-72B, and Seed 2.0 Pro show models extract some facts, but troubleshooting stays hard with minimal gains over text-only baselines. The benchmark aims to advance ultrasound training, guidance, and automation.

Catalog record

CREATORXucheng Wang, BS, Xiaoman Zhang, PhD, Sung Eun Kim, MD, Ankit Pal, BS, Pranav Rajpurkar, PhD
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45041
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.10916

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ReXSonoVQA: A Video QA Benchmark for Procedure-Centric Ultrasound Understanding Xucheng Wang, BS, Xiaoman Zhang, PhD, Sung Eun Kim, MD, Ankit Pal, BS, Pranav Rajpurkar, PhD Abstract Ultrasound acquisition requires skilled probe manipulation and real-time adjustments. Vision-language models (VLMs) could enable autonomous ultrasound systems, but existing benchmarks evaluate only static images, not dynamic procedural understanding. We introduce ReXSonoVQA , a video QA benchmark with 514 video clips and 514 questions (249 MCQ, 265 free-response) targeting three competencies: Action-Goal Reasoning, Artifact Resolution & Optimization, and Procedure Context & Planning. Zero-shot evaluation of Gemini 3 Pro, Qwen3.5-397B, LLaVA-Video-72B, and Seed 2.0 Pro shows VLMs can extract some procedural information, but troubleshooting questions remain challenging with minimal gains over text-only baselines, exposing limitations in causal reasoning. ReXSonoVQA enables developing perception systems for ultrasound training, guidance, and robotic automation. 1 [2pt] 1 [2pt] 1 [2pt] 1 [2pt] 1 [2pt] Data and Code Availability The dataset comprises publicly available ultrasound demonstration videos sourced from YouTube. We provide YouTube URLs with timestamps and regions of interest, following established practices in video benchmark curation (e.g., HowTo100M ( 16 ) , AVOS ( 6 ) ). All video frames shown in figures throughout this paper correspond to entries in the dataset; source URLs are provided in the repository. The ReXSonoVQA’s dataset and code are publicly available at https://github.com/rajpurkarlab/RexSonoVQA . Institutional Review Board (IRB) IRB was not required for this study as the data consist exclusively of publicly available demonstration videos sourced from YouTube. 1 Introduction Ultrasound is widely used because it is portable, safe, and can provide real time imaging, but its acquisition is operator-dependent: obtaining clinically usable views requires skilled, adaptive control of probe position, orientation, pressure, and device settings ( 24 ) . This dependence limits both the scalability of ultrasound services and the consistency of scan quality, motivating growing interest in intelligent ultrasound assistance: from real-time guidance systems that support novice operators to fully autonomous robotic scanning ( 10 ; 3 ; 11 ; 7 ) . For such systems to succeed, their perception modules must understand not just what is visible in static frames, but how to obtain and refine it: how probe maneuvers map to imaging goals, how operators troubleshoot artifacts, and how protocols progress. Vision Language Models (VLMs) have emerged as a promising foundation for such perception systems ( 5 ; 23 ) and are increasingly evaluated on medical imaging tasks ( 9 ; 4 ; 15 ) . However, existing evaluation has focused almost exclusively on static interpretation spanning from identifying anatomy, recognizing views to detecting pathology from individual images ( 13 ) . This leaves a critical gap: automation systems require temporal and causal reasoning about actions, troubleshooting, and protocol execution that static benchmarks cannot assess ( 18 ) . Figure 1: A ReXSonoVQA example: Type 3 (Procedure Context & Planning, Free-Response) question requiring identification of the screening objective and anatomical transition during a transverse sweep. Gemini 3 Pro correctly identifies the anatomical transition (tendon to muscle) but fails to specify the correct screening objective, receiving a partial score (1/2). More MCQs and free-responses examples see Appendix B . Table 1: Comparison of ReXSonoVQA with existing medical vision-language benchmarks. Dynamic Reasoning : Requires understanding of temporal processes, actions, and causal relationships. Diagnosis : Evaluates pathology identification or diagnostic interpretation. Blind Baseline : Includes text-only evaluation to verify questions require visual input. ✓ indicates present; ✗ indicates absent. Benchmark Modality # Items Diagnosis US- Specific Multi- Organ Dynamic Reasoning MCQ Free- Response Blind Baseline MedMCQA Text 193,155 ✓ ✗ ✓ ✗ ✓ ✗ ✗ GMAI-MMBench Image 26,000 ✓ ✗ ✓ ✗ ✓ ✗ ✗ OmniMedVQA Image 127,995 ✓ ✗ ✓ ✗ ✓ ✗ ✗ VQA-RAD Image 3,515 ✓ ✗ ✓ ✗ ✓ ✓ ✗ ReXVQA Image 696,000 ✓ ✗ ✗ ✗ ✓ ✗ ✗ ReXrank Image 600,000+ ✓ ✗ ✗ ✗ ✗ ✓ ✗ U2-Bench Image 7,241 ✓ ✓ ✓ ✗ ✓ ✓ ✗ ReXSonoVQA Video 514 ✗ ✓ ✓ ✓ ✓ ✓ ✓ We introduce ReXSonoVQA , the first video-based benchmark for procedure-centric ultrasound understanding, comprising 514 video clips paired with 249 multiple choice (MCQ) and 265 free-response questions spanning six clinical categories. ReXSonoVQA evaluates dynamic procedural reasoning via three cognitive tasks: (1) Action–Goal Reasoning , (2) Artifact Resolution \& Optimization , and (3) Procedure Context \& Planning . (Fig. 1 ) Our benchmark’s construction pipeline leverages instructional videos with timestamped transcripts, applying quality control of QA including blind(text-only) solvability screening and distractor refinement to ensure video is required for answering the question. (Fig. 2 ) While existing VLMs excel at static frame interpretation, dynamic procedural understanding requires frontier video-language models. We evaluate Gemini 3 Pro, Qwen3.5-397B, LLaVA-Video-72B, and Seed 2.0 Pro, all supporting native video input. Other leading commercial VLMs (GPT-5.2 ( 20 ) , Claude Opus 4.5 ( 1 ) ) lack native video support and showed poor temporal reasoning with image sequences in exploratory tests. Evaluation under paired conditions (with video vs. text-only) reveals that even with frontier video-language models, substantial gaps remain: Type 2 questions prove most challenging, exposing critical limitations in causal troubleshooting reasoning essential for real-world ultrasound automation. 2 Related Work Vision-Language Models in Medical Imaging. Medical VLMs have advanced medical image interpretation by integrating visual and textual understanding into unified frameworks ( 14 ; 17 ; 19 ; 26 ; 25 ) . Recent ultrasound-specific models, including FetalCLIP ( 15 ) and LLAUS ( 7 ) , have demonstrated strong performance in aligning ultrasound features with clinical text and performing static anatomy identification. However, existing medical VLMs focus primarily on static, single-frame interpretation, such as identifying anatomy, recognizing views, or detecting pathology from individual images, rather than reasoning about dynamic procedural content. Moreover, most leading commercial VLMs including GPT-5.2 ( 20 ) and Claude Opus 4.5 ( 1 ) do not support native video input. Exploratory evaluation using sequential image inputs showed these models failed to perform temporal reasoning effectively (Fig. A6 , Appendix B ), motivating our focus on models with native video understanding (Gemini 3 Pro, Qwen3.5-397B, LLaVA-Video-72B, and Seed 2.0 Pro) and extended context windows necessary for analyzing procedural ultrasound sequences. Ultrasound Automation and Robotics. Ultrasound’s operator dependence has driven research toward robotic standardization ( 18 ) . Recent work has shifted from rule-based visual servoing toward learning-based autonomy: UltraBot employs imitation learning for autonomous carotid scanning ( 10 ) , USPilot uses LLM-enhanced graph planning to translate natural language into robotic actions ( 3 ) , and Sonomate provides real-time scanning guidance through visually grounded language models ( 8 ) . Despite this progress, a fundamental challenge persists: current perception systems lack the deep procedural understanding required for adaptive, real-world deployment. Successful automation demands more than anatomy recognition but requires models reason about probe manipulation strategies, diagnose and resolve image quality issues in real-time, adapt protocols to patient-specific constraints, and make decisions about scan progression. These capabilities remain largely unevaluated in existing benchmarks. Figure 2: End-to-end pipeline for constructing ReXSonoVQA: (1) Task Definition, (2) Data Curation & Ground Truth Construction, (3) Prompt Refinement and Quality Control Loop, and (4) Benchmark Construction & Evaluation. Medical AI Benchmarks. Current medical AI benchmarks fall into two main categories: identification/classification and reasoning/generation tasks. Text-centric

Read the complete text