HAIBU-ReMUD: Reasoning Multimodal Ultrasound Dataset and Model Bridging to General Specific Domains

Shijie Wang, Yilun Zhang, Zeyu Lai, Dexing Kong · 2025

Read HAIBU-ReMUD: Reasoning Multimodal Ultrasound Dataset and Model Bridging to General Specific Domains by Shijie Wang, Yilun Zhang, Zeyu Lai, Dexing Kong (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-07837.txt.

Introduction

Shijie Wang, Yilun Zhang, Zeyu Lai, and Dexing Kong build training data to teach AI models the language of medical ultrasound. General multimodal models falter in narrow fields that lack tidy image-text data. Ultrasound knowledge sits scattered in books, guidelines, and reports, often as PDFs or scans. Their 2025 pipeline turns such sources into four-part items: image, question, reasoning trace, and answer.

The resulting set, ReMUD, holds over forty-five thousand items, both reasoning and plain forms. A model tuned on it beat general models at ultrasound tasks. The team released the data, the building code, and the model weights. They pitch the recipe as a fix for data shortage in other narrow fields too.

Catalog record

CREATORShijie Wang, Yilun Zhang, Zeyu Lai, Dexing Kong
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-07837

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

HAIBU-ReMUD: Reasoning Multimodal Ultrasound Dataset and Model Bridging to General Specific Domains Shijie Wang, Yilun Zhang, Zeyu Lai, Dexing Kong Abstract Multimodal large language models (MLLMs) have shown great potential in general domains but perform poorly in some specific domains due to a lack of domain-specific data, such as image-text data or vedio-text data. In some specific domains, there is abundant graphic and textual data scattered around, but lacks standardized arrangement. In the field of medical ultrasound, there are ultrasonic diagnostic books, ultrasonic clinical guidelines, ultrasonic diagnostic reports, and so on. However, these ultrasonic materials are often saved in the forms of PDF, images, etc., and cannot be directly used for the training of MLLMs. This paper proposes a novel image-text reasoning supervised fine-tuning data generation pipeline to create specific domain quadruplets (image, question, thinking trace, and answer) from domain-specific materials. A medical ultrasound domain dataset ReMUD is established, containing over 45,000 reasoning and non-reasoning supervised fine-tuning Question Answering (QA) and Visual Question Answering (VQA) data. The ReMUD-7B model, fine-tuned on Qwen2.5-VL-7B-Instruct, outperforms general-domain MLLMs in medical ultrasound field. To facilitate research, the ReMUD dataset, data generation codebase, and ReMUD-7B parameters will be released at https://github.com/ShiDaizi/ReMUD , addressing the data shortage issue in specific domain MLLMs. I Introduction In general domains, image-text data exists in abundance, like web images along with their corresponding captions and contexts. Benefit from this, multimodal large language models (MLLMs) instructions-tuned by leveraging multimodal inputs, such as LLaVA [ 1 ] and GPT-4o [ 2 ] , have displayed remarkable zero-shot task completion performance across a wide range of user-oriented vision-language tasks, for instance, image understanding and reasoning. The rapid development of MLLMs has provided more possibilities for expansion into broader fields and application scenarios [ 3 , 4 , 5 ] . Despite demonstrating excellent vision-language abilities in general domains, MLLMs’ performance in specific domains often shows notable deficiencies [ 6 ] . Training data generated by publicly available data usually lacks domain-specific knowledge, especially in medical domain, resulting in a lack of domain-specific expertise. General domain MLLMs may produce incorrect responses or complete hallucinations, which are not allowed to appear in medical domain [ 7 ] . Pure text-based large language models (LLMs) can be self-supervised trained through autoregression, predicting the next word. However, in contrast, MLLMs require the pre-preparation of relevant multimodal data for training. As a consequence, preparing data for multimodal training demands substantial human and financial resources. Thus, we ask: Do we have a simple and automated way to construct a large amount of multimodal data in specific domain? Fig. 1: Statistical overview of ReMUD. ReMUD incorporates ultrasound knowledge across various anatomical regions, enabling the model to acquire comprehensive and extensive ultrasound expertise. In this paper, we propose a novel image-text data generation pipeline to create specific domain quadruplets (image, question, thinking trace and answer) from scratch. In a specific domain, there are a large number of professional books and papers, which are often stored in the form of PDF. These materials contain a wealth of curated image-text information. The images along with their corresponding text descriptions are scattered throughout these resources. Taking the medical ultrasound domain as an instance, we utilized the bounding box function of Qwen2.5-VL [ 8 ] and the multimodal image-text recognition and generation capacities of GPT-4o, Gemini-2.0-Flash-Thinking-Exp [ 9 ] to establish a multimodal dataset ReMUD specific to the ultrasound domain. Subsequently, by performing supervised fine-tuning with Qwen2.5-VL-7B-Instruct serving as the base model, the fine-tuned model ReMUD-7B was developed. This model, equipped with reasoning ability, performs better than general-domain multimodal large language models on the test dataset USTQ-Knowledge and UVQA-Diagnosis we established. Specifically, our paper makes the following contributions: • Specific domain quadruplets generation pipeline. We present a novel data generation pipeline to create domain specific quadruplets (image, question, thinking trace and answer) from domain specific materials which can ensure the accuracy of professional knowledge. This requires no manual annotations or pre-generated dataset and it is applicable to various general specific domains. • ReMUD. We establish the first open-source medical ultrasound domain supervised fine-tuning (SFT) dataset ReMUD, which contains 45,000+ reasoning and non-reasoning SFT QA and VQA data (see Fig. 1 ). It incorporates text Question Answering data, Visual Question Answering data, reasoning data, and non-reasoning data. Two test datasets USTQ-Knowledge and UVQA-diagnos have been compiled to evaluate the ultrasound capabilities. The reasoning model ReMUD-7B, which is fine-tuned by ReMUD on Qwen2.5-VL-7B-Instruct, outperforms other general-domain MLLMs. • Open-source. To facilitate research of specific domain and medical ultrasound field, we will release the following assets to the public: the ReMUD instruction-following dataset, the codebase for data generation and the parameters of ReMUD-7B. II Related work Fig. 2: Flowchart illustrating the process of visual question answering (VQA) data generation, incorporating data collection, image grounding, VQA creation, and Data Cleaning. II-A Multimodal large language model Multimodal Large Language Models (MLLMs) have witnessed a surge in research recently [ 10 , 8 , 11 , 12 , 13 ] . In architecture, they typically integrate pre-trained modality encoders (e.g., Vision Transformer visual encoder and its variants [ 14 , 15 , 16 ] ), pre-trained LLMs (such as LLaMA [ 17 ] , Vicuna [ 18 ] , and Qwen series [ 19 , 20 ] ), and modality interfaces for multimodal interaction [ 21 ] . Training MLLMs involves pre-training with large scale image-text data for modality alignment [ 1 ] , instruction-tuning to enhance task generalization, and alignment tuning to match human preferences. Evaluation methods include closed-set evaluations on task-specific datasets and open-set assessments like manual scoring, model scoring, and case studies. Additionally, extended techniques such as Multimodal In-Context Learning (M-ICL) [ 22 ] , Multimodal Chain of Thought (M-CoT) [ 23 ] , and LLM-aided visual reasoning [ 24 ] have been developed to enhance MLLMs’ capabilities. However, challenges remain, including handling long-context multimodal information [ 25 ] and improving instruction-following [ 26 ] , presenting opportunities for future research. In ultrasound field, LLaVA-Ultra [ 27 ] is the first ultrasound multimodal large language model that trained by professional ultrasound multi-modal data, but its data and model parameters are not open-sourced. II-B Visual Question Answering data generation Manually creating Visual Question Answering (VQA) datasets is a time-consuming and expensive process [ 28 ] . Template-based methods have been developed to address VQA data shortage. They generate QA pairs using templates, creating datasets like CLEVR [ 29 ] and MIMIC-Diff-VQA [ 30 ] . But models trained on such data perform poorly on complex human-written questions [ 31 ] . Later, some studies use image descriptions and LLMs to generate questions and answer by using LLMs’ high contextual understanding and reasoning ability to answer the question. For instance, Path-VQA [ 32 ] is obtained by extracting pathology images and captions from publicly-available textbooks and digital libraries, using natural language processing techniques to generate question-answer pairs from the captions. Recently, there are even more MLLMs [ 2 , 9 ] to assist in the generation of VQA data. II-C Test-time reasoning Test-time reasoning research has explored diverse methods to scale test-time compute. Parallel methods like majority voting [ 33 , 34 ] and Best-of-N [ 35 , 36 ]

Read the complete text