CaReAQA: A Cardiac and Respiratory Audio Question Answering Model for Open-Ended Diagnostic Reasoning
Tsai-Ning Wang, Lin-Lin Chen, Neil Zeghidour, Aaqib Saeed · 2025
Read CaReAQA: A Cardiac and Respiratory Audio Question Answering Model for Open-Ended Diagnostic Reasoning by Tsai-Ning Wang, Lin-Lin Chen, Neil Zeghidour, Aaqib Saeed (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-01199.txt.
Introduction
Heart and lung sounds carry rich diagnostic clues, but reading them usually needs a trained ear or narrow classifiers trained on scarce labels. Tsai-Ning Wang and co-authors build CaReAQA in this 2025 paper, a model that pairs a foundation audio encoder with a language model so it can answer open-ended questions about such sounds.
They also release CaReSound, a benchmark of annotated recordings with paired questions and answers. The model scored well on open diagnostic questions and carried over decently to fixed-choice tasks on unseen data. The authors present audio-language reasoning as a scalable aid for clinical decisions.
Catalog record
| CREATOR | Tsai-Ning Wang, Lin-Lin Chen, Neil Zeghidour, Aaqib Saeed |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-01199
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CaReAQA: A Cardiac and Respiratory Audio Question Answering Model for Open-Ended Diagnostic Reasoning Tsai-Ning Wang, Lin-Lin Chen, Neil Zeghidour, Aaqib Saeed Abstract Medical audio signals, such as heart and lung sounds, play a crucial role in clinical diagnosis. However, analyzing these signals remains challenging: traditional methods rely on handcrafted features or supervised deep learning models that demand extensive labeled datasets, limiting their scalability and applicability. To address these issues, we propose CaReAQA 1 1 1 Dataset and pretrained models are available at https://huggingface.co/datasets/tsnngw/CaReSound and CaReAQA model at https://huggingface.co/tsnngw/CaReAQA . , an audio-language model that integrates a foundation audio model with the reasoning capabilities of large language models, enabling clinically relevant, open-ended diagnostic responses. Alongside CaReAQA, we introduce CaReSound, a benchmark dataset of annotated medical audio recordings enriched with metadata and paired question-answer examples, intended to drive progress in diagnostic reasoning research. Evaluation results show that CaReAQA achieves 86.2% accuracy on open-ended diagnostic reasoning tasks, outperforming baseline models. It also generalizes well to closed-ended classification tasks, achieving an average accuracy of 56.9% on unseen datasets. Our findings show how audio-language integration and reasoning advances medical diagnostics, enabling efficient AI systems for clinical decision support. † † volume: 287 † † year: 2025 † † workshop: Conference on Health, Inference, and Learning (CHIL) 2025 Figure 1 : Spectrograms(left) and waveforms(right) paired with diagnostic question-and-answer outputs, demonstrating the model’s ability to analyze and answer questions about normal and abnormal heart and respiratory sounds. Figure 2 : Overview of CaReAQA. Our multimodal framework combining audio and text data for clinical sound analysis, enabling symptom classification, disease diagnosis, and diagnostic reasoning with a large language model. 1 Introduction Medical audio data, including cardiac and respiratory sounds, is a rich source of physiological information and plays a crucial role in diagnosing a wide range of health conditions. For example, heart murmurs can indicate structural cardiac abnormalities, while respiratory sounds, such as wheezes or crackles, are often early indicators of conditions like asthma, chronic obstructive pulmonary disease (COPD), or pulmonary fibrosis. Analyzing these sounds more effectively could significantly improve diagnostic accuracy and deepen our understanding of underlying health issues. Traditional medical audio analysis uses labor-intensive, less adaptable handcrafted features and signal processing techniques. Supervised deep learning models have demonstrated strong performance but require large amounts of annotated data, which are costly and time-consuming to collect. Meanwhile, general-purpose audio-language models, though powerful, lack the domain-specific knowledge needed to capture the unique properties of medical audio. As a result, these models often fall short in identifying critical patterns essential for distinguishing pathological from non-pathological sounds and mainly are limited to close-ended classification tasks. Despite advancements in multimodal AI, medical audio remains an underexplored domain compared to visual and textual data. Current models are often optimized for broad, non-specialized tasks and insufficiently tailored to the demands of healthcare diagnostics. Furthermore, while versatile models trained on diverse datasets have shown improvements across a wide range of tasks, their lack of specialization limits their utility in clinical applications. To tackle these challenges, we introduce CaReAQA, a Cardiac and Respiratory Audio Question Answering model tailored for open-ended question answering in medical diagnostics. By integrating a self-supervised audio foundation model with the advanced reasoning capabilities of large language models (LLMs), CaReAQA generates clinically relevant and context-aware diagnostic responses that adapt to the complexity of real-world medical scenarios. Unlike traditional models limited to predefined tasks or fixed outputs, CaReAQA provides the flexibility and depth required to address diverse and nuanced diagnostic challenges effectively. Our contributions are as follows: • We introduce a novel audio-language model designed specifically for medical diagnostic question-answering task, integrating self-supervised audio encoder with the reasoning capabilities of large language models to produce open-ended diagnostic answers. • We introduce CaReSound, a new benchmark dataset comprising of diverse public medical audio recordings, such as respiratory and cardiac sounds, annotated with detailed metadata and question-answer pairs. • We establish a robust evaluation framework to assess the model’s performance across a variety of tasks, including open-ended question answering and closed-ended classification tasks. Our experimental evaluation demonstrates that CaReAQA significantly outperforms strong baselines on open-ended diagnostic reasoning tasks across diverse datasets. By accurately interpreting complex patterns in medical audio and generalizing effectively to unseen data, our model addresses the unique challenges of medical diagnostics that general-purpose audio models fail to meet. Our work not only lays the groundwork for advanced diagnostic tools but also highlights its potential to transform health monitoring by enabling accurate, efficient, and accessible diagnostic support, ultimately improving patient outcomes and aiding clinicians in decision-making. We emphasize that CaReAQA is not intended for direct clinical deployment but a step toward advancing auscultation-based diagnostic reasoning. 2 Related Work 2.1 General-Purpose Audio-Language Models Recent advances in audio-language models have significantly enhanced the integration of auditory perception and reasoning, enabling LLMs to handle diverse audio-related tasks. For instance, the LTU (Listen, Think, Understand) model ( Gong et al., 2023 ) excels in both closed and open-ended question-answering tasks. Instruction-based frameworks like Pengi ( Deshmukh et al., 2023 ) improve task versatility by framing diverse tasks as instruction-response problems, while models like GAMA ( Ghosh et al., 2024 ) and AudioGPT ( Huang et al., 2024 ) leverage pre-trained audio models for reasoning over complex auditory inputs. AudioGPT, in particular, excels in multi-turn dialogues for general audio tasks but relies on predefined interfaces like ChatGPT and external systems (e.g., ASR or task-specific models) for audio processing. In contrast, our model natively processes audio and text inputs within a unified framework, eliminating reliance on external systems or pipelines. By focusing on medical audio data, it addresses the unique challenges of health diagnostics and generates tailored open-ended responses, bridging a critical gap left by general-purpose models. 2.2 Domain-Specific Audio and Multimodal Applications Domain-specific applications of audio-language models remain relatively under-explored. Efforts like RespLLM ( Zhang et al., 2024b ) have begun addressing this gap by leveraging pretrained large language models and cross-modal attention mechanisms to fuse audio and text representations for classification tasks. However, its scope remains focused on respiratory sounds, leaving broader health diagnostics unaddressed. Likewise, RespLLM is limited to binary classification tasks, providing only 1 (“yes”) or 0 (“no”) outputs through a basic linear layer without support for question answering. In contrast, our work is built to handle open-ended question answering. This broader focus makes our approach more adaptable to complex challenges in audio-based health monitoring, going beyond the narrow scope of binary classification. OPERA ( Zhang et al., 2024a ) introduces a self-supervised framework for respiratory audio, leveraging large-scale unlabeled datasets to learn robust acoustic representations. While effective for tasks like health condition inference and lung function estimation, it relies on training or fine-tuning a classifier on top of the encoder and remains limited to respiratory audio. Additionally, it lacks the ability to perform audio-language reasoning or generate open-ended diagnostic responses. In contrast, our approach overcomes these limitations and extends to cardiac sounds, enabling a broader and more integrated analysis of acoustic biomarkers. Medical visual question answering (VQA) systems have demonstrated significant potential in automating clinical reasoning by answering questions based on medical images ( Van Sonsbeek et al., 2023 ) . However, while these systems excel at leveraging visual data for open-ended reasoning, they overlook