Au-M-ol: A Unified Model for Medical Audio and Language Understanding

Meizhu Liu, Nistha Mitra, Paul Li, Amine Abdaoui, Adam Ledyard, Tao Sheng, [0.5em] Oracle AI Science · 2026

Read Au-M-ol: A Unified Model for Medical Audio and Language Understanding by Meizhu Liu, Nistha Mitra, Paul Li, Amine Abdaoui, Adam Ledyard, Tao Sheng, [0.5em] Oracle AI Science (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-23284.txt.

Introduction

Liu and five colleagues present Au-M-ol, a multimodal model that extends large language models with audio processing for clinical tasks. An audio encoder extracts sound features from medical speech, an adaptation layer maps them into LLM input space, and a pretrained LLM transcribes and interprets clinical language. The design lets the model read spoken medical content directly, improving accuracy and robustness. On medical transcription, it cuts word error rate by 56 percent against leading baselines, holding up under noise, specialised terms, and speaker variation.

Catalog record

CREATORMeizhu Liu, Nistha Mitra, Paul Li, Amine Abdaoui, Adam Ledyard, Tao Sheng, [0.5em] Oracle AI Science
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE26018
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.23284

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Au-M-ol: A Unified Model for Medical Audio and Language Understanding Meizhu Liu, Nistha Mitra, Paul Li, Amine Abdaoui, Adam Ledyard, Tao Sheng, [0.5em] Oracle AI Science Abstract In this work, we present Au-M-ol, a novel multimodal architecture that extends Large Language Models (LLMs) with audio processing. It is designed to improve performance on clinically relevant tasks such as Automatic Speech Recognition (ASR). Au-M-ol has three main components: (1) an audio encoder that extracts rich acoustic features from medical speech, (2) an adaptation layer that maps audio features into the LLM input space, and (3) a pretrained LLM that performs transcription and clinical language understanding. This design allows the model to interpret spoken medical content directly, improving both accuracy and robustness. In experiments, Au-M-ol reduces Word Error Rate (WER) by 56% compared to state-of-the-art baselines on medical transcription tasks. The model also performs well in challenging conditions, including noisy environments, domain-specific terminology, and speaker variability. These results suggest that Au-M-ol is a strong candidate for real-world clinical applications, where reliable and context-aware audio understanding is essential. 1 Introduction The global healthcare system has faced mounting pressure in recent years, with the COVID-19 pandemic intensifying deep-rooted structural challenges 24 ; 15 . Chronic issues such as overcrowded hospitals, lengthy patient waitlists, and critical shortages of healthcare professionals have worsened, leading to widespread burnout and increasing attrition among clinicians 24 ; 30 . As healthcare systems struggle to meet rising demand with limited resources, the need for scalable, intelligent solutions to enhance operational efficiency and support clinical workflows has become more urgent than ever. Artificial intelligence (AI), particularly the latest large language models (LLMs), offers transformative potential for healthcare delivery. These models can streamline documentation, improve clinical decision-making, and enable new forms of interaction with medical data. However, current LLMs and multimodal models remain limited in their applicability to complex clinical environments—particularly when audio is involved. Models such as Whisper 21 , Audio Qwen 6 , MaLa-ASR 27 , and Macaw-LLM 16 have demonstrated strong performance in general-purpose audio transcription. Building on this foundation, a growing body of research has investigated joint modeling of audio and text. Approaches like Speech-LLaMA 26 , mSLAM 3 , SpeechUT 29 , SSR 25 and some others 17 ; 13 leverage cross-modal alignment techniques or adapter modules to fuse acoustic and linguistic representations. Meanwhile, architectures such as the Perceiver 11 ; 14 ; 8 and Dual-decoder Transformer 28 aim to efficiently integrate heterogeneous modalities at scale. More recent models, including the Cascaded Cross-Modal Transformer 23 , show promise in higher-level tasks such as emotion and intent recognition from audio-text pairs. Figure 1: The overview of the Au-M-ol model architecture. Despite these advances, such models are typically trained and evaluated on general-domain data and often fall short in the medical domain, where language tends to be more complex and highly specialized, and clinical accuracy is paramount. Whisper, while highly capable at speech-to-text transcription, does not capture essential clinical audio biomarkers—such as intonation, breathing patterns, or acoustic irregularities—that are critical for diagnosing conditions like Parkinson’s disease or respiratory disorders. Similarly, Audio Qwen 6 incorporates multimodal signals but lacks the semantic grounding necessary for interpreting domain-specific intent. Models like SSR and SpeechUT focus on modality alignment but have not been applied to high-stakes clinical applications where interpretability and robustness are essential. Med-PaLM and Med-PaLM 2 9 are healthcare-focused models that achieve strong performance in medical question answering, reasoning, and knowledge retrieval. Despite their textual reasoning capabilities, these models are unable to process or transcribe audio, limiting their applicability in clinical scenarios that require automatic speech recognition (ASR). Some recent approaches, such as The Sound of Healthcare 1 , combine ASR with LLMs in a sequential pipeline to improve transcription quality for medical audio. While effective, this two-stage architecture introduces several limitations: • High computational cost: Running separate ASR and LLM models increases inference time and hardware requirements. • Scalability challenges: Maintaining two independent models complicates deployment across large healthcare systems. • Fragmented processing: Passing intermediate outputs between models introduces potential failure points and data inefficiencies. These limitations can hinder the responsiveness and scalability of such systems, restricting their adoption in time-sensitive, real-world healthcare applications. More recently, Whispering-LLaMA 22 integrates audio embeddings from Whisper directly into LLaMA via cross-attention adapters. This allows LLaMA to reason over both audio and text simultaneously, resulting in more accurate transcriptions, particularly in challenging scenarios. However, because both the Whisper decoder and the LLaMA decoder are involved, two transformer decoding passes with cross-modal attention are required, which can slow down real-time applications. To address these limitations, we propose Au-M-ol, a novel multimodal architecture that unifies audio and text processing into a single, efficient encoder-decoder framework. Our model integrates an audio encoder with a decoder-based LLM via an adaptation layer that aligns acoustic features with language embeddings. This tight integration allows the model to perform transcription and understanding in a unified manner, significantly reducing latency and computational overhead. Unlike traditional cascaded systems, Au-M-ol is lightweight, easier to fine-tune, and capable of real-time inference. It outperforms state-of-the-art baselines in medical transcription tasks, achieving significant transcription accuracy in medical audios. The remainder of the paper is organized as follows. Section 2 presents the details of the proposed model. Section 3 describes the experimental setup, including the datasets, training procedures, and evaluation results. Section 4 provides concluding remarks, Section 5 talks about possible future directions and Section 6 discusses the limitations of our work. 2 Proposed Model The core of the model is a lightweight, robust encoder-decoder transformer architecture that extracts health-related information from speech and processes it alongside text using a decoder-based language model. This unified design eliminates the need for separate systems for transcription, documentation, and diagnostics, significantly simplifying workflows and reducing complexity. 2.1 Model Architecture Au-M-ol is a unified encoder-decoder architecture that integrates speech and text understanding within a single model. The model consists of a Whisper encoder, a lightweight adaptation layer, and a decoder-only language model (LLaMA-3.1-8B). Unlike prior work that adopts a sequential ASR-to-LLM pipeline 1 ; 9 , our model jointly processes audio (e.g., prosody, speech patterns) and text (e.g., clinical notes) in a single end-to-end framework. This integration removes the need for separate ASR and language modeling components, reducing complexity and improving efficiency. The overall architecture is shown in Figure 1 . We describe each component in detail below. 2.2 Audio Encoder We employ a Whisper-based audio encoder (Whisper-Large-V2) 21 to transform raw audio waveforms into high-level feature representations. The encoder is composed of a stack of Transformer blocks, each consisting of multi-head self-attention, feed-forward layers, layer normalization, and residual connections. Prior to encoding, audio is preprocessed through several steps. First, the waveform is resampled to 16 kHz. It is then converted into a log-Mel spectrogram using a short-time Fourier transform (STFT) with a window size of 25 ms and a hop (stride) of 10 ms. The spectrogram is mapped into 80 Mel frequency bins, resulting in a perceptually-informed representation aligned with human auditory perception. Each input covers a fixed duration of 30 seconds, yielding a spectrogram tensor of shape (80, 3000), where 3000 corresponds to the number of frames. The spectrograms are normalized per frequency bin, and positional encodings are added to

Read the complete text