Towards Fusion of Neural Audio Codec-based Representations with Spectral for Heart Murmur Classification via Bandit-based Cross-Attention Mechanism
Orchid Chetia, Girish*, Mohd Mujtaba, Swarup Ranjan, Priyabrata, Santanu, Arun Balaji, Rajesh · 2025
Read Towards Fusion of Neural Audio Codec-based Representations with Spectral for Heart Murmur Classification via Bandit-based Cross-Attention Mechanism by Orchid Chetia, Girish*, Mohd Mujtaba, Swarup Ranjan, Priyabrata, Santanu, Arun Balaji, Rajesh (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-01148.txt.
Introduction
Orchid Chetia Phukan and seven co-authors join two kinds of sound traits to sort heart murmurs: coded traits from a learned audio codec, and classic traits of tone. Their paper is from two thousand twenty-five.
The team holds the two kinds fill each other's gaps. The coded traits catch fine sound marks, such as shifts of rhythm. The tone traits hold the form of the sound's waves, such as its spread of energy, which is key for heart sounds. Their frame, BAOMI, fuses the two by a bandit-based focus. An agent gives more weight to the heads that count most in a multi-head focus net. Tests back the join.
Catalog record
| CREATOR | Orchid Chetia, Girish*, Mohd Mujtaba, Swarup Ranjan, Priyabrata, Santanu, Arun Balaji, Rajesh |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 22 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-01148
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Towards Fusion of Neural Audio Codec-based Representations with Spectral for Heart Murmur Classification via Bandit-based Cross-Attention Mechanism Orchid Chetia, Girish*, Mohd Mujtaba, Swarup Ranjan, Priyabrata, Santanu, Arun Balaji, Rajesh Abstract In this study, we focus on heart murmur classification (HMC) and hypothesize that combining neural audio codec representations (NACRs) such as EnCodec with spectral features (SFs), such as MFCC, will yield superior performance. We believe such fusion will trigger their complementary behavior as NACRs excel at capturing fine-grained acoustic patterns such as rhythm changes, spectral features focus on frequency-domain properties such as harmonic structure, spectral energy distribution crucial for analyzing the complex of heart sounds. To this end, we propose, BAOMI , a novel framework banking on novel bandit-based cross-attention mechanism for effective fusion. Here, a agent provides more weightage to most important heads in multi-head cross-attention mechanism and helps in mitigating the noise. With BAOMI , we report the topmost performance in comparison to individual NACRs, SFs, and baseline fusion techniques and setting new state-of-the-art. Phukan* Akhtar* Behera Mallick Roy Buduru Sharma IIIT-DelhiIndia UPESIndia V.B.S.P.UIndia Independent ResearcherIndia Dhineu SolutionsIndia University of TartuEstonia Plaksha UniversityIndia keywords Heart Murmur Classification, Neural Audio Codecs, Spectral Features † † email: Correspondence: [email protected] † † footnotetext: * Contributed equally as a first authors. 1 Introduction Cardiovascular diseases (CVDs) remain the leading cause of global mortality, claiming millions of lives each year [ 1 ] . Early diagnosis is vital to improving patient outcomes; however, traditional diagnostic methods, such as manual auscultation, are dependent on clinician expertise and prone to inconsistencies. Phonocardiograms (PCGs), which capture heart sounds as audio signals, offer a non-invasive, cost-effective approach for early diagnosis, identifying abnormal acoustic patterns like pathological murmurs [ 2 ] . Advances in ML have enabled the automated analysis of PCGs, enhancing diagnostic accuracy and broadening accessibility [ 3 ] . Despite these gains, challenges related to noise interference, recording variability, and the detection of subtle anomalies persist, underscoring the need for more robust methods. In this work, we focus on Heart Murmur Classification (HMC). HMC focuses specifically on identifying and categorizing heart murmurs, which are abnormal sounds resulting from turbulent blood flow in the heart or nearby vessels. It is central to cardiac diagnostics, aiming to identify abnormal patterns within PCGs indicative of cardiovascular conditions. Early approaches relied on handcrafted spectral features, such as MFCCs and time-frequency representations, to differentiate normal and pathological heart sounds [ 4 ] . Such features were intially modeled using classical ML techniques such as SVM, kNN [ 5 ] and later on, followed by the use of DL models such as CNN [ 6 ] and Transformer [ 7 ] . Further, Alkhodari et al. [ 8 ] proposed the use of ensemble of transformer networks for HMC. Tsai et al. [ 9 ] proposed the use of capsule-based network that utilizes dynamic routing with MFCC features. Zhang et al. [ 10 ] gave a novel parallel branch convolution and self-attention based architecture with monte-carlo dropout for HMC. Recently researchers have shown the effectiveness of using neural audio codec representations (NACRs) for heart sound analysis [ 11 ] . Here, they explored the use of NACRs extract from neural audio codecs (NACs) such as EnCodec, DAC, and so on for classfying heart sounds as normal or abnormalities. Also, NACRs have shown promises in various audio and speech processing tasks such as speech separation [ 12 ] , environmental sound classification and various speech and audio processing applications [ 13 ] . These NACRs have shown remarkable performance in these tasks as they preserve critical acoustic details while filtering out noise. However, previous works on HMC haven’t focused on combining NACRs with spectral features (SFs) such as MFCC and LFCC that were traditionally used. We believe such fusion will lead to further improvement in HMC performance. So, in this study we explore such fusion for HMC and hypothesize that it will bring out their complementary strengths: NACRs excel in capturing intricate acoustic patterns such as subtle rhythm changes, while SFs emphasize frequency-domain characteristics, including harmonic structures and spectral energy distributions, which are vital for understanding the complexity of heart sounds. To the best of our knowledge, we are the first study exploring fusion of NACRs and SFs for HMC. We introduce BAOMI (Fusion using BA ndit-based Cr O ss-Attention M echan I sm), a novel framework for effective fusion of NACRs and SFs, which utilizes a novel bandit-based cross-attention mechanism. BAOMI employs an agent to identify the most relevant attention heads and providing them more weightage within the multi-head cross-attention mechanism, reducing the impact of noise. Using BAOMI , we achieve superior performance compared to individual NACRs and SFs, strong baseline fusion methods and we establish state-of-the-art (SOTA) results against existing methods from previous works. To summarize, the main contributions of the paper are: • We propose, BAOMI for effective fusion of NACRs with SFs. It introduces a novel bandit-based cross-attention mechanism that dynamically identifies and utilizes the most effective attention heads for fusion. • With BAOMI , we achieve superior results compared to models using NACRs or SFs individually, baseline fusion techniques and establishes new SOTA performance for HMC, outperforming previous methods. • The superior performance showed by the combination of NACRs and SFs through BAOMI can be attributed to the emergence of complementary behavior by NACRs and SFs where NACRs focuses on the fine-grained acoustic details while SFs focuses on frequency-domain properties critical for heart sound analysis. The source code and trained model checkpoints for this work can be found at: https://github.com/Helix-IIIT-Delhi/BAOMI-Heart_Murmur Figure 1 : Proposed Framework: BAOMI 2 Neural Audio Codec In this section, we discuss the SOTA NACs considered in our study. EnCodec [ 14 ] : It generates compact, quantized audio representations through a streaming encoder-decoder architecture utilizing RVQ. By incorporating multi-scale spectrogram adversarial losses and Transformer-based entropy coding, EnCodec strikes a balance between compression efficiency and audio quality. Its bandwidth ranges from 1.5 to 24 kbps with sample rates up to 48 kHz. We use the EnCodec versions that requires that input to resample to 24 kHz 1 1 1 https://huggingface.co/facebook/encodec_24khz (EnCodec24) and 48 (EnCodec48) kHz 2 2 2 https://huggingface.co/facebook/encodec_48khz . Speech Tokenizer 3 3 3 https://github.com/ZhangXInFD/SpeechTokenizer [ 15 ] : It is optimized for speech language models, merging semantic and acoustic tokens into a unified representation. Utilizing a RVQ-based encoder-decoder framework, its hierarchical design encodes semantic content in early layers, followed by acoustic features in later layers. It offers reconstruction quality comparable to EnCodec. Descript Audio Codec (DAC) 4 4 4 https://github.com/descriptinc/descript-audio-codec [ 16 ] : It provides highly efficient audio compression, achieving remarkable quality retention. It compresses 44.1 kHz audio up to 90 times into discrete tokens at 8 kbps. It employs multi-period and multi-band Short-Time Fourier Transform (STFT) discriminators to minimize high-frequency artifacts, while multi-scale STFT and mel-spectral loss functions enhance fidelity. SNAC 5 5 5 https://github.com/hubertsiuzdak/snac [ 17 ] : It introduces a multi-scale approach to audio compression, employing RVQ across different temporal resolutions. This layered quantization captures both fine and broad audio details, enhancing perceptual quality with local windowed attention. We use the SNAC versions that requires input audios to be sampled 24 kHz (SNAC24) and 32 kHz (SNAC32). Before