A Human-Inspired Decoupled Architecture for Efficient Audio Representation Learning

\authorblockN Harunori Kawano \authorrefmark 1 and Takeshi Sasaki \authorrefmark 2 \authorblockA \authorrefmark 1 Faculty of Engineering and Information Technology, University of Technology Sydney Sydney, Australia E-mail: [email protected] \authorblockA \authorrefmark 2 Dept. of Electrical Engineering and Computer Science, Shibaura Institute of Technology Tokyo, Japan E-mail: [email protected] · 2026

Read A Human-Inspired Decoupled Architecture for Efficient Audio Representation Learning by \authorblockN Harunori Kawano \authorrefmark 1 and Takeshi Sasaki \authorrefmark 2 \authorblockA \authorrefmark 1 Faculty of Engineering and Information Technology, University of Technology Sydney Sydney, Australia E-mail: [email protected] \authorblockA \authorrefmark 2 Dept. of Electrical Engineering and Computer Science, Shibaura Institute of Technology Tokyo, Japan E-mail: [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-26098.txt.

Introduction

Kawano and Sasaki present HEAR, a compact model for learning from audio inspired by human hearing. Standard transformers are too big and costly for small devices. HEAR splits the work into two parts. An acoustic model pulls out local sound features. A task model then weaves in global meaning. A distilled tokenizer supports masked audio modeling during training. HEAR needs only 15 million parameters and 9.47 GFLOPs per run. That is a small share of the usual 85 to 94 million parameters. Scores stay competitive across audio classification tests.

Catalog record

CREATOR\authorblockN Harunori Kawano \authorrefmark 1 and Takeshi Sasaki \authorrefmark 2 \authorblockA \authorrefmark 1 Faculty of Engineering and Information Technology, University of Technology Sydney Sydney, Australia E-mail: [email protected] \authorblockA \authorrefmark 2 Dept. of Electrical Engineering and Computer Science, Shibaura Institute of Technology Tokyo, Japan E-mail: [email protected]
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE23707
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.26098

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Human-Inspired Decoupled Architecture for Efficient Audio Representation Learning \authorblockN Harunori Kawano \authorrefmark 1 and Takeshi Sasaki \authorrefmark 2 \authorblockA \authorrefmark 1 Faculty of Engineering and Information Technology, University of Technology Sydney Sydney, Australia E-mail: [email protected] \authorblockA \authorrefmark 2 Dept. of Electrical Engineering and Computer Science, Shibaura Institute of Technology Tokyo, Japan E-mail: [email protected] Abstract While self-supervised learning (SSL) has revolutionized audio representation, excessive parameterization and quadratic computational cost of standard Transformers limit their deployment on resource-constrained devices. To address this bottleneck, we propose HEAR (Human-inspired Efficient Audio Representation), a novel decoupled architecture. Inspired by the human cognitive ability to isolate local acoustic features from global context, HEAR splits the processing pipeline into two dedicated modules: an Acoustic Model for local feature extraction and a Task Model for global semantic integration. Coupled with an Acoustic Tokenizer trained via knowledge distillation, our approach enables robust Masked Audio Modeling (MAM). Extensive experiments demonstrate that HEAR requires only 15M parameters and 9.47 GFLOPs for inference, operating at a fraction of the computational cost of conventional foundation models (which typically require 85M–94M parameters). Despite this high efficiency, HEAR achieves highly competitive performance across diverse audio classification benchmarks. The code and pre-trained models are available at https://github.com/HarunoriKawano/HEAR . 1 Introduction Self-supervised learning (SSL) has revolutionized representation learning in computer vision [ 4 , 3 , 13 ] and natural language processing [ 9 , 18 , 8 ] . Building on these foundational breakthroughs, SSL has emerged as the standard paradigm for audio representation, effectively mitigating the reliance on large-scale labeled data. Following pioneering models [ 15 , 6 ] like wav2vec 2.0 [ 2 ] that established the effectiveness of SSL primarily for speech recognition, spectrogram-based architectures [ 16 , 1 , 7 ] such as AST [ 11 ] and SSAST [ 12 ] have expanded this success to diverse downstream tasks. While these pre-training paradigms achieve excellent recognition accuracy across diverse tasks, they substantially increase model sizes and computational cost. This creates a critical bottleneck for on-device execution and real-time processing under limited resources. A primary cause of this excessive parameterization and computational overhead is the homogeneous scaling of standard Transformer architectures [ 23 ] . To capture both fine-grained acoustic details and high-level semantics simultaneously, existing foundation models naively stack deep, heavily parameterized layers. This monolithic approach inherently leads to massive model sizes. Furthermore, their self-attention mechanism scales quadratically ( O(N^{2}) ) with the input sequence length N . This structural limitation compounds the computational burden, presenting a major technical barrier for processing long and continuous audio inputs. To overcome this computational cost, we draw inspiration from the human auditory system. Rather than processing continuous waveforms entirely as global information, human hearing develops through two distinct stages: an innate capability to extract localized acoustic features, and a subsequently acquired ability to interpret high-level, context-dependent semantics. Building on this cognitive structure, we propose HEAR (Human-inspired Efficient Audio Representation), a novel framework that significantly reduces computational costs while maintaining high representational capacity. HEAR partitions the architecture into two modules: an Acoustic Model that focuses exclusively on extracting local acoustic properties, and a Task Model that integrates global context for specific downstream tasks. This decoupling not only circumvents the quadratic bottleneck by restricting the initial processing to local features, but also prevents model enlargement by allowing each specialized module to remain highly compact. Furthermore, we incorporate the concept of information compression, mirroring how humans convert continuous acoustic signals into discrete semantic units like phonemes. By employing SSL that targets semantically condensed discrete labels rather than a continuous feature space, our Acoustic Model robustly acquires underlying sound structures without overfitting to superficial noise. Figure 1: Overview of the pre-training pipeline for the proposed HEAR framework. (a) Knowledge-Distilled Tokenizer Training: An acoustic tokenizer is trained to encode continuous mel-spectrogram patches into discrete semantic tokens, optimized via signal reconstruction, codebook diversity, and knowledge distillation from a pre-trained teacher model. (b) MAM: Using the frozen tokenizer, the Acoustic Model is pre-trained to predict the discrete tokens of independent random masked patches, enabling robust local feature extraction. The main contributions of this work are summarized as follows: • We propose HEAR, a computationally efficient architecture inspired by the human auditory system that structurally decouples local feature extraction (Acoustic Model) and global task adaptation (Task Model). • We introduce an Acoustic Tokenizer based on Gumbel-Softmax [ 17 ] and knowledge distillation [ 14 ] to emulate the discrete interpretation of audio signals, enabling robust Masked Audio Modeling (MAM). • We demonstrate that HEAR requires only 15M parameters and 9.47 GFLOPs, operating at a fraction of the computational cost of existing foundation models (typically 85M–94M parameters). Despite this highly compact size, HEAR maintains comparable accuracy across multiple audio classification tasks. 2 Methods 2.1 Architecture Overview The architecture of HEAR is grounded in the decoupled cognitive processing of the human auditory system. Specifically, it comprises two primary modules: an Acoustic Model responsible for extracting local acoustic features, and a Task Model that integrates these feature sequences to adapt to specific downstream tasks. In the initial inference stage, the input audio waveform is converted into a log-mel spectrogram and normalized using instance-specific mean and standard deviation. Subsequently, we apply patch splitting along the time axis with a kernel size of 2 and a stride of 2, yielding a 1D input token sequence. To avoid the quadratic computational complexity ( O(N^{2}) ) bottleneck with respect to sequence length, the Acoustic Model divides the input audio into fixed-length local chunks of 6 seconds and processes each independently. Because the relative distance and sequential relationships between adjacent acoustic events play an essential role in forming local patterns, we employ a Transformer encoder incorporating relative position representations [ 21 ] as the backbone of the Acoustic Model. The sequence of local features extracted by the Acoustic Model is ultimately fed into the Task Model. To strictly validate the core concept of decoupling feature extraction from task adaptation, we employ a standard Transformer architecture for the Task Model. The Task Model applies a global self-attention mechanism to the input local feature sequence, integrating the global context across the entire sequence. This maps the universal acoustic representations acquired by the Acoustic Model into the high-level semantics required for specific downstream tasks, yielding the final prediction. To fully exploit the potential of the proposed decoupled architecture, we design a three-stage training pipeline: 1. Acoustic Tokenizer Training: Acquiring high-quality discrete representations from continuous audio via signal reconstruction and knowledge distillation. 2. Acoustic Model Pre-training: Learning local feature extraction capabilities through MAM, which predicts the discrete labels generated by the tokenizer. 3. Downstream Adaptation: Combining the pre-trained Acoustic Model with the Task Model to perform fine-tuning via a feature fusion process. We detail each phase below. 2.2 Knowledge-Distilled Tokenizer Training The first stage of our proposed framework, Acoustic Tokenizer pre-training, aims to extract semantically rich discrete representations from continuous audio signals. As illustrated in Fig. 1 (a), this framework builds upon the VQ-VAE architecture [ 22 ] . It consists of a Tokenizer that encodes input signals into latent representations and a Decoder that reconstructs the original acoustic features. Similar to the Acoustic Model, both the Tokenizer and Decoder employ Transformers with

Read the complete text