Representation-Regularized Convolutional Audio Transformer for Audio Understanding

Bing Han, Chushu Zhou, Yifan Yang, Wei Wang, Chenda Li, Wangyou Zhang, Yanmin Qian · 2026

Read Representation-Regularized Convolutional Audio Transformer for Audio Understanding by Bing Han, Chushu Zhou, Yifan Yang, Wei Wang, Chenda Li, Wangyou Zhang, Yanmin Qian (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-21612.txt.

Introduction

Bing Han, Chushu Zhou, Yifan Yang, Wei Wang, Chenda Li, Wangyou Zhang, and Yanmin Qian improve self-supervised learning for audio. Methods that bootstrap their own targets have made strong progress on sound understanding. Most work at one level of detail only, which limits how they capture sound's mix of fine and broad structure.

Bootstrapping from scratch is also slow and costly to converge. The authors add representation-level guidance to a convolutional audio transformer. Regularizing the learned features should speed training and enrich the detail captured. Better audio models serve speech, music, and sound-event tools alike.

Catalog record

CREATORBing Han, Chushu Zhou, Yifan Yang, Wei Wang, Chenda Li, Wangyou Zhang, Yanmin Qian
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-21612

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Representation-Regularized Convolutional Audio Transformer for Audio Understanding Bing Han, Chushu Zhou, Yifan Yang, Wei Wang, Chenda Li, Wangyou Zhang, Yanmin Qian Abstract Bootstrap-based Self-Supervised Learning (SSL) has achieved remarkable progress in audio understanding. However, existing methods typically operate at a single level of granularity, limiting their ability to model the diverse temporal and spectral structures inherent in complex audio signals. Furthermore, bootstrapping representations from scratch is computationally expensive, often requiring extensive training to converge. In this work, we propose the Convolutional Audio Transformer (CAT), a unified framework designed to address these challenges. First, to capture hierarchical audio features, CAT incorporates a Multi-resolution Block that aggregates information across varying granularities. Second, to enhance training efficiency, we introduce a Representation Regularization objective. Drawing inspiration from generative modeling, this auxiliary task guides the student model by aligning its predictions with high-quality semantic representations from frozen, pre-trained external encoders. Experimental results demonstrate that CAT significantly outperforms baselines on audio understanding benchmarks. Notably, it achieves competitive performance on the AudioSet 20k dataset with 5 times faster convergence than existing methods. Codes and checkpoints will be released soon at https://github.com/realzhouchushu/CAT . 1 Introduction Self-supervised learning (SSL) has become a cornerstone of representation learning, enabling models to extract meaningful patterns from unlabeled data across various modalities. In the audio domain, SSL has successfully mitigated the reliance on large-scale manual annotations, producing transferable representations for downstream tasks. Among these, bootstrap-based approaches, which employ a teacher-student framework to predict latent representations—have shown remarkable efficacy. State-of-the-art (SOTA) models such as data2vec series 5 ; 4 , EAT 10 , ATST 29 and M2D 37 leverage this paradigm to achieve impressive results in audio understanding tasks. Despite these advancements, current bootstrap-based audio SSL methods face two critical limitations. First, they often overlook the inherent hierarchical nature of audio signals. Audio events span diverse temporal and spectral scales, ranging from transient acoustic textures to long-term semantic contexts. Existing methods typically process audio at a single, fixed level of granularity, limiting their ability to capture these multi-scale structures effectively. Second, training these models from scratch is inherently inefficient. Relying solely on internal consistency between the teacher and student networks requires extensive computational resources and prolonged training periods to bootstrap high-quality representations from random initialization. Figure 1: CAT Pre-training Architecture Overview. The model follows a student-teacher bootstrap paradigm. The Student Encoder processes a masked spectrogram, while the Teacher Encoder (updated via Exponential Moving Average, EMA) receives the unmasked input. The training objective is composed of three parts: (1) Patch-level Loss ( L_{p} ): The student projector predicts the teacher’s latent representations for masked patches; (2) Global Loss ( L_{g} ): Aligns the global CLS token of the student with the teacher’s aggregated features; (3) Representation Loss ( L_{r} ): A regularization term that aligns intermediate representations from the student encoder with high-quality features extracted from a frozen external audio encoder. To address the efficiency bottleneck, we draw inspiration from recent advances in generative modeling, REPA 52 . These studies demonstrate that generative models (e.g., diffusion models 21 ; 44 ) converge significantly faster and generate higher-quality content when guided by representations from pre-trained encoders. We argue that the core objective of bootstrap SSL, predicting masked representations is fundamentally an implicit generative task. Just as diffusion models benefit from perceptual guidance, the “reconstruction” process in SSL can be accelerated by leveraging external semantic knowledge. Instead of learning solely from scratch, the student model can benefit from “standing on the shoulders of giants” by aligning with mature representations from external models. Guided by this insight, we propose the Convolutional Audio Transformer (CAT), a unified framework designed to enhance both the granularity and efficiency of audio representation learning. To tackle the granularity issue, CAT incorporates a Multi-resolution Block. Unlike traditional single-layer patch embeddings, this module utilizes hierarchical convolutional layers to extract and aggregate features at multiple temporal and frequency scales, aligning with the multi-scale characteristics of audio signals. Simultaneously, to improve training efficiency, we introduce a Representation Regularization objective. We treat the student branch as a generative generator and align its intermediate features with high-quality representations from frozen, well-pretrained external encoders (e.g., CLAP 51 , Audio-MAE 24 , AST 16 ). This auxiliary task provides stable semantic guidance, effectively ”shortcutting” the early stages of representation learning. Our experimental evaluations on audio and speech datasets validate the effectiveness of this approach. CAT not only achieves superior performance on audio understanding tasks compared to existing baselines but also demonstrates remarkable efficiency, highlighting the power of combining multi-resolution processing with representation regularization. Our main contributions are summarized as follows: • We propose the Convolutional Audio Transformer (CAT), which replaces the standard patch embedding with a Multi-resolution Block to capture audio information across varying granularities. • We introduce Representation Regularization to audio SSL, formulating the masked prediction task as a generative process guided by external pre-trained encoders. This strategy significantly improves representation quality and training stability. • We demonstrate that CAT establishes new state-of-the-art (SOTA) results on standard audio understanding benchmarks with 5 times training speed-up. 2 Related Work 2.1 Pre-trained Audio Encoder Supervised pre-training, contrastive pre-training and self-supervised pre-training are the three widely adopted approaches for audio encoder pre-training. Supervised pre-training usually relies on collected labels to guide model training. Representations extracted from supervised learning methods such as PANN 26 , AST 16 and HTS-AT 8 perform well on various tasks. For contrastive pre-training, CLAP 51 consists of dual encoders—one for audio and one for text—trained jointly to align audio representations with their corresponding natural language descriptions. However, these methods typically require large amounts of labeled data, which limits their scalability. Self-supervised learning (SSL) methods have recently gained more attention, as they can learn high-quality representations while eliminating the need for labeled data. Some self-supervised methods including MAE-AST 3 , Audio-MAE 24 and MaskSpec 12 follow the fashion of MAE 20 , using features of unmasked patches to predict masked audio patches. Another popular type of methods is bootstrap-based self-supervised methods. These approaches such as data2vec 5 , EAT 10 and A-JEPA 14 adopt a student-teacher framework where the student model learns to predict representations generated by a slowly updated teacher model. Bootstrap-based self-supervised methods have demonstrated promising performance on various audio understanding tasks. 2.2 Representation Learning as Auxiliary Tasks Beyond traditional training paradigms, recent studies in generative modeling have explored using external representations to guide the learning process. In the domain of image generation, REPA 52 demonstrated that diffusion models—which generate images from noise—converge significantly faster when their intermediate features are regularized to align with representations from pre-trained encoders (e.g., DINOv2 40 ). Subsequent works have extended this concept: iREPA 43 emphasizes the importance of spatial structural information, SARA 7 introduces multi-level representation alignment to optimize the diffusion training process and accelerate convergence, SoftREPA 28 applies it to multimodal text-to-image generation, and Semantic-VAE 39 utilizes alignment to enhance speech synthesis quality. We draw a parallel between these generative tasks and bootstrap-based SSL. We argue that the core objective of Masked SSL—predicting masked representations from partial inputs—is effectively an implicit generative task. Despite this theoretical connection, its potential to accelerate convergence and improve feature quality in understanding tasks (e.g., audio classification) remains underexplored. Our work bridges this gap by integrating

Read the complete text