JOINT FEATURE AND OUTPUT DISTILLATION FOR LOW-COMPLEXITY ACOUSTIC SCENE CLASSIFICATION
2025
Read JOINT FEATURE AND OUTPUT DISTILLATION FOR LOW-COMPLEXITY ACOUSTIC SCENE CLASSIFICATION (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-19557.txt.
Introduction
Haowen Li and co-authors report a compact system for naming the place a sound recording was made in, in this 2025 challenge paper. Phones and small devices need scene tools that stay light, yet big teacher models hold the best knowledge. Their method trains a small student model with help from two large teachers at once.
The student learns both from the teachers' final answers and from their inner features. On a standard city-sounds dataset, the submitted systems reached accuracy near fifty-nine percent. The report was prepared for a 2025 challenge task on low-complexity sound scene naming, where model size is strictly limited.
Catalog record
| YEAR | 2025 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 14 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-19557
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
JOINT FEATURE AND OUTPUT DISTILLATION FOR LOW-COMPLEXITY ACOUSTIC SCENE CLASSIFICATION Abstract This report presents a dual-level knowledge distillation framework with multi-teacher guidance for low-complexity acoustic scene classification (ASC) in DCASE2025 Task 1. We propose a distillation strategy that jointly transfers both soft logits and intermediate feature representations. Specifically, we pre-trained PaSST and CP-ResNet models as teacher models. Logits from teachers are averaged to generate soft targets, while one CP-ResNet is selected for feature-level distillation. This enables the compact student model (CP-Mobile) to capture both semantic distribution and structural information from teacher guidance. Experiments on the TAU Urban Acoustic Scenes 2022 Mobile dataset (development set) demonstrate that our submitted systems achieve up to 59.30% accuracy. 1 1 1 https://github.com/HaoWLee/dcase2025_task1_inference Index Terms — Acoustic Scene Classification, Knowledge Distillation, Data Augmentation, Feature Distillation 1 Introduction Acoustic Scene Classification (ASC) aims to identify the environment in which an audio recording was captured, such as a street, shopping mall, or park, based on its acoustic characteristics [ 1 , 2 ] . The DCASE 2025 Challenge Task 1 focuses on developing low-complexity ASC models that are robust to domain shifts across mobile recording devices and diverse urban environments. The challenge emphasizes generalization across devices under strict constraints on model size and computational cost. The task uses the TAU Urban Acoustic Scenes 2022 Mobile dataset [ 3 ] , which contains approximately 64 hours of audio recordings under 10 acoustic scenes. Compared with previous editions, the 2025 challenge emphasizes on robustness to unseen-device conditions and data efficiency. Specifically, models must be trained using only a 25% subset of the official training set. In addition, submitted systems must not exceed 128kB parameter memory and 30M multiply-accumulate operations (MACs) per inference pass. To address these challenges, knowledge distillation (KD) has been widely adopted to train compact student networks under the supervision of larger teacher networks [ 4 , 5 ] . KD was first introduced by Hinton et al. [ 6 ] as a technique to compress large models into smaller ones by transferring soft target distributions. Since then, KD has evolved into a general learning paradigm with various forms of knowledge, including output logits, intermediate features, attention maps to relational structures [ 7 ] . In DCASE challenges, prior work has largely focused on output-level distillation using logits, CPJKU’s submission in DCASE2023 achieved strong performance under low-complexity constraints [ 8 ] . However, few studies have explored feature-level supervision, which has been shown to offer additional benefits in general deep learning settings [ 9 , 10 ] . Models such as the Patchout Spectrogram Transformer (PaSST) [ 11 ] and Convolutional Patch-ResNet (CP-ResNet) [ 12 ] have demonstrated strong performance in ASC and served as effective teachers for compact CNN-based architectures [ 8 ] , and early investigations into model frameworks for ASC [ 13 , 14 ] have also offered guidance for this work. In this work, we employ a dual-level knowledge distillation framework that combines output-level and feature-level supervision to improve the training of a compact CP-Mobile [ 15 ] student model. Specifically, we ensemble multiple high-performing teacher models (CP-ResNet and PaSST) to provide complementary guidance through soft target distributions. Inspired by advances in feature-based distillation such as FitNets [ 9 ] and SimKD [ 10 ] , we further align intermediate representations between a designated teacher and the student network to enhance structural transfer. This report is organized as follows. Section 2 describes the input feature extraction process and data augmentation techniques. Section 3 details the knowledge distillation framework, including student-teacher architecture, feature matching strategies, and teacher model training procedures. Section 4 reports the configuration and evaluation results of submitted systems for DCASE 2025 Task 1. Finally, Section 5 summarizes the key findings and concludes the report. 2 Data preprocessing and augmentation 2.1 Preprocessing All models operate on 32 kHz audio. Log-Mel spectrograms are extracted using configurations customized for each model to balance time-frequency resolution and computational efficiency. Figure 1: Overview of the joint feature and logit distillation framework. Teacher A-D provide ensembled soft targets for output-level distillation, and teacher D supervises feature-level distillation. Teacher models: We use 2 architectures as teacher models: PaSST and CP-ResNet. For each architecture, we employ different spectrogram preprocessing settings to emphasize either frequency or temporal resolution: • PaSST-1: 1024-point FFT, 800-point window, 320-point hop, 128 Mel bins • PaSST-2: 4096-point FFT, 800-point window, 320-point hop, 128 Mel bins • CP-ResNet-1: 4096-point FFT, 3072-point window, 750-point hop, 256 Mel bins • CP-ResNet-2: 4096-point FFT, 3072-point window, 500-point hop, 256 Mel bins Student model: CP-Mobile [ 15 ] uses the same 32 kHz audio input and adopts a 4096-point FFT, 3072-point window, and a 500-point hop size, producing 256 Mel bins. Compared to CP-ResNet, this configuration provides improved temporal resolution, which aligns better with the receptive fields of lightweight models under complexity constraints. 2.2 Data Augmentation To improve model generalization under limited supervision and domain shift, three augmentation strategies are employed: time-domain rolling, frequency-domain MixStyle (Freq-MixStyle), and device impulse response (DIR) convolution. Time Roll: Input waveforms are circularly shifted along the time axis to introduce temporal variability while preserving semantic content. For example, CP-Mobile and PaSST-1 apply a shift of up to 312 ms (10,000 samples at 32 kHz), while other models adopt a shorter shift of 125 ms (4,000 samples). Freq-MixStyle: Following the domain generalization framework MixStyle [ 16 ] , we adopt a frequency-wise variant tailored to log-Mel spectrograms. With probability p , sample-wise channel statistics are interpolated using a Beta distribution with parameter \alpha_{\text{mix}} to perturb style-related information and improve cross-device robustness. This augmentation is only applied to CP-ResNet teachers. DIR Augmentation: To simulate the acoustic coloration introduced by different devices, waveforms are convolved with DIRs sourced from the MicIRP dataset [ 17 ] . Each training example undergoes DIR-based augmentation with a specified probability to encourage invariance to microphone and channel characteristics. The augmentation configurations for each model are summarized in Table 1 . Table 1: Data augmentation configurations for teacher and student models. Model Time Roll DIR Prob. Freq-MixStyle PaSST-1 312 ms 0.6 ( \alpha_{\text{mix}} = 0.4, p = 0.4) PaSST-2 125 ms 0.4 ( \alpha_{\text{mix}} = 0.4, p = 0.8) CP-ResNet-1 125 ms 0.4 ( \alpha_{\text{mix}} = 0.4, p = 0.8) CP-ResNet-2 125 ms 0.6 ( \alpha_{\text{mix}} = 0.3, p = 0.4) CP-Mobile 312 ms 0.6 None 3 Training and knowledge distillation 3.1 Teacher Model Training All teacher models are trained independently on the 25% subset of the TAU Urban Acoustic Scenes 2022 Mobile dataset, using their respective log-Mel spectrogram configurations detailed in Section 2 . All teachers adopt Freq-MixStyle augmentation, while DIR convolution is selectively applied to improve robustness against microphone variability. We train 4 teacher models: 2 based on PaSST and 2 on CP-ResNet. For each architecture, one variant uses a spectrogram configuration emphasizing high frequency resolution (longer FFT and window), and the other favors higher temporal resolution (shorter window and hop size). This dual-resolution setup provides complementary time-frequency perspectives for KD. To improve teacher model generalization and stability of soft targets, we apply model soup [ 18 ] within each teacher model. Specifically, we select the top 5 checkpoints after training convergence and compute the average of their weights. This simple