SE-AGCNet: An End-to-End Framework for Joint Speech Enhancement and Loudness Control in Meeting Scenarios

Jinming, Wei, Xionghu, Eng Siong · 2026

Read SE-AGCNet: An End-to-End Framework for Joint Speech Enhancement and Loudness Control in Meeting Scenarios by Jinming, Wei, Xionghu, Eng Siong (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-25959.txt.

Introduction

Jinming Zhang and colleagues merge two audio jobs that meeting software usually runs in sequence: speech enhancement, which cleans noise from voices, and automatic gain control, which evens out loudness. Running gain first can amplify the noise, while cleaning first can erase quiet speech. Their SE-AGCNet framework trains both tasks together in one network, designed for meetings where speakers vary widely in volume.

Synergy between the tasks is the point: enhancement that protects quiet speech gives the gain stage something to work with. A companion data pipeline, SE-AGC-DataGen, simulates training material, and scoring uses standard loudness measures including integrated loudness in LUFS and loudness range. Across their experiments the joint system reaches target loudness reliably while improving speech quality and speech-recognition accuracy over strong separate-module baselines.

Catalog record

CREATORJinming, Wei, Xionghu, Eng Siong
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE23 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.25959

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SE-AGCNet: An End-to-End Framework for Joint Speech Enhancement and Loudness Control in Meeting Scenarios Jinming, Wei, Xionghu, Eng Siong Abstract Conventional audio pipelines typically treat speech enhancement (SE) and automatic gain control (AGC) as discrete modules, which often limits overall performance. For instance, applying AGC before SE may inadvertently amplify background noise, while prioritizing SE tends to over-suppress low-volume speech. To address these limitations, we propose SE-AGCNet, an end-to-end framework that jointly optimizes SE and AGC. Tailored for meeting scenarios with significant volume variations, SE-AGCNet leverages the synergy between the two tasks: SE preserves quiet speech, thereby facilitating effective volume adjustment by the AGC component. Furthermore, we propose a specialized data simulation pipeline, SE-AGC-DataGen, and incorporate standardized loudness evaluation metrics: integrated loudness (LUFS), short-term loudness (St LUFS), and LRA. Experiments show that SE-AGCNet consistently achieves target loudness while improving speech quality and ASR accuracy over competitive baselines. Zhang Rao Zhong Chng keywords Speech Enhancement, Automatic Gain Control, Joint Training, Meeting Room Acoustics † † address: 1 Zhejiang University, China 2 Nanyang Technological University, Singapore 3 Hunan University, China † † email: [email protected], [email protected] 1 Introduction The audio front-end is traditionally characterized by the ”3A algorithms”: Acoustic Echo Cancellation (AEC), Noise Suppression (also referred to as speech enhancement), and Automatic Gain Control (AGC). Conventional audio pipelines typically implement AGC [ 19 , 6 , 1 , 8 , 23 , 22 , 18 , 12 , 11 , 2 , 13 , 17 ] and Speech Enhancement (SE) [ 10 , 25 , 24 ] as separate, cascaded modules, a design that introduces several limitations. When AGC precedes SE, it amplifies speech and noise indiscriminately, thereby reducing the signal-to-noise ratio and complicating subsequent denoising. When applied after SE, AGC performance becomes highly dependent on SE quality, as residual noise is often unintentionally amplified. A further challenge arises from SE models’ tendency to over-suppress low-volume speech. Jointly training SE and AGC provides a principled solution to the limitations of cascaded pipelines. Through simultaneous optimization, the SE module is encouraged to preserve low-volume speech while concentrating on noise suppression, with the assurance that volume normalization will be managed by the AGC module. This synergistic framework mitigates two key issues: it reduces the risk of SE misclassifying far-field or quiet speech as noise, and it prevents AGC from inadvertently amplifying residual background noise. Recent studies have attempted to integrate AEC, SE, and AGC within unified frameworks. For example, [ 21 ] introduced a unified 3A framework but treated AGC merely as a detached post-processing module without specifying its underlying algorithm. Similarly, [ 27 ] explored the joint training of AEC, SE, and AGC. However, their AGC targets were generated by a proprietary in-house AGC tool, which may cap the model’s achievable performance and hinder reproducibility. To address these limitations, this paper proposes SE-AGCNet, a deep-learning-based framework for joint SE and AGC. The proposed approach is designed for meeting-room scenarios, where audio quality is often degraded by background noise, reverberation, and volume discrepancies arising from differences in distance from the microphone, variations in speaking style, and head movements or shifts in body position. Experimental results demonstrate that SE-AGCNet achieves superior performance in both SE and AGC tasks, as well as in downstream Automatic Speech Recognition (ASR) tasks, achieving significant reductions in WER/CER while maintaining optimal loudness characteristics compared with conventional methods. Our main contributions are summarized as follows: • We propose SE-AGCNet 1 1 1 Code and demo: https://jinming00.github.io/SE-AGCNet/ , a flexible framework for joint, end-to-end training of SE and AGC. This framework supports integration with diverse existing speech enhancement architectures. • We propose SE-AGC-DataGen, a comprehensive and reproducible data simulation pipeline that addresses the lack of publicly available AGC datasets and enables joint SE-AGC training. • We introduce standardized loudness evaluation metrics (integrated loudness (LUFS), short-term loudness (St LUFS), and Loudness Range (LRA)) based on ITU-R BS.1770 [ 7 ] and EBU R128 [ 5 ] to enable more objective and perceptually meaningful evaluation of AGC performance. Figure 1: Overview of SE-AGCNet architecture. The system processes input audio through speech enhancement and automatic gain control modules in a joint training framework. SE Target Audio: clean and volume-unbalanced speech for SE module training; AGC Target Audio: clean and volume-balanced speech for AGC module training. 2 Proposed SE-AGCNet Figure 1 illustrates the SE-AGCNet architecture. Our approach employs a two-stage architecture that jointly optimizes speech enhancement and automatic gain control in the time-frequency domain, transforming a noisy and volume-unbalanced speech waveform y\in\mathbb{R}^{L} into a clean and volume-balanced output \hat{x}\in\mathbb{R}^{L} , where L denotes the waveform length. Specifically, we first extract the magnitude spectrum Y_{m}\in\mathbb{R}^{T\times F} and phase spectrum Y_{p}\in\mathbb{R}^{T\times F} from the input waveform y through STFT, where T and F denote the number of frames and frequency bins, respectively. The speech enhancement module processes the noisy input to produce enhanced magnitude spectrum \hat{X}_{se,m}\in\mathbb{R}^{T\times F} and enhanced phase spectrum \hat{X}_{se,p}\in\mathbb{R}^{T\times F} : \hat{X}_{se,m},\hat{X}_{se,p}=\text{SE}(Y_{m},Y_{p};\theta_{se}) (1) where \theta_{se} denotes the speech enhancement model parameters. The enhanced magnitude spectrum is subsequently RMS-normalized and processed by the AGC module to generate a volume-balanced magnitude spectrum: \hat{X}_{agc,m}=\text{AGC}(\text{RMS-norm}(\hat{X}_{se,m});\theta_{agc}) (2) where \theta_{agc} denotes the AGC module parameters. Finally, the enhanced and volume-balanced waveform \hat{x} is reconstructed via ISTFT using the AGC-processed magnitude spectrum and the enhanced phase from the SE module. SE-AGCNet addresses two fundamental challenges: (1) the SE module performs noise suppression while preserving low-volume speech; and (2) the AGC module handles loudness adjustment. Through joint training, these modules operate synergistically in an end-to-end optimization framework. 2.1 Speech Enhancement Model We adopt MP-SENet as the SE backbone and keep its original architecture and training setup (16 kHz sampling rate, 400-sample STFT window, 100-sample hop size, 400-point FFT, and 2-second training segments). Our only modification is an asymmetric reweighting strategy [ 20 ] . Specifically, the MP-SENet loss definition remains unchanged, but for each time-frequency bin (t,f) , if the predicted magnitude is lower than the target magnitude (i.e., \hat{X}_{se,m}^{t,f}<X_{se,target,m}^{t,f} ), we multiply the corresponding SE loss contribution by \alpha=10.0 ; otherwise, we keep its original weight. This imposes a 10\times stronger penalty on over-suppression. During SE training, the target is clean but volume-unbalanced speech, so the model learns noise suppression while preserving loudness variation for the downstream AGC module. 2.2 AGC Module The AGC module performs volume adjustment by processing the RMS-normalized enhanced magnitude spectrum from MP-SENet to produce volume-balanced outputs. 2.2.1 Architecture Design The AGC module takes the RMS-normalized enhanced magnitude spectrum \hat{X}_{se,m}^{norm}\in\mathbb{R}^{T\times F} from the SE module as input and produces volume-balanced magnitude spectrum \hat{X}_{agc,m}\in\mathbb{R}^{T\times F} as output. During training, the AGC target audio is the clean and volume-balanced magnitude spectrum X_{agc,target,m} . The AGC module consists of three components: frequency-domain convolutional processing, bidirectional LSTM processing, and spectral reconstruction. The frequency-domain component extracts features through two 2D convolution layers: H_{1}=\text{ReLU}(\text{BN}(\text{Conv2D}(\hat{X}_{se,m}^{norm}))) (3) H_{conv}=\text{ReLU}(\text{BN}(\text{Conv2D}(H_{1}))) (4) where H_{conv}\in\mathbb{R}^{T\times F\times 16} represents frequency-aware features extracted by 16-channel convolutions with 3\times 3 kernels. The bidirectional LSTM processes temporal sequences by reshaping the features to (T,F\times 16) and using a 2-layer BiLSTM with hidden size 256 per direction. The LSTM output is then projected back to F\times 16 through linear layers before reconstruction. Finally, the reconstruction component employs transposed convolutions: \hat{X}_{agc,m}=\text{ReLU}(\text{ConvT}(\text{ReLU}(\text{BN}(\text{ConvT}(H_{lstm}))))) (5) with progressive channel reduction ensuring

Read the complete text