Reducing Linguistic Hallucination in LM-Based Speech Enhancement via Noise-Invariant Acoustic-Semantic Distillation DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; Under review; 2026 8082 CCS: Computing methodologies Natural language generation
Zheng Wang, Xiaobin Rong, Hang Su, Tianyi Tan, Junnan Wu, Lichun Fan, Zhenbo Luo, Jian Luan, Jing Lu · 2026
Read Reducing Linguistic Hallucination in LM-Based Speech Enhancement via Noise-Invariant Acoustic-Semantic Distillation DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; Under review; 2026 8082 CCS: Computing methodologies Natural language generation by Zheng Wang, Xiaobin Rong, Hang Su, Tianyi Tan, Junnan Wu, Lichun Fan, Zhenbo Luo, Jian Luan, Jing Lu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-08608.txt.
Introduction
Zheng Wang and nine coauthors clean up noisy speech with a language model while stopping it from inventing words. Under heavy noise, such enhancers can output smooth speech that says the wrong thing. Their system, L3-SE, learns conditioning signals that stay stable in noise by copying two clean targets at once: one for sound quality, one for word content.
A high fidelity audio codec built from layered speech features feeds the generator with clean tokens. Tests show fewer invented words and stronger content faithfulness than earlier language model enhancers, especially in low signal and echoing rooms, while sound quality stays competitive. Speech technology gains reliability where noise is worst.
Catalog record
| CREATOR | Zheng Wang, Xiaobin Rong, Hang Su, Tianyi Tan, Junnan Wu, Lichun Fan, Zhenbo Luo, Jian Luan, Jing Lu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.08608
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Reducing Linguistic Hallucination in LM-Based Speech Enhancement via Noise-Invariant Acoustic-Semantic Distillation DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; Under review; 2026 8082 CCS: Computing methodologies Natural language generation Zheng Wang, Xiaobin Rong, Hang Su, Tianyi Tan, Junnan Wu, Lichun Fan, Zhenbo Luo, Jian Luan, Jing Lu Abstract . Language model (LM)-based speech enhancement (SE) can generate natural-sounding speech, but under severe noise it often suffers from unreliable conditioning, leading to perceptually plausible yet linguistically incorrect outputs. To address this issue, we propose L3-SE, a noise-invariant acoustic-semantic distillation framework for reducing linguistic hallucination in LM-based SE. The proposed method learns a noise-invariant conditioning encoder from noisy speech by jointly distilling two complementary clean-speech targets: an acoustic target for reconstruction fidelity and a semantic target for linguistic consistency. The resulting noise-invariant acoustic-semantic representations are used to condition a decoder-only autoregressive language model, which predicts clean acoustic tokens that are decoded into enhanced speech. To support high-quality generation, we further employ a high-fidelity codec built on learnable weighted WavLM layer representations as the discrete acoustic interface. By improving the reliability of conditioning under adverse conditions, the proposed framework substantially reduces hallucination and improves content faithfulness. Experiments show that the proposed method consistently outperforms prior LM-based speech enhancement baselines on linguistic consistency metrics, with especially clear gains under low-SNR and reverberant conditions, while maintaining competitive perceptual quality. Audio samples are available at https://max1wz.github.io/L3-SE-Demo-Page/ . The complete source code will be released after the manuscript is accepted. 1. Introduction Speech enhancement (SE) aims to recover clean speech from noisy mixtures, improving perceptual quality and intelligibility for both human listening and downstream applications such as speech communication, automatic speech recognition (ASR) ( 3 ) , and speaker verification ( 21 ) . Deep-learning-based SE methods generally fall into two families: discriminative (or predictive) ( 50 ; 62 ) and generative approaches. While discriminative models can suppress noise effectively, they often over-suppress speech components under severe noise or reverberation, degrading naturalness ( 51 ; 52 ) . In contrast, generative SE produces enhanced speech via generation formulations, including diffusion models ( 25 ; 32 ) , flow-matching models ( 51 ; 26 ) , and language model (LM)-based approaches ( 60 ; 20 ; 57 ) , demonstrating strong capabilities in synthesizing speech with superior perceptual quality. However, generative SE also introduces a fundamental challenge: perceptual plausibility does not necessarily imply content faithfulness ( 44 ) . When the conditioning information extracted from distorted speech is unreliable, the generative prior may dominate and produce outputs that sound clean but deviate from the original spoken content, for example through word substitutions, deletions, or other semantic distortions. This failure mode, referred to as linguistic hallucination ( 41 ) , becomes especially pronounced under severe noise and reverberation. Therefore, for generative SE, the central issue is not only whether the model can generate natural speech, but whether its conditioning representations remain robust enough to preserve the original linguistic content. Among generative approaches, LM-based speech enhancement (SE) is particularly promising because autoregressive language models can capture long-range temporal dependencies and high-level linguistic structure. Existing LM-based methods mainly follow either a single-stage or a two-stage design. In single-stage pipelines, degraded-speech representations are directly used to condition an LM for token prediction ( 20 ; 57 ) . While simple and effective, this design exposes the generator to corrupted conditioning, making it more vulnerable to linguistic hallucination under adverse conditions. In two-stage pipelines, the intermediate representation is first refined and then used to guide token generation in a second stage ( 60 ; 33 ; 59 ) . This coarse-to-fine strategy suggests that improving conditioning before generation can be beneficial. However, existing methods typically focus on only one type of representation, either acoustic representations derived from neural audio codecs (NACs) ( 24 ) or semantic representations extracted by ASR or self-supervised learning (SSL) speech encoders ( 37 ; 4 ; 16 ) . This is limiting for hallucination-resistant LM-based SE, since acoustic representations support reconstruction fidelity, speaker characteristics, and prosody, whereas semantic representations are crucial for maintaining linguistic consistency and preventing content drift. A related line of work improves downstream robustness by learning noise-invariant speech representations from corrupted inputs ( 48 ; 65 ; 41 ; 14 ) . While these studies show that robust upstream representations can substantially benefit downstream performance, they are still typically centered on a single aspect of the speech signal, most often semantic or contextualized representations for robust recognition and invariance. For LM-based SE, however, reducing linguistic hallucination requires both semantic robustness and acoustic robustness. This motivates our design of a noise-invariant encoder jointly constrained by acoustic and semantic targets, rather than optimizing either aspect alone. We address this problem with L3-SE , a framework for L owering L inguistic hallucination in L M-based S peech E nhancement via noise-invariant acoustic-semantic distillation. Starting from a single SSL encoder, WavLM ( 4 ) , we build two task-specialized teacher models, WavCodec and WavS2T, which share the same WavLM backbone but are optimized for different objectives. WavCodec is trained with a speech codec objective, while WavS2T is trained with a speech-to-transcript objective. Rather than manually selecting a fixed backbone layer, each teacher learns its own task-specialized layer aggregation, yielding complementary representations that emphasize either low-level acoustic detail or high-level linguistic content. We then use the encoder parts of WavCodec and WavS2T as the acoustic and semantic teacher encoders, respectively, freeze them, and train a noisy-speech student, NI-Encoder, to match the corresponding clean-speech teacher targets through representation-level alignment losses. NI-Encoder follows the same overall architectural form as the teachers, namely a shared WavLM backbone followed by acoustic and semantic heads, and produces acoustic-semantic representations from the same noisy input. In this way, low-level acoustic fidelity and high-level linguistic consistency are jointly imposed on a single noise-invariant encoder, providing more reliable conditioning for downstream generation. Built on the learned noise-invariant acoustic-semantic representations, we further employ an LM to generate clean acoustic tokens for speech reconstruction. The resulting L3-SE framework contains three components: (i) NI-Encoder, which produces noise-invariant acoustic-semantic conditioning representations; (ii) an autoregressive LM that predicts clean acoustic tokens; and (iii) the WavCodec decoder, which converts predicted tokens into waveforms. Here, WavCodec primarily serves as a discrete acoustic interface for LM-based generation and waveform reconstruction, while the main contribution of this work lies in improving the robustness and faithfulness of the conditioning pathway through acoustic-semantic distillation. By conditioning the LM on the distilled representations, L3-SE reduces linguistic hallucination and improves content faithfulness while preserving competitive perceptual quality. The main contributions of this work are summarized as follows: • Noise-invariant acoustic-semantic distillation for LM-based speech enhancement. We propose a joint distillation strategy that learns noise-robust conditioning representations by simultaneously matching acoustic and semantic clean-speech targets, improving both reconstruction fidelity and linguistic consistency under adverse conditions. • A shared-backbone teacher-student design for reducing linguistic hallucination. We construct acoustic and semantic teachers on top of the same SSL backbone and use them to supervise a student encoder with acoustic and semantic branches, enabling paired conditioning representations for LM-based speech enhancement. • An LM-based speech enhancement framework with improved content faithfulness. Built on NI-Encoder and a discrete