Model as Loss: A Self-Consistent Training Paradigm
Saisamarth Rajesh, Milos, Andrew · 2025
Read Model as Loss: A Self-Consistent Training Paradigm by Saisamarth Rajesh, Milos, Andrew (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-21156.txt.
Introduction
Saisamarth Rajesh Phaye and co-authors propose a new way to train speech cleaning models in this 2025 paper. Such models strip noise from recorded speech. Training often relies on hand-built loss rules or on features from a large speech model. The authors argue that these rules miss fine traits of clean speech that matter for good output.
Their method, called Model as Loss, uses the model's own encoder as the guide. That encoder already learned the traits that mark clean, clear speech for the task at hand. Training then pushes the decoder to make output that matches those traits in the clean reference. This self-guided setup aims for speech that sounds more natural and keeps the details listeners need.
Catalog record
| CREATOR | Saisamarth Rajesh, Milos, Andrew |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 23 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-21156
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Model as Loss: A Self-Consistent Training Paradigm Saisamarth Rajesh, Milos, Andrew Abstract Conventional methods for speech enhancement rely on handcrafted loss functions (e.g., time or frequency domain losses) or deep feature losses (e.g., using WavLM or wav2vec), which often fail to capture subtle signal properties essential for optimal performance. To address this, we propose Model as Loss, a novel training paradigm that utilizes the encoder from the same model as a loss function to guide the training. The Model as Loss paradigm leverages the encoder’s task-specific feature space, optimizing the decoder to produce output consistent with perceptual and task-relevant characteristics of the clean signal. By using the encoder’s learned features as a loss function, this framework enforces self-consistency between the clean reference speech and the enhanced model output. Our approach outperforms pre-trained deep feature losses on standard speech enhancement benchmarks, offering better perceptual quality and robust generalization to both in-domain and out-of-domain datasets. Phaye Cernak Harper Audio Machine LearningLogitech keywords speech enhancement, noise reduction, deep feature loss, loss functions † † email: (sphaye, mcernak, aharper)@logitech.com 1 Introduction Speech enhancement has long been a challenging problem, with applications in telecommunication, hearing aids, and robust automatic speech recognition (ASR) [ 1 , 2 , 3 , 4 , 5 ] . A critical component in training enhancement models is the choice of loss function, which directly influences the quality and generalization of enhanced output [ 6 , 7 ] . Conventional loss functions, such as time or spectrogram-domain losses [ 8 ] , often do not fully capture the complex relationships between noisy and clean speech. For instance, spectrogram loss treats all frequency bins equally, which can result in overemphasis of less perceptually relevant regions while underweighting crucial frequencies important for speech intelligibility [ 9 ] . Although there are perceptually sensitive losses such as mel-spectral losses [ 8 ] or PMSQE [ 9 ] , they overly compress the signal, limiting the preservation of fine-grained details crucial for maintaining speech intelligibility, achieving suboptimal performance [ 10 ] . Pre-trained deep feature losses [ 6 ] , such as those derived from WavLM [ 11 ] or Wav2Vec [ 12 ] , have gained popularity due to their ability to incorporate perceptual and contextual information into training objectives. Recent work by Babaev et al. [ 3 ] shows that WavLM’s intermediate convolutional features have a high correlation to speech enhancement, as compared to its transformer layers. WavLM has also been shown to be superior to Wav2Vec 2.0 [ 13 ] when used as a loss function. However, these methods are often optimized for tasks such as ASR or phoneme recognition, which may not align with speech enhancement objective. Such losses might prioritize linguistic content while ignoring residual noise components critical to the enhancement task. Moreover, these pre-trained neural networks, when used as loss functions, can suffer from limited sensitivity to noise, as the extracted features may focus on abstract representations rather than task-specific properties [ 14 ] . Baseline Baseline {}_{\text{10epochs}} Baseline {}_{\text{wavlm}} Ours {}_{\text{mal}-\text{frozen-fe}} Ours {}_{\text{mal}-\text{frozen}} Ours {}_{\text{mal}-\text{dynamic}} Noisy Spectrogram 1 Enhancement 10 Enhancements 100 Enhancements Figure 1: Comparison of models using iterative enhancement. Each row represents the number of iterative enhancements applied, where the output of the previous enhancement step is used as input for the next step. The columns show different models being compared. Figure 2: An illustration of the Model as Loss paradigm, showcasing the \mathcal{L}_{\text{mal}-{\text{dynamic}}} variation. To address these limitations, we propose a novel training paradigm, Model as Loss (MAL) , which uses the encoder of the same model as a loss function to guide the training of the decoder. It involves training a model with conventional loss functions and then using the trained encoder’s embeddings as a loss function for the next stage. This approach aligns the loss function with the downstream task, leveraging the encoder’s ability to extract task-specific features while ensuring contextual and hierarchical understanding of the signal. Unlike traditional methods that rely on external pre-trained models or handcrafted losses, this paradigm exploits the encoder’s specialization in processing noisy speech and its inherent ability to prioritize perceptually relevant signal components. Our proposed method has several advantages. First, the encoder’s feature space is inherently tailored to the enhancement task, capturing both global and local signal features crucial for noise suppression. Second, using the encoder as a loss function enforces a feedback loop that aligns training and inference dynamics, improving generalization to unseen noise types. Third, this approach ensures relevance across all frequency bins by leveraging the encoder’s weighting of spectral components based on their contribution to the task, avoiding the pitfalls of uniformly weighted losses. Finally, the self-consistency of the model provides a robust foundation for optimizing the decoder, leading to superior performance and perceptual quality. In this paper, we present a detailed analysis of the MAL paradigm, including its theoretical foundations and practical implementation. Through extensive experiments, we demonstrate that our approach outperforms conventional pre-trained deep feature losses and hand-crafted loss functions on standard speech enhancement benchmarks. The results highlight the potential of using the model itself as a loss function, offering a new perspective on loss design in machine learning. 2 Methodology In the realm of speech enhancement, models typically comprise an encoder and one or more decoders [ 5 ] . For simplicity, we will refer to this setup as a single encoder-decoder system moving forward. The encoder’s job is to extract relevant features from the noisy signal, which are then used by the decoder to synthesize the enhanced signal. The usual approach for training encoder-decoder models involves minimizing a loss function \mathcal{L} between the clean reference speech and the model output. For example, an L1 loss between the clean and enhanced signal in spectral domain using Short-time Fourier Transform (STFT) can be represented as: \mathcal{L}_{\text{spectral}}=\|\text{STFT}(\mathbf{y}_{\text{clean}})-\text{STFT}(\mathbf{y}_{\text{enhanced}})\|_{1} (1) where y clean is the clean reference speech and y enhanced is the model output when the noisy speech is used as input to the model. The objective is to minimize the difference between the model’s output and the clean speech, typically measured on a spectrogram or multiple spectrograms of different resolutions. Once we train a model with some loss \mathcal{L} to convergence, we know that for any given pair (y enhanced , y clean ), \mathcal{L} will be minimal. However, this does not imply that any other loss \mathcal{L} new will also be minimal. Our search is to find the ideal loss function \mathcal{L} ideal , such that, once trained to convergence, for any given mathematical function \mathcal{F} , we get the minimum loss: \mathcal{L}_{\text{F}}=\|\mathcal{F}(\mathbf{y}_{\text{clean}})-\mathcal{F}(\mathbf{y}_{\text{enhanced}})\|_{1} (2) Babaev et al. [ 3 ] propose a Signal-to-Noise (SNR) Rule , which suggests that as more noise is added to speech (lowering the SNR), feature representations should move farther apart in the embedding space. Extending this idea, intuitively, once the model is trained to satisfy equation ( 2 ), y enhanced and y clean should be identical points in the embedding space. Assuming equation ( 2 ) holds, if we input y clean to the trained model, it should return y clean . Moreover, y enhanced when used as input should again return y enhanced , which is y clean . This would ensure the stability of