Stack Less, Repeat More: A Block Reusing Approach for Progressive Speech Enhancement

Jangyeon, Ui-Hyeop, Jaehyun, Hyung-Min · 2025

Read Stack Less, Repeat More: A Block Reusing Approach for Progressive Speech Enhancement by Jangyeon, Ui-Hyeop, Jaehyun, Hyung-Min (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-19401.txt.

Introduction

Jangyeon Kim and three co-authors clean up noisy speech with a reused processing block in a 2025 paper. Speech enhancement usually stacks many different blocks to build deep representations. Stacking adds parameters and redundancy, even though later stages mostly refine what earlier ones did.

Their method repeats one block many times instead, keeping the entry and exit stages shallow. Each pass refines the noisy input a bit more in place. Tests show the number of passes matters more than having many distinct blocks. Deepening the entry and exit stages added little once reuse was in place. The design offers a leaner path to clear speech for calls and recordings.

Catalog record

CREATORJangyeon, Ui-Hyeop, Jaehyun, Hyung-Min
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE21 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-19401

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Stack Less, Repeat More: A Block Reusing Approach for Progressive Speech Enhancement Jangyeon, Ui-Hyeop, Jaehyun, Hyung-Min Abstract This paper presents an efficient speech enhancement (SE) approach that reuses a processing block repeatedly instead of conventional stacking. Rather than increasing the number of blocks for learning deep latent representations, repeating a single block leads to progressive refinement while reducing parameter redundancy. We also minimize domain transformation by keeping an encoder and decoder shallow and reusing a single sequence modeling block. Experimental results show that the number of processing stages is more critical to performance than the number of blocks with different weights. Also, we observed that the proposed method gradually refines a noisy input within a single block. Furthermore, with the block reuse method, we demonstrate that deepening the encoder and decoder can be redundant for learning deep complex representation. Therefore, the experimental results confirm that the proposed block reusing enables progressive learning and provides an efficient alternative for SE. Kim Shin Ko Park Sogang UniversityRepublic of Korea Sogang UniversityRepublic of Korea keywords speech enhancement, block reusing, progressive refinement, dual-path architecture, parameter efficiency † † email: {jykim97, dmlguq123, jhko, hpark}@sogang.ac.kr 1 Introduction † † footnotetext: *Equal contribution. † † footnotetext: † Corresponding Author. With the rapid development of deep learning, speech enhancement (SE) has been significantly improved as a critical pre-processing step in various applications. In particular, from the conventional real-valued masking in time-frequency (TF) with Short-Time Fourier Transform (STFT), the estimation of complex mask has led to substantial improvement of SE [ 1 , 2 ] . Furthermore, from U-Net-based architectures [ 1 , 2 , 3 , 4 , 5 ] , dual-path modeling in the TF domain further boosted performance in both speech separation (SS) [ 6 , 7 , 8 ] and SE [ 9 , 10 , 11 , 12 ] . More recently, in addition to mask-based approaches, studies based on direct spectral mapping also has been explored, which eliminates the need for masking by outputting enhanced signals either partially [ 11 , 12 ] or in full [ 7 , 8 ] . Progress in dual-path modeling has been driven by the development of sequence modeling capacity of Transformer-based architectures [ 8 , 9 , 10 , 11 ] . Transformer-based dual-path models have demonstrated remarkable performance improvements by stacking multiple layers. At the same time, while model performance has continued to improve, there has been a notable shift where model sizes have generally decreased, whereas computational complexity has increased. This trend is evident in the transition from U-Net-based architectures to Transformer-based TF models. It is also reported that the extent of feature processing, influenced by computational complexity, plays a more critical role than the complexity of domain transitions between different representations with a number of parameters and non-linear projections in the model [ 13 ] . Based on this observation, we consider an alternative approach where we reuse a processing block repeatedly instead of stacking multiple processing blocks. These weight sharing across layers have been actively explored in automatic speech recognition [ 14 , 15 ] or natural language processing [ 16 , 17 ] . However, since these tasks often involve transforming input into different domains, sharing weights can lead to performance limitations [ 18 ] . To mitigate this, partial weight sharing [ 14 ] or adapter-based mechanisms [ 18 , 19 , 20 ] have been introduced. On the other hand, in SS, such iterative block reusing technique has been successfully applied [ 21 , 22 ] and shown to be effective. This is likely because SS operates within the same domain for both input and output. When the input and output domains remain the same, we hypothesize that progressive refinement through iterative block reuse can serve as a more efficient alternative to learning deep latent representations using a model with multiple blocks. Motivated by this, we introduce the block reusing method in SE. Compared to conventional stacking, reusing a block does not result in significant performance degradation in SE. Furthermore, we investigated the impact of the number of stacked and repeated blocks. Empirically, the number of processing stages itself plays a more critical role for performance than the number of blocks with unique weights. Also, our experiments show that training with a repeated block naturally leads a single block to progressively refine the input in contrast to conventional stacked architectures. Moreover, a deep encoder-decoder structure for learning deeper latent representations can be redundant. To validate our hypothesis, we selected MP-SENet [ 11 ] as a baseline which has a convolutional encoder-decoder (CED) architecture that has been widely adopted in recent models [ 1 , 2 , 5 , 10 ] . Then, we minimized domain transformation by keeping CED shallow while increasing the repetition of a single Transformer-based dual-path block in the intermediate layer. Notably, the proposed model retains competitive performance despite the reduction in parameters and computational complexity. This demonstrates that reusing blocks enables progressive feature refinement while significantly reducing parameters, making it an efficient approach for SE tasks without relying on deep latent features learned through a deep encoder and decoder. (a) (b) Figure 1 : Comparison of block (a) stacking and (b) repeating 2 The proposed method The SE task fundamentally involves generating an output signal that retains only the desired speech components while remaining in the same domain as the input. Therefore, rather than learning complex representations with multiple blocks, a progressive refinement approach through iterative processing with a single block can serve as an effective strategy. 2.1 Overview of SE network As illustrated in Figure 1 (a), conventional SE methods are often based on the CED architecture with stacked sequence modeling blocks [ 10 , 11 , 12 ] . When a noisy signal is given in the TF domain with T frames and F frequency bins, the noisy input is encoded as a feature representation with the shape of {C\hskip-1.42262pt\times\hskip-1.42262ptT\hskip-1.42262pt\times\hskip-2.84526ptF} in TF domain, where C is the feature dimension. The encoded feature is then processed by a stack of B sequence modeling blocks. Finally, the decoder transforms the output feature back to its original shape to estimate either the mask value or the direct speech signal. 2.2 The proposed block reusing method Instead of stacking multiple blocks, we propose using a single sequence modeling block repeatedly as shown in Figure 1 (b), which may lead to progressive enhancement without explicit intermediate loss. In particular, this strategy, often referred to as the unfolding method, can have some variants depending on the information fusion scheme between processing stages [ 21 ] . When using the output of a shared block as input in subsequent iterations, we can simply reuse the output of the previous stage as input to the next stages as a direct connection (DC). Meanwhile, we can also add initial feature to the each stage output as a summation connection (SC) to encourage leveraging the input feature. On the other hand, simply reusing the single block can be challenging for more complex noisy inputs. Therefore, we can consider repeating stacks of blocks to enhance modeling capacity of sequence block. 2.3 Dual-path TF model for SE To validate and analyze our block reuse method,

Read the complete text