MFLA: Monotonic Finite Look-ahead Attention for Streaming Speech Recognition

Yinfeng, Huiyan, Chenyang, Manhong, Yutao, Xingyang, Yanmin · 2025

Read MFLA: Monotonic Finite Look-ahead Attention for Streaming Speech Recognition by Yinfeng, Huiyan, Chenyang, Manhong, Yutao, Xingyang, Yanmin (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-03722.txt.

Introduction

Xia and co-authors adapt a famous speech-recognition model for live transcription, in a paper published in 2025. Big pre-trained models transcribe beautifully but want the whole utterance before answering, which live captioning cannot wait for. The team fine-tunes the model under a prefix-to-prefix scheme: partial audio in, partial text out. A mechanism borrowed from neural firing builds a near-monotone alignment between sounds and letters as speech unfolds.

Their attention design lets each word look back freely but only a little way forward, capping delay. A wait-k decoding rule, which reads a few chunks before writing, keeps decoding simple and consistent. The result is streaming recognition without training a model from scratch. The authors test the approach on standard speech sets. Reusing a giant model's ear for live work is the paper's contribution.

Catalog record

CREATORYinfeng, Huiyan, Chenyang, Manhong, Yutao, Xingyang, Yanmin
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE22 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-03722

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MFLA: Monotonic Finite Look-ahead Attention for Streaming Speech Recognition Yinfeng, Huiyan, Chenyang, Manhong, Yutao, Xingyang, Yanmin Abstract Applying large pre-trained speech models like Whisper has shown promise in reducing training costs for various speech tasks. However, integrating these models into streaming systems remains a challenge. This paper presents a novel prefix-to-prefix training framework for streaming recognition by fine-tuning the Whisper. We introduce the Continuous Integrate-and-Fire mechanism to establish a quasi-monotonic alignment between continuous speech sequences and discrete text tokens. Additionally, we design Monotonic Finite Look-ahead Attention, allowing each token to attend to infinite left-context and finite right-context from the speech sequences. We also employ the wait- k decoding strategy to simplify the decoding process while ensuring consistency between training and testing. Our theoretical analysis and experiments demonstrate that this approach achieves a controllable trade-off between latency and quality, making it suitable for various streaming applications. Xia Li Le Wang Sun Ma Qian † 1 Honor Device Co, Ltd, China 2 Auditory Cognition and Computational Acoustics Lab MoE Key Lab of Artificial Intelligence, AI Institute School of Computer Science, Shanghai Jiao Tong University, Shanghai, China keywords Streaming speech recognition, Whisper, Monotonic attention † † email: {xiayinfeng,lihuiyan}@honor.com, {yanminqian}@sjtu.edu.cn 1 Introduction As a framework for weakly supervised pre-training on large-scale datasets, Whisper [ 1 ] has shown strong performance in multilingual recognition, but reveals a significant inference delay. Although methods such as knowledge distillation [ 2 , 3 ] and speculative decoding [ 4 ] , have been proposed to improve inference speed, they do not alter the fundamental nature of the system as a sequence-to-sequence model, limiting its applicability only to offline systems. In contrast, online (streaming) recognition systems employing prefix-to-prefix models are capable of satisfying latency requirements in certain specific scenarios, such as real-time subtitles. However, integrating the Whisper model into streaming systems presents significant challenges, primarily due to asynchronous processing problem [ 5 ] and unreliable boundary transcription. The challenge of asynchronous processing arises from the fundamental discrepancy between training and inference conditions: conventional sequence-to-sequence models leverage the full source context during training, while online systems must generate predictions incrementally based on partial source inputs. This inherent mismatch can be effectively addressed through monotonic attention mechanisms. Motivated by the observation of a roughly monotonic alignment between inputs and outputs, Raffel et al. [ 6 ] proposed a differentiable approach that enables end-to-end alignment learning during training and linear-time decoding through hard monotonic constraints. Subsequent advancements introduced three principal variants: Monotonic Infinite Lookback Attention (MILkA) [ 7 ] , Monotonic Chunkwise Attention (MoChA) [ 8 ] , and Monotonic Multihead Attention (MMA) [ 9 ] . Online recognition systems typically process input as fixed-length speech chunks, which can lead to unreliable transcription at chunk endpoints due to random truncation. To address these boundary-induced errors, the improved wait- k policy [ 10 ] delays processing until the first k chunks have been received and then output at a fixed rate r ; Macháček et al. integrated the Local Agreement policy [ 11 ] with self-adaptive latency into Whisper, identifying the longest common prefix between two consecutive chunks as stable hypotheses; Simul-Whisper [ 12 ] halted decoding at the appropriate time and discarded unreliable transcriptions, this dual-protection strategy further minimized the risk of performance degradation in online systems. Although previous studies have proposed various solutions, these approaches often remain one-dimensional and struggle to balance latency and quality in streaming speech recognition systems. A key challenge in predicting the current token is its heavy influence from boundary ambiguity and acoustic similarity, which makes it inherently dependent on the appropriate context of the speech sequence. This dependency frequently leads to a predictable degradation in recognition quality when using conventional monotonic attention mechanisms. Furthermore, the improved wait- k policy required setting a fixed output rate r prior to decoding, making recognition latency and quality sensitive to variations in speaking speed and presence of silence segments, respectively. The Local Agreement policy introduced a higher fixed delay, while Simul-Whisper was hindered by a complex online decoding pipeline. These limitations highlight the need for more robust and flexible solutions for streaming speech recognition systems. In this paper, we propose a novel prefix-to-prefix fine-tuning approach based on the pre-trained Whisper model, resulting in the fine-tuned model named Streaming-Whisper. The key contributions of this paper are summarized as follows: 1. We introduce a predictor based on the Continuous Integrate-and-Fire (CIF) mechanism to estimate the number of target tokens, thereby establishing a quasi-monotonic alignment between continuous speech sequences and discrete tokens. 2. We develop Monotonic Finite Look-ahead Attention (MFLA) to enable each token to dynamically attend to both the infinite left-context and the finite right-context windows, which transforms the training paradigm from conventional sequence-to-sequence to a more efficient prefix-to-prefix framework. 3. We adopt the efficient wait- k decoding strategy, which not only eliminates the complexity associated with additional decoding processes but also achieves a superior trade-off between latency and recognition quality compared to the state-of-the-art Local Agreement policy. 2 Methods 2.1 Overview As shown in Figure 1 , the proposed Streaming-Whisper includes three modules: encoder, decoder, and predictor. The encoder converts the input speech sequence X=\{x_{1},x_{2},...,x_{T}\} into a hidden state sequence H=\{h_{1},h_{2},...,h_{T}\} , and defines h_{1:T}=\mathit{f}(x_{1:T}) ; the decoder employs the hidden states to produce the output sequence Y=\{y_{1},y_{2},...,y_{N}\} through an autoregressive process, and defines y_{i}=\mathit{g}(y_{i-1},h_{1:T}) . The predictor is adopted to establish the number of target tokens and guide the generation of MFLA, which will be discussed in detail in Section 2.2 and Section 2.3 . Figure 1: Structure of the proposed Streaming-Whisper. 2.2 Predictor The predictor consists of two linear layers and two ReLU activation layers, which can predict the token weight \alpha_{1:T} of each hidden state h_{1:T} . The expression of the predictor function is defined as \alpha_{j}=e(h_{j}) . We accumulate the weights \alpha_{1:T} to determine the number of target tokens, and the loss function is defined as the Mean Relative Error (MRE) loss. Subsequently, we introduce the CIF mechanism to establish a quasi-monotonic alignment between continuous speech signals and their corresponding discrete tokens, which delineates the temporal boundaries (left and right) for each target token. This mechanism provides three significant benefits: (1) during the training phase, it manages the finite right-context window to train online speech recognition systems by guiding the generation of MFLA; (2) during the incremental decoding stage, it allows for tracking the streaming decoding process and stopping it at the appropriate time to prevent unreliable boundary transcriptions; (3) it can monitor the entire decoding trajectory, thus mitigating common decoding repetition problems [ 13 ] . According to [ 14 , 15 ] , the weight \alpha is scaled by the target length N during training, while weight \alpha is used directly during inference. 2.3 Monotonic Attention Figure 2: MoChA in encoder and MFLA in decoder. In an offline speech recognition system, the encoder processes the entire speech sequence as input and outputs a corresponding sequence of hidden states; consequently, its self-attention mechanism operates under full-attention during training. In contrast, online systems employ encoders that sequentially process a series of fixed-size speech chunks. To adapt to this setting, we replace the conventional convolutional layers in front of the Whisper’s encoder transformer with causal convolution layers. Additionally,

Read the complete text