Improving endpoint detection in end-to-end streaming ASR for conversational speech
Anandh C, Karthik Pandia Durai, Jeena Prakash, Manickavela Arumugam, Kadri Hacioglu, S. Pavankumar Dubagunta, Andreas Stolcke, Shankar Venkatesan, Aravind Ganapathiraju · 2025
Read Improving endpoint detection in end-to-end streaming ASR for conversational speech by Anandh C, Karthik Pandia Durai, Jeena Prakash, Manickavela Arumugam, Kadri Hacioglu, S. Pavankumar Dubagunta, Andreas Stolcke, Shankar Venkatesan, Aravind Ganapathiraju (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-17070.txt.
Introduction
Anandh C and co-authors sharpen endpoint detection for streaming speech recognition, in a 2025 paper. A voice assistant must know the instant you stop talking: cut early and your words are lost; wait and the reply lags. Transducer recognizers, the streaming standard, tend to emit words late, and that delay poisons the endpoint call.
Their fixes attack both faults. An end-of-word token after each word, trained with a delay penalty, makes the recognizer commit sooner. A helper network supplies trustworthy frame-level speech activity, grounding the endpoint decision in the audio itself. Tested on the Switchboard corpus of phone conversations, the package steadies endpointing against both premature cuts and sluggish replies.
Catalog record
| CREATOR | Anandh C, Karthik Pandia Durai, Jeena Prakash, Manickavela Arumugam, Kadri Hacioglu, S. Pavankumar Dubagunta, Andreas Stolcke, Shankar Venkatesan, Aravind Ganapathiraju |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 20 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-17070
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Improving endpoint detection in end-to-end streaming ASR for conversational speech Anandh C, Karthik Pandia Durai, Jeena Prakash, Manickavela Arumugam, Kadri Hacioglu, S. Pavankumar Dubagunta, Andreas Stolcke, Shankar Venkatesan, Aravind Ganapathiraju Abstract ASR endpointing (EP) plays a major role in delivering a good user experience in products supporting human or artificial agents in human-human/machine conversations. Transducer-based ASR (T-ASR) is an end-to-end (E2E) ASR modelling technique preferred for streaming. A major limitation of T-ASR is delayed emission of ASR outputs, which could lead to errors or delays in EP. Inaccurate EP will cut the user off while speaking, returning incomplete transcript while delays in EP will increase the perceived latency, degrading the user experience. We propose methods to improve EP by addressing delayed emission along with EP mistakes. To address the delayed emission problem, we introduce an end-of-word token at the end of each word, along with a delay penalty. The EP delay is addressed by obtaining a reliable frame-level speech activity detection using an auxiliary network. We apply the proposed methods on Switchboard conversational speech corpus and evaluate it against a delay penalty method. keywords endpointing, latency, E2E ASR, RNN-T, Zipformer, transducer † † address: 1 Uniphore Software Systems, India 2 Indian Institute of Technology Madras, India 3 Uniphore Software Systems, USA † † email: [email protected], [email protected] 1 Introduction Speech endpointing, i.e., the detection of end-of-speech in a turn, is an important aspect of conversational speech recognition. Two-party human-to-human/machine conversations in an industrial setting, such as a customer speaking to a call center agent, or a customer speaking to a voice bot, require precise endpointing such that the live transcription is organized into correct turns during the conversation, which in-turn leads to timely and accurate extraction of further information. In the case of bot-driven calls, effective endpointing improves task completion rate. Streaming ASR systems should not only achieve a low word error rate (WER) but also exhibit low latency for a good user experience. To achieve this, the choice of the model architecture and the choice of endpointing approach plays an important role. End-to-end (E2E) ASR models have become the de facto standard ASR models for industrial use cases. Connectionist temporal classification (CTC) [ 1 ] , recurrent neural network transducer (RNN-T) [ 2 ] , and attention-based encoder-decoder (AED) [ 3 ] are the main E2E ASR techniques used. An encoder is used in all three, while the approaches differ in their decoding methods. CTC assumes conditional independence of the output symbols in the loss function, which is seen as a limitation of the technique. AED uses an encoder in output, and monotonic cross-attention [ 4 ] is used to obtain the alignment between inputs and outputs. Generally, for AED-based ASR, a CTC model is utilised as an auxiliary model to obtain a good alignment between input frames and output symbols [ 5 ] . In RNN-T, the current output is conditioned on the previous outputs. An autoregressive network is used on the output tokens. The representation obtained from this network, together with the encoder representation, is used in a joiner network that predicts the next output token. More recently, RNN encoders have been replaced by transformer-based encoders such as Conformer [ 6 ] and Zipformer [ 7 ] . We refer to such an architecture as transducer-based ASR (T-ASR). In streaming use cases, the technique should not only support streaming capabilities, but also exhibit low latency for a good user experience. Monotonic attention has been used to support streaming in AED. However, T-ASR has been shown to outperform AED in terms of latency and throughput [ 8 ] . T-ASR is a widely used streaming E2E ASR technique. T-ASR still suffers from latency problems due to the delayed emission of output tokens [ 9 , 10 , 11 ] . [ 12 ] constrained the training alignments to be closer to a predefined reference obtained through forced alignment, thereby limiting the token emission delay. Similarly, [ 13 ] used a training loss that encouraged advanced predictions of tokens as compared to a forced-aligned reference. [ 14 ] penalised late prediction of tokens by defining an expected latency for each frame. Work such as [ 10 , 11 ] use losses that encourage non-blank token predictions. User latency is affected by delayed emission and delayed endpointing, though delayed endpointing can also be an effect of delayed emission. In the T-ASR framework, to address latency and endpointing jointly, [ 9 , 15 ] used a special token to detect the end of a sentence (EOS) and by penalising its late prediction in the training loss. In [ 16 ] , a binarised (speech/non-speech) state sequence from the joiner output was used to achieve a forced alignment at the frame level. The corresponding probabilities were obtained using an independent long-short-term memory (LSTM) network, which used encoder outputs as its inputs. Here, the last senone state is considered as an EOS marker. [ 17 ] also used two parallel networks, one for ASR prediction and another for endpointing. The endpoint model was trained to predict endpoints using either acoustic features or latent representations from a shared layer. In addition to using the final silence state to detect end of a query (EOQ), the authors proposed using a special EOQ token in the transcripts, thereby detecting EOQ both using acoustics and text. In this work, we make three contributions: (A) Using encoder embeddings and a light-weight network to reliably estimate the endpoints; (B) using an end-of-word token to avoid word fragments while doing aggressive endpointing; (C) novel endpointing strategies that improve the trade-off between latency and word error rate. Existing endpointing methods for E2E ASR are typically evaluated on voice search tasks or on the Librispeech [ 18 ] read speech dataset. We evaluate our methods on spontaneous speech taken from the Switchboard conversational telephone speech corpus [ 19 ] . A Zipformer transducer T-RNN [ 7 ] is used for all our experiments. The rest of the paper is organized as follows. Section 2 describes the proposed methodology for endpoint detection in detail. The experimental setup, results, and analysis are found in Section 3 . Section 4 offers conclusions. 2 Proposed methods We now describe the problem of endpointing delay and the proposed methods to address it. 2.1 Endpointing delay in transducer-based ASR In ASR systems, the trailing silence (non-speech following a speech region) is generally used to detect endpoint, though contextual information from the preceding speech region and conversation can also help predict ends of speaker turns [ 20 , 21 ] . In T-ASR, non-speech is subsumed by the blank token. The training loss function mandates that a single non-blank token be emitted for every occurrence of a sub-word unit in speech. This means that the majority of the frame-level predictions are blank tokens around these non-blank emissions, in both speech and non-speech regions. Thus, to detect non-speech, the model provides no alternative but to search for the timestamps—or regions—that have no “occasional” non-blank tokens emitted. This is achieved by setting a threshold on the number of consecutive blank tokens emitted, above which an endpoint is declared [ 22 ] . This formulates endpoint prediction as a detection problem, where a