Effective Context in Neural Speech Models

Yen, Sharon, Hao · 2025

Read Effective Context in Neural Speech Models by Yen, Sharon, Hao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-22487.txt.

Introduction

Yen, Goldwater, and Tang gauge how much past speech their net speech tools use. Long reach in such tools is prized, and ways to stretch it abound. Yet how much reach a tool in fact uses had gone un-gauged. Their term for used reach is the live reach, in the text. Two new gauges size that live reach for a tool. Speech chains of the Transformer type take the test in the work.

For taught tools, live reach tracks the task in fine terms. Tone track needs least, then phone sorts, then word sorts most. For self-taught tools, live reach grows in the first coats most. It stays short past those coats, near the phone tool mark. Since reach stays short, HuBERT can stream with no build change. No fresh tune is need for that stream mode, they show.

Catalog record

CREATORYen, Sharon, Hao
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE23 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-22487

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Effective Context in Neural Speech Models Yen, Sharon, Hao Abstract Modern neural speech models benefit from having longer context, and many approaches have been proposed to increase the maximum context a model can use. However, few have attempted to measure how much context these models actually use, i.e., the effective context. Here, we propose two approaches to measuring the effective context, and use them to analyze different speech Transformers. For supervised models, we find that the effective context correlates well with the nature of the task, with fundamental frequency tracking, phone classification, and word classification requiring increasing amounts of effective context. For self-supervised models, we find that effective context increases mainly in the early layers, and remains relatively short—similar to the supervised phone model. Given that these models do not use a long context during prediction, we show that HuBERT can be run in streaming mode without modification to the architecture and without further fine-tuning. Meng Goldwater Tang nocounter]The Centre for Speech Technology ResearchUniversity of EdinburghUnited Kingdom keywords interpretability, self-supervised learning, probing † † email: [email protected], [email protected], [email protected] 1 Introduction The recent success of speech and language processing systems can be largely attributed to better modeling of context. Various speech tasks benefit from contextualized speech embeddings learned with self-supervision [ 1 , 2 , 3 , 4 , 5 ] . Automatic speech recognition, summarization, question answering, and language modeling have all been shown to improve when models are engineered to have a wider context [ 6 , 7 , 8 , 9 , 10 , 11 ] . However, observing an improved performance does not necessarily imply that a wider context is actually used. Failing to properly attribute where the improvement is from gives the false impression that longer context is always better. It is possible that a wider context is not even necessary to achieve competitive performance. In this work, we distinguish between the context a model has access to (i.e., the context by design) and the context that a model actually uses (the effective context). The amount of context available to the model can be increased by changing the model architecture [ 9 , 10 , 11 ] . However, the context that the model actually uses will be affected by various other design choices, such as the data and the training algorithm. LSTMs, for example, have access to the entire history but also have difficulty making full use of it [ 12 ] . Several studies have presented indirect evidence that Transformers, despite having access to the entire input sequence, do not make use of the full context. They approach this problem by modifying the model architecture [ 13 , 14 , 15 ] or by analyzing attention maps [ 13 , 16 ] . A few other studies [ 17 , 18 , 19 , 20 , 21 ] , though not directly studying the amount of context, have probing results that suggest the limited use of context in Transformers. Overall, these approaches are ad-hoc and limited either to a particular model architecture or to a particular task. In this work, we approach the problem of measuring effective context from first principles. If a change to a part of the input does not affect the output much, then that part of the input is effectively not used by the model and hence not in the effective context. We propose two approaches, one based on truncation of the input and the other based on the Jacobian with respect to the input. The truncation approach is more intuitive—truncating input frames not in the effective context should hardly affect the output. The Jacobian approach (similar to gradients) measures the effect of infinitesimal changes on the input. Both approaches are model-agnostic, not tied to specific loss functions, do not require labels, and can be applied to any layer of a network. We apply the proposed approaches to measure effective context of speech Transformers, as they are the dominant architecture in both supervised and self-supervised settings. In the supervised setting, we find that the amount of effect context used in Transformers for predicting f0, phones, and words increases in that order. In the self-supervised setting, we find that the effective context of self-supervised Transformers, specifically wav2vec 2.0 [ 3 ] , HuBERT [ 4 ] , and WavLM [ 5 ] , is shorter than that for predicting phones in the supervised setting. We also observe a clear correlation between the amount of effective context and probing performance on phones and words. Given the relatively short effective context of self-supervised Transformers, limiting the history and lookahead of these models should not hurt their performance much. This implies that we can readily run pretrained Transformers in low-latency (400ms) streaming mode without any modification to the architecture or further training. Using a simple probing classifier, our experiments show that, compared to a phone error rate of 11.9% when using full utterances, we only observe a 0.6% degradation with the full history, and a 1.5% degradation with a 2s history. 2 Measuring Effective Context We define effective context based on the following principle. For an input utterance of T frames x_{1},\dots,x_{T} , we are interested in how a function f changes as we change the frames. At a high level, if changes made to frame x_{t} do not change the output of f by much, then x_{t} is not part of the effective context of f . The function f can be the computation from the input to any of the intermediate layers, or all the way to the loss. The output of f typically has multiple frames, i.e., h_{1},\dots,h_{T}=f(x_{1},\dots,x_{T}) , and we will use [f(x_{1},\dots,x_{T})]_{t} to denote h_{t} . Below we detail two changes that can made to the frames, leading to the truncation approach and the Jacobian approach. 2.1 The Truncation Approach If we are interested in whether a frame x_{t} is in the effective context or not, the most intuitive approach is to remove x_{t} from the input to f . Removing single frames is not very meaningful as, in speech, a lot can be inferred from neighboring frames. Instead, for models that can make use of the full context, we make the assumption (which we will later verify) that f has a symmetric effective context. We choose a window of size 2W+1 frames centered at t and truncate the frames outside of the window. We then compute \displaystyle d\Bigg(\Big[f(x_{1},\dots,x_{T})\Big]_{t},\Big[f(x_{t-W},\dots,x_{t+W})\Big]_{t}\Bigg) (1) for t=1,\dots,T of an utterance and average over a data set, where d is a distance function that can be the \ell_{2} distance or a performance metric, such as phone probing error rate. The truncation approach has been inspired by similar ideas used to study the context of text-based language models [ 22 , 23 , 24 ] . Figure 1: Examples illustrating influence. Left: The x-axis is the time point \tau of the input utterance x , and the y-axis is the calculated influence value s(t,\tau) for the timepoint t=300 (red dot). Right: The relative influence S(\sigma) (normalized), where the x-axis is the time shift to the center frame. . 2.2 The Jacobian Approach Instead of explicitly truncating, another approach is to make changes to the input. Formally,

Read the complete text