CUPE: Contextless Universal Phoneme Encoder for Language-Agnostic Speech Processing 1 1 1 Accepted in: 8th International Conference on Natural Language and Speech Processing (ICNLSP 2025)
Abdul Rehman, Jian-Jun Zhang, Xiaosong Yang · 2025
Read CUPE: Contextless Universal Phoneme Encoder for Language-Agnostic Speech Processing 1 1 1 Accepted in: 8th International Conference on Natural Language and Speech Processing (ICNLSP 2025) by Abdul Rehman, Jian-Jun Zhang, Xiaosong Yang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-15316.txt.
Introduction
Abdul Rehman, Jian-Jun Zhang, and Xiaosong Yang built a speech model in 2025 that reads single speech sounds without using the words around them. Most sound recognizers need long stretches of speech and rules tied to one language. Their model, called CUPE, looks at a window of about one sound in length.
Each short window is handled on its own, and the model has fewer parts than rival tools. Still, tests across many languages showed results close to those rivals. Tests on a phonetic corpus from a California university, with languages the model had not been trained on, also went well. The result suggests basic sound patterns shared across languages can carry much of the work.
Catalog record
| CREATOR | Abdul Rehman, Jian-Jun Zhang, Xiaosong Yang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 31 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-15316
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CUPE: Contextless Universal Phoneme Encoder for Language-Agnostic Speech Processing 1 1 1 Accepted in: 8th International Conference on Natural Language and Speech Processing (ICNLSP 2025) Abdul Rehman, Jian-Jun Zhang, Xiaosong Yang Abstract Universal phoneme recognition typically requires analyzing long speech segments and language-specific patterns. Many speech processing tasks require pure phoneme representations free from contextual influence, which motivated our development of CUPE - a lightweight model that captures key phoneme features in just 120 milliseconds, about one phoneme’s length. CUPE processes short, fixed-width windows independently and, despite fewer parameters than current approaches, achieves competitive cross-lingual performance by learning fundamental acoustic patterns common to all languages. Our extensive evaluation through supervised and self-supervised training on diverse languages, including zero-shot tests on the UCLA Phonetic Corpus, demonstrates strong cross-lingual generalization and reveals that effective universal speech processing is possible through modeling basic acoustic patterns within phoneme-length windows. 1 Introduction Current speech processing systems depend heavily on contextual information, creating a double-edged sword for certain tasks. While extensive context provides crucial bias toward appropriate attention mechanisms, it simultaneously makes it nearly impossible to isolate individual speech units—particularly allophones—from their contextual embeddings. Modern systems such as derivatives of wav2vec 2.0 Baevski et al. 2020 typically analyze 300-2500ms of speech, incorporating extensive language-specific patterns and contextual dependencies. While effective for automatic speech recognition, this approach entangles phonetic content with contextual information, making it extremely difficult to disentangle the acoustic properties that define individual speech sounds. The necessity for contextless processing emerges from two critical considerations: alignment precision and representation purity. Extended temporal windows (e.g., 500ms) reduce inter-frame discriminability as individual frame representations become increasingly influenced by surrounding context. Optimal alignment performance requires maximally discriminative frame-level representations, where each frame maintains distinct characteristics. As context window length increases, the transformer’s attention mechanism progressively attenuates frame-specific features through contextual averaging, resulting in diminished temporal resolution. For paralinguistic tasks, contextless models function as quantization preprocessing stages. When frame-level embeddings encode predominantly contextual rather than local information, this homogenization undermines the model’s capacity to capture subtle local acoustic variations essential for allophone analysis and speaker-specific phonetic characterization. Our empirical results directly challenge the assumption that more context is always better—models using 120ms of speech windows actually perform on-par if not better than those using full word context across multiple evaluation scenarios, while simultaneously providing access to pure phonemic representations less contaminated by contextual dependencies. Our work makes three key contributions. First, we demonstrate that universal phoneme recognition can be achieved effectively with just 120ms of context, a fraction of the 300-2500ms typically used in current approaches. Second, we introduce CUPE, a lightweight architecture (30M parameters) that achieves competitive performance through focused local feature extraction. Third, we provide a feature extraction method that captures pure phonemic representations by eliminating contextual dependencies, leading to cleaner and more interpretable phoneme embeddings across languages. By operating on brief windows—approximately the duration of a typical phoneme Crystal and House 1988 , CUPE learns language-agnostic acoustic features that characterize phonemes universally. This focus on fundamental acoustic patterns, independent of language-specific context, enables robust cross-lingual generalization and, crucially, provides access to clean allophonic representations that are essential for understanding speaker-specific phonetic variations. The contextless nature of our approach enables several practical applications: • Timestamps Alignment : Generating time-aligned transcripts from raw text and audio. This task is critical for training downstream text-to-speech models. Since this is the main application for phoneme recognition, it helps to have as little context information in each frame so that there is a sharper contrast between frames for precise boundary detection. • Speech style learning : It serves as a foundational allophone encoder. Embeddings of each frame can be used to generate acoustically pure allophone variants of base phonemes. This is also useful for training downstream text-to-speech tasks which currently rely on IPA dictionaries or sub-word tokens. • Robust phoneme verification : Complementing traditional ASR systems by detecting and correcting errors that arise from over-reliance on language context. • Cross-linguistic research : Generating language-agnostic phoneme representations that facilitate multilingual studies and enable more accurate speech disorder diagnostics. Through extensive evaluation, we validate CUPE (Contextless Universal Phoneme Encoder), an architecture that deliberately restricts analysis to short windows. Our results demonstrate that this constrained approach matches or exceeds the performance of context-heavy models (XLS-R Babu et al. 2022 ) across diverse languages while using an order of magnitude fewer parameters and providing clean, context-independent phonemic representations suitable for allophone analysis. Figure 1: The windowing approach restricts the model’s context for better localized learning, therefore, generalizing better across languages without learning longer patterns. 2 Contextless Universal Phoneme Model Analysis of our evaluation datasets (Table 2 ) shows phoneme durations averaging 80ms (range: 62-107ms), consistent with Crystal and House’s findings of 70-120ms for English phonemes Crystal and House 1988 . Our architecture processes acoustic features through Conv1D layers at 13.1ms per frame, with a 120ms window and 80ms stride to capture 1-2 phonemes per window. This approach provides precise frame-level analysis while maintaining phoneme-level context, departing from traditional methods that rely on broader windows. To preserve acoustic continuity across overlapping windows, we implement a cosine-based weighting mechanism for feature fusion. The complete model architecture is illustrated in Figure 1 , with detailed specifications provided in Table 1 . 2.1 Window Slicer The Window Slicer module addresses the fundamental challenge of processing continuous speech signals by segmenting raw waveforms (16 kHz) into overlapping windows. This design enables localized feature extraction while preserving temporal continuity at boundaries. Using a 120 ms window size with an 80 ms stride provides sufficient context for phonetic events while reducing computational complexity from \mathcal{O}(T^{2}) to \mathcal{O}(W^{2}) , where T is the total sequence length and W is the window size. Given an input audio signal \mathbf{x}\in\mathbb{R}^{B\times T} , where B is the batch size and T is the total number of samples in the input sequence ( T=\text{sample\_rate}\times\text{duration} ): w_{b,i}(t)=x_{b}(t+is),\quad t\in[0,W-1] (1) where b\in[0,B-1] is the batch index, i\in[0,N-1] is the window index, t is the time index within each window, W=1920 is the window size (120 ms \times 16 kHz), s=1280 is the stride length (80 ms \times 16 kHz), and N=\left\lfloor\frac{T-W}{s}\right\rfloor+1 is the number of windows. 2.2 Feature Extractor Drawing from raw waveform processing techniques Dai et al. 2017 ; Schneider et al. 2019 , our feature extraction stage implements a hierarchical CNN architecture that processes raw waveforms directly. This design, detailed in Table 1 , captures increasingly abstract representations while maintaining computational efficiency. Following the success of Squeeze-and-Excitation Networks Hu et al. 2018 in speech recognition Han et al. 2020 , we incorporate adaptive channel-wise recalibration through frequency attention. The architecture separates temporal and spectral processing streams, inspired by multi-stream approaches Han et al. 2021 , to capture both evolving acoustic patterns and frequency relationships. 2.3 Windowwise Transformer Our transformer encoder layers process independent fixed windows instead of the whole clip, modifying the contextual processing of standard transformers Vaswani et al. 2017 . This approach represents a departure from traditional speech transformers by restricting context to local windows, ensuring that phoneme recognition decisions rely on relevant local context. Our preliminary experiments showed a tendency to overfit with larger transformer layers, leading us to maintain a light architecture (13M