LightBeam: An Accurate and Memory-Efficient CTC Decoder for Speech Neuroprostheses

Ebrahim, Junlin, Nima, Jonathan C · 2026

Read LightBeam: An Accurate and Memory-Efficient CTC Decoder for Speech Neuroprostheses by Ebrahim, Junlin, Nima, Jonathan C (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-14002.txt.

Introduction

Feghhi and co-authors present LightBeam, a decoder that turns brain signals into text for speech neuroprostheses. The standard decoder for this task needs about 320 GB of memory, which shuts out many users and labs. LightBeam needs only about 10 GB and drops the old WFST design. It folds a language model into beam search with delayed fusion, so no large n-gram model is needed. On the Brain-to-Text '24 and '25 benchmarks, it matches the best reported results.

Catalog record

CREATOREbrahim, Junlin, Nima, Jonathan C
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE23805
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.14002

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LightBeam: An Accurate and Memory-Efficient CTC Decoder for Speech Neuroprostheses Ebrahim, Junlin, Nima, Jonathan C Abstract A promising pathway for restoring communication in patients with dysarthria and anarthria is speech neuroprostheses, which directly decode speech from cortical neural activity. Two benchmarks, Brain-to-Text ’24 and ’25, released intracranial recordings from patients with dysarthria along with a baseline algorithm trained with Connectionist Temporal Classification (CTC). Despite significant innovation on these benchmarks, all leading published prior work relies on a WFST-based CTC decoder that requires ∼ {\sim} 320 GB of RAM. These memory requirements limit accessibility for both patients and researchers. Here, we propose LightBeam , a non-WFST based CTC decoder that requires only ∼ {\sim} 10 GB of RAM and achieves state-of-the-art performance on both benchmarks. LightBeam achieves this by integrating an LLM into the beam-search process via delayed fusion, obviating the prior need for using a large N-gram LM. LightBeam is implemented in Python and is open-source. Feghhi Hu Hadidi Kao keywords Brain-Computer Interface, Speech Neuroprostheses, CTC, Delayed Fusion † † address: 1 Neuroscience Interdepartmental Program 2 Department of Electrical & Computer Engineering 3 Department of Computer Science University of California Los Angeles † † email: [email protected] 1 Introduction Conditions such as amyotrophic lateral sclerosis (ALS), cerebral palsy, or brainstem stroke can lead to dysarthria and anarthria. Dysarthria and anarthria are speech production disorders in which the neural activity generating speech plans in the motor cortex remains intact, yet the transmission of these motor plans to the vocal tract or the vocal tract musculature itself is impaired [ 17 ] . These impairments manifest as slow or slurred speech in dysarthria, and a complete loss of the ability to speak in anarthria. Speech neuroprostheses offer a promising path toward restoring communication for patients with these disorders, bypassing the vocal tract by decoding speech directly from neural activity [ 19 , 4 , 2 , 15 ] . Two benchmarks, Brain-to-Text ’24 (B2T ’24; [ 20 ] ) and ’25 (B2T ’25; [ 3 ] ), have accelerated the development of speech neuroprostheses. Each competition provided intracranial electrode recordings from the speech motor cortex of a distinct participant with ALS while they attempted to repeat text prompts. Along with the neural dataset, the organizers released a baseline algorithm consisting of two components. The first component, the CTC encoder, converts neural activity into phoneme-level logits using a Gated Recurrent Unit (GRU) architecture trained with CTC loss [ 9 ] . The second component, the CTC decoder, converts logits into words by running beam search on a weighted finite-state transducer (WFST) graph integrated with a large 5-gram LM. Since LLMs cannot be represented as WFST graphs, an LLM (OPT 6.7B; [ 21 ] ) rescores beams at the end of the trial. This CTC decoder requires {\sim} 320 GB of RAM. While significant progress has been made on these benchmarks, existing top-performing published algorithms rely on the memory-intensive baseline WFST-based CTC decoder [ 12 , 5 , 22 ] . These memory requirements present a bottleneck for deployment on local, resource-constrained devices. Local deployment is ideal as it preserves user privacy, minimizes transmission latency, and maximizes accessibility for patients and the broader research community. In this study, we present LightBeam , an accurate and memory-efficient CTC decoder for speech neuroprostheses. The core insight behind LightBeam is that rather than only using an LLM for second-pass rescoring, we integrate it within the beam search process (i.e. first-pass decoding) at fixed time intervals via delayed fusion [ 11 ] . By doing so, we reduce the computational burden of preserving likely beam hypotheses on the N-gram LM, and replace the large 5-gram LM with a relatively small 4-gram LM. LightBeam reduces memory requirements from {\sim} 320 GB to {\sim} 10 GB of RAM. Furthermore, it performs significantly better than the baseline WFST-based decoder on both benchmarks when paired with either the baseline GRU or the time-masked Transformer encoder introduced by Feghhi et al. [ 5 ] . When applying generative error correction, LightBeam achieves state-of-the-art (SOTA) performance compared to published results by Zhang et al. [ 22 ] (that also use generative error correction) on both benchmarks. While LightBeam requires making more frequent calls to an LLM, real-time factor (RTF) values are well below 1, so it remains viable for clinical use. Furthermore, LightBeam is open-source and implemented in Python, making it easily modifiable and deployable in modern research environments, with code provided here . 2 Related Works Prior work attempted to entirely bypass the WFST-based decoder by projecting neural activity into the embedding space of an LLM, and fine-tuning the LLM to directly generate text [ 6 , 22 ] . While this multi-modal LLM approach has been successful in automatic speech recognition (ASR) [ 12 ] , it performs substantially worse than the WFST decoder in this context, likely due to small neural dataset sizes [ 22 ] . One reason researchers prioritized building multi-modal LLMs over integrating them into the beam-search process may be that the majority of open-source CTC toolkits only support N-gram LM integration [ 14 , 7 , 10 ] . While the Flashlight decoder on PyTorch does support neural network integration, it is CPU-based and thus cannot support LLM inference [ 13 ] . Hori et al. [ 11 ] were the first to demonstrate that LLMs could be effectively used in first-pass decoding without additional projection layers; however, they released no open-source implementation. Furthermore, their results are shown in the context of ASR with encoders that output sub-word tokens and are already accurate without external LM integration. By contrast, CTC encoders for neural speech decoding typically output phonemes and heavily rely on external LM guidance due to small dataset sizes. 3 Methods 3.1 Dataset Our analyses utilize the neural datasets from the Brain-to-Text ’24 (B2T ’24) and ’25 (B2T ’25) benchmarks [ 19 , 2 ] . The B2T ’24 dataset consists of 12,100 sentences recorded from participant T12 over 25 sessions spanning 4 months, using 128-electrode Utah arrays implanted in the ventral motor cortex. The B2T ’25 dataset includes 10,948 sentences from participant T15 across 45 sessions over 20 months, recorded with 256 electrodes in the ventral motor cortex. Across both datasets, neural features consist of thresholded spike counts and spiking-band power [ 16 ] . These signals were binned in 20 ms windows and z-scored within blocks. Additional details regarding these datasets, including train, val, and test split construction are included in Supplementary Section A as well as the original studies. 3.2 Encoder models We trained the baseline GRU encoders using the hyperparameters and codebases provided by the B2T ’24 and ’25 benchmark organizers. The B2T ’24 GRU was bidirectional (non-streaming), while the B2T ’25 GRU was unidirectional. Additionally, we evaluated LightBeam using the time-masked Transformer, which outperforms the baseline GRU on B2T ’24 [ 5 ] and achieves state-of-the-art published results when combined with self-supervised pretraining [ 22 ] . For B2T ’24, we log-transformed the neural data following Feghhi et al. [ 5 ] ; for B2T ’25, we utilized the z-scored data provided by the organizers without further transformation. To address the challenge of the unbounded KV-cache in self-attention—which complicates clinical deployment—we trained

Read the complete text