Neural Signals Generate Clinical Notes in the Wild
Jathurshan Pradeepkumar, Zheng Chen, Jimeng Sun · 2026
Read Neural Signals Generate Clinical Notes in the Wild by Jathurshan Pradeepkumar, Zheng Chen, Jimeng Sun (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-22197.txt.
Introduction
Jathurshan Pradeepkumar, Zheng Chen, and Jimeng Sun turn brain recordings directly into clinical notes. Writing reports that summarize abnormal patterns and diagnostic findings from long-term EEG recordings is slow, expert work. Their 2026 preprint curates a large clinical EEG dataset to attack the problem.
The dataset pairs nearly ten thousand reports with roughly eleven thousand hours of recordings from just over nine thousand patients. With that scale, models can learn to generate clinical notes from raw neural signals in real-world settings, not just in the lab. Automated drafting could cut neurologists' reporting burden, though the generated notes would still need expert review before guiding care.
Catalog record
| CREATOR | Jathurshan Pradeepkumar, Zheng Chen, Jimeng Sun |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 75 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-22197
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Neural Signals Generate Clinical Notes in the Wild Jathurshan Pradeepkumar, Zheng Chen, Jimeng Sun Abstract Generating clinical reports that summarize abnormal patterns, diagnostic findings, and clinical interpretations from long-term EEG recordings remains labor-intensive. We curate a large-scale clinical EEG dataset with 9,922 9{,}922 reports paired with approximately 11,000 11{,}000 hours of EEG recordings from 9,048 9{,}048 patients. We therefore develop CELM, the first clinical EEG-to-Language foundation model capable of summarizing long-duration, variable-length EEG recordings and performing end-to-end clinical report generation at multiple scales, including recording description, background activity, epileptiform abnormalities, events/seizures, and impressions. Experimental results show that, with patient history supervision, our method achieves 70 % 70\% – 95 % 95\% average relative improvements in standard generation metrics (e.g., ROUGE-1 and METEOR) from 0.2 0.2 – 0.3 0.3 to 0.4 0.4 – 0.6 0.6 . In the zero-shot setting without patient history, CELM attains generation scores in the range of 0.43 0.43 – 0.52 0.52 , compared to baselines of 0.17 0.17 – 0.26 0.26 . CELM integrates pretrained EEG foundation models with language models to enable scalable multimodal learning. We release our model and benchmark construction pipeline at 1 Introduction Electroencephalography (EEG) records long-term, real-time neuronal activity with millisecond-level temporal resolution and is widely used in hospitals for neurological diagnosis, such as epilepsy ( Kotoge et al. 2025 ) and sleep health ( Yang et al. 2023 ) . However, writing clinical reports to summarize diagnostic findings over long-term EEG recordings remains a labor-intensive process. Typically, neurologists visually inspect EEG signals to assess brain activity and then compose clinical reports that document abnormal patterns (i.e., EEG phenotypes) and their clinical interpretations ( Tatum IV et al. 2016 ) . Moreover, this task requires substantial domain knowledge and extensive clinical experience, which in turn incur long-term training costs and require continual updates as new EEG phenotypes or waveforms are identified ( Biswal et al. 2019 ) . To support this profound clinical practice, several automated clinical EEG report generation models based on advanced deep learning techniques have been proposed ( Biswal et al. 2019 ; Lee et al. 2023 ; Yin & Shin 2025 ) . These methods aim to encode EEG recordings into latent representations that capture temporal structure and clinically relevant phenotypes, which are then decoded into clinical reports. However, existing methods remain limited in several aspects. ① Conceptually, some works formulate report generation as a phenotype classification or retrieval problem followed by text decoding. They do not model EEG-to-report generation as an end-to-end learning task, and the misalignment between classification and generation objectives does not guarantee optimal report generation. ② Methodologically, these methods operate on short EEG segments and fixed-context templates. They fail to model long-term temporal context and global diagnostic reasoning required for interpreting long-duration EEG data. Moreover, fixed-context templates prevent multi-granularity report generation beyond predefined categories. ③ Practically, these methods are developed as task-specific models, each tailored to a narrow reporting objective (e.g., impression-only reports) ( Lee et al. 2023 ) . In contrast, clinical practice requires multi-level reporting, where neurologists routinely construct overall EEG summaries and section-wise outputs, including impressions, interpretations, and event or seizure annotations. Present work. We introduce CELM , which, to the best of our knowledge, is the first clinical EEG-to-language foundation model (ELM) capable of generating EEG reports at multiple scales. We train and evaluate CELM on 9{,}922 clinical reports paired with approximately 11{,}000 hours of EEG recordings from 9{,}048 patients, spanning both clinical context–conditioned and zero-context settings, and covering key report sections including EEG description, background activity, epileptiform abnormalities, events/seizures, and impressions/interpretations. Our motivation stems from recent advances in multimodal foundation models that enable text generation from non-text modalities (e.g., images and speech) ( Bordes et al. 2024 ; Shen et al. 2025 ) . Recent studies have begun to incorporate EEG signals into language-based modeling frameworks ( Ndir et al. 2025 ; Gijsen & Ritter 2025 ) . However, they primarily leverage text supervision to enhance EEG classification and do not support EEG-to-text report generation. To tackle aforementioned limitations, the contributions of this paper lie in: ❶ Conceptually, we formulate neurological clinical report generation as an EEG-to-Language foundation modeling problem. CELM enables end-to-end training and generation by directly translating raw EEG recordings into clinical text, without relying on intermediate phenotype classification or template-based pipelines. ❷ Methodologically, we introduce the first EEG-to-language multimodal framework, comprising three core components: (i) epoch-aggregated tokenization that produces compact representations from variable-length recordings, (ii) sequence-aware alignment that captures inter-epoch temporal dependencies, and (iii) prompt fusion for conditional report generation. Importantly, our model design harnesses the representation capabilities of pretrained EEG foundation models and LLMs while enabling flexible generation across multiple clinical settings. ❸ Practically, we introduce the first ELM benchmark construction pipeline based on the Harvard Electroencephalography Database ( Sun et al. 2025 ; Zafar et al. 2025 ) , which contains approximately 12\mathrm{k} clinical reports paired with about 100\mathrm{k} hours of EEG recordings from 10k patients. Rather than requiring access to the raw, unstructured 200 TB corpus and associated metadata, our pipeline enables efficient filtering of EEG recordings matched to reports. We release all source code to support further evaluation and development. Experimental results show that CELM consistently outperforms all baselines across all settings. With patient history, it achieves 70\% – 95\% average relative improvements, with standard generation metrics (e.g., METEOR ( Banerjee & Lavie 2005 ) and ROUGE-1 ( Lin 2004 ) ) from 0.2 – 0.3 to 0.4 – 0.6 across diverse cases. In the zero-context setting without patient history, CELM achieves generation scores in the range of 0.43 – 0.52 , compared to baselines of 0.17 – 0.26 . We provide multi-perspective case analyses to shed light on future ELM development. 2 Related Work EEG-to-Language Modeling. Existing work at the intersection of EEG and natural language can be broadly categorized into two paradigms: (1) EEG-to-language decoding and (2) text-enhanced EEG representation learning. The goal of EEG-to-language decoding is to reconstruct textual content from concurrent EEG recordings of subjects reading or imagining speech ( Herff et al. 2015 ; Wang & Ji 2022 ) . This line of work spans both invasive approaches using electrocorticography (ECoG) ( Makin et al. 2020 ; Willett et al. 2023 ) and non-invasive methods based on scalp EEG ( Wang & Ji 2022 ; Duan et al. 2023 ; Zhou et al. 2024 ) . However, these methods assume precise EEG–text alignment, whereas clinical EEG consists of heterogeneous events embedded in long, continuous recordings spanning hours to days ( Sun et al. 2025 ; Zafar et al. 2025 ) . The second paradigm leverages clinical notes as auxiliary supervision to enhance EEG representation learning, rather than decoding language directly from neural activity. Inspired by vision–language pretraining frameworks ( Radford et al. 2021 ) , a recent work ( Ndir et al. 2025 ) aligns EEG data with clinical report text in a shared feature space. However, it remains focused on discriminative objectives and does not support automated clinical report generation. Clinical EEG Report Generation. An early attempt at EEG report generation, EEGtoText ( Biswal et al. 2019 ) , proposed a two-stage pipeline that first classifies EEG phenotypes, and then generates report text via a