SYNAPSE: Synergizing an Adapter and Finetuning for High-Fidelity EEG Synthesis from a CLIP-Aligned Encoder

Jeyoung Lee, Hochul Kang · 2025

Read SYNAPSE: Synergizing an Adapter and Finetuning for High-Fidelity EEG Synthesis from a CLIP-Aligned Encoder by Jeyoung Lee, Hochul Kang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-17547.txt.

Introduction

Jeyoung Lee and Hochul Kang generate images from brain activity with a two-stage framework called SYNAPSE. Reading images out of electroencephalography is hard: the signals are noisy, spatially coarse, and vary strongly between people. Earlier systems bolted EEG features onto pretrained image generators through elaborate alignment pipelines, ending up large and hard to interpret. SYNAPSE first trains an EEG autoencoder whose latent space is aligned with the CLIP image-text space, combining reconstruction with cross-modal alignment so the representation carries meaning.

In the second stage, that encoder is frozen and plugged into a lightly adapted Stable Diffusion model, so few parameters need training. On the CVPR40 benchmark, the method reached state-of-the-art fidelity with better reconstruction efficiency than prior EEG-to-image models, and it generalized across subjects. The authors argue that reconstructing what the brain perceives, not what it classifies, is the key.

Catalog record

CREATORJeyoung Lee, Hochul Kang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-17547

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SYNAPSE: Synergizing an Adapter and Finetuning for High-Fidelity EEG Synthesis from a CLIP-Aligned Encoder Jeyoung Lee, Hochul Kang Abstract Recent progress in diffusion-based generative models has enabled high-quality image synthesis conditioned on diverse modalities. Extending such models to brain signals could deepen our understanding of human perception and mental representations. However, electroencephalography (EEG) presents major challenges for image generation due to high noise, low spatial resolution, and strong inter-subject variability. Existing approaches—such as DreamDiffusion, BrainVis, and GWIT—primarily adapt EEG features to pre-trained Stable Diffusion models using complex alignment or classification pipelines, often resulting in large parameter counts and limited interpretability. We introduce SYNAPSE, a two-stage framework that bridges EEG signal representation learning and high-fidelity image synthesis. In Stage1, a CLIP-aligned EEG autoencoder learns a semantically structured latent representation by combining signal reconstruction and cross-modal alignment objectives. In Stage2, the pretrained encoder is frozen and integrated with a lightweight adaptation of Stable Diffusion, enabling efficient conditioning on EEG features with minimal trainable parameters. Our method achieves a semantically coherent latent space and state-of-the-art perceptual fidelity on the CVPR40 dataset, outperforming prior EEG-to-image models in both reconstruction efficiency and image quality. Quantitative and qualitative analyses demonstrate that SYNAPSE generalizes effectively across subjects, preserving visual semantics even when class-level agreement is reduced. These results suggest that reconstructing what the brain perceives, rather than what it classifies, is key to faithful EEG-based image generation. 1 Introduction Recent advances in diffusion-based generative models have enabled high-quality conditional image synthesis across diverse modalities. Stable Diffusion (SD) 26 established a scalable foundation for multimodal generation, motivating research into image synthesis conditioned on novel signals, including brain activity. Translating brain signals into visual representations could provide insight into human perception and cognition, with potential applications in mental health and dream interpretation. Most prior work has focused on reconstructing images from functional magnetic resonance imaging (fMRI) signals 9 ; 24 ; 25 , achieving promising results. However, fMRI acquisition is expensive, slow, and physically demanding, motivating research on Electroencephalography (EEG) as a more accessible alternative. EEG-based generation presents distinct challenges due to its high noise level, low spatial resolution, and strong inter-subject variability. A core difficulty lies in bridging the modality gap—mapping noisy EEG signals to a structured latent space that captures visual semantics. Early attempts using Generative Adversarial Networks (GANs) 5 produced low-resolution outputs with limited fidelity. DreamDiffusion 1 and BrainVis 4 later investigated EEG-conditioned SD to reconstruct visual experiences. Although these studies opened an important research direction, their designs did not explicitly account for the temporal and structural characteristics of EEG. DreamDiffusion directly projected EEG features into the SD latent space, leading to unstable modality alignment and a large parameter overhead due to MAE pretraining. BrainVis adopted a cascaded diffusion framework that first performs EEG-to-label classification, substitutes each predicted label with a corresponding word token, and encodes it through a frozen CLIP 23 text encoder to condition the second-stage diffusion. This indirect conditioning simplifies EEG supervision but restricts the expressiveness of continuous EEG representations, constraining the generation process to discrete label semantics. Moreover, both pipelines employ large parameter counts—MAE pretraining in DreamDiffusion and cascaded diffusion stages in BrainVis—while their reported FID scores remain relatively high. Existing EEG datasets such as EEGCVPR40 31 ; 20 also provide limited data per subject, particularly for Subject 4, which has been used as a standard benchmark in previous studies 1 ; 4 ; 18 . Such scarcity and inter-subject variability make single-subject learning prone to overfitting and hinder generalization, further highlighting the need for a multi-subject framework. Recent work such as GWIT 18 leveraged ControlNet 35 to simplify the training pipeline and achieve high classification accuracy of generated images. However, its comparison on EEGCVPR40 was still limited to a single-subject setting, and the ControlNet-based design reduced interpretability while maintaining relatively high FID scores. These limitations suggest that previous methods primarily adapted EEG into existing SD frameworks rather than tailoring the generation process to EEG as a structured modality. Additional alignment or classification-based conditioning increases model complexity and obscures how EEG features contribute to image synthesis. This motivates a framework that directly embeds EEG representations into a semantically consistent space shared with CLIP 23 , eliminating the need for post-hoc alignment networks. To address these challenges, we propose SYNAPSE, an EEG-conditioned image generation framework designed for multi-subject learning and efficient training. SYNAPSE incorporates a hybrid autoencoder that extracts compact and interpretable condition vectors with reduced parameters. The encoder is integrated into Stable Diffusion through lightweight selective finetuning, enabling end-to-end training on a single RTX 3090 GPU. Furthermore, by combining IP-Adapter 34 and Classifier-Free Guidance (CFG) 7 , SYNAPSE achieves substantially lower FID scores and improved perceptual fidelity across subjects compared to existing EEG-based models. 2 Related Work This paper is related to two main areas: multimodal image generation and neuroscience-based image generation. 2.1 Multimodal Image Generation The emergence of SD 26 brought significant advances in multimodal image generation. Its core design leverages Contrastive Language-Image Pre-Training (CLIP) 23 for cross-modal representation learning, inspiring numerous Vision-Language Models (VLMs). Subsequent versions, including SD v2 v3, improved fidelity through larger OpenCLIP 10 embeddings, diffusion transformers, and flow-matching 16 objectives. SDXL 22 introduced a two-stage architecture that separates the base and refiner models for better adherence to complex prompts, while SD-Turbo 29 applied adversarial diffusion distillation to enable near real-time generation. Beyond architecture, many approaches focus on efficient finetuning for customized generation. Dreambooth 27 enabled subject-specific synthesis, while Low-Rank Adaptation (LoRA) 8 introduced parameter-efficient finetuning (PEFT) strategies, later extended by DoRA 17 and MoRA 11 . However, these methods remain primarily text-conditioned. IP-Adapter 34 addressed image-based conditioning, providing lightweight finetuning that maintains visual consistency and operates efficiently on consumer GPUs (e.g., RTX 3090). Wang et al. 32 proposed selective finetuning, adapting specific diffusion components for efficient task-specific learning. Aligning the generated image with the conditioning signal remains a central challenge. Classifier guidance 19 improves conditional adherence but requires an external classifier. CFG 7 achieves similar effects via sampling without extra supervision. Building on these foundations, our framework integrates SD v2.1, CFG, IP-Adapter, and selective finetuning to effectively incorporate EEG-based conditioning. Figure 1 : Qualitative comparison of SYNAPSE with state-of-the-art EEG-to-image generation models across representative classes. Each row displays generated samples for ’Airliner’, ’Jack O’Lantern’, and ’Panda’ from a different model (Dream Diffusion 1 , BrainVis 4 , GWIT 18 , and Ours). While prior methods show limitations in structural coherence (e.g., ’Airliner’) or semantic fidelity (e.g., ’Panda’), our model (Ours) consistently generates images with higher perceptual quality and closer adherence to the visual features of the original stimuli. This visually supports the state-of-the-art FID score reported in Table 3 . 2.2 Neuroscience-based Image Generation Neuroscience-based image generation can be broadly categorized into fMRI-based and EEG-based approaches. For fMRI, Rakhimberdina et al. 9 reviewed deep learning techniques for natural image reconstruction, and Huang et al. 24 surveyed methods for decoding visual information using machine learning. Other studies focused on specific domains, such as face reconstruction 25 . These works demonstrate the relative success of fMRI-based image generation. In contrast, EEG-based models initially received less attention due to significant challenges, including data scarcity and a low signal-to-noise ratio. A foundational dataset for this task, often referred to as EEGCVPR40

Read the complete text