Mind2Matter: Creating 3D Models from EEG Signals

Xia Deng, Shen Chen, Jiale Zhou, Lei Li · 2025

Read Mind2Matter: Creating 3D Models from EEG Signals by Xia Deng, Shen Chen, Jiale Zhou, Lei Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-11936.txt.

Introduction

Xia Deng and co-authors turn brain wave reads of a seen thing into a full three depth model of it, in a paper from 2025. Past work leans on costly brain scans that can not run live. Wave caps are cheap, safe, and quick, but their blur kept three depth builds out of reach till now.

A wave coder first pulls sight marks from the wave stream in space and time. A large tongue model then turns those marks into words and cues for the build. Shape blobs grown from the cues take the last form, steered by a plan of where parts sit. Tests show the builds catch the main shape and sense of the seen thing. The team aims at brain links and fake worlds.

Catalog record

CREATORXia Deng, Shen Chen, Jiale Zhou, Lei Li
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-11936

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Mind2Matter: Creating 3D Models from EEG Signals Xia Deng, Shen Chen, Jiale Zhou, Lei Li Abstract . The reconstruction of 3D objects from brain signals has gained significant attention in brain-computer interface (BCI) research. Current research predominantly utilizes functional magnetic resonance imaging (fMRI) for 3D reconstruction tasks due to its excellent spatial resolution. Nevertheless, the clinical utility of fMRI is limited by its prohibitive costs and inability to support real-time operations. In comparison, electroencephalography (EEG) presents distinct advantages as an affordable, non-invasive, and mobile solution for real-time brain-computer interaction systems. While recent advances in deep learning have enabled remarkable progress in image generation from neural data, decoding EEG signals into structured 3D representations remains largely unexplored. In this paper, we propose a novel framework that translates EEG recordings into 3D object reconstructions by leveraging neural decoding techniques and generative models. Our approach involves training an EEG encoder to extract spatiotemporal visual features, fine-tuning a large language model to interpret these features into descriptive multimodal outputs, and leveraging generative 3D Gaussians with layout-guided control to synthesize the final 3D structures. Experiments demonstrate that our model captures salient geometric and semantic features, paving the way for applications in brain-computer interfaces (BCIs), virtual reality, and neuroprosthetics. Our code is available in https://github.com/sddwwww/Mind2Matter . DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Computing methodologies Computer graphics CCS: Human-centered computing Human computer interaction (HCI) Figure 1. Illustration from visual stimuli to 3D model generation. A subject receives a visual input(left), and EEG signals are recorded. These signals are processed, interpreted as text (middle), and used to generate a 3D model of the scene (right), translating brain activity into a visual representation. 3D object reconstruction using EEG signals is depicted. 1. Introduction Decoding brain signals to reconstruct 3D objects represents a transformative frontier in brain-computer interface (BCI) research ( Brumberg et al., 2018 ; Pandarinath et al., 2017 ) , with profound implications for human-computer interaction, neuroscience, and assistive technologies. By translating neural activity into tangible 3D representations, such advancements could enable intuitive control of virtual environments, facilitate neuroprosthetic design ( Borton et al., 2013 ) , and deepen our understanding of how the brain encodes complex visual perceptions ( Naselaris et al., 2009 ; Horikawa and Kamitani, 2017 ) . Unlike traditional input methods that rely on manual or verbal commands, brain signal decoding offers a direct, non-invasive pathway to bridge the gap between mental imagery and digital output. This capability is particularly valuable for applications requiring real-time interaction, such as immersive virtual reality ( Slater and Sanchez-Vives, 2016 ) and creative 3D modeling ( Nishino et al., 2001 ) , where seamless integration of human intent and computational systems is paramount. Despite the promise of neural decoding, existing research has predominantly focused on reconstructing perceptual experiences using functional magnetic resonance imaging (fMRI). Leveraging fMRI’s high spatial resolution, many studies have successfully mapped brain activity to visual reconstructions ( Takagi and Nishimoto, 2023 ; Chen et al., 2023b ; Xia et al., 2024 ; Scotti et al., 2023 ; Gao et al., 2024a ; Gao et al., 2025 ) . However, the clinical utility of fMRI remains substantially limited by its prohibitive cost, lack of portability, and insufficient temporal resolution to support real-time operation ( Lan et al., 2023 ) . In contrast, electroencephalography (EEG) represents a cost-effective, portable, and non-invasive neuroimaging modality that offers distinct advantages for visual reconstruction and real-time applications ( Heckenlively and Arden, 2006 ) . This study pioneers a novel approach for reconstructing three-dimensional objects from EEG signals evoked by visual stimuli, utilizing 3D Gaussian splatting ( Kerbl et al., 2023 ) to bridge the gap between neural activity and structured volumetric representations. The endeavor confronts substantial methodological challenges stemming from fundamental limitations of scalp-recorded EEG. Unlike fMRI, EEG struggles with low spatial resolution and a low signal-to-noise ratio, which complicates the extraction of robust neural representations. Additionally, EEG signals often fail to effectively capture high-level semantic information, and the complexity of generative models further hinders precise learning of structural details ( Hebb, 2005 ) . While EEG-based decoding has made strides in 2D image generation ( Kavasidis et al., 2017 ; Mishra et al., 2023 ; Bai et al., 2023 ; Singh et al., 2024 ; Pan et al., 2024 ) , the extension to veridical 3D object recovery remains largely unexplored territory. To overcome these challenge, we propose Mind2Matter, a novel two-stage framework that bridges neural decoding and 3D generation through hierarchical semantic understanding. In the first stage, we train a deep multi-channel neural encoder to capture localized semantic features from EEG recordings. This encoder effectively extracts spatiotemporal patterns, which are then projected into a labeled embedding space to fine-tune a language model. By aligning EEG embeddings with semantic representations, the fine-tuned language model generates detailed textual descriptions of the observed visual stimuli, such as object identities and their spatial relationships. This cross-modal translation leverages the language model’s inherent world knowledge ( Cai et al., 2025 ; Li, 2024 ) to compensate for EEG’s semantic sparsity, effectively converting noisy neural patterns into structured object descriptions. While most methods focus on reconstructing single objects, human visual perception is intrinsically holistic, simultaneously encoding multiple objects along with their functional properties and spatial configurations. In the second stage of Mind2Matter, we leverage a scene-level text-to-3D framework based on generative 3D Gaussian techniques ( Chen et al., 2024b ; Liu et al., 2024a ; Yan et al., 2024 ) to synthesize complex 3D scenes. This framework takes the generated textual descriptions as input and produces high-fidelity, coherent 3D reconstructions featuring multiple objects with precise interactions. By incorporating layout-guided control, our approach ensures that the synthesized 3D scenes accurately reflect the spatial and semantic relationships described in the text, overcoming the limitations of EEG’s low spatial resolution. Through this two-stage pipeline, Mind2Matter not only demonstrates the feasibility of EEG-based 3D reconstruction but also establishes a scalable pathway for real-time BCI applications, with potential impacts in virtual reality, neuroprosthetics, and intuitive 3D design. In summary, this paper makes the following contributions: • We presents Mind2Matter, a novel framework for EEG-to-3D scene reconstruction, which enables the extraction of semantic and spatial features from EEG signals to reconstruct complex, multi-object 3D scenes. • A two-stage neural encoding-decoding framework is proposed, facilitating the implicit encoding of EEG signals into textual descriptions and the subsequent decoding into coherent 3D scenes with precise object interactions. • Experimental results demonstrate that the proposed method can effectively reconstruct complex 3D scenes with multiple objects using EEG signals, highlighting its potential for real-time BCI applications. 2. Related Work 2.1. Generating texts from brain activities The generation of textual descriptions from brain activities has emerged as a promising direction in brain-computer interface (BCI) research, aiming to translate neural signals into interpretable semantic outputs. Early research ( Biswal et al., 2019 ; Srivastava et al., 2020 ; Liu et al., 2024b ) predominantly focused on closed-vocabulary approaches, wherein the decoding process was restricted to a predefined set of words or phrases. Notably, Think2type ( Srivastava et al., 2020 ) converted activity intentions of disabled individuals into Morse code

Read the complete text