Revisiting Your Memory: Reconstruction of Affect-Contextualized Memory via EEG-guided Audiovisual Generation
Joonwoo Kwon, Heehwan Wang, Jinwoo Lee, Sooyoung Kim, Shinjae Yoo, Yuewei Lin, Jiook Cha · 2024
Read Revisiting Your Memory: Reconstruction of Affect-Contextualized Memory via EEG-guided Audiovisual Generation by Joonwoo Kwon, Heehwan Wang, Jinwoo Lee, Sooyoung Kim, Shinjae Yoo, Yuewei Lin, Jiook Cha (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-05296.txt.
Introduction
Joonwoo Kwon and co-authors tackle a new task: rebuilding a person's recalled memory as sound and video, guided by feelings read from brain waves. Volunteers recalled personal memories while their brain signals were recorded, along with text, images, and music tied to each memory. Nine people took part in building the data set.
The team's three-step system first decodes how feelings shift over time during recall, then builds matched audio and video that follow the same path. Their decoder tracked feeling shifts with high accuracy. Viewers said the rebuilt clips matched their recalled feelings, and clips built from true feeling paths beat clips built from shuffled ones. The work links brain decoding with personal media tools.
Catalog record
| CREATOR | Joonwoo Kwon, Heehwan Wang, Jinwoo Lee, Sooyoung Kim, Shinjae Yoo, Yuewei Lin, Jiook Cha |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 24 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-05296
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Revisiting Your Memory: Reconstruction of Affect-Contextualized Memory via EEG-guided Audiovisual Generation Joonwoo Kwon, Heehwan Wang, Jinwoo Lee, Sooyoung Kim, Shinjae Yoo, Yuewei Lin, Jiook Cha Abstract In this paper, we introduce RecallAffectiveMemory , a novel task designed to reconstruct autobiographical memories through audio-visual generation guided by affect extracted from electroencephalogram (EEG) signals. To support this pioneering task, we present the EEG-AffectiveMemory dataset, which encompasses textual descriptions, visuals, music, and EEG recordings collected during memory recall from nine participants. Furthermore, we propose RYM (Recall Your Memory), a three-stage framework for generating synchronized audio-visual contents while maintaining dynamic personal memory affect trajectories. Experimental results indicate that our method can faithfully reconstruct affect-contextualized audio-visual memory across all subjects, both qualitatively and quantitatively, with participants reporting strong affective concordance between their recalled memories and the generated content. Our approaches advance affect decoding research and its practical applications in personalized media creation via neural-based affect comprehension. Introduction Figure 1: Study overview. (a) Conceptual scheme of RecallAffectiveMemory task. (b) The EEG-AffectiveMemory dataset collection process with nine participants recalling their memories. Autobiographical memories, constructed from sensory impressions and affective states, are central to shaping our sense of self. When familiar sights or sounds activate these memories, they bridge past and present experiences, allowing us to relive both sensations and affects. Affective memory recall is a core area of research in cognitive and affective neuroscience, providing insights into how affect dynamics shape the richness and retrieval of personal experiences. Research has established that affect plays a crucial role in forming memories composed of visual and auditory information ( LaBar and Cabeza 2006 ; Jäncke 2008 ; Buchanan 2007 ) . Autobiographical memory recall, in particular, may effectively evoke robust affective states, underscoring the deep interconnection between affect and memory ( Siedlecka and Denson 2019 ) . Despite the co-occurrence of various affects during memory recall, prior studies have mainly focused on static self-reported measures, offering limited insights into the temporal dynamics of affect ( Mills and D’Mello 2014 ; Onton and Makeig 2009 ) . Recent research has shifted toward examining how multiple affective states change and interact over time in naturalistic scenarios ( Kuppens and Verduyn 2017 ) . This approach employs stimuli such as movies or music to investigate both the neural substrates of affective states and their influence on sensory processing and memory formation ( McClay, Sachs, and Clewett 2023 ; Camacho et al. 2022 ) . This evolving focus has advanced affect decoding research, with studies demonstrating the successful application of machine learning for decoding affective states from neural signals, including movie viewing, music listening, and memory recall ( Saarimäki 2021 ; Chanel et al. 2009 ; Iacoviello et al. 2015 ; Placidi et al. 2015 ; Dar et al. 2024 ) . While EEG-based affect decoding shows promise for real-world applications, its untapped potential in decoding the rich and dynamic affective states evoked during autobiographical memory-recall remains a critical and largely unexplored frontier. Understanding these dynamics and developing methods for their neural decoding could lead to breakthroughs in uncovering the fundamental role of recalling unforgettable memories in everyday affective experiences. In parallel, affect-guided content generation has rapidly evolved, opening new possibilities for transforming affect-contextualized memories into multimedia experiences. Early studies demonstrated basic affect-guided neural style transfer in images ( Huang et al. 2018a ) and music ( Sigel, Zhou, and Luo 2021 ) , while recent advances have enabled more nuanced control of affective expression through affect-specific text prompts ( Son et al. 2023 ; Weng et al. 2023 ; Agostinelli et al. 2023 ; Copet et al. 2024 ) . Furthermore, recent generative models ( Rombach et al. 2022 ; Feng et al. 2022 ; Ma et al. 2024 ; Liu et al. 2022 ; Parmar et al. 2024 ) support more sophisticated control over the generation process. Building on these advancements, the integration of generative models with affect decoding offers a novel opportunity to create multimedia content that preserves both the content and affective context of autobiographical memories. Given the critical role of affective states in shaping sensory processing during memory formation and recall, this integration has the potential to transform personalized entertainment and media creation, providing individuals with a deeply resonant experience where their autobiographical memories are enriched by AI-generated content. Here, we aim to decode temporal dynamics of affective states from EEG during autobiographical memory recall and develop a system for generating affectively resonant videos accompanied by music from personal memories using generative artificial intelligence (genAI). To support this task, we present EEG-AffectiveMemory dataset, which captures multi-modal data including text descriptions, sketch paintings, associated musical pieces, and EEG signals during memory recall. Of note, our experimental protocol specifically targets affective state transitions by having participants recall episodes containing mixed affects (e.g., ” sad but cozy ” and ” happy but lonely ”) and report real-time affective states during memory recall and generated content viewing. This design enables simultaneous tracking of dynamic affective changes alongside their neural correlates. Leveraging this comprehensive dataset, we propose RYM , a three-stage multi-modal framework that decodes dynamic affective transitions from neural signals and generates synchronized audio-visual content reflecting the affective trajectories during memory recall. To summarize, our work has three main contributions. • For the first time, we introduce a novel task and comprehensive multi-modal dataset for studying affect-contextualized autobiographical memories. This dataset will be accessible on our project page. • We successfully decode dynamic affective states from EEG signals during memory recall, capturing temporal affective trajectories. • We propose RYM, a simple yet effective framework that faithfully generates synchronized audio-visual content while preserving the affective dynamics of personal memories. Related Works Affect Decoding using Human Brain Signal. Machine learning models have shown increasing success in decoding affect from neural signals (EEG, fMRI) ( Saarimäki 2021 ) . While traditional studies used controlled stimuli, research has shifted toward naturalistic approaches using movies and music ( Koelstra et al. 2011 ; Zheng and Lu 2015 ) . EEG has emerged as a preferred modality due to its high temporal resolution and accessibility ( Al-Nafjan et al. 2017 ; Vempati and Sharma 2023 ) . Several influential open-source datasets (DEAP, SEED, DREAMER) combine EEG with naturalistic stimuli, with DREAMER demonstrating portable EEG feasibility ( Koelstra et al. 2011 ; Zheng and Lu 2015 ; Katsigiannis and Ramzan 2017 ) . Recent datasets explore increasingly natural contexts: social video watching (AMIGOS) ( Miranda-Correa et al. 2018 ) and facial expression recording (PME4) ( Chen, Ro, and Zhu 2022 ) . These resources have advanced both neural pattern analysis and affect dynamics decoding using machine learning and deep learning models ( Saarimäki 2021 ; Vempati and Sharma 2023 ) . Despite these advances, the neural dynamics of autobiographical memory recall, a powerful affect elicitor, remain largely unexplored ( Siedlecka and Denson 2019 ) . While early studies showed potential in decoding memory-induced affect using conventional machine learning ( Chanel et al. 2009 ) , subsequent research demonstrated successful affect classification from EEG signals ( Iacoviello et al. 2015 ; Placidi et al. 2015 ) . Recent work shows deep learning models outperform conventional machine learning approaches in memory-induced affect classification ( Dar et al. 2024 ) , while