EmoMind: Decoding Affective Captions from Human Brain fMRI
Bilal A. Mohammed, Lin Gu, Ruogu Fang · 2026
Read EmoMind: Decoding Affective Captions from Human Brain fMRI by Bilal A. Mohammed, Lin Gu, Ruogu Fang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-16739.txt.
Introduction
Mohammed, Gu and Fang present EmoMind, the first end-to-end system that decodes affective captions directly from fMRI brain signals. It first finds a neutral scene description from brain-decoded visual features, then rewrites it with a 34-dimensional emotion vector decoded from the same recording. Guidance during training balances content preservation against feeling. Across two emotion fMRI datasets, EmoMind beats label-prompted GPT-4 on all three validation axes, with the largest gains on person-specific feeling structure.
Catalog record
| CREATOR | Bilal A. Mohammed, Lin Gu, Ruogu Fang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52842 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.16739
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
EmoMind: Decoding Affective Captions from Human Brain fMRI Bilal A. Mohammed, Lin Gu, Ruogu Fang Abstract Decoding visual experience from brain activity has advanced substantially, but current brain-to-text systems largely recover semantic content while discarding affect. Additionally, language models can generate emotional text when prompted with categorical labels, but such labels collapse rich inter-subject variability into coarse discrete bins. We present EmoMind, the first end-to-end pipeline for decoding affective captions directly from fMRI signals. EmoMind first retrieves a semantically grounded neutral scene description from brain-decoded visual features, then rewrites it using a continuous 34-dimensional emotion vector decoded from the same fMRI recording. To control the balance between content preservation and affective expression, we train the rewriter with classifier-free guidance against an identity-preserving null branch, enabling smooth interpolation between semantic fidelity and affective expressivity. We evaluate affective caption generation with a three-axis validation framework spanning subject-specificity, structural geometry, and causal control. We further augment this framework with a synthetic-brain substitution test that probes robustness to the measurement apparatus, and we benchmark each axis against GPT-4 prompted with brain-decoded top-5 emotion labels as a strong discrete baseline. Across two independent emotion fMRI datasets, EmoMind significantly outperforms label-prompted GPT-4 on all three axes, with the largest gains on metrics that require person-specific affective structure rather than population-level emotion aggregation. These results establish continuous brain-decoded affect as a viable control signal for individualized affective caption generation and open new directions for studying individual affective brain organisation. 1 Introduction Advances in non-invasive neuroimaging techniques, like functional magnetic resonance imaging (fMRI), have allowed researchers to measure patterns of brain activity linked to high-level cognitive functions such as language processing and emotion perception. Converting neural signals into emotion expressions that are embedded in natural language provides a window onto how the brain organises affective experience at the individual level. Additionally, it gives a substrate for translational applications in mental health where clinically meaningful signals are precisely the inter-individual variability that population-average readouts miss. Figure 1: (a) Existing approaches decode the scene without affect [ 1 , 2 , 3 ] or condition LLMs on shared categorical labels [ 4 , 5 , 6 ] . (b) Ours. Per-subject fMRI is decoded along two paths – semantic retrieval for the neutral caption x and a per-subject ridge for the continuous 34-D Cowen & Keltner vector e_{s} . The axis-token rewriter (Section 3.2 ) maps (x,e_{s}) to a subject-specific affective caption. In recent years, deep learning has driven substantial progress in generating diverse outputs from brain signals including reconstruction of visual scenes [ 3 , 7 ] , speech [ 2 ] , and linguistic descriptions [ 1 , 8 ] . Recent studies have further demonstrated open-vocabulary fMRI-to-text generation by integrating LLMs directly into the decoding pipeline [ 9 , 10 , 11 , 12 , 13 , 14 , 15 ] . However, as Figure 1 a illustrates, current approaches still face limitations in handling the affective dimension of perception. MindCaptioning trains a decoder to generate fluent descriptions of perceived video content, but the supervision targets are scene captions written without emotional content [ 1 ] . The semantic decoder of Tang et al. 2023 reconstructs continuous perceived or imagined speech from cortical activity, but also without an affect-conditioned output channel. UniCoRN couples a brain encoder to a frozen LLM and produces open-vocabulary text from fMRI, but the generation objective is purely semantic [ 9 ] . BrainX similarly aligns brain features with a multimodal language model and shows cross-subject text generation while remaining affect-blind [ 15 ] . Controlled generation methods such as CTRL and PPLM can steer LLM outputs toward emotion but only via a small set of categorical control codes, which collapses inter-subject variability into a discrete set [ 4 , 5 , 6 ] . Therefore, it is essential to incorporate insights from how the brain actually represents affect into the design of brain-conditioned emotional generation models. Neural representations of affect in the human brain are distributed, high-dimensional, and individually variable. Decades of affective neuroscience have shown that emotional experience is encoded in distributed activation patterns spanning cortical and subcortical networks [ 16 , 17 ] . These patterns vary continuously rather than partitioning into discrete categories [ 18 ] and individual subjects produce distinct neural signatures even for identical naturalistic stimuli [ 19 , 20 ] . A continuous high-dimensional vector decoded from brain activity can in principle preserve these three properties. A top-5 categorical emotion label cannot. We present the first brain-driven emotional captioning pipeline. As shown in Figure 1 b, per-subject fMRI is decoded along two parallel streams. A neutral scene description is first obtained from brain-decoded language features using retrieval-based brain captioning approaches [ 1 , 21 ] . This caption is then paired with a continuous 34-dimensional emotion vector decoded by a subject-specific ridge model. The caption-emotion pair is passed to a trained rewriter based on an encoder-decoder language model [ 22 ] , which uses learned emotion-axis conditioning inspired by style-token and control-code methods [ 23 , 4 ] . The rewriter produces an emotion-styled caption while allowing inference-time control over the balance between semantic content and affect via classifier-free guidance trained with an identity-preserving null branch. Our contributions are: (i) the first brain-driven emotional captioning pipeline conditioned on a continuous decoded emotion vector; (ii) a three-axis validation framework (subject-specificity, structural geometry, causal control) augmented by a TRIBE v2 substitution probe that diagnoses representational gaps in current video-to-fMRI encoders; (iii) head-to-head against strongly prompted GPT-4 across two independent fMRI datasets, with ours dominating on all three axes. 2 Related works Brain decoding from fMRI. Prior work decodes both semantic content from naturalistic fMRI using retrieval-based and generative pipelines [ 2 , 1 , 8 , 3 ] , and emotion as categorical or dimensional variables from distributed neural activity [ 19 , 16 , 24 , 17 , 20 ] . The first line recovers textual or semantic embeddings but focuses on scene content rather than emotional framing; the second treats emotion as a target for analysis, not as a control signal for generation. Our work preserves semantic decoding while using decoded emotion as the conditioning signal that drives caption generation directly. Controllable text generation. Machine learning methods for controllable generation steer sentiment, style, or emotion through prompts, control codes, or classifier-free guidance [ 25 , 26 , 4 , 5 , 27 , 28 , 6 ] . These methods assume the control signal is externally specified by the user or task [ 29 , 30 ] . In contrast, our setting derives the control signal from decoded neural activity, making the problem one of brain-conditioned control rather than user-conditioned generation. Prompting LLMs with emotion. LLMs are highly responsive to emotional content in prompts. EmotionPrompt [ 31 ] shows that injecting emotional stimuli into instructions yields measurable gains across 45 tasks, and prompt-sentiment framing similarly modulates model behaviour [ 32 ] . These results establish text-based emotion conditioning as a credible baseline for affective generation, yet all such methods specify the conditioning signal externally as labels or text descriptions and cannot read out a target subject’s continuous, person-specific affective state. Our strongest baseline prompts GPT-4 with