Moving Beyond Diffusion: Hierarchy-to-Hierarchy Autoregression for fMRI-to-Image Reconstruction

Xu Zhang, Ruijie Quan, Wenguan Wang, Yi Yang · 2025

Read Moving Beyond Diffusion: Hierarchy-to-Hierarchy Autoregression for fMRI-to-Image Reconstruction by Xu Zhang, Ruijie Quan, Wenguan Wang, Yi Yang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-22335.txt.

Introduction

Xu Zhang, Ruijie Quan, Wenguan Wang, and Yi Yang rebuild viewed images from brain activity without leaning on diffusion models. Reading a picture out of fMRI signals usually routes the brain data through one fixed neural embedding that steers a diffusion generator from start to finish. A single static hint throws away the layered structure of vision and mismatches what each generation stage actually needs.

Their framework, MindHier, goes coarse to fine in the autoregressive style instead. A hierarchical encoder pulls multi-level embeddings out of the fMRI signal, an alignment scheme ties each level to the matching level of a vision model's features. A scale-aware guidance strategy feeds each embedding in at the matching scale of reconstruction. The authors present the design as both more efficient and closer to how the brain itself builds a scene, layer upon layer.

Catalog record

CREATORXu Zhang, Ruijie Quan, Wenguan Wang, Yi Yang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-22335

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Moving Beyond Diffusion: Hierarchy-to-Hierarchy Autoregression for fMRI-to-Image Reconstruction Xu Zhang, Ruijie Quan, Wenguan Wang, Yi Yang Abstract Reconstructing visual stimuli from fMRI signals is a central challenge bridging machine learning and neuroscience. Recent diffusion-based methods typically map fMRI activity to a single high-level embedding, using it as fixed guidance throughout the entire generation process. However, this fixed guidance collapses hierarchical neural information and is misaligned with the stage-dependent demands of image reconstruction. In response, we propose MindHier, a coarse-to-fine fMRI-to-image reconstruction framework built on scale-wise autoregressive modeling. MindHier introduces three components: a Hierarchical fMRI Encoder to extract multi-level neural embeddings, a Hierarchy-to-Hierarchy Alignment scheme to enforce layer-wise correspondence with CLIP features, and a Scale-Aware Coarse-to-Fine Neural Guidance strategy to inject these embeddings into autoregression at matching scales. These designs make MindHier an efficient and cognitively-aligned alternative to diffusion-based methods by enabling a hierarchical reconstruction process that synthesizes global semantics before refining local details, akin to human visual perception. Extensive experiments on the NSD dataset show that MindHier achieves superior semantic fidelity, 4.67 × \times faster inference, and more deterministic results than the diffusion-based baselines. 1 Introduction Reconstructing visual stimuli from fMRI signals stands as a fundamental challenge at the intersection of computer vision and cognitive neuroscience. Recent breakthroughs in this field have been largely driven by diffusion-based methods {}_{\!} ( Xia et al. 2024a ; Ozcelik & VanRullen 2023 ; Gong et al. 2025 ; Scotti et al. 2023 ; Tian et al. 2025 ; Scotti et al. 2024 ; Fang et al. 2023 ; Quan et al. 2024 ; Takagi & Nishimoto 2023 ; Mai & Zhang 2023 ; Li et al. 2025 ; Wang et al. 2024 ; Sun et al. 2023 ; Huo et al. 2024 ; Chen et al. 2023b ) , which typically align the encoded fMRI embedding with a high-level representation from multimodal models such as CLIP {}_{\!} ( Radford et al. 2021 ) . Once aligned, this single high-level feature serves as a fixed guidance signal throughout the entire diffusion process, progressively transforming random Gaussian noise into a reconstructed image (Fig.​ 1 (a)). Despite achieving impressive results, relying solely on a static high-level feature to guide the entire generation pipeline introduces two fundamental limitations. First , fMRI signals are inherently hierarchical, where different brain regions capture coarse semantic content as well as fine-grained perceptual details {}_{\!} ( Naselaris et al. 2009 ) . However, current methods collapse this rich, multi-level information into a single vector, leading to its underutilization. Second , the guidance is temporally invariant, yet generative models operate in a dynamic multi-stage manner: early stages require global semantic constraints, while later stages demand precise structural cues {}_{\!} ( Rissanen et al. 2023 ; Dieleman 2024 ) . A fixed neural guidance is often redundant in the early phase and insufficient in the later phase, creating a mismatch between representation and generation. Beyond these limitations , diffusion models themselves provide limited control points for injecting stage-aware guidance {}_{\!} ( Zhou et al. 2024 ) , which hinders the effective exploitation of hierarchical brain features. Figure 1: Comparison of fMRI-to-image reconstruction pipelines. (a) Prior diffusion-based methods utilize a fixed neural feature to guide the reconstruction. (b) In contrast, MindHier employs scale-aware guidance, leveraging hierarchical neural features to first establish a low-resolution overview (“Forest”) before progressively refining local details (“Trees”) at higher resolutions. To address these challenges, we propose MindHier, a new coarse-to-fine fMRI-to-image reconstruction framework built upon scale-wise autoregressive modeling {}_{\!} ( Tian et al. 2024 ) , moving beyond diffusion-based pipelines (Fig.​ 1 (b)). MindHier integrates three key designs. (i) Hierarchical fMRI Encoder : a unified encoder that transforms fMRI signals into a hierarchy of embeddings, capturing the full spectrum of neural information from global semantics to local details. (ii) Hierarchy-to-Hierarchy Alignment : a dual-objective training scheme that enforces hierarchy-to-hierarchy alignment between fMRI embeddings and CLIP features, pairing shallow layers with low-level visual features for structural fidelity and deeper layers with high-level features for semantic coherence. (iii) Scale-Aware Coarse-to-Fine Neural Guidance : a principled strategy that injects the hierarchical embeddings into autoregression at matching scales. Specifically, the high-level semantic embeddings from the fMRI encoder supervise the small-scale ( low-resolution ) stage to establish a semantically coherent global layout, while lower-level embeddings are progressively injected at larger scales ( higher resolution ) to refine structures and enrich textures. Together, these components enable MindHier to leverage the coarse-to-fine nature of visual autoregression, yielding reconstructions that achieve strong semantic fidelity and competitive structural faithfulness compared to diffusion-based baselines. MindHier achieves state-of-the-art high-level metrics on the NSD dataset {}_{\!} ( Allen et al. 2022 ) , e . g ., the highest CLIP score of 96.4% and the lowest SwAV distance of 0.329. Beyond empirical improvements, MindHier offers several deeper advantages. First , its coarse-to-fine generation naturally echoes the “ Forest before Trees ” principle from cognitive neuroscience ( Navon 1977 ) , whereby human perception prioritizes global structure before resolving local details. By seeding reconstructions with high-level semantic embeddings and progressively refining fine-grained cues, MindHier translates this perceptual hierarchy into a computational framework. Second , the coarse-to-fine autoregressive modeling substantially enhances efficiency: MindHier achieves significant speedups by allocating most computation to low-resolution scales. For example, reconstructing a high-fidelity visual stimulus takes just 2.64 seconds per image, which is 4.67× faster than MindEye2 {}_{\!} ( Scotti et al. 2024 ) ). Third , MindHier produces more stable and consistent reconstructions. Unlike diffusion-based pipelines that are inherently stochastic due to random noise initialization, MindHier anchors its generation process directly to fMRI-derived embeddings, initializing with high-level embeddings and conditioning subsequent scales on lower-level ones. In a nutshell, our contributions are three-fold: • We identify a fundamental mismatch between the fixed neural guidance and the dynamic nature of image reconstruction. To address this, we introduce MindHier, a coarse-to-fine autoregressive framework that dynamically tailors neural guidance to the specific demands of each generation stage, from establishing a global semantic layout to rendering fine-grained details. • We propose Hierarchy-to-Hierarchy Alignment to disentangle the fMRI signal into hierarchical neural features, and Scale-Aware Guidance to strategically inject these features into the corresponding scales of the autoregressive generator for high-fidelity image reconstruction. • The empirical results demonstrate that our cognitively-aligned framework sets a new state of the art in semantic fidelity and structural faithfulness. Furthermore, it achieves a significant inference speedup and consistent reconstruction of visual stimuli compared to diffusion-based methods. 2 Related Work fMRI-to-Image Reconstruction. The reconstruction of visual stimuli from brain activity is a pivotal challenge at the intersection of cognitive neuroscience and computer vision. Early works {}_{\!} ( Kay et al. 2008 ; Miyawaki et al. 2008 ; Nishimoto et al. 2011 ) correlate fMRI signals with handcrafted image features, while subsequent advances {}_{\!} ( Horikawa & Kamitani 2017 ) improve this brain-visual association by using sparse linear regression to predict features from early CNN layers. The advent of Variational Autoencoders {}_{\!} ( Kingma & Welling 2013 ) and Generative Adversarial Networks {}_{\!} ( Goodfellow et al. 2014 ) enables more direct pixel-level reconstruction {}_{\!} ( Seeliger et al. 2018 ; Ren et al. 2021 ; Ozcelik et al. 2022 ) . However, these methods often yield ambiguous images that lack

Read the complete text