Optimal Transport for Brain-Image Alignment: Unveiling Redundancy and Synergy in Neural Information Processing

Yang Xiao, Wang Lu, Ruimeng Ye, Xiaolong Ma · 2025

Read Optimal Transport for Brain-Image Alignment: Unveiling Redundancy and Synergy in Neural Information Processing by Yang Xiao, Wang Lu, Ruimeng Ye, Xiaolong Ma (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-10663.txt.

Introduction

Xiao and a team including Ma use optimal transport to align brain and image signals in twenty twenty-five. Artificial networks are inspired by the brain and may help decode brain activity. Older alignment methods use mean squared error, which matches points one by one. That ignores global structure and yields coarse readings.

Their method builds a transport plan between brain voxel and image embeddings. Controlling how much mass moves reduces the effect of redundant information. The aligned brain signals feed a large language model for brain captioning. Results beat the prior best method across ten metrics in single and cross subject tests. Masking and dimension reduction experiments reveal redundancy and synergy in brain processing. The authors argue transport gives a more precise alignment.

Catalog record

CREATORYang Xiao, Wang Lu, Ruimeng Ye, Xiaolong Ma
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-10663

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Optimal Transport for Brain-Image Alignment: Unveiling Redundancy and Synergy in Neural Information Processing Thanks: Corresponding Author Yang Xiao, Wang Lu, Ruimeng Ye, Xiaolong Ma Abstract The design of artificial neural networks (ANNs) is inspired by the structure of the human brain, and in turn, ANNs offer a potential means to interpret and understand brain signals. Existing methods primarily align brain signals with stimulus signals using Mean Squared Error (MSE), which focuses only on local point-wise alignment and ignores global matching, leading to coarse interpretations and inaccuracies in brain signal decoding. In this paper, we address these issues through optimal transport (OT) and theoretically demonstrate why OT provides a more effective alignment strategy than MSE. Specifically, we construct a transport plan between brain voxel embeddings and image embeddings, enabling more precise matching. By controlling the amount of transport, we mitigate the influence of redundant information. We apply our alignment model directly to the Brain Captioning task by feeding brain signals into a large language model (LLM) instead of images. Our approach achieves state-of-the-art performance across ten evaluation metrics, surpassing the previous best method by an average of 6.11% in single-subject training and 3.81% in cross-subject training. Additionally, we have uncovered several insightful conclusions that align with existing brain research. We unveil the redundancy and synergy of brain information processing through region masking and data dimensionality reduction visualization experiments. We believe our approach paves the way for a more precise understanding of brain signals in the future. The code is available soon. 1 Introduction The structure of the human brain has inspired the development of neural networks [ 51 ] , and in recent years, research on brain-computer interfaces has gained increasing attention. Researchers have been exploring ways to leverage neural networks to decode brain signals and better understand brain function. Among various neural recording techniques, functional Magnetic Resonance Imaging (fMRI) provides an indirect measure of neural activity by capturing fluctuations in blood oxygen levels. With advances in deep learning, researchers have attempted to align fMRI signals with neural representations in pre-trained models to enhance brain decoding capabilities [ 41 , 57 ] . As the main approach to interpreting brain region signals, brain-image alignment combines artificial intelligence and neuroscience, allowing us to decipher semantic information from neural activities in response to different types of stimuli [ 71 , 70 , 75 , 2 , 69 , 20 , 77 , 52 , 35 , 57 ] . By projecting data in a shared embedding space, information from different modalities can be mapped to the same dimensional space, ensuring semantic consistency [ 49 , 30 , 26 , 68 ] . By aligning fMRI with the image using deep learning methods, researchers implemented the aligned model for downstream tasks, such as Visual Description [ 71 , 72 ] , Image Generation [ 35 , 57 , 72 ] , Object Detection [ 71 ] , Image Retrieval [ 72 ] . These advances open up new possibilities for understanding human cognition and developing brain-computer interfaces. Figure 1 : The left is MSE Heatmap and the right is OT Heatmap. Although brain-image alignment has made significant progress, previous research methods have not considered the differences between brain signal representations and real-world physical signal representations. These methods mainly focus on point-to-point alignment, leading to misalignment and redundancy. Specifically, most alignment approaches rely on mean squared error (MSE) loss or contrastive InfoNCE loss [ 42 ] . However, these batch-level loss functions primarily focus on minimizing direct distance metrics, potentially overlooking complex non-linear relationships in cross-modal alignment. As shown in Fig. 1 , MSE can only capture point-wise relationships resulting in highlighted correlations only along the diagonal, while failing to capture global relationships, leading to weak correlations in other areas. Consequently, the underlying mechanisms of alignment remain insufficiently explored [ 59 ] . To solve the above issues, we analyzed the mathematical principle of MSE alignment. Different modalities can naturally be interpreted as different probability distributions, and aligning these distributions can be formulated as an Optimal Transport (OT) problem [ 62 ] . We found that MSE is equivalent to a special case of OT, namely the problem of optimizing a quadratic cost function (Euclidean distance) when the transport plan is fixed to the identity matrix. However, MSE does not consider whether a better matching strategy exists. Instead, it directly computes the error based on point-wise matching, which can easily lead to local optima. In contrast, general optimal transport builds upon MSE by allowing more flexible matching between different points and considering the alignment of two distributions from a global perspective. Compared to MSE, this approach is more effective in achieving a globally optimal solution. Thus, we propose to use the OT method to achieve brain-image alignment. We can obtain an optimized transport plan by optimizing the cost matrix, which depends on the distance between one distribution and another. Hence, the optimization of the OT problem is equivalent to learning the valid representations in machine learning [ 29 ] . We constructed a new loss function based on OT (See in Fig. 2 ). We proved that MSE loss is a special case of OT loss. As shown in the OT heatmap (See in Fig. 1 ), compared to MSE, the OT heatmap highlights points outside the diagonal, indicating that it also captures relationships beyond point-wise correspondences. Therefore, OT not only considers the point-wise alignment of MSE but also takes into account the impact of the overall distribution. By optimizing the loss function, we achieved the SOTA results on downstream tasks much better than MSE loss. We also verify the existing research conclusions about the brain in cognitive neuroscience from the perspective of deep learning. The brain processes information in both redundant and synergistic ways [ 36 ] . Receiving both types of information means that the brain can maintain robustness by utilizing redundant information while achieving better processing performance through synergistic. Redundancy means that increasing the amount of transport information does not always yield optimal outcomes, as excessive information flow may introduce noise or inefficiencies. Therefore, we explored the impact of restricting the mass of transport information on alignment evaluation (See in Fig. 5 ). At the same time, the Synergistic nature of brain function is closely tied to its modular organization [ 61 ] . Different brain areas (called Brodmann’s Areas) process different semantic information (See in Fig. 3 ) [ 8 ] . These areas are universally recognized and widely utilized as a standardized anatomical framework for mapping neuropsychological functions within the cerebral cortex [ 61 ] . Based on the brain activation, NSD dataset divides the effectively activated cortex into five major brain regions (A brain region consists of multiple brain areas.): 1) Vision: V1, V2, V3, and V4. 2) Body: EBA, FBA-1, FBA-2, and mTL-bodies (’mid temporal lobe bodies’). 3) Face: OFA, FFA-1, FFA-2, and mTL-faces (’mid temporal lobe faces’). 4) Place: OPA, PPA, and RSC. 5) Word: OWFA, VWFA-1, VWFA-2, mfs-words (’mid fusiform sulcus words’), and mTL-words (’mid temporal lobe words’). On this basis, we not only analyzed the redundancy (See in Tab. 2 and Fig. 6 ) and the synergy (See in Tab. 2 and Fig. 6 )

Read the complete text