Cross-Attention fusion of MRI and Jacobian maps for Alzheimer’s Disease Diagnosis
Shijia Zhang, MBI, Xiyu Ding, MS, Brian Caffo, PhD, Junyu Chen, PhD, Xinyi Zhang, PhD, Hadi Kharrazi, PhD, Zheyu, Wang PhD · 2025
Read Cross-Attention fusion of MRI and Jacobian maps for Alzheimer’s Disease Diagnosis by Shijia Zhang, MBI, Xiyu Ding, MS, Brian Caffo, PhD, Junyu Chen, PhD, Xinyi Zhang, PhD, Hadi Kharrazi, PhD, Zheyu, Wang PhD (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-00586.txt.
Introduction
Zhang and colleagues present a study from twenty twenty-five on spotting Alzheimer's disease from brain scans. Early spotting matters, since brain loss cannot be undone once it sets in. Standard scans show light and dark tissue patterns, but early shifts are subtle. A second map type shows where the brain has shrunk or grown in shape. The team mixes both map types with a method that lets each guide the other. They test it on a large public dataset for this disease.
Their mix beat two other ways of joining the maps. It did best at telling disease cases from healthy ones. It also used far fewer parts than two big rival models. The authors say shape maps add real value to standard scans.
Catalog record
| CREATOR | Shijia Zhang, MBI, Xiyu Ding, MS, Brian Caffo, PhD, Junyu Chen, PhD, Xinyi Zhang, PhD, Hadi Kharrazi, PhD, Zheyu, Wang PhD |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 21 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-00586
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Cross-Attention fusion of MRI and Jacobian maps for Alzheimer’s Disease Diagnosis Shijia Zhang, MBI, Xiyu Ding, MS, Brian Caffo, PhD, Junyu Chen, PhD, Xinyi Zhang, PhD, Hadi Kharrazi, PhD, Zheyu, Wang PhD Abstract Early diagnosis of Alzheimer’s disease (AD) is critical for intervention before irreversible neurodegeneration occurs. Structural MRI (sMRI) is widely used for AD diagnosis, but conventional deep learning approaches primarily rely on intensity-based features, which require large datasets to capture subtle structural changes. Jacobian determinant maps (JSM) provide complementary information by encoding localized brain deformations, yet existing multimodal fusion strategies fail to fully integrate these features with sMRI. We propose a cross-attention fusion framework to model the intrinsic relationship between sMRI intensity and JSM-derived deformations for AD classification. Using the Alzheimer’s Disease Neuroimaging Initiative (ADNI) dataset, we compare cross-attention, pairwise self-attention, and bottleneck attention with four pre-trained 3D image encoders. Cross-attention fusion achieves superior performance, with mean ROC-AUC scores of 0.903 (±0.033) for AD vs. cognitively normal (CN) and 0.692 (±0.061) for mild cognitive impairment (MCI) vs. CN. Despite its strong performance, our model remains highly efficient, with only 1.56 million parameters—over 40 times fewer than ResNet-34 (63M) and Swin UNETR (61.98M). These findings demonstrate the potential of cross-attention fusion for improving AD diagnosis while maintaining computational efficiency. 1 Introduction Alzheimer’s disease (AD) is the most common cause of dementia, leading to irreversible neurodegeneration that ultimately affect patients’ daily function and places a significant burden on caregivers and society [ 1 ] . With no cure currently available, early diagnosis before irreversible brain structure change occur is crucial for potential interventions to delay disease progression, manage symptoms and improve patient outcomes. Brain atrophy, a hallmark of AD, has been suggested as a potential biomarker for early AD detection, as alterations in brain structure may precede years before noticeable cognitive symptoms [ 19 ] . Structural magnetic resonance imaging (sMRI) is the primary imaging modality for assessing brain atrophy and neurodegenerative changes due to its superior tissue contrast and high spatial resolution [ 6 ] . Consequently, sMRI has been intensively explored for early AD diagnosis, particularly with the advancements of modern machine learning and computational power. Deep learning methods based on sMRI, particularly convolutional neural networks (CNN), have shown promise in the diagnosis of AD [ 6 , 8 , 14 ] . However, current approaches face several challenges. First, the majority of current deep learning studies in AD rely on 2D MRI slices, requiring manual selection of informative cross-sections, which may introduce bias and limit the model’s ability to capture 3D neurodegenerative patterns. Second, these methods predominantly rely on intensity-based features, which contain rich structural information but often require large sample sizes to effectively capture subtle, localized deformation features – particularly in 3D whole-brain analysis. Third, a few studies incorporate other imaging modalities, such as PET, or clinical and genetic factors like age and APOE status, to improve AD classification [ 9 , 15 , 18 , 20 ] . While these multi-modal approaches can enhance diagnostic accuracy, PET scans are often unavailable in routine clinical settings due to cost and accessibility constraints, and clinical assessments, including cognitive tests and genetic screening, typically occur later in the diagnostic process. As a result, relying on these features may lead to overly optimistic performance estimates and inflate the true predictive value of MRI-based models for early AD diagnosis. Alternatively, Jacobian determinant maps (JSM) provide a compact and effective representation of morphological transformations in the brain by quantifying localized expansion or contraction at each voxel relative to a standard template [ 3 ] . JSMs encode structural deformations, allowing for efficient feature extraction and potentially improved sensitivity to neurodegenerative changes, particularly with moderate sample sizes as often seen in AD cohort studies. For instance, hippocampal atrophy in AD manifests not only as changes in signal intensity but also as localized tissue contraction, which JSMs can directly quantify [ 17 ] . Previous studies have trained CNN models using whole-brain Jacobian domain features at the subject level to diagnose AD with surprisingly high performance [ 2 ] . Given these advantages, integrating JSM into deep learning frameworks could potentially enhance sensitivity to structural alterations, particularly in 3D MRI studies with moderate sample size. Only a few studies have explored the integration of deformative characteristics with sMRI intensity maps for AD diagnosis. Most existing approaches treat modalities independently, either by concatenating the input level, where the raw sMRI and Jacobian maps are merged at the input stage and fed into a shared feature extractor, or by employing conventional early or late fusion strategies [ 2 , 15 ] . However, such approaches fail to account for the intrinsic relationship between sMRI and JSM, which originates from the corresponding sMRI scan and provides complementary information about structural changes. In this paper, we explore the use of attention mechanisms [ 23 ] , particularly cross-attention [ 10 ] , as a framework for modeling intrinsic interactions between sMRI and JSM. We leverage the cross-attention mechanisms by employing JSM to detect deformation regions and direct the extraction of anatomically meaningful sMRI intensity patterns linked to AD. Although cross-attention has been successfully applied to integrate multimodal imaging data, such as sMRI and PET [ 9 ] , for AD diagnosis, to our knowledge, it has not been previously utilized for fusion of JSM and sMRI in AD classification. Given the complementary nature of these modalities – where sMRI provides intensity-based structural information and JSM encodes localized deformation patterns – the attention mechanism offers a potential approach to effectively model their interdependencies while preserving their distinct characteristics. To evaluate the feasibility of this fusion strategy, we utilize the Alzheimer’s Disease Neuroimaging Initiative (ADNI) database [ 13 ] and employ a simple CNN encoder integrated with cross-attention. While transformer-based architectures are increasingly explored in medical imaging, CNNs remain well-suited for feature extraction due to their strong spatial inductive biases and relatively low data requirements. Given the moderate sample sizes typically available in AD datasets, a CNN backbone enables us to examine the benefits of cross-attention without introducing excessive computational complexity or requiring large-scale pertaining. This study provides an initial assessment of cross-attention fusion for JSM-MRI in AD classification. By leveraging the complementary strengths of both modalities, we explore whether this integration enhances the model’s ability to distinguish between AD, mild cognitive impairtment (MCI), and cognitive normal (CN) subjects. Our findings offer insights into improving the accuracy and reliability of early AD diagnosis, as well as contributing to the broader development of multimodal deep learning techniques for neuroimaging. 2 Methods 2.1 Data preparation The study used T1-weighted sMRI data from the ADNI database [ 13 ] . A total of 818 subjects from ADNI-1 were included, including 224 patients with AD, 362 cases of MCI, and 232 controls of CN. Inclusion criteria were: (i) 55 years of age, (ii) diagnosis of CN, AD, and MCI is closet date within 6 months of image acquired date, and (3) availability of high-resolution (1×1×1 mm 3 ) sMRI scans. Each participant was represented by their earliest available image. Demographic and clinical characteristics are summarized in Table 1 . Table 1: Demographics and MMSE a Scores by diagnosis group. Group n Avg. Age