Local Temporal Feature Enhanced Transformer with ROI-rank Based Masking for Diagnosis of ADHD
Byunggun Kim 1, Younghun Kwon 1, 2] · 2025
Read Local Temporal Feature Enhanced Transformer with ROI-rank Based Masking for Diagnosis of ADHD by Byunggun Kim 1, Younghun Kwon 1, 2] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-11474.txt.
Introduction
Byunggun Kim and Younghun Kwon built a transformer model in 2025 to diagnose ADHD from resting brain scans. The scans record blood-flow signals across brain regions while a person lies still. Their model hunts for the space-time patterns that mark the disorder, in children and adults alike.
Three ideas shape the model. A convolutional front end builds richer region features. Attention works in short time windows to catch local signal habits. A masking trick favors the regions most tied to each other. Tested on a public set of 939 people, it beat other transformer variants on accuracy and related scores. The authors say the favored regions double as candidate biomarkers for the disorder.
Catalog record
| CREATOR | Byunggun Kim 1, Younghun Kwon 1, 2] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-11474
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Local Temporal Feature Enhanced Transformer with ROI-rank Based Masking for Diagnosis of ADHD Byunggun Kim 1, Younghun Kwon 1, 2] Abstract In modern society, Attention-Deficit/Hyperactivity Disorder (ADHD) is one of the common mental diseases discovered not only in children but also in adults. In this context, we propose a ADHD diagnosis transformer model that can effectively simultaneously find important brain spatiotemporal biomarkers from resting-state functional magnetic resonance (rs-fMRI). This model not only learns spatiotemporal individual features but also learns the correlation with full attention structures specialized in ADHD diagnosis. In particular, it focuses on learning local blood oxygenation level dependent (BOLD) signals and distinguishing important regions of interest (ROI) in the brain. Specifically, the three proposed methods for ADHD diagnosis transformer are as follows. First, we design a CNN-based embedding block to obtain more expressive embedding features in brain region attention. It is reconstructed based on the previously CNN-based ADHD diagnosis models for the transformer. Next, for individual spatiotemporal feature attention, we change the attention method to local temporal attention and ROI-rank based masking. For the temporal features of fMRI, the local temporal attention enables to learn local BOLD signal features with only simple window masking. For the spatial feature of fMRI, ROI-rank based masking can distinguish ROIs with high correlation in ROI relationships based on attention scores, thereby providing a more specific biomarker for ADHD diagnosis. The experiment was conducted with various types of transformer models. To evaluate these models, we collected the data from 939 individuals from all sites provided by the ADHD-200 competition. Through this, the spatiotemporal enhanced transformer for ADHD diagnosis outperforms the performance of other different types of transformer variants. (77.78ACC 76.60SPE 79.22SEN 79.30AUC) 1 Introduction Attention- deficit/hyperactivity disorder (ADHD) is a common mental disease that occurs from children to adults [ 1 , 2 , 3 ] . It causes difficulties in many ways among daily life. In the recent, the number of patients diagnosed with ADHD has been increasing [ 4 , 5 ] . However, the exact cause of ADHD has not yet been discovered, either clinically or psychologically [ 6 ] . Recently, the use of Functional Magnetic Resistance Imaging (fMRI) data as a new approach to finding biomarkers of ADHD has attracted attention [ 7 , 8 , 9 , 10 , 11 ] . This is because fMRI contain biological activation information according to the location of the brain and has shown meaningful results in studies of various neuropsychiatric diseases [ 7 , 8 ] . It has become possible to find reliable biomarkers of neuropsychiatric diseases by using the Resting-State Functional Connectivity (RSFC) that analyzes regional correlations in brain [ 12 ] . However, traditional RSFC analysis uses only a small amount of data obtained from a specific site, making it difficult to detect heterogeneity across individual studies [ 13 , 14 ] . Figure 1: The enhanced ADHD diagnosis transformer model architecture. It modified three modules (CNN-based embedding block, local temporal attention, and ROI-rank based masking) from baseline transformer. Therefore, most of studies [ 15 , 16 ] are a shift toward using a large number of rs-fMRI data samples collected from various sites to find more general biomarkers. Especially, deep learning models have shown great potential in discovering new biomarkers for ADHD [ 17 , 18 , 19 , 20 ] . Subsequently, the development of an ADHD diagnostic model using rs-fMRI has led to new understanding. Specifically, deep learning-based ADHD diagnosis models have proposed the architectures for extracting temporal or spatial features from rs-fMRI [ 18 , 21 , 19 , 20 , 22 , 23 ] . These usually consist of Convolutional Neural Network (CNN) and Recurrent Neural Network (RNN) to extract brain functions contained in the Blood Oxygen Level Dependent (BOLD) signal. However, due to structural limitations, they had difficulty to train spatiotemporally global features from rs-fMRI data. Accordingly, the attention mechanism [ 24 ] , which can learn important features through relationships, is being actively used in learning rs-fMRI data [ 25 , 26 , 27 , 28 ] . Among them, models that modify the transformer structure to make full use of the attention mechanism for spatiotemporal feature extraction of rs-fMRI are increasing. However, since rs-fMRI data has a four-dimension (x,y,z,t) , the attention mechanism optimized only for learning the partial relation of rs-fMRI has been proposed. To overcome these limitations, we constructed an encoder-decoder transformer model capable of learning the relationship between the full-range of rs-fMRI. This enables to learn independent relationship within spatial or temporal features. Also, it can attend to relation between spatial and temporal features. In addition, we proposed three modifications of the transformer structure suitable for ADHD diagnosis. These are one embedding method constructed based on 1D-CNN and two attention methods specialized for each feature: local temporal attention for temporal feature and ROI-rank based masking for spatial feature. Specifically, local temporal attention is an attention that uses window masking to restrict only the relationships between short term BOLD signals. This show that heavily focusing on relationship within local BOLD signals is important in first stage to learn temporal features. And BOLD signal learning is more effective by hierarchically applying full temporal attention after local temporal attention. Next, ROI-rank based masking is an attention method that effectively select a small number of ROIs based on attention scores. Therefore, it learns only the relationships between ROIs that more relevant to ADHD differentiation. This provides clues to support the assumption that patients with ADHD may have more sparsely strong functional connectivity in brain than healthy control (HC). In summary, the overall structure is shown in figure 1 . This new ADHD diagnosis transformer showed notable biomarker simultaneously not only in BOLD signal but also in ROI that more important than other. To prove this new ADHD diagnosis transformer’s performance, we collect data from multiple sites provided by the ADHD-200 competition. Also, we evaluate the various ablation type of models to prove that our proposed methods can more effectively diagnosis the ADHD. Through spatiotemporally enhanced transformer for diagnose ADHD, we achieved best diagnostic performance (77.78ACC 76.60SPE 79.22SEN 79.30AUC). The contents of each section of the paper are as follows. In Section 2 , we investigated the studies to find biomarkers of ADHD and the development of a deep learning-based ADHD diagnostic model. In section 3 , we explain that our proposed the transformer model and modified methods for ADHD diagnosis. In section 4 , we describe the specific experiments using the ADHD200 corpus and the analysis. Finally, in section 5 , we conclude this study. 2 Related Works 2.1 Traditional Study with rs-fMRI for ADHD It is difficult to find the clear evidence of ADHD. However, with the advent of the fMRI, which can indirectly obtain brain activity information from patients in a noninvasive manner, many of studies have shown meaningful results to find biomarkers of ADHD. Especially, with rs-fMRI, Default Mode Network (DMN), a network analysis that focus on the resting state brain activity, contributed to find the important brain regions that distinguish between focused and non-focused states of the brain [ 29 , 30 , 31 ] . These regions are also closely related to mental disorders that