Why is Attention Sparse in Particle Transformer?
Timothy Legge, Aaron Wang, Jacob Ortiz, Victor Limouzi, Zihan Zhao, Abhijith Gandrakota, Elham E. Khoda, Jennifer Ngadiuba, Javier Duarte, Richard Cavanaugh · 2025
Read Why is Attention Sparse in Particle Transformer? by Timothy Legge, Aaron Wang, Jacob Ortiz, Victor Limouzi, Zihan Zhao, Abhijith Gandrakota, Elham E. Khoda, Jennifer Ngadiuba, Javier Duarte, Richard Cavanaugh (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-00210.txt.
Introduction
Jet tagging sorts the sprays of particles produced in collider collisions, and transformer models now lead the field. Timothy Legge and a team of co-authors ask why attention in the Particle Transformer is so sparse, in a 2025 study. Its attention pattern is nearly binary, which is striking.
The team investigates where that sparsity comes from and whether it encodes physically meaningful correlations. The Particle Transformer is a leading model at the CERN Large Hadron Collider. Understanding its inner behavior helps physicists trust its decisions and may point to simpler, faster models that keep the same physical insight. Sparse attention is cheaper to compute, so the pattern may be a feature, not a bug.
Catalog record
| CREATOR | Timothy Legge, Aaron Wang, Jacob Ortiz, Victor Limouzi, Zihan Zhao, Abhijith Gandrakota, Elham E. Khoda, Jennifer Ngadiuba, Javier Duarte, Richard Cavanaugh |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 19 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-00210
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Why is Attention Sparse in Particle Transformer? Timothy Legge, Aaron Wang, Jacob Ortiz, Victor Limouzi, Zihan Zhao, Abhijith Gandrakota, Elham E. Khoda, Jennifer Ngadiuba, Javier Duarte, Richard Cavanaugh Abstract Transformer-based models have achieved state-of-the-art performance in jet tagging at the CERN Large Hadron Collider (LHC), with the Particle Transformer (ParT) representing a leading example of such models. A striking feature of ParT is its sparse, nearly binary attention structure, raising questions about the origin of this behavior and whether it encodes physically meaningful correlations. In this work, we investigate the source of ParT’s sparse attention by comparing models trained on multiple benchmark datasets and examine the relative contributions of the attention term and the physics-inspired interaction matrix before softmax. We find that binary sparsity arises primarily from the attention mechanism itself, with the interaction matrix playing a secondary role. Moreover, we show that ParT is able to identify key jet substructure features, such as leptons in semileptonic top decays, even without explicit particle identification inputs. These results provide new insight into the interpretability of transformer-based jet taggers and clarify the conditions under which sparse attention patterns emerge in ParT. Our code is available here . FERMILAB-PUB-25-0820-CMS-LDRD-PPD 1 Introduction Machine learning (ML) has become essential for analyzing the massive datasets from the CERN Large Hadron Collider (LHC) [ 8 ] . Jet tagging, or the classification of collimated streams of particles that result from high-energy collisions at LHC, has proven to be a task highly suitable to ML. Among the most promising approaches to jet tagging are graph neural networks (GNNs) and transformers, which capture correlations among particles in jets. GNNs, such as ParticleNet [ 17 ] , reached state-of-the-art jet tagging by treating jets as unordered particle clouds [ 23 , 6 ] . More recently, transformer models [ 24 ] , which have transformed language and vision processing [ 20 , 7 , 1 ] , have been adapted to collider physics, where their ability to model long-range dependencies provides a powerful tool for identifying jet substructure signatures [ 22 , 15 , 21 , 9 ] . A leading example is the Particle Transformer (ParT) [ 19 ] , which augments multi-head attention with physics informed pairwise particle interaction features. This has made ParT one of the most effective jet taggers to date. Interpretability is crucial in high-energy physics to ensure ML models capture physically meaningful correlations rather than dataset-specific artifacts. Attention provides a natural interpretability handle: particle–particle attention maps can reveal what transformers learn [ 4 ] . Understanding when and why attention becomes sparse may further guide model optimization, clarifying the role of physics-inspired interactions and suggesting opportunities to simplify architectures and make them more efficient. 2 Related Work ML methods, particularly graph neural networks (GNNs), have been widely applied to particle physics problems at the CERN LHC [ 23 , 6 ] . ParticleNet [ 17 ] , a GNN designed for jet tagging, achieved state-of-the-art performance by treating jets as particle clouds. Recent studies have also interpreted such models. For example, Mokhtar et al. [ 14 ] employed layerwise relevance propagation to construct edge relevancy, uncovering how ParticleNet distinguishes three-prong hadronic top quark decays. These efforts highlight the growing emphasis on explainability in ML for high-energy physics. Building on this line of work, attention-based transformers have emerged as powerful alternatives to GNNs for jet tagging. ParT [ 16 , 25 ] extends the standard attention mechanism with physics-inspired pairwise particle interactions. While delivering excellent performance, ParT also raises important interpretability questions. Wang et al. [ 25 ] analyzed ParT’s attention maps and particle-pair correlations in the \eta – \phi plane, revealing a sparse, nearly binary attention pattern where each particle focuses on at most one other particle. Their subjet-level studies further showed that ParT attends to leptons in semileptonic top decays and captures both intra- and inter-subjet structures in hadronic top and Higgs decays, suggesting that the model learns physically meaningful jet substructure. In line with these efforts, our work investigates the origin of ParT’s sparse attention by systematically comparing datasets and quantifying the roles of traditional attention and the physics-inspired interaction matrix. (a) Quark–Gluon Dataset Attention Distribution (b) Top Landscape t\to bqq^{\prime} Attention Distribution (c) JetClass Full t\to bqq^{\prime} Attention Distribution (d) JetClass Kinematic t\to bqq^{\prime} Attention Distribution (e) JetClass t\to bqq^{\prime} (f) JetClass t\to b\ell\nu (g) JetClass Kinematic t\to bqq^{\prime} (h) JetClass Kinematic t\to b\ell\nu (i) JetClass Kinematic Proportion of Attention to Lepton in t\to b\ell\nu Figure 1: Comparison of ParT attention distributions across datasets (a–d), \eta – \phi particle attention maps (e–h), and the distribution of attention scores to the lepton in t\to b\ell\nu with ParT trained on JetClass Kinematic (i). \eta – \phi maps use pre-softmax attention values without the interaction matrix included. Symbols: (✖: muon, ▲: charged hadron, ▼: neutral hadron, ⚫: photon, ✚: electron). Transparency scales with particle p_{\mathrm{T}} , while line intensity reflects attention scores. 3 Datasets and Model This method compares the attention heatmaps and interpretability of multiple different datasets. We use the JetClass dataset [ 19 ] , the Top Landscape dataset [ 12 ] , and the Quark–Gluon dataset as our benchmark datasets for comparative studies. Datasets We evaluate models on three widely used jet-tagging benchmarks. The JetClass dataset [ 18 ] comprises 100M training, 5M validation, and 20M test jets across 10 classes from quarks, gluons, and decays of W , Z , Higgs, and top quarks ( q/g , H\to b\bar{b} , H\to c\bar{c} , H\to gg , H\to 4q , H\to\ell\nu qq^{\prime} , t\to bqq^{\prime} , t\to b\ell\nu , W\to qq^{\prime} , Z\to q\bar{q} ). Each jet contains up to 128 particles (avg. 30–50) with 17 features spanning kinematics, PID, and displacement. The Top Landscape Tagging benchmark [ 11 ] provides 2M jets (1.2M train, 0.4M val/test) distinguishing t\to bqq^{\prime} decays from light-quark/gluon backgrounds, described only by constituent four-vectors with 7 kinematic features. Finally, the Quark–Gluon Tagging benchmark [ 13 ] consists of 2M jets (1.6M train, 0.2M val/test) for discriminating quark- from gluon-initiated jets, using both kinematic and PID information. Model Architecture ParT is a state-of-the-art jet tagging model, which introduces a modified attention mechanism, called particle multi-head attention (P-MHA). Given a particle-level representation of a jet with N constituents, x\in\mathbb{R}^{N\times d} , the attention is computed as \displaystyle\begin{split}\text{P-MHA}(x)&=\text{concat}(\text{head}_{1},\dots,\text{head}_{h})W^{O}\\ \text{head}_{i}&=\text{softmax}\left(\frac{xW_{i}^{W}(xW_{i}^{K})^{\intercal}}{\sqrt{d_{k}}}+U_{i}\right)xW_{i}^{V},\end{split} (1) where U_{i}\in\mathbb{R}^{N\times N} encodes pairwise particle interactions. These are learned from a set of kinematic features (\ln\Delta,\ln k_{\text{T}},\ln z,\ln m^{2}) described in Refs. [ 5 , 19 ] that are encoded through convolutional layers. We use pretrained ParT weights from Ref. [ 19 ] for models on the JetClass dataset, and train from scratch on the Top and Quark–Gluon datasets. 4 Results Distribution of Attention Scores We visualize the distribution of attention scores across all heads for models trained on the full JetClass feature set and on JetClass with only kinematic features, as shown in images (a – d) of Figure 1 . For both cases, as well as for the Quark–Gluon dataset, ParT produces a very sparse, nearly binary attention map. In contrast, the Top Landscape dataset does not exhibit this binary behavior. Since the Top Landscape dataset contains only kinematic information and no particle identification (PID) features,