Interpreting Transformers for Jet Tagging
Aaron Wang, Abhijith Gandrakota, Jennifer Ngadiuba, Vivekanand Sahu, Priyansh Bhatnagar, Elham E Khoda, Javier Duarte · 2024
Read Interpreting Transformers for Jet Tagging by Aaron Wang, Abhijith Gandrakota, Jennifer Ngadiuba, Vivekanand Sahu, Priyansh Bhatnagar, Elham E Khoda, Javier Duarte (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-03673.txt.
Introduction
Aaron Wang and six co-authors open the black box of a transformer model that classifies particle jets at the Large Hadron Collider, in this 2024 paper. When protons collide, they spray jets of particles, and telling which particle made which jet is a core task of collider physics. The model, ParT, is excellent at it, but nobody knew what it had learned.
Reading the model's attention maps, the team found a stark pattern: each particle attends to at most one other particle, a binary, pairing behavior nobody designed in. The model also shifts its focus between important particles and clumps depending on how the jet's parent decayed. Much of what it learned matches the classic substructure measures physicists invented by hand. Interpretation of this kind builds trust in machine taggers, and hints at leaner architectures for the next generation of detectors.
Catalog record
| CREATOR | Aaron Wang, Abhijith Gandrakota, Jennifer Ngadiuba, Vivekanand Sahu, Priyansh Bhatnagar, Elham E Khoda, Javier Duarte |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 16 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-03673
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Interpreting Transformers for Jet Tagging Aaron Wang, Abhijith Gandrakota, Jennifer Ngadiuba, Vivekanand Sahu, Priyansh Bhatnagar, Elham E Khoda, Javier Duarte Abstract Machine learning (ML) algorithms, particularly attention-based transformer models, have become indispensable for analyzing the vast data generated by particle physics experiments like ATLAS and CMS at the CERN LHC. Particle Transformer (ParT), a state-of-the-art model, leverages particle-level attention to improve jet-tagging tasks, which are critical for identifying particles resulting from proton collisions. This study focuses on interpreting ParT by analyzing attention heat maps and particle-pair correlations on the η \eta - ϕ \phi plane, revealing a binary attention pattern where each particle attends to at most one other particle. At the same time, we observe that ParT shows varying focus on important particles and subjets depending on decay, indicating that the model learns traditional jet substructure observables. These insights enhance our understanding of the model’s internal workings and learning process, offering potential avenues for improving the efficiency of transformer architectures in future high-energy physics applications. FERMILAB-CONF-24-0868-CMS-LDRD 1 Introduction Machine learning (ML) algorithms are becoming crucial for effectively analyzing the enormous data produced by particle physics experiments like ATLAS and CMS at the CERN LHC [ 1 ] . Attention-based transformer models [ 2 ] , with their ability to effectively capture and weigh the relative importance of different elements in input data, have revolutionized various domains, including natural language processing and computer vision [ 3 , 4 , 5 ] . The ability of the attention mechanism to discern intricate correlations in the collimated spray of particles jets initiated from massive particle decays has proven invaluable in searches for new physics and standard model measurements [ 6 ] . High-energy collisions of protons at LHC can create new unstable particles which then decay and produce sprays of outgoing particles, called jets, which are observed by these experiments [ 7 ] . Jet tagging, or the process of identifying the particle that initiates this spray, is a critical step in data analysis at the LHC [ 8 , 9 ] . One such state-of-the-art model for jet tagging is Particle Transformer (ParT) [ 10 ] , which is a transformer variant that leverages physics-inspired particle-particle pairwise features (or “interactions”) to augment the attention mechanism. Although this is one of the best performing jet-tagging models, there is effort required to interpret the underlying workflow and why the model performs so well. The attention-based architecture of transformers may offer deeper insight into the inner workings of neural networks [ 11 ] . In the case of a jet tagger, the attention matrix captures the particle-to-particle correlations potentially making these ML models more interpretable. These attention scores between particles highlight the most important connections between particles necessary for classification. This helps us to understand whether the neural network is learning the physics we know. In addition, looking at which parts of the input the neural network focuses on can help us improve the computational performance by pruning unnecessary parts of the neural network architecture, resulting in a more efficient jet tagging model. Related Work This paper builds on prior research that has applied machine learning, specifically graph neural networks (GNNs), to particle physics problems at the CERN LHC [ 12 , 13 ] . Reference [ 14 ] attempts to explain how ParticleNet [ 8 ] , a GNN designed for jet tagging learns to classify three-prong hadronic top quark decays using edge relevancy ( R ) graphs based on layerwise relevance propagation. Similarly, we visualize the attention scores between pairs of particles in order to investigate how ParT classifies jets. 2 Methods This study uses the pre-trained ParT model, trained using the JetClass dataset [ 10 , 15 ] , which consists of 100 million examples of jets for training, 5 million for validation, and 20 million for testing, containing 10 classes of jets with different decay modes of quarks, gluons, W and Z bosons, Higgs bosons, and top quarks. The classes are q/g , H\to b\bar{b} , H\to c\bar{c} , H\to gg , H\to 4q , H\to\ell\nu qq^{\prime} , t\to bqq^{\prime} , t\to b\ell\nu , W\to qq^{\prime} , and Z\to q\bar{q} , indicating the decays to combination of light quarks ( q ), charm quark ( c ), bottom quark ( b ), and lepton ( \ell ). There are 17 input features divided into 3 categories: kinematics, particle ID, and trajectory displacement. Each jet is composed of a maximum of 128 particles, ordered by p_{\text{T}} , with an average of 30–50 particles per jet. ParT introduces a modified attention mechanism, called particle multihead attention (P-MHA). Given a particle-level representation of a jet with N constituents with each constituent having d features, x\in\mathbb{R}^{N\times d} , the attention is computed as \displaystyle\begin{split}\text{P-MHA}(x)&=\text{concat}(\text{head}_{1},\dots,\text{head}_{h})W^{O}\\ \text{head}_{i}&=\text{softmax}\left(\frac{xW_{i}^{W}(xW_{i}^{K})^{\intercal}}{\sqrt{d_{k}}}+U_{i}\right)xW_{i}^{V},\end{split} (1) where U_{i}\in\mathbb{R}^{N\times N} is a single feature for each pair of particles, which can be concatenated into a larger structure, known as the pairwise interaction matrix U\in\mathbb{R}^{N\times N\times d^{\prime}} . In ParT, the number of pairwise features d^{\prime} must be equal to the number of heads h , which is chosen to be 8. In addition, d_{k}=16 so that the overall output dimension is the same as the input dimension d=hd_{k}=128 . The pairwise interaction matrix U is learned by applying a 4-layer pointwise MLP with (64, 64, 64, 8) channels and GELU nonlinearity from the 4 features motivated by Ref. [ 16 ] , (\ln\Delta,\ln k_{\text{T}},\ln z,\ln m^{2}) , where \displaystyle\begin{split}\Delta&=\sqrt{(y_{a}-y_{b})^{2}+(\phi_{a}-\phi_{b})^{2}},\\ k_{\text{T}}&=\min(p_{\text{T},a},p_{\text{T},b})\Delta,\\ z&=\min(p_{\text{T},a},p_{\text{T},b})/(p_{\text{T},a}+p_{\text{T},b}),\\ m^{2}&=(E_{a}+E_{b})^{2}-\|\mathbf{p}_{a}+\mathbf{p}_{b}\|^{2},\end{split} (2) y_{i} is the rapidity, \phi_{i} is the azimuthal angle, p_{\text{T},i}=(p_{x,i}^{2}+p_{y,i}^{2})^{1/2} is the transverse momentum, and \mathbf{p}_{i}=(p_{x,i},p_{y,i},p_{z,i}) is the momentum 3-vector and \|\cdot\| is the norm, for i=a , b . We use the exact pre-trained model weights provided from the Particle Transformer repository [ 10 ] . 3 Results Figure 1: Distribution of attention scores (left) illustrating the binary nature. Heat map of attention scores for example jets in the t\to b\ell\nu (center) and t\to bqq^{\prime} classes (right) between the constituent particles. Distribution of Attention Scores We visualize the attention scores across all heads in a heat map and observe that the ParT model’s attention scores exhibit a nearly binary (on/off) pattern. As shown in Fig. 1 , the distribution of attention scores reveals that most values are either close to one or zero. This binary characteristic indicates that particles generally attend to, at most one other particle. This behavior raises the intriguing question of whether the model is capable of capturing underlying physical laws and how we might interpret the specific physics the model is learning from the data. The binary nature of the attention weights also suggests potential for more efficient transformer models by reducing the computational complexity of attention mechanisms, focusing only on key interactions between particles. Particle Attention Graphs We decluster each jet into a specific number of subjets using the k_{\text{T}} algorithm [ 17 ] implemented in the FastJet package [ 18 ] with Python bindings [ 19 ] . We cluster into two subjets for leptonic top quark decays ( t\to b\ell\nu ), three subjets for hadronic top decays ( t\to bqq^{\prime} ), and four subjets for Higgs decays into four quarks ( H\to 4q ). For each process, we visualize the attention scores by representing