What exactly did the Transformer learn from our physics data?

Martin Erdmann, Niklas Langner, Josina Schulte, Dominik Wirtz · 2025

Read What exactly did the Transformer learn from our physics data? by Martin Erdmann, Niklas Langner, Josina Schulte, Dominik Wirtz (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-21042.txt.

Introduction

Martin Erdmann and co-authors ask what a transformer truly learned from cosmic ray simulations in a 2025 paper. Networks that excel at science tasks can still hide their inner logic. Two probes test whether the learned features make physical sense.

First, the position codes trained on air showers, which spread in round symmetry, were inspected for that pattern. Second, the attention weights laid on particles traced back to a galaxy catalog were mapped. In both cases the network had picked up plausible, physics-shaped features on its own. The authors read this as hope that such nets learn nature, not noise.

Catalog record

CREATORMartin Erdmann, Niklas Langner, Josina Schulte, Dominik Wirtz
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE23 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-21042

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

What exactly did the Transformer learn from our physics data? Martin Erdmann, Niklas Langner, Josina Schulte, Dominik Wirtz Abstract Transformer networks excel in scientific applications. We explore two scenarios in ultra-high-energy cosmic ray simulations to examine what these network architectures learn. First, we investigate the trained positional encodings in air showers which are azimuthally symmetric. Second, we visualize the attention values assigned to cosmic particles originating from a galaxy catalog. In both cases, the Transformers learn plausible, physically meaningful features. keywords Deep Learning, Transformer, Attention, Embedding, Positional Encoding 1 Introduction With the initial implementation of the Transformer, it was found to deliver excellent benchmark results for text translation tasks Vaswani et al. (2017) . This was followed shortly afterwards by successful applications of the Transformer to image recognition Dosovitskiy et al. (2020) and then as generative models with great impact when opened to the public OpenAI (2022) . Due to its superior performance, the Transformer became established in science too and has replaced earlier approaches in a number of physics applications Qu et al. (2022) ; Finke et al. (2023) ; Abdul Halim et al. (2023) ; Wu et al. (2025) ; Brehmer et al. (2024) . For a collection on Transformer models developed in particle physics refer to Feickert and Nachman (2021) . For text- and image-based challenges, understanding the very good performance has been analysed in two aspects: 1) Positional encoding, that is information about arrangement patterns in the input data. 2) Attention, that concerns evaluation of correlations between input information. Both are convincingly demonstrated in self-attentions and translations, where annotated texts show that, after appropriate training, the Transformer evaluates connections between word sequences largely correctly. In scientific applications, the performance of Transformers is systematically evaluated by means of accuracy analyses or receiver operating characteristic (ROC) curves. However, it is often challenging to understand why Transformers can achieve outstanding performance. Recently, for a jet application in particle collisions, attention heat maps and particle-pair correlations have been used to demonstrate that the network learns traditional jet substructure observables Wang et al. (2024) . In this paper we show two applications from the physics of the highest-energy cosmic rays, where we want to understand through targeted visualizations what the excellent performance of the Transformers is based on. The scientific applications have two different objectives visualized in Fig. 1 . The first is to investigate about exploiting the azimuthal symmetry of a hexagon-shaped sensor array. Here we focus on the aspect of positional encoding. The second case is about the assignment of cosmic particles to given galaxies or to background. Here we investigate the attentions of the particles. Both studies are based on simulation data. Figure 1: ultra-high-energy cosmic rays propagate from their origin to an observatory on Earth. Two different Transformer networks are investigated here, first the hexagonal arrangement of the observatory’s sensors, and second the displacement of the cosmic rays owing to magnetic fields. 2 Positional encoding Scientific context ultra-high-energy cosmic rays are very likely composed of ionized atomic nuclei with masses ranging from protons up to iron. These cosmic particles are not observed directly but rather indirectly, through collisions with atoms in the Earth’s atmosphere (Fig. 1 ). These collisions generate so-called air showers, comprising billions of secondary particles. Detection of these air showers relies on a calorimetric measurement in which the Earth’s atmosphere serves as the absorber material, and readout is performed by a single layer of sensors at ground level. This setup is inherently stochastic: the relative detector coverage on the ground is minimal (on the order of 10^{-5} ), yet this low coverage remains fully sufficient because of the large number of secondary particles produced. In the single readout layer, sensors are typically arranged in a hexagonal pattern (Fig. 1 ). Each sensor records both the temporal profile of incoming secondary particles and the signal amplitudes. To characterize the kinematics of the cosmic particles, information from all sensors is merged. Quantities of the cosmic particles of interest are: energy, arrival direction (represented by zenith and azimuth), and mass-related information through the shower depth and the muon count. While direction and energy can successfully be reconstructed with rule-based algorithms, the shower profile and muon count—which are key for determining the mass—are quantities where connectionist methods such as deep neural networks (e.g., Transformers) show potential advantages over rule-based approaches. Details of how these observables are determined can be found in publications of the Pierre Auger Observatory Aab et al. (2015) ; Aab et al. (2021a) ; Aab et al. (2021b) ; Abdul Halim et al. (2023) ; Abdul Halim et al. (2025) . Research on Transformer In this work, we focus on whether a Transformer, through its training process, exploits the symmetry of the hexagonal sensor arrangement to reconstruct cosmic-ray characteristics. Air shower physics is essentially rotationally symmetric with respect to the azimuth of the cosmic-ray arrival direction, although small deviations may arise—for instance, due to the Earth’s magnetic field. Rotational symmetry in azimuth can be captured by neural network architectures that employ so-called hexaconv convolutions Hoogeboom et al. (2018) , in which filters are rotated six times in accordance with the hexagonal geometry. By contrast, the Transformer architecture presented here does not include any explicit information about symmetries. However, the Transformer could in principle encode domain-specific symmetries contained in the training data within its positional encodings. Figure 2 illustrates for a subset of nine sensors the positional encoding step within the complete Transformer architecture, developed for the conditions of the Pierre Auger Observatory Abdul Halim et al. (2023) . Note that the two-dimensional structure of the sensor array is flattened into a one-dimensional vector, i.e. it is to be re-learned by trainable positional encoding. Prior to this encoding step, the sensor signals in the time domain have already been analysed, so that the most relevant information for each sensor is distilled into d=130 latent variables. A trainable positional encoding of the same dimension d is then added to each sensor’s set of d latent variables, preserving the dimensionality for the subsequent network stage that ultimately infers the mass-related parameters. For additional details on the network architecture, refer to Abdul Halim et al. (2023) . Figure 2: Positional Encoding part of the Transformer architecture for reconstructing information on the cosmic particle mass. Exemplified are nine sensors (circles, central sensor in blue) with their information coded in 130 latent variables (gray boxes). The red part shows the trainable Positional Encoding vectors of equal dimension which are added to the latent variables. Here, we interpret the positional encodings for each sensor that were adapted during the network’s training to retrieve information on the particle mass. Typically, in an air shower event, one of the sensors exhibits the strongest signal, which we take then as the geometric center of the sensor array. The sensors in the immediately neighboring hexagonal ring record smaller signals, and sensors in successive outer rings are struck even less. Our primary interest lies in the similarity of the d=130 -dimensional positional encoding vectors across sensors. As a measure of similarity, we calculate the scalar product normalized to unity (similar to the cosine of the angle between vectors) by comparing the positional encoding vector \vec{n} of one

Read the complete text