Attention Trajectories as a Diagnostic Axis for Deep Reinforcement Learning

Charlotte Beylier [email protected], Hannah Selder [email protected], Arthur Fleig [email protected], Simon M. Hofmann [email protected], Nico Scherf [email protected] · 2025

Read Attention Trajectories as a Diagnostic Axis for Deep Reinforcement Learning by Charlotte Beylier [email protected], Hannah Selder [email protected], Arthur Fleig [email protected], Simon M. Hofmann [email protected], Nico Scherf [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-20591.txt.

Introduction

Beylier, Selder, Fleig, Hofmann, and Scherf track where learning agents look as they train. Their 2025 paper builds a method to study deep reinforcement learning through attention. Agents post high scores while their true strategies stay hidden. Which inputs an agent relies on, and when that changes, matters. Saliency maps highlight the inputs behind each choice.

The method pools saliency at object and modality levels. Those pooled profiles form trajectories of attention over training. Atari games and custom Pong setups supplied test agents. Muscle-driven biomechanical user simulations added a harder case. Each algorithm showed its own attention biases in the traces. Some agents exploited reward loopholes the designers never intended. Others overfit to redundant sensory channels that added nothing. The attention patterns matched measurable behavior differences closely. Several saliency methods cross-checked the findings for robustness. The authors propose attention as a third diagnostic axis beside scores.

Catalog record

CREATORCharlotte Beylier [email protected], Hannah Selder [email protected], Arthur Fleig [email protected], Simon M. Hofmann [email protected], Nico Scherf [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE73 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-20591

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Attention Trajectories as a Diagnostic Axis for Deep Reinforcement Learning Charlotte Beylier [email protected], Hannah Selder [email protected], Arthur Fleig [email protected], Simon M. Hofmann [email protected], Nico Scherf [email protected] Abstract While deep reinforcement learning agents demonstrate high performance across domains, their internal decision processes remain difficult to interpret when evaluated only through performance metrics. In particular, it is poorly understood which input features agents rely on, how these dependencies evolve during training, and how they relate to behavior. We introduce a scientific methodology for analyzing the learning process through quantitative analysis of saliency. This approach aggregates saliency information at the object and modality level into hierarchical attention profiles, quantifying how agents allocate attention over time, thereby forming attention trajectories throughout training. Applied to Atari benchmarks, custom Pong environments, and muscle-actuated biomechanical user simulations in visuomotor interactive tasks, this methodology uncovers algorithm-specific attention biases, reveals unintended reward-driven strategies, and diagnoses overfitting to redundant sensory channels. These patterns correspond to measurable behavioral differences, demonstrating empirical links between attention profiles, learning dynamics, and agent behavior. To assess robustness of the attention profiles, we validate our findings across multiple saliency methods and environments. The results establish attention trajectories as a promising diagnostic axis for tracing how feature reliance develops during training and for identifying biases and vulnerabilities invisible to performance metrics alone. 1 Introduction Deep reinforcement learning (DRL) has achieved remarkable success in domains ranging from robotics to simulated users to healthcare ( 19 ; 34 ; 29 ; 15 ; 47 ) . Despite these advances, the learning process of DRL agents remains poorly understood. In particular, it is unclear how agents learn to rely on specific features when making decisions, how these dependencies evolve during training, and what factors influence this process. This opacity hinders not only scientific progress for improving how DRL agents learn, but also the safe and reliable deployment of DRL in real-world settings. Among existing interpretability tools, saliency maps are used to highlight the input features most relevant for a model’s prediction ( 61 ; 48 ; 22 ) . Their post-hoc nature allows them to be applied to diverse architectures and problem settings, offering a versatile means of analyzing deep neural networks. In DRL, they have been used to visualize action-relevant regions and to expose issues such as observational overfitting ( 42 ; 52 ) . However, their potential to study the learning process of DRL agents and extract global patterns of attention remains largely underexplored. The majority of prior work focuses on developing new saliency methods 17 ; 26 ; 42 ; 32 ; 23 , while only a few studies apply existing techniques to analyze DRL agents 35 ; 55 ; 59 , and even fewer attempt to extract global patterns of attention 18 ; 32 , that is to say, consistent, system-level regularities in how attention is allocated across agents over time. Indeed, saliency maps are typically applied to a few isolated frames from already trained agents followed by a human interpretation of the agent’s strategy. While such analyses are valuable for improving saliency techniques and providing a glimpse of what features DRL agents rely on, they do not extract information about global patterns of attention and provide very limited insight into how feature dependencies change during training. Furthermore, this qualitative and case-based usage is prone to post-hoc storytelling and confirmation bias, a limitation highlighted by 6 , who found that only 7% of interpretability claims were experimentally verified. Here, we propose a scientific methodology, exploiting existing saliency tools, to extract global patterns of attention trajectory in DRL agents and connecting them to agent’s behavior through controlled experiments. Our approach focuses on quantifying and analyzing the evolution of agents’ attention throughout training and on identifying the factors that shape it. This systematic, quantitative approach reduces the risk of post-hoc storytelling and cognitive bias. We demonstrate the utility of this methodology through three case studies: (i) a comparative analysis of attention trajectories across four classic DRL algorithms (A2C, PPO, DQN, and QR-DQN) in Atari games, revealing algorithm-specific attention profiles indicative of robustness; (ii) a custom Pong environment designed to isolate the influence of reward shaping, showing how rewards shape both attention and strategy; and (iii) a biomechanical simulation with multimodal sensory inputs including vision and proprioception, where we analyze how attention is distributed across input channels in sequential tasks and diagnose overfitting to redundant sensory channels. Robustness checks across multiple saliency methods further assess the consistency of our findings. In summary, our contributions include: • A measurement tool, the hierarchical-attention profile (or h-profile), which quantifies how agents allocate attention to input features by aggregating saliency maps into consistent, structured, and interpretable representations. • An empirical analysis of how these attention profiles evolve during training and how they are influenced by key factors: the learning algorithm, the reward function, and the properties of the environment, on three case studies – from ATARI games to biomechanical simulations with multimodal sensory inputs. • A systematic connection between attention and observable behavior established through controlled experiments designed to reveal how changes in feature reliance manifest in the agent’s behavior. • An open-source implementation of our methodology to aid transparency and reproducibility, will be freely available upon publication. Taken together, these results demonstrate that our empirical methodology provides reliable, reproducible insights into how feature reliance emerges, evolves, and is shaped by key factors. By offering a systematic approach to saliency maps we enable the extraction of global attention patterns across algorithms, environments, and reward functions, advancing interpretability beyond isolated case studies of individual agents. In addition to making those systems more transparent, our approach acts as a promising diagnostic axis for identifying algorithmic biases, misallocated attention, and overfitting behaviors invisible to performance metrics alone. 2 Related Work In this section, we review existing work on saliency maps and their application to reinforcement learning, first outlining general methods and then discussing prior efforts to explain RL agents, highlighting the lack of systematic and quantitative analyses that motivates our work. Saliency maps Saliency maps are a class of explainable AI methods designed to highlight the input features that most strongly influence a model’s predictions, originally developed to explain predictions of image classifiers in computer vision. They can reveal undesirable effects, such as the Clever Hans effect, where a classifier bases its decision on spurious cues, e.g., a label printed at the bottom of an image rather than the object itself ( 32 ) . They can also uncover harmful biases where networks make unfair associations between concepts ( 13 ) . Saliency map methods can be broadly grouped into three categories: gradient-based 50 ; 51 methods such as Grad and SmoothGrad, perturbation-based 17 ; 42 methods, and Layer-wise Relevance Propagation (LRP) 7 ; 32 . Gradient-based methods estimate the significance of each input pixel on the network’s decision-making process by analyzing the gradient with respect to these pixels. In contrast, perturbation-based methods estimate this influence by individually altering each pixel and observing the resulting effect on the network’s choice of action. Finally, LRP is a backpropagation-based relevance method that iteratively redistributes prediction scores backward through the network using conservation rules, ensuring that total relevance is preserved. This yields saliency maps that are typically sharper and

Read the complete text