A Loss Landscape Visualization Framework for Interpreting Reinforcement Learning: An ADHDP Case Study Journal: Acta Astronautica

Jingyi Liu, Jian Guo, Eberhard Gill · 2026

Read A Loss Landscape Visualization Framework for Interpreting Reinforcement Learning: An ADHDP Case Study Journal: Acta Astronautica by Jingyi Liu, Jian Guo, Eberhard Gill (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-14600.txt.

Introduction

Liu, Guo and Gill extend their critic landscape method into a full framework for reinforcement learning. A case study on ADHDP spacecraft attitude control anchors the work. Four views make up the framework. One rebuilds the critic match loss surface in 3D. One maps actor loss with the critic frozen. One tracks time, Bellman error and policy weights along training. One maps TD error across states. Variant comparisons show how stabilizers and target updates reshape learning stability.

Catalog record

CREATORJingyi Liu, Jian Guo, Eberhard Gill
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50402
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.14600

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Loss Landscape Visualization Framework for Interpreting Reinforcement Learning: An ADHDP Case Study Journal: Acta Astronautica Jingyi Liu, Jian Guo, Eberhard Gill Abstract Reinforcement learning algorithms have been widely used in dynamic and control systems. However, interpreting their internal learning behavior remains a challenge. In the authors’ previous work, a critic match loss landscape visualization method was proposed to study critic training. This study extends that method into a framework which provides a multi-perspective view of the learning dynamics, clarifying how value estimation, policy optimization, and temporal-difference (TD) signals interact during training. The proposed framework includes four complementary components: a three-dimensional reconstruction of the critic match loss surface that shows how TD targets shape the optimization geometry; an actor loss landscape under a frozen critic that reveals how the policy exploits that geometry; a trajectory combining time, Bellman error, and policy weights that indicates how updates move across the surface; and a state–TD map that identifies the state regions that drive those updates. The Action-Dependent Heuristic Dynamic Programming (ADHDP) algorithm for spacecraft attitude control is used as a case study. The framework is applied to compare several ADHDP variants and shows how training stabilizers and target updates change the optimization landscape and affect learning stability. Therefore, the proposed framework provides a systematic and interpretable tool for analyzing reinforcement learning behavior across algorithmic designs. 1 Introduction Reinforcement learning (RL) algorithms have attracted researchers’ attention in recent years. They have also been applied in dynamics and control problems. Reinforcement learning control algorithms can be used in systems with uncertainties. For example, in an Active Debris Removal (ADR) mission, post-capture attitude control of the combined spacecraft encounters the challenges of system uncertainties, such as uncertain inertial parameters. In this case, reinforcement learning algorithms [ 9 , 17 , 13 ] , which rely on the sampled data during the control process, rather than the model information, have also been used. Although reinforcement learning has achieved success in various control tasks, it doesn’t always perform as expected. RL algorithms are optimized within a training environment, which is a simulation of a specific physical system. The learned policies for one RL algorithm may not transfer effectively to new or slightly different systems [ 11 ] . In some cases, even a minor change in a system parameter can cause a previously well-tuned algorithm to fail [ 12 ] . RL’s performance sensitivity to the training environment motivates the need to interpret the algorithm’s performance. Understanding RL’s training behavior is important for revealing the learning mechanism and thus improving performance robustness. Recent studies have explored various approaches to interpret reinforcement learning behavior in control systems. For instance, some works visualize the state–action value function to reveal the learned control patterns [ 3 ] , while others extract symbolic representations of the policy [ 4 ] . Methods such as sensitivity analysis, Jacobian-based visualization, and Koopman operator approximation [ 14 ] have also been introduced to relate RL policies to interpretable dynamical structures. In the authors’ previous work, a visualization method that maps the critic match loss landscape was proposed to analyze the critic behavior during the training of actor–critic reinforcement learning algorithms. However, a single loss landscape alone cannot fully capture the interactions among the modules in the RL agent. Therefore, a more comprehensive and structured visualization framework is needed to support the interpretation of training dynamics and performance variations across reinforcement learning algorithms. To help demonstrate the interpretation framework, one RL algorithm should be chosen as the sample. The Adaptive Dynamic Programming (ADP) method was proposed to obtain nearly optimal control strategies based on reinforcement learning (RL) and actor-critic (AC) architecture, which has received considerable attention in the past decades [ 7 , 10 , 19 , 16 , 20 ] . Among the ADP method, action-dependent heuristic dynamic programming (ADHDP), a form of Q-learning, has been been applied in post-capture control for the combined spacecraft [ 18 , 2 ] . A notable characteristic of ADHDP is that it maintains a simpler structure, which allows faster training and adaptation in online implementation. On this basis, ADHDP is considered here to evaluate the potential of online reinforcement learning methods for attitude control of rigid combined spacecraft with unknown inertia. The ADHDP algorithm has shown its usage in the work mentioned above. Yet in practice, training instablities are often observed. Standard scalar metrics such as reward or TD error do not reveal exactly why training fails. These methods don’t give enough information on the training information of the algorithm. Furthermore, when the ADHDP algorithm is equipped with deep learning techniques, such as a target neural network and different loss calculation methods, the underlying training procedure is not carefully studied. As a result, the ADHDP algorithm is used as a sample to explore how an interpretation framework can be applied to interpret the training process and control behavior of the ADHDP algorithm. In this work, a visualization framework is proposed to interpret the training behavior of actor–critic reinforcement learning algorithms. The framework reshapes the critic match loss using the one-step temporal difference (TD) error, enabling the observation of critic convergence trends throughout training. To analyze the actor’s adaptation process, an actor loss landscape is constructed to illustrate how policy updates respond to the critic’s evaluation. A time–TD–actor weight plot is introduced to capture how the policy evolves with the change of Bellman error, while a state–TD plot identifies the state regions that contribute most to TD fluctuations. Together, these visualizations form an integrated framework for diagnosing the interaction between actor, critic, and TD error during reinforcement learning training. ADHDP variants with different deep learning techniques are selected as samples for the demonstration of the interpretation framework. In this work, Section 4.2 introduces the visualization framework and the ADHDP variants. Section 4.3 presents the visualization results and analysis for ADHDP variants’ performance using landscape and trajectory analyses. Section 4.4 is the conclusion. 2 Method In this section, the visualization framework for interpreting RL algorithms is introduced. The ADHDP algorithm is also given, which is used as an object to be interpreted using the visualization framework. 2.1 Visualizing the loss function In reinforcement learning, neural networks are employed to approximate functions, using a large number of parameters. The loss landscape, describing how the loss changes with respect to these parameters, provides insights into characteristics such as flatness, sharpness, local minima, and saddle points. Yet, the high dimensionality of the parameter space renders direct visualization intractable. To overcome this issue, the Contour Plots and Random Directions method has been introduced [ 8 ] . This approach selects two directions, \bm{\delta} and \bm{\eta} , and evaluates the loss across the 2-D plane they span. The resulting visualization corresponds to a projected slice of the overall loss landscape. The loss on this plane is defined as f(\alpha,\beta)=\mathcal{L}(\bm{\zeta}^{*}+\alpha\bm{\delta}+\beta\bm{\eta}), (1) where \bm{\zeta}^{*} denotes the reference point in the parameter space, and \alpha and \beta are the step sizes along the two selected directions. The function \mathcal{L} represents the loss value evaluated at each displaced position. This formulation transforms the high-dimensional optimization landscape into a three-dimensional surface, where \bm{\delta} and \bm{\eta} define the two axes, (\alpha,\beta) are the coordinates in this plane, and f(\alpha,\beta) represents

Read the complete text