Assuring the Safety of Reinforcement Learning Components: AMLAS-RL

Calum Corrie Imrie 1 , Ioannis Stefanakos 1 , Sepeedeh Shahbeigi 1 , Richard Hawkins 1 , and Simon Burton 1 · 2025

Read Assuring the Safety of Reinforcement Learning Components: AMLAS-RL by Calum Corrie Imrie 1 , Ioannis Stefanakos 1 , Sepeedeh Shahbeigi 1 , Richard Hawkins 1 , and Simon Burton 1 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-08848.txt.

Introduction

Calum Corrie Imrie and co-authors adapt a safety assurance method so it can vouch for learning systems that act by trial and error, in a 2025 paper. Reinforcement learning suits robots and vehicles whose surroundings defy tidy models, but safety-critical use demands proof, not just good averages. Existing safe-learning tricks add constraints during training, yet they do not build a structured safety case across the system's whole life.

Their adaptation, AMLAS-RL, extends a supervised-learning assurance method into an iterative cycle for reinforcement learning. A running example, a wheeled vehicle driving to a goal without collisions, shows the steps. The framework generates assurance arguments stage by stage, so gaps surface early. The authors aim it at engineers who must convince regulators, not just themselves, that a learning component is safe.

Catalog record

CREATORCalum Corrie Imrie 1 , Ioannis Stefanakos 1 , Sepeedeh Shahbeigi 1 , Richard Hawkins 1 , and Simon Burton 1
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-08848

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Assuring the Safety of Reinforcement Learning Components: AMLAS-RL Calum Corrie Imrie 1 , Ioannis Stefanakos 1 , Sepeedeh Shahbeigi 1 , Richard Hawkins 1 , and Simon Burton 1 Abstract The rapid advancement of machine learning (ML) has led to its increasing integration into cyber-physical systems (CPS) across diverse domains. While CPS offer powerful capabilities, incorporating ML components introduces significant safety and assurance challenges. Among ML techniques, reinforcement learning (RL) is particularly suited for CPS due to its capacity to handle complex, dynamic environments where explicit models of interaction between system and environment are unavailable or difficult to construct. However, in safety-critical applications, this learning process must not only be effective but demonstrably safe. Safe-RL methods aim to address this by incorporating safety constraints during learning, yet they fall short in providing systematic assurance across the RL lifecycle. The AMLAS methodology offers structured guidance for assuring the safety of supervised learning components, but it does not directly apply to the unique challenges posed by RL. In this paper, we adapt AMLAS to provide a framework for generating assurance arguments for an RL-enabled system through an iterative process; AMLAS-RL . We demonstrate AMLAS-RL using a running example of a wheeled vehicle tasked with reaching a target goal without collision. I Introduction Reinforcement learning (RL) is a machine learning (ML) technique focused on training agents to generate optimal policies for interacting with their operating environment [ 1 ] . Inspired by behavioural psychology, RL mimics the human trial-and-error process. An agent takes actions within an environment, observes the resulting outcomes, and learns from feedback in the form of a reward signal. Over time, the agent discovers optimal strategies to achieve long-term objectives. RL excels in complex, uncertain, or changing environments, often where human decision-making becomes infeasible due to high dimensionality or delayed consequences. This makes it especially valuable in cyber-physical systems (CPS), where intelligent decision-making must be integrated with real-time control of physical processes. Applications in domains such as smart energy systems [ 2 ] and autonomous driving [ 3 ] can benefit from RL’s capacity for self-directed learning, adaptability, and long-term optimisation. While traditional ML models often require extensive annotation, RL agents learn by interacting with simulated or real environments, reducing manual effort [ 4 ] . Furthermore, RL’s ability to integrate human feedback, when available, makes it flexible for incorporating domain expertise. Despite its strengths, RL also presents challenges that limit its widespread deployment in real-world CPS. A major limitation is the high cost or risk associated with real-world experimentation, including domains such as robotics, healthcare, or autonomous driving, trial-and-error learning can result in unsafe behaviours. This makes simulation-based training a critical component in RL research, allowing agents to learn in virtual environments before being deployed in the real-world [ 5 ] . Another significant challenge lies in the interpretability and transparency of RL agents. Many RL algorithms, particularly deep-RL, learn complex, non-linear policies encoded in neural networks, making it difficult to understand or justify specific decisions. This “black-box” nature raises concerns in safety-critical applications and can undermine trust in deployed systems [ 6 ] . As a result, developing explainable RL remains an active research area focused on producing models whose decision-making processes are more transparent and understandable to humans. To address the assurance challenges unique to RL, we adapt the Assurance of Machine Learning in Autonomous Systems ( AMLAS ) framework [ 7 ] for RL components. This adaptation, referred to as AMLAS-RL , introduces a structured, iterative methodology for generating assurance arguments tailored specifically to RL-enabled components. AMLAS-RL accounts for distinctive RL characteristics, such as policy optimisation, state-action feedback loops, and reward-driven behaviour, across six defined stages spanning from safety scoping to model deployment. This paper presents three key contributions. First, we extend the AMLAS methodology to support RL applications by defining AMLAS-RL as an assurance framework that integrates seamlessly into RL development lifecycles. Second, we detail each stage of AMLAS-RL , showing how core assurance activities, are adapted to reflect RL’s interactive and dynamic nature. Finally, we demonstrate the practical applicability of AMLAS-RL through a case study involving a simulated autonomous vehicle tasked with reaching a goal while satisfying key safety requirements: avoiding energy depletion, limiting time in unsafe zones, and preventing collisions with obstacles. II Background II-A Reinforcement Learning RL problems are commonly modelled using Markov Decision Processes (MDPs) [ 8 ] , which provide a mathematical framework for sequential decision-making under uncertainty. This section formalises the MDP model and introduces key theoretical constructs that underpin the learning and optimisation processes in RL. The behaviour of an MDP is governed by a policy , which resolves nondeterminism by selecting an action to execute in each state. Policies can be classified according to the information they use: a memoryless (or Markov ) policy bases its decision solely on the current state, a finite-memory policy uses a bounded portion of past observations, and an infinite-memory policy may consider the entire history of interactions. A policy \pi defines the agent’s behaviour as a mapping from states to probability distributions over actions. In the memoryless case, this is represented as \pi:S\rightarrow\mathsf{Dist}(A) , where \mathsf{Dist}(A) denotes the set of probability distributions over the action space A . The performance of a policy is typically evaluated using value functions . The state-value function V^{\pi}(s) represents the expected return when starting from state s and following policy \pi , while the action-value function Q^{\pi}(s,a) gives the expected return after taking action a in state s and subsequently following \pi . The goal in RL is often to compute an optimal policy \pi^{*} that maximises the expected cumulative discounted reward over time. While standard RL aims to maximise expected return, it does not inherently account for the safety of intermediate decisions or the potential risks associated with exploration. Safe-RL [ 9 ] addresses this limitation by integrating safety constraints into the learning process. These constraints are often formalised using cost functions, risk metrics, or probabilistic guarantees, and the objective becomes to learn a policy that not only performs well but also satisfies specified safety criteria. A common approach is to augment the MDP framework into a constrained Markov Decision process (CMDP), where the agent must optimise its return while ensuring that expected cumulative costs remain below a predefined threshold. Alternative formulations involve risk-sensitive objectives, such as minimising the variance of returns or bounding the probability of catastrophic failures. Safe-RL is particularly critical in domains where unsafe actions can have serious consequences, such as autonomous driving [ 10 ] , healthcare [ 11 ] , and robotics [ 12 ] . It often relies on techniques like reward shaping, constrained optimisation, shielding, or simulation-based pretraining to balance exploration with risk mitigation. II-B Safety Assurance of Machine Learning Conventional functional safety standards have traditionally failed to address the unique characteristics of ML-based software. Salay et al. [ 13 ] analysed ISO 26262—the functional safety standard for road vehicles—and proposed adaptations to better support the integration of ML technologies. Burton et al. [ 14 ] investigated the challenges in assuring the safety of ML-based highly automated driving systems, proposing a contract-based framework to demonstrate the fulfilment of safety-related guarantees under defined assumptions. There are even approaches

Read the complete text