Safe Deployment of Offline Reinforcement Learning via Input Convex Action Correction

Alex Durkin, Jasper Stolte, Matthew Jones, Raghuraman Pitchumani, Bei Li, Christian Michler, Mehmet Mercangöz · 2025

Read Safe Deployment of Offline Reinforcement Learning via Input Convex Action Correction by Alex Durkin, Jasper Stolte, Matthew Jones, Raghuraman Pitchumani, Bei Li, Christian Michler, Mehmet Mercangöz (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-22640.txt.

Introduction

Alex Durkin and six co-authors make offline reinforcement learning safer to deploy in a chemical plant. Offline learning builds a control policy from past plant data, avoiding risky live experiments. The test bed is a stirred-tank reactor running a heat-releasing polymerization, a process that can run away if mishandled. Their 2025 paper builds a simulation of the reactor with its kinetics, energy balance, and limits, in a standard format. Three plant scenarios are covered, starting with startup.

A correction step then bends the learned policy's actions toward safe ones, using a convexity trick that keeps the fix fast and predictable. Policies can thus be bold in simulation yet guarded in the plant. The authors target a real gap: most offline methods ignore deployment safety.

Catalog record

CREATORAlex Durkin, Jasper Stolte, Matthew Jones, Raghuraman Pitchumani, Bei Li, Christian Michler, Mehmet Mercangöz
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE71 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-22640

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Safe Deployment of Offline Reinforcement Learning via Input Convex Action Correction Alex Durkin, Jasper Stolte, Matthew Jones, Raghuraman Pitchumani, Bei Li, Christian Michler, Mehmet Mercangöz Abstract Abstract Offline reinforcement learning (offline RL) offers a promising framework for developing control strategies in chemical process systems using historical data, without the risks or costs of online experimentation. This work investigates the application of offline RL to the safe and efficient control of an exothermic polymerisation continuous stirred-tank reactor. We introduce a Gymnasium-compatible simulation environment that captures the reactor’s nonlinear dynamics, including reaction kinetics, energy balances, and operational constraints. The environment supports three industrially relevant scenarios: startup, grade change down, and grade change up. It also includes reproducible offline datasets generated from proportional-integral controllers with randomised tunings, providing a benchmark for evaluating offline RL algorithms in realistic process control tasks. We assess behaviour cloning and implicit Q-learning as baseline algorithms, highlighting the challenges offline agents face, including steady-state offsets and degraded performance near setpoints. To address these issues, we propose a novel deployment-time safety layer that performs gradient-based action correction using input convex neural networks (PICNNs) as learned cost models. The PICNN enables real-time, differentiable correction of policy actions by descending a convex, state-conditioned cost surface, without requiring retraining or environment interaction. Experimental results show that offline RL, particularly when combined with convex action correction, can outperform traditional control approaches and maintain stability across all scenarios. These findings demonstrate the feasibility of integrating offline RL with interpretable and safety-aware corrections for high-stakes chemical process control, and lay the groundwork for more reliable data-driven automation in industrial systems. Keywords Offline Reinforcement Learning, Process Control, Safety, Convex Neural Networks 1 Introduction The application of reinforcement learning (RL, [ 1 ] ) to industrial process control holds promise for improving efficiency, adaptability, and long-term performance. However, real-world deployment of RL policies in such settings remains rare, particularly in high-stakes domains like chemical processing, power generation, and autonomous manufacturing, due to strict safety and stability requirements [ 2 ] . This concern is heightened in the context of offline RL, where policies are trained entirely from historical data with no opportunity for online trial-and-error [ 3 ] . While this mitigates the risk of unsafe exploration during training, it also introduces the challenge of verifying and correcting policy behaviour at deployment time, when unseen disturbances or distributional shifts can lead to failure. In industrial systems, a poorly chosen control action can have serious consequences, including damaging equipment, violating regulatory constraints, or causing cascading system instability. Offline-trained RL agents, optimised for historical performance, may produce unsafe or unstable actions when operating in regions underrepresented in the training data. Therefore, ensuring safety at deployment becomes a critical bottleneck in applying RL to real-world process control. Polymerisation grade transitions are critical but challenging operations in continuous reactors, requiring changes in feed composition, temperature, and other conditions to switch from one polymer product to another [ 4 ] . These transitions are nonlinear, constrained, and economically sensitive: off-specification material generated during the switch is wasteful, while long transition times reduce throughput [ 5 , 6 ] . Managing grade changes efficiently demands controllers that balance safety, product quality, and operational speed—making them an ideal testbed for intelligent control methods such as RL. Control theory offers powerful tools to address this challenge. In particular, the concepts of asymptotic stability and regions of attraction are foundational in guaranteeing that a control policy not only keeps the system bounded but also drives it to a desired operating regime [ 7 ] . These properties are often certified using Lyapunov functions, which provide a scalar energy-like measure that decreases over time under the system dynamics, ensuring stability and safe recoverability from perturbations [ 8 ] . Inspired by this perspective, our approach introduces a deployment-time safety mechanism for offline RL agents using partially input convex neural networks (PICNNs, [ 9 ] ) as learned cost models. These models are constructed to be convex in the control actions and expressive over the state, enabling efficient, gradient-based correction of unsafe actions. When the offline RL policy proposes an action that may compromise stability or violate safety constraints, the PICNN model provides a locally optimal, safe correction by descending its convex cost surface. Although not explicitly trained as Lyapunov functions, the structure of these models ensures they behave like a learned control energy landscape that discourages unsafe actions and guides the system toward safe operating regions. This architecture is particularly well-suited for process control applications, where control inputs must be interpretable, verifiable, and responsive to safety constraints. Our method acts as a safety layer over arbitrary offline policies, enabling stable and conservative control corrections in real time without requiring modifications to the underlying policy or additional interaction with the environment. This paper proceeds with a summary of the related work in literature followed by a summary of our contributions. Section 2 provides some background on offline RL and PICNNs for safe control. Section 3 describes the benchmark environment and datasets used for evaluation. Section 4 details the offline training of RL agents, including behaviour cloning and implicit Q-learning as well as the PICNN cost model. Section 5 outlines our novel online action correction algorithm. Section 6 presents the experimental results, comparing the performance of offline RL agents with and without PICNN-based action correction. Finally, Section 7 concludes with a discussion of the findings and future work. 1.1 Related Work A range of methods has been proposed to incorporate safety into RL, especially in the context of continuous control. A prominent direction is the use of Lyapunov-based techniques to constrain policy updates or verify system stability. For example, [ 10 ] introduced a model-based RL framework that leverages Lyapunov functions to guarantee safe learning. Their method models uncertainty in the dynamics using Gaussian processes to estimate dynamics and enforce stability through Lyapunov constraints. Other efforts apply control barrier functions (CBFs) to ensure constraint satisfaction through quadratic programming [ 11 ] , and some integrate CBFs with control Lyapunov functions to achieve safety and stability simultaneously [ 12 ] . These techniques offer strong guarantees but often assume access to accurate dynamics models and are primarily applied during training or online control. Another branch of safe RL research focuses on constrained policy optimisation. Constrained policy optimisation [ 13 ] ensures monotonic policy improvement while respecting safety constraints in expectation. Other works adopt primal-dual or Lagrangian methods to dynamically balance reward maximisation and constraint enforcement [ 14 ] . In the offline RL setting, safety becomes even more critical due to the absence of exploration. Conservative Q-learning [ 15 ] , implicit Q-learning [ 16 ] , and model-based offline policy optimisation [ 17 ] address distributional shift and unsafe extrapolation by constraining policies to remain close to the behaviour policy or trusted regions of the dataset. While these methods reduce the risk of catastrophic actions during inference, they lack mechanisms to actively correct unsafe outputs at deployment. Other approaches learn explicit safety critics or constraint models. For example, Lyapunov-based actor-critic methods [ 18 ] train neural approximations of Lyapunov functions to evaluate and constrain policy updates. Risk-sensitive methods model quantiles or variance of returns to encode safety preferences [ 19

Read the complete text