Learning When to Act: Communication-Efficient Reinforcement Learning via Run-Time Assurance
Adam Haroon, Erick J. Rodríguez-Seda, Cody Fleming, Tristan Schuler, [4pt] Department of Mechanical Engineering, Iowa State University, Ames, IA, USA · 2026
Read Learning When to Act: Communication-Efficient Reinforcement Learning via Run-Time Assurance by Adam Haroon, Erick J. Rodríguez-Seda, Cody Fleming, Tristan Schuler, [4pt] Department of Mechanical Engineering, Iowa State University, Ames, IA, USA (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-12561.txt.
Introduction
Haroon and three colleagues teach one policy to decide what to do and when to act, cutting talk in safe learning. A run-time safety layer steps in with a one-step Lyapunov prediction and a precomputed LQR backup. That beats methods that enforce safety only on average. The policy learns control and timing together under a pointwise Lyapunov shield around a known equilibrium. On pendulum, cart-pole and quadrotor tests, it stretches mean sample gaps by roughly two to three and a half times over a Lyapunov-triggered baseline.
Catalog record
| CREATOR | Adam Haroon, Erick J. Rodríguez-Seda, Cody Fleming, Tristan Schuler, [4pt] Department of Mechanical Engineering, Iowa State University, Ames, IA, USA |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 101844 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.12561
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Learning When to Act: Communication-Efficient Reinforcement Learning via Run-Time Assurance Adam Haroon, Erick J. Rodríguez-Seda, Cody Fleming, Tristan Schuler, [4pt] Department of Mechanical Engineering, Iowa State University, Ames, IA, USA Abstract Safe reinforcement learning (RL) typically asks what an agent should do. We ask when it needs to act, and show that a single policy can jointly learn control inputs and communication-efficient timing decisions under a pointwise Lyapunov safety shield. We scope the framework to stabilization around a known equilibrium, where CARE-based LQR backups, Lyapunov certificates, and classical Lyapunov-STC are well defined, enabling a clean comparison against the analytical baseline. A run-time assurance (RTA) layer overrides the policy pointwise via a one-step-ahead Lyapunov prediction and a precomputed LQR backup, providing a strictly stronger guarantee than constrained MDP methods that enforce safety only in expectation. On an inverted pendulum, cart–pole, and planar quadrotor, the learned policy achieves 1.91 × 1.91\times , 1.45 × 1.45\times , and 3.51 × 3.51\times higher mean inter-sample interval (MSI) than a classical Lyapunov-triggered baseline; a fixed LQR controller at the same average rate is unstable on all three plants, showing that adaptive timing, not a lower average rate, is what makes sparsity safe. A CARE-derived Lyapunov reward transfers across environments without redesign, with a single weight w c w_{c} controlling the stability–communication tradeoff; ablations confirm the RTA shield is essential, with its removal reducing MSI by 1.27 1.27 – 1.84 × 1.84\times and degrading state norms. A preference-conditioned extension recovers the full tradeoff frontier from a single model at 2 11 \tfrac{2}{11} of training compute, and SAC experiments confirm the results are algorithm-agnostic across discrete and continuous domains. A 12-state 3D quadrotor case study extends the framework to higher-dimensional systems where classical STC design is analytically intractable: a SAC agent reaches MSI = 0.302 =0.302 s ( 94 % 94\% of τ max \tau_{\max} ) at 0 % 0\% RTA, while classical Lyapunov-STC remains pinned at τ min \tau_{\min} and a fixed-rate LQR controller at the same average interval crashes within two control updates. Robustness to ± 30 % \pm 30\% plant-mass variation and additive disturbances confirms graceful degradation, with the RTA absorbing what the learned policy cannot. 1 Introduction Safe reinforcement learning (RL) has made substantial progress on what an agent should do, but has largely ignored the question of when it needs to act. Fixed-rate control executes at every timestep regardless of whether the state has changed meaningfully; in safety-critical cyber-physical systems, this is expensive, consuming sensing, computation, and bandwidth on every update. Lifting this assumption without sacrificing safety is the central problem we address. The control theory literature studies this as self-triggered control (STC) and event-triggered control (ETC) [ 16 , 20 , 6 , 13 , 30 ] : classical analytical methods determine the next update time from a Lyapunov or model-based triggering rule, providing strong stability guarantees but becoming difficult to apply to nonlinear, underactuated, or high-dimensional dynamics where the linearized one-step prediction leaves significant communication savings on the table. Learning-based approaches relax this conservatism by approaching the admissibility boundary empirically [ 7 , 28 , 11 , 29 , 3 , 24 , 27 ] , but existing formulations either omit formal safety guarantees on the triggering decision [ 7 , 28 ] , rely on model-based learning of the dynamics [ 11 ] , or address an interaction-cost continuous-time setting without a hard timing-safety mechanism [ 27 ] . Our framing differs on multiple axes: we operate in discrete-time STC where \tau_{k} is selected at each decision instant, with both discrete-action (DQN) and continuous-action (SAC) agents; we are model-free with a precomputed Linear-Quadratic Regulator (LQR) backup controller; and we provide pointwise Lyapunov-decreasing safety via a hard RTA override, a formal guarantee that expectation-level constraint methods such as Lagrangian-relaxation [ 5 , 2 ] cannot provide by construction (Section 5.5 ). \begin{overpic}[figures/RTA_Blank.png] \put(16.0,20.0){\pic@makebox@{\makebox}{}(0.0,0.0){Backup}} \put(16.0,15.0){\pic@makebox@{\makebox}{}(0.0,0.0){Controller}} \put(51.0,17.0){\pic@makebox@{\makebox}{}(0.0,0.0){RL Agent}} \put(85.0,56.0){\pic@makebox@{\makebox}{}(0.0,0.0){Environment}} \put(53.0,56.0){\pic@makebox@{\makebox}{}(0.0,0.0){ZOH}} \put(16.0,58.0){\pic@makebox@{\makebox}{}(0.0,0.0){RTA}} \put(86.0,40.0){\footnotesize$\bm{r}(t),\bm{o}(t)$} \put(61.0,57.5){\footnotesize$\bm{u}(t)$} \put(72.0,2.5){\footnotesize$r_{k},\bm{o}_{k}$} \put(36.0,2.0){\footnotesize$\bm{x}_{k}$} \put(36.0,36.0){\footnotesize$\bm{u}_{k},\tau_{k}$} \put(18.0,30.0){\footnotesize$\bm{u}_{k},\tau_{\min}$} \put(37.0,57.0){\footnotesize$\bm{u}_{k}$} \put(68.0,35.0){\footnotesize$\tau_{k}$} \put(11.0,32.0){\rotatebox{90.0}{\tiny$|\hat{q}_{k+1}|>\theta_{\mathrm{RTA}}$}} \end{overpic} Figure 1: Proposed self-triggered control framework. The RL agent outputs a joint action (\bm{u}_{k},\tau_{k}) . The RTA evaluates the one-step-ahead safety predicate |\hat{q}_{k+1}|>\theta_{\mathrm{RTA}} on the linearized prediction of the next state and, if violated, substitutes the LQR backup at \tau_{\min} . The selected input is held by a zero-order hold (ZOH) until the next sampling instant t_{k}+\tau_{k} . We propose a unified RL framework (Fig. 1 ) that jointly selects the control input and the next inter-sample interval, supervised by an RTA layer acting as a safety shield [ 17 , 19 , 25 ] . The RTA uses a precomputed LQR backup and a one-step-ahead safety prediction to override unsafe actions, with each backup intervention provably unsaturated and Lyapunov-decreasing [ 4 , 21 , 10 ] . Episodes are time-bounded (not step-bounded), so a larger MSI directly reduces the number of agent decisions per episode of fixed duration: the operational quantity that communication efficiency targets. The RTA differs fundamentally from constrained MDP methods [ 5 , 2 , 14 , 9 , 12 ] , which enforce safety only in expectation; we demonstrate empirically that a Lagrangian-DQN baseline using the identical constraint predicate achieves 21 – 52\% lower MSI than RL-STC while accumulating up to 35\% hard safety violations at the best checkpoint (Section 5.5 ). The framework is scoped to stabilization around a known equilibrium where CARE, Lyapunov analysis, and classical STC are well defined, enabling clean comparison; we evaluate on three lower-dimensional plants (inverted pendulum, CartPole, planar quadrotor) plus a 12-state 3D quadrotor case study (Section 6 ) where analytical STC design is intractable. Main contributions. (1) A joint RL formulation that simultaneously learns control inputs and inter-sample intervals under a hard pointwise safety certificate (Proposition 1 ); unlike prior shielded RL that enforces safety at fixed time steps, our shield acts on the timing decision \tau_{k} itself. A single weight w_{c} traces the full stability/communication tradeoff frontier. (2) A CARE-derived Lyapunov reward that transfers across SISO and MIMO environments without redesign. (3) Systematic empirical validation on three lower-dimensional plants: 1.45 – 3.51\times MSI gains over Classical Lyapunov-STC, ablations isolating the RTA’s role, a Lagrangian-DQN comparison, a preference-conditioned extension at \tfrac{2}{11} compute, SAC confirming algorithm-agnosticism, and robustness under \pm 30\% mass variation and additive disturbances. (4) A 12-state 3D quadrotor case study showing the framework scales to higher-dimensional MIMO systems where analytical STC design is intractable and where Proposition 1 ’s formal certificate does not extend at the chosen \tau_{\min} , testing the framework’s empirical reach beyond where the certificate applies (MSI =0.302\,\mathrm{s} , 94\% of \tau_{\max} , 0\% RTA; Classical STC pinned at \tau_{\min} ; B2 unstable within two control updates). 2 Problem Formulation 2.1 Evaluation Scope The framework targets stabilization around a known equilibrium. Three structural conditions are required for the safety certificates developed below: (i) a valid local linearization, (ii) a positive-definite quadratic Lyapunov function, and (iii) accuracy of the one-step-ahead prediction in ( 2 ). Stabilization around a known equilibrium guarantees all three: the linearization at the equilibrium is well posed, the CARE solution yields V(x)=x^{\top}Px that is positive-definite about that point, and the prediction is accurate within its neighborhood. Propositions 1 and 2 therefore hold under this scope. 2.2 Self-Triggered Control Consider a