Learned Lyapunov Certificates for Safe Residual Reinforcement Learning on Euler–Lagrange Systems

Giansalvo Cirrincione, Adriano Fagiolini · 2026

Read Learned Lyapunov Certificates for Safe Residual Reinforcement Learning on Euler–Lagrange Systems by Giansalvo Cirrincione, Adriano Fagiolini (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-06934.txt.

Introduction

Giansalvo Cirrincione and Adriano Fagiolini combine learning with a classic adaptive controller for mechanical systems such as robot arms. Their design adds three learned parts: a safety measure, a policy adding small torque corrections, and a network estimating missed dynamics. A simple filter checks every output against safety.

The authors prove stability and convergence results for the combined system under stated conditions. On a two-joint arm with friction and changing loads, the learned safety measure drives most of the gain, cutting tracking error notably. A seven-joint robot study shows the full pipeline still converges, while also exposing a warm-start problem worth noting.

Catalog record

CREATORGiansalvo Cirrincione, Adriano Fagiolini
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE96 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.06934

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Learned Lyapunov Certificates for Safe Residual Reinforcement Learning on Euler–Lagrange Systems Giansalvo Cirrincione, Adriano Fagiolini Abstract We augment the Slotine–Li adaptive controller for Euler–Lagrange systems with three learned components: a structured-quadratic Lyapunov function V ψ V_{\psi} whose positive-definiteness follows from a Cholesky parameterization, a residual Soft Actor–Critic policy that adds bounded torque corrections to the analytic baseline, and a physics-informed neural network that estimates unmodeled dynamics. A closed-form safety filter, derived from the single affine constraint V ˙ ψ + α ​ V ψ ≤ 0 \dot{V}_{\psi}+\alpha V_{\psi}\leq 0 , projects every policy output onto the safe set without requiring an online QP solver. We prove: global feasibility of the filter under a drift-decay condition on the control-degeneracy set; exponential stability under exact shielding, with a robust extension whose margin depends on the PINN approximation error; almost-sure convergence of the three-timescale policy–certificate–multiplier updates to a KKT point; and a PAC generalization bound for the certificate over compacts. On a 2-DOF manipulator with nonlinear friction and variable payload, the learned certificate accounts for most of the empirical gain: tracking error drops by 41% on nominal friction and 24% on aggressive friction at the centroid of the training distribution. A 7-DOF scalability study on a Franka Emika Panda confirms clean convergence of the full pipeline at industrial scale, identifies the conditions under which gains over exact model-based baselines should and should not be expected, and documents a warm-start pathology of the learned certificate that has practical implications for deployment. KEYWORDS: safe reinforcement learning, learned Lyapunov certificates, closed-form safety filters, Euler–Lagrange systems, residual adaptive control, quadratic programming, robust stability 1 Introduction The adaptive controller of Slotine and Li 1987 for Euler–Lagrange systems B(q)\ddot{q}+C(q,\dot{q})\dot{q}+G(q)=Y(q,\dot{q},\ddot{q})\,\pi=\tau (1) produces asymptotic tracking via the certainty-equivalence torque \tau=Y\hat{\pi}-K_{d}s , the sliding variable s=\dot{e}+\Lambda e , and the Lyapunov candidate V=\tfrac{1}{2}s^{\top}Bs+\tfrac{1}{2}\tilde{\pi}^{\top}K_{\pi}^{-1}\tilde{\pi} . The guarantees rest on the linear-in-parameters regressor Y\pi and on a priori knowledge of the mass matrix, Coriolis terms, and gravity up to the unknown \pi . In practice, joint elasticity, friction hysteresis, payload-dependent couplings, and soft-body effects resist clean parameterization. The adaptive law cannot exploit historical data or transfer across tasks. This paper replaces the analytic Lyapunov candidate and the fixed adaptation law with learned counterparts while retaining the closed-loop stability guarantees. The approach rests on three components ( Figure 1 ): • A structured-quadratic Lyapunov function V_{\psi}(\mathbf{x})=\mathbf{x}^{\top}(L_{\psi}L_{\psi}^{\top}+\epsilon I)\mathbf{x} , positive-definite by construction, enforced at inference by a closed-form safety filter. • A Soft Actor–Critic policy that outputs a bounded residual torque on top of the Slotine–Li law. • A physics-informed neural network whose approximation error appears explicitly in the stability margin. Contributions. (1) The structured-quadratic parameterization yields a single-constraint QP with closed-form solution, globally feasible under a verifiable drift-decay condition ( Theorem 4 ). (2) Exponential stability holds under exact shielding ( Corollary 6 ); a robust counterpart absorbs the PINN error with margin \|\nabla V_{\psi}\|\cdot\|\hat{h}-h\| ( Theorem 13 ). (3) The joint policy–certificate–multiplier updates converge almost surely to a KKT point ( Theorem 10 ). (4) A PAC generalization bound certifies the learned V_{\psi} over compacts ( Proposition 14 ). (5) An ablation on a 2-DOF manipulator isolates the learned certificate as the dominant source of gain ( +41\% nominal, +24\% aggressive friction). (6) A 7-DOF scalability study confirms clean convergence at industrial scale and identifies certificate warm-start as a failure mode. Learned components Environment \dot{\mathbf{x}}=h(\mathbf{x})+g(\mathbf{x})\tau PINN f_{\theta} \hat{\pi}_{\theta},\;r_{\theta} SAC policy \pi_{\phi} (\phi_{\text{actor}},\phi_{\text{critic}}) Safety shield QP, closed-form V_{\psi}(\mathbf{x})=\mathbf{x}^{\top}P_{\psi}(\mathbf{x})\mathbf{x} P_{\psi}=L_{\psi}L_{\psi}^{\top}+\epsilon I Dual ascent \mu\leftarrow[\mu+\eta_{\mu}(\hat{L}_{\text{lyap}}-d)]_{+} \mathbf{x} \tau_{\text{raw}} \tau^{\star} h_{\theta} (q,\dot{q},\ddot{q},\tau) \nabla V_{\psi} a,b,c \dot{\mathbf{x}}_{\theta} violation \mu Figure 1: Architecture overview. The SAC policy \pi_{\phi} proposes torques projected onto the safe half-space via a closed-form QP derived from the learned Lyapunov function V_{\psi} ; the PINN f_{\theta} supplies the dynamics model. Solid arrows: forward flow; dashed: dual path. The rest of the paper is organized as follows. Section 2 positions the contribution against the literature on neural Lyapunov control, safe reinforcement learning (RL), and PINN-based robot control. Section 3 formalizes the framework. Section 4 states and proves the stability results. Section 7 describes the experimental protocol. Section 9 concludes. 2 Related Work Four research strands are relevant: adaptive control of manipulators, neural Lyapunov methods, safe reinforcement learning, and physics-informed modeling. 2.1 Adaptive control of manipulators The passivity-based adaptive controller of Slotine and Li 1987 exploits the linear-in-parameters property of rigid-body dynamics and the skew-symmetry of \dot{B}-2C to derive an exponentially convergent certainty-equivalence law. Subsequent extensions accommodate friction, flexible joints, and task-space formulations [ 21 , 27 ] . The linear-in-parameters assumption, however, is restrictive: unmodeled nonlinearities such as stick-slip friction, backlash, or payload-dependent couplings require either overparameterized regressors or robustifying terms that sacrifice performance for worst-case guarantees. The proposed framework preserves the Slotine–Li structure as a warm-start (through supervised pretraining of V_{\psi} on the analytic candidate) but allows the learned components to generalize the adaptation mechanism to non-parametric residuals. 2.2 Neural Lyapunov control The idea of learning a Lyapunov function jointly with a controller was popularized by Chang et al. 2019 , who combined a multilayer-perceptron (MLP) candidate with \tanh activation and a falsifier (an SMT-style counterexample generator) to produce verified regions of attraction for nonlinear polynomial systems. Zhou et al. 2022 extended this paradigm to settings where the dynamics are themselves unknown and learned by a neural network, providing closed-loop stability guarantees for the approximated system and a certified transfer to the true dynamics via Lipschitz bounds. Wu et al. 2024 relaxed the global Lyapunov-decrease constraint to a region-of-attraction-specific one, enabling larger verifiable ROAs via empirical falsification with strategic regularization. Dawson et al. 2022a combined neural Lyapunov with barrier certificates for safe nonlinear control, while Richards et al. 2018 learned the Lyapunov function adaptively from data with statistical stability certificates. All of these works share a common trait: the Lyapunov function is parameterized by a generic architecture (tanh-MLP or input-convex neural network, ICNN) and its positive-definiteness is either enforced through training losses or verified a posteriori. In contrast, this paper adopts a structured-quadratic form V_{\psi}(\mathbf{x})=\mathbf{x}^{\top}P_{\psi}(\mathbf{x})\mathbf{x} with P_{\psi} built from a Cholesky factor, which gives positive-definiteness by construction and — critically — ensures that the safety-filter QP is feasible under a verifiable drift-decay condition ( Theorem 4 ). The design choice is motivated not by expressiveness (ICNNs are strictly more expressive) but by the safety-filter viewpoint: feasibility failures in the QP would halt deployment, and empirically-verified feasibility is not sufficient in a safety-critical context. 2.3 Safe and constrained reinforcement learning The Constrained Markov Decision Process (CMDP) framework [ 2 ] and its deep-RL instantiations have grown into a rich literature. Achiam et al. 2017 introduced Constrained Policy Optimization (CPO), a trust-region method with per-update safety guarantees. Lagrangian relaxations, trading off formal guarantees for simplicity, were proposed in TRPO-Lag and PPO-Lag [ 24 ] and refined by proportional–integral–derivative (PID) Lagrangian updates [ 28 ] to damp multiplier oscillations. Tessler et al. 2019 established the actor-critic formulation with learnable penalty; Paternain et al. 2019 proved zero duality gap for constrained RL under mild conditions, legitimizing the primal-dual approach. A parallel line of work treats safety as a hard runtime constraint via control barrier functions (CBFs), originally formalized by Ames et al. 2017 . Cheng et al. 2019 combined model-free RL

Read the complete text