Constraint-Enhanced Reinforcement Learning Based on Dynamic Decoupled Spherical Radial Squashing
Qijun Liao, Zhaoxin Yu, Jue Yang, [4pt] School of Mechanical Engineering, University of Science, Technology Beijing · 2026
Read Constraint-Enhanced Reinforcement Learning Based on Dynamic Decoupled Spherical Radial Squashing by Qijun Liao, Zhaoxin Yu, Jue Yang, [4pt] School of Mechanical Engineering, University of Science, Technology Beijing (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-04185.txt.
Introduction
Qijun Liao, Zhaoxin Yu, and Jue Yang address a safety limit for robots that learn by trial and error. Each joint can only change speed or position by a set amount at each step. Those limits differ a lot from joint to joint, so a single round limit wastes much of the space a robot could safely use.
Their method gives each joint its own changing radius, matched to where the robot now stands. In simulated robot tests it kept every step within the hard limits, earned the highest task scores, and needed no extra solver while the robot was running.
Catalog record
| CREATOR | Qijun Liao, Zhaoxin Yu, Jue Yang, [4pt] School of Mechanical Engineering, University of Science, Technology Beijing |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 90 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.04185
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Constraint-Enhanced Reinforcement Learning Based on Dynamic Decoupled Spherical Radial Squashing Qijun Liao, Zhaoxin Yu, Jue Yang, [4pt] School of Mechanical Engineering, University of Science, Technology Beijing Abstract When deploying reinforcement learning policies to physical robots, actuator rate constraints—hard limits on how fast each joint can move per control step—are unavoidable. These limits vary substantially across joints due to differences in motor inertia, power bandwidth, and transmission stiffness, creating pronounced heterogeneity that existing methods fail to handle geometrically: the per-joint feasible region forms a high-dimensional box in action-increment space, yet QP projection and spherical parameterization methods impose isotropic ball-shaped constraints, exponentially under-covering the true feasible set as heterogeneity grows. This paper proposes Dynamic Decoupled Spherical Radial Squashing (DD-SRad), which resolves this mismatch by computing a position-adaptive radius independently for each actuator, achieving tight alignment with the true per-joint feasible region. DD-SRad satisfies per-step hard constraints with probability 1, preserves well-conditioned gradients throughout training, and admits exact policy gradient backpropagation with zero runtime solver overhead. MuJoCo benchmark experiments demonstrate the highest task return at zero constraint violation—matching the unconstrained upper bound—with 30%–50% improvement in constraint-space coverage over spherical baselines. High-fidelity IsaacLab simulations with Unitree H1 and G1 humanoid robots confirm end-to-end optimality parameterized directly from official joint specifications, validating a systematic pathway from hardware datasheets to safe deployment. Code is publicly available at https://anonymous.4open.science/r/DD-SRad . 1 Introduction Reinforcement learning has become a powerful tool for training robotic control policies, enabling legged robots and manipulators to acquire agile behaviors through large-scale simulation. However, deploying these policies on physical hardware requires more than high task rewards: the action sequence produced by the policy must be executable by real actuators at every control step. A fundamental requirement is satisfying actuator rate constraints, namely hard limits on how much each joint’s position command can change per control step, imposed by motor inertia, power amplifier bandwidth, and transmission stiffness. Violating these constraints can cause commands to be silently discarded, trigger overcurrent protection, or even lead to permanent joint damage. Critically, these limits differ substantially across joints: for the Unitree H1 humanoid, hip joints allow 2.2 \times the rate of ankle joints, while industrial six-axis manipulators can exceed a 5:1 proximal-to-distal ratio [ 1 ] . This heterogeneity across actuators is not a design defect, but a structural reality that any RL algorithm targeting real-world deployment must address. Heterogeneous per-step rate constraints impose a structure that existing approaches fail to handle simultaneously across three dimensions. The constraint |a_{t}^{i}-a_{t-1}^{i}|\leq\delta^{i} —where a_{t}^{i} denotes the i -th joint’s position command at step t and \delta^{i}>0 its per-joint rate limit—defines a feasible set \mathcal{F}_{t}=\prod_{i}[a_{t-1}^{i}-\delta^{i},\,a_{t-1}^{i}+\delta^{i}] in action-increment space: an \ell_{\infty} hyperrectangle whose side lengths 2\delta^{i} vary across dimensions and drift with each control step. Yet each major solution family sacrifices a different desideratum: hierarchical MPC/QP methods incur runtime solver overhead and training-deployment inconsistency, precluding end-to-end optimization [ 2 , 3 , 4 , 5 , 6 , 7 ] ; CMDP-based safe RL methods offer only expected-form guarantees that permit transient per-step violations [ 8 , 9 , 10 , 11 , 12 , 13 ] ; and action parameterization methods either impose isotropic \ell_{2} -ball constraints that compress the reachable set under heterogeneous \delta^{i} [ 14 ] , or assume a static feasible set incompatible with the per-step drift of \mathcal{F}_{t} [ 15 , 16 , 17 ] . Under heterogeneous constraints, the reachable-set volume of \ell_{2} spherical parameterization is only (\min_{i}\delta^{i})^{d}/\prod_{i}\delta^{i} of the \ell_{\infty} feasible set—a ratio that degrades exponentially with dimension and heterogeneity. No existing method simultaneously achieves hard per-step constraint satisfaction, exact \ell_{\infty} coverage of \mathcal{F}_{t} , and end-to-end training without runtime solver overhead. Table 1 summarizes this comparison systematically. Table 1 : Comparison of Rate-Constraint Handling Methods. Method Hard Per-Step Guarantee Heterogeneous Rate-Limit Support Exact Box-Shaped Geometry Time-Varying Feasible Set No Runtime Solver Off-Policy End-to-End RL+MPC/QP [ 2 , 3 ] \checkmark Partial ✗ \checkmark ✗ ✗ CMDP (CPO [ 8 ] , FOCOPS [ 10 ] ) ✗ ✗ – – \checkmark \checkmark TN-SAC [ 17 ] \checkmark \checkmark \checkmark ✗ \checkmark \checkmark SRad [ 14 ] \checkmark ✗ ✗ ✗ \checkmark \checkmark BoxPre+ [ 14 ] \checkmark \checkmark ✗ \checkmark ✗ \checkmark FlowPG [ 15 ] / ARAM [ 16 ] \checkmark \checkmark \checkmark ✗ \checkmark ✗ DD-SRad (Ours) \checkmark \checkmark \checkmark \checkmark \checkmark \checkmark Hard Per-Step Guarantee : every control step satisfies the actuator rate limits. Heterogeneous Rate-Limit Support : different joints may have different rate limits. Time-Varying Feasible Set : the valid action region changes with the previous action. To fill this gap, this research proposes Dynamic Decoupled Spherical Radial Squashing (DD-SRad) , a smooth analytic action parameterization that simultaneously achieves hard per-step constraint satisfaction, exact \ell_{\infty} coverage of the feasible set, and end-to-end gradient backpropagation with zero runtime solver overhead. Instead of applying a single global radius, DD-SRad computes a position-adaptive effective radius R_{\text{eff}}^{i} independently for each action dimension, so that the reachable set tightly covers the full \ell_{\infty} feasible box \mathcal{F}_{t} rather than an inscribed sphere. This per-dimension decoupling eliminates the dimensional coupling inherent to \ell_{2} squashing and provably satisfies per-step hard constraints with probability 1. Because the parameterization is fully smooth and analytic, it admits exact policy gradient backpropagation via the chain rule with zero solver overhead, and plugs directly into off-policy backbones such as SAC and TD3 without architectural changes. The contributions of this paper are as follows: • Geometric alignment : DD-SRad achieves exact \ell_{\infty} coverage of the feasible set via per-dimension adaptive radius, recovering volume systematically lost by \ell_{2} baselines. • Theoretical guarantees : This research proves per-step hard constraint satisfaction with probability 1 and establishes Jacobian condition number bounds governed by the per-dimension limits. • End-to-end compatibility : The smooth analytic form supports exact backpropagation with no runtime solver. • Empirical validation : DD-SRad achieves the highest return at zero violation on MuJoCo benchmarks and transfers directly to Unitree H1/G1 locomotion in IsaacLab using official rate specifications. 2 Methodology 2.1 Preliminaries Spherical Radial Squashing (SRad). Kasaura et al. [ 14 ] define the mapping: a=c_{s}+R\cdot\frac{u}{\sqrt{1+\|u\|^{2}}} (1) where u\in\mathbb{R}^{d} is the latent action (network output), c_{s} is the ball center, and R>0 is the global radius. This mapping strictly guarantees \|a-c_{s}\|<R , with \nabla_{u}a nonzero everywhere. However, the global radius R induces dimensional coupling: if any dimension approaches the boundary, R must be reduced globally, restricting all other dimensions’ exploration. In the differential action space, SRad uses R=\min_{i}\delta^{i} , giving an \ell_{2} ball that mismatches the \ell_{\infty} feasible set geometry; no globally valid and tight R exists under heterogeneous \boldsymbol{\delta} . This geometric mismatch motivates DD-SRad’s design: computing R_{\text{eff}}^{i} independently per dimension so that the reachable set aligns exactly with \mathcal{F}_{t} (Theorem 2.7 ), while the smooth per-dimension Jacobian preserves full gradient information under heterogeneous \boldsymbol{\delta} (Proposition 2.5 ). This work addresses reinforcement learning with rate constraints, where the state space is \mathcal{S}\subseteq\mathbb{R}^{n} , the action space is \mathcal{A}=\prod_{i=1}^{d}[a_{\min}^{i},a_{\max}^{i}] , the reward function is r:\mathcal{S}\times\mathcal{A}\to\mathbb{R} , and the transition dynamics are P(s_{t+1}|s_{t},a_{t}) . The rate constraints require: |a_{t}^{i}-a_{t-1}^{i}|\leq\delta^{i},\quad\forall i\in\{1,\ldots,d\},\ \forall t\geq 1 (2) where \boldsymbol{\delta}=[\delta^{1},\ldots,\delta^{d}]\in\mathbb{R}_{++}^{d} , allowing significant heterogeneity ( \delta^{i}\neq\delta^{j} ). State augmentation \tilde{s}_{t}=(s_{t},a_{t-1}) resolves the induced non-Markovianity (Appendix B.1 ). Standard conditions assumed: bounded r ,