Physics-Informed Policy Optimization via Analytic Dynamics Regularization
Namai Chandra, Liu Mohan, Zhihao Gu, Lin Wang † · 2026
Read Physics-Informed Policy Optimization via Analytic Dynamics Regularization by Namai Chandra, Liu Mohan, Zhihao Gu, Lin Wang † (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-14469.txt.
Introduction
Chandra and co-authors add physics rules to reinforcement learning for robot control. Actor-critic methods learn well but need many samples and can break physics. Their PIPER framework adds a soft physics term to the policy loss. The term is a differentiable Lagrangian residual taken from the robot simulator file. It nudges updates toward motions that fit the dynamics. No simulator or core algorithm changes are needed. Tests show gains in learning speed, stability and control accuracy, the authors report.
Catalog record
| CREATOR | Namai Chandra, Liu Mohan, Zhihao Gu, Lin Wang † |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37033 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.14469
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Physics-Informed Policy Optimization via Analytic Dynamics Regularization Namai Chandra, Liu Mohan, Zhihao Gu, Lin Wang † Abstract Reinforcement learning (RL) has achieved strong performance in robotic control; however, state-of-the-art policy learning methods, such as actor-critic methods, still suffer from high sample complexity and often produce physically inconsistent actions. This limitation stems from neural policies implicitly rediscovering complex physics from data alone, despite accurate dynamics models being readily available in simulators. In this paper, we introduce a novel physics-informed RL framework, called PIPER , that seamlessly integrates physical constraints directly into neural policy optimization with analytical soft physics constraints . At the core of our method is the integration of a differentiable Lagrangian residual as a regularization term within the actor’s objective. This residual, extracted from a robot’s simulator description, subtly biases policy updates towards dynamically consistent solutions. Crucially, this physics integration is realized through an additional loss term during policy optimization, requiring no alterations to existing simulators or core RL algorithms . Extensive experiments demonstrate that our method significantly improves learning efficiency, stability, and control accuracy, establishing a new paradigm for efficient and physically consistent robotic control. 1 Introduction Reinforcement learning (RL) has achieved strong performance in robotic control, establishing itself as a dominant paradigm for solving complex manipulation and locomotion tasks. State-of-the-art actor-critic algorithms, most notably Proximal Policy Optimization (PPO) ( 23 ) , Soft Actor-Critic (SAC) ( 11 ) , and Twin Delayed Deep Deterministic Policy Gradient (TD3) ( 8 ) , have demonstrated the ability to learn sophisticated behaviors directly from high-dimensional sensor observations. However, despite these empirical successes, data-driven RL still contends with high sample complexity and often produces physically inconsistent actions. This persistent limitation raises a critical research question: How can we bridge the gap between data-driven policy learning and the well-defined physical laws that govern robotic systems? Figure 1 : Comparison of architecture and performance between SAC and Physical-Informed SAC . (a) The standard SAC framework relying solely on the MuJoCo simulator, resulting in High Error in the robotic arm task. (b) The Physical-Informed SAC framework which incorporates an Analytic Physics & Constraints module, leading to Low Error and precise control. This limitation stems from neural policies implicitly rediscovering complex physics from data alone, despite accurate dynamics models being readily available in simulators. Standard model-free RL treats the environment as a ”black box”, mapping states to rewards without leveraging the underlying mechanical structure. This tabula rasa approach forces the agent to spend millions of timesteps relearning basic concepts such as inertia, gravity, and momentum conservation via trial-and-error. As illustrated in Fig. 1 , policies learned in this manner often exhibit high-frequency oscillation or ”jitter”, as the neural network exploits simulator integration errors rather than learning smooth, energy-efficient motion. While a modern physics engine like MuJoCo ( 26 ) explicitly encodes Euler-Lagrange dynamics ( M,C,G ), standard algorithms discard this rich information during optimization. Figure 2 : Overview of the physics-informed actor–critic reinforcement learning framework. The actor policy generates control actions based on the current system state, while the critic network evaluates action quality by estimating the value function. A PINN, coupled with a high-fidelity simulator (MuJoCo), embeds prior physical laws and system dynamics into the learning loop, providing physically consistent state transitions and constraints. The interaction between data-driven policy optimization and physics-based modeling improves training stability, sample efficiency, and generalization performance compared to purely data-driven reinforcement learning approaches. In this paper, we introduce PIPER (Physics-Informed Policy Optimization with Analytical Dynamic Regularization), a novel framework that seamlessly integrates physical constraints directly into neural policy optimization. By ”analytical,” we refer to the explicit utilization of the simulator’s internal physics dynamics file to derive governing constraints, employing them as soft physics constraints to ground the policy in the system’s true mechanics. Our key idea is to move beyond the black-box assumption by providing the policy with a dense, structure-aware gradient that complements the sparse reward . This formulation is inspired by recent advances in physics-informed learning ( 21 ; 19 ) and structured architectures ( 10 ) , but addresses the open challenge of integrating such priors into model-free policy gradients without requiring rigid safety layers ( 4 ) or handcrafted controllers ( 14 ) . At the core of our method is the integration of a differentiable Lagrangian residual as a regularization term within the actor’s objective. By leveraging the exact Euler-Lagrange terms ( M,C,G ) extracted from the robot’s simulator description at runtime, we construct a residual vector that quantifies the precise violation of the equations of motion for any given action. This residual subtly biases policy updates towards dynamically consistent solutions. Crucially, this physics integration is realized through an additional loss term during policy optimization, requiring no alterations to existing simulators or core RL algorithms . This mechanism acts as a differentiable ”physics coach,” guiding the policy search manifold toward feasible regions without enforcing hard constraints that could hinder exploration. Extensive experiments demonstrate that our method significantly improves learning efficiency, stability, and control accuracy. We validate our approach on a curriculum of robotic tasks ranging from kinematic reaching to complex object manipulation. Our results show substantial gains across all metrics: success rates improve by over 7% in contact-rich tasks (FetchSlide), final precision improves by up to 79.5% (PIPER-TD3), learning stability increases by 42% (PickAndPlace), and sample efficiency improves by over 45% (PIPER-SAC). By grounding the policy search in analytic mechanics, PIPER establishes a new paradigm for efficient and physically consistent robotic control. In summary, our contributions are as follows: • Physics-Informed Policy Optimization with Explicit Residuals (PIPER): We introduce a framework that regularizes reinforcement learning using analytic dynamics. By minimizing a differentiable Lagrangian residual, we enforce physical consistency directly within the policy optimization landscape. • Plug-and-Play Dynamics Extraction: We propose an “Dynamics Oracle” that extracts exact inertial properties ( M,C,G ) from MuJoCo XML descriptors at runtime, enabling physics-aware learning on arbitrary kinematic chains without manual derivation. • Algorithm-Agnostic Applicability: We demonstrate that PIPER is compatible with a wide range of algorithms, including on-policy (PPO), off-policy (TD3), and distributional methods (SAC, TQC), without modifying their core update rules. • Robustness in Contact-Rich Regimes: We validate our approach on a curriculum of robotic tasks ranging from reaching to complex object manipulation (Push, Slide). Our results show that physics regularization significantly improves sample efficiency and asymptotic precision, outperforming state-of-the-art baselines in sparse-reward settings. 2 Related Works Model-Free Actor–Critic Reinforcement Learning. Model-free RL has become the dominant paradigm for continuous control in robotics. Early actor–critic algorithms such as Deep Deterministic Policy Gradient (DDPG) ( 17 ) enabled learning of continuous control policies directly from high-dimensional observations, but suffered from instability and overestimation bias. Proximal Policy Optimization (PPO) ( 23 ) stabilized on-policy updates through a clipped surrogate objective, making it one of the most widely adopted algorithms in robotics and simulation. Twin Delayed Deep Deterministic Policy Gradient (TD3) ( 8 ) reduced overestimation errors in off-policy learning by employing twin Q-networks and delayed policy updates. Soft Actor–Critic (SAC) ( 11 ) further improved sample efficiency and exploration by incorporating entropy regularization into the actor–critic framework. Despite their empirical success, these model-free algorithms treat the environment as a black box and ignore known physical structures of robotic systems.