Physics-informed Value Learner for Offline Goal-Conditioned Reinforcement Learning
Vittorio Giammarino, Ruiqi Ni, Ahmed H. Qureshi · 2025
Read Physics-informed Value Learner for Offline Goal-Conditioned Reinforcement Learning by Vittorio Giammarino, Ruiqi Ni, Ahmed H. Qureshi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-06782.txt.
Introduction
Robots that learn goals from logged data alone get a physics-based nudge from Vittorio Giammarino and two co-authors in a 2025 reinforcement learning paper. Offline goal-based learning suits steering and movement, where live trial and error is costly or unsafe. Sparse data coverage and long time spans make the learning shaky.
The authors add a regularizer to value learning derived from the Eikonal equation, a formula describing how travel cost grows with distance. Unlike plain penalties used just to steady training, this one comes from continuous-time optimal control and pushes the learned value function to match true cost-to-go geometry. It plugs into current temporal-difference methods. Combined with a hierarchical implicit Q-learning algorithm, the regularized version improved results and reach, with the largest gains when separate data segments must be stitched together and in large steering tasks.
Catalog record
| CREATOR | Vittorio Giammarino, Ruiqi Ni, Ahmed H. Qureshi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-06782
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Physics-informed Value Learner for Offline Goal-Conditioned Reinforcement Learning Vittorio Giammarino, Ruiqi Ni, Ahmed H. Qureshi Abstract Offline Goal-Conditioned Reinforcement Learning (GCRL) holds great promise for domains such as autonomous navigation and locomotion, where collecting interactive data is costly and unsafe. However, it remains challenging in practice due to the need to learn from datasets with limited coverage of the state-action space and to generalize across long-horizon tasks. To improve on these challenges, we propose a Physics-informed (Pi) regularized loss for value learning, derived from the Eikonal Partial Differential Equation (PDE) and which induces a geometric inductive bias in the learned value function. Unlike generic gradient penalties that are primarily used to stabilize training, our formulation is grounded in continuous-time optimal control and encourages value functions to align with cost-to-go structures. The proposed regularizer is broadly compatible with temporal-difference-based value learning and can be integrated into existing Offline GCRL algorithms. When combined with Hierarchical Implicit Q-Learning (HIQL), the resulting method, Physics-informed HIQL (Pi-HIQL), yields significant improvements in both performance and generalization, with pronounced gains in stitching regimes and large-scale navigation tasks. Code is available at link 1 1 1 https://github.com/VittorioGiammarino/PI-HIQL . 1 Introduction In recent years, many of the most effective machine learning paradigms have capitalized on vast amounts of unlabeled or weakly labeled data. Similarly, in dynamic systems learning, Offline GCRL has emerged as a pivotal framework, enabling the use of large-scale, multitask datasets without requiring explicit reward annotations. Specifically, Offline RL [ 1 , 2 ] leverages passively collected trajectories to learn control policies, offering great promise for applications such as autonomous navigation, locomotion, and manipulation, where interactive training is usually costly and unsafe. GCRL [ 3 , 4 ] extends this capability by enabling learning across diverse datasets without explicit rewards. Despite its potential, Offline GCRL faces significant challenges, including accurate Goal-Conditioned Value Function (GCVF) estimation from limited data, policy extraction from imperfect value functions, and generalization to unseen state-goal pairs [ 5 ] . Among these issues, GCVF estimation remains the most fundamental, as improvements in this area can enhance both policy extraction and generalization, ultimately advancing the entire field of Offline GCRL. Pi inductive biases offer a promising direction for improving GCVF estimation in the offline setting. As demonstrated in prior work [ 6 ] , Pi methods can introduce physically or geometrically meaningful structure into the learned value function, enhancing sample efficiency and generalization. In Fig. 1 , we illustrate a representative GCRL task in which an agent must navigate from various starting positions in a maze to a specified goal. Fig. 1 b shows the contour plot of a GCVF learned by a non-Pi state-of-the-art (SOTA) algorithm. The resulting value function fails to robustly encode obstacle constraints, leading to suboptimal policies that often fail to reach the goal. These limitations motivate the use of Pi regularizers as a principled means to incorporate structural priors into value learning, and thus improve GCVF estimation and policy reliability in complex environments. The primary contribution of this work is the introduction of a Pi Eikonal regularizer for GCVF estimation in Offline GCRL tasks. Inspired by the Eikonal PDE [ 7 ] , this regularizer imposes a distance-like cost-to-go structure on the learned GCVF, serving as an effective inductive bias during training. By enforcing this structure, the regularizer improves value estimation accuracy and promotes generalization to unseen states, while also reducing the number of required training steps compared to non-Pi approaches (see Fig. 1 a). In contrast to Hamilton-Jacobi-Bellman (HJB) PDE-based methods [ 6 ] , which require explicit system dynamics and often suffer from numerical instability [ 8 , 9 ] , our method is model-free and easy to implement. Empirically, it outperforms both HJB-regularized and unregularized baselines while adding only minimal computational overhead. To validate the effectiveness of our Eikonal regularizer, we integrate it into the HIQL framework [ 10 ] , a current SOTA algorithm for Offline GCRL. We refer to our method as Pi-HIQL. This choice is motivated by HIQL’s strong baseline performance, making it an ideal candidate for demonstrating the benefits of our approach. In our evaluation, conducted on the challenging OGbench benchmark [ 11 ] , we compare Pi-HIQL against Quasimetric RL (QRL) [ 12 ] , Contrastive RL (CRL) [ 13 ] , and standard HIQL. Pi-HIQL consistently outperforms or matches the performance of HIQL and other baselines across most tasks, achieving SOTA results particularly in large-scale navigation and trajectory stitching scenarios. These gains underscore the utility of the Eikonal regularizer in enhancing GCVF estimation and overall Offline GCRL performance, with limited exceptions in tasks involving complex object interactions. (a) Pi-HIQL (ours) (b) HIQL [ ] Figure 1 : Contour plots of the GCVF for antmaze-giant-navigate-v0 in [ 11 ] , learned after 100,000 training steps by our Pi algorithm Pi-HIQL , and the standard HIQL . The plots are generated by varying the agent’s center of mass x - y coordinates while keeping all other states fixed. Recall that the policy \pi is trained to move the agent in the direction that maximizes the GCVF. The effects of the Eikonal regularizer are evident in Fig. 1 a, where the contour plot closely follows the maze structure, in contrast to Fig. 1 b, where the learned GCVF ignores the maze structure. 2 Related work To the best of our knowledge, Pi regularization for value estimation has only recently been explored. Notably, Lien et al. 2024 propose an Offline RL objective derived from the HJB equation in continuous-time optimal control [ 8 , 9 ] , aiming to enforce first-order derivative consistency within the critic network. In contrast, we introduce a simpler, model-free Pi regularizer for GCVF learning, based on the residual of the Eikonal PDE. We show that this regularizer induces a distance-like structure in the value function and integrates naturally into standard GCRL pipelines. While gradient norm penalties—closely related to the Eikonal PDE residual—have been employed in generative modeling [ 14 ] and, more recently, in model-based RL to regularize Q-functions and mitigate overfitting [ 15 ] , their application in GCRL remains, to our knowledge, unexplored. In contrast to these prior methods, which primarily aim to stabilize training, our regularizer is designed to inject a structural inductive bias into the GCVF, thereby improving sample efficiency and generalization. To the best of our knowledge, this work presents the first use of the Eikonal PDE as a regularization objective in value-based RL, and its first practical deployment in the Offline GCRL setting. More broadly, there has been a growing interest in incorporating physical priors and geometric, distance-like structures into RL algorithms, especially in model-based or Koopman-inspired frameworks [ 16 , 17 ] . These approaches typically rely on known system dynamics or exploit symmetry groups in state space, whereas our method operates directly on the GCVF in a fully model-free setting. In parallel, the use of non-Pi constraints for value learning has been extensively studied in Offline RL. Many approaches constrain learned policies to remain close to the behavior policy, either through explicit density modeling [ 18 , 19 , 20 ] or implicit divergence constraints [ 21 , 22 ] . Others directly regularize the Q-function to assign low values to