Accelerating Visual-Policy Learning through Parallel Differentiable Simulation

Haoxiang You, Yilang Liu, Ian Abraham · 2025

Read Accelerating Visual-Policy Learning through Parallel Differentiable Simulation by Haoxiang You, Yilang Liu, Ian Abraham (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-10646.txt.

Introduction

Haoxiang You, Yilang Liu, and Ian Abraham speed up robot policy learning from camera views in 2025. Their method pairs a physics simulator that supports gradient flow with direct gradient steps on the policy. Image rendering is split off from the gradient path, which drops cost and smooths the update size.

On standard visual control tasks run on fast GPU simulators, training time fell a lot against rival methods. Final scores beat all baselines in the tests. On a humanoid walk task the method gained four times the return and learned a run policy in four hours on one GPU.

Catalog record

CREATORHaoxiang You, Yilang Liu, Ian Abraham
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE60 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-10646

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Accelerating Visual-Policy Learning through Parallel Differentiable Simulation Haoxiang You, Yilang Liu, Ian Abraham Abstract In this work, we propose a computationally efficient algorithm for visual policy learning that leverages differentiable simulation and first-order analytical policy gradients. Our approach decouple the rendering process from the computation graph, enabling seamless integration with existing differentiable simulation ecosystems without the need for specialized differentiable rendering software. This decoupling not only reduces computational and memory overhead but also effectively attenuates the policy gradient norm, leading to more stable and smoother optimization. We evaluate our method on standard visual control benchmarks using modern GPU-accelerated simulation. Experiments show that our approach significantly reduces wall-clock training time and consistently outperforms all baseline methods in terms of final returns. Notably, on complex tasks such as humanoid locomotion, our method achieves a 4 × 4\times improvement in final return, and successfully learns a humanoid running policy within 4 hours on a single GPU. Videos and code are available on https://haoxiangyou.github.io/Dva_website/ 1 Introduction Learning to control robots from visual inputs is a key challenge in robotics, with the potential to enable a wide range of real-world applications, ranging from autonomous driving and home service robots to industrial automation. Most methods for learning visual policies fall into two categories: imitation learning and reinforcement learning (RL). Imitation learning trains policies by mimicking expert demonstrations, which are typically collected via human operation ( Bojarski et al. 2016 ; Kendall et al. 2019 ) or teleoperation systems ( Chi et al. 2024 ; Black et al. 2024 ) . When expert demonstrations are scarce or difficult to obtain, visual policies can instead be learned through RL ( Mnih et al. 2015 ; Hafner et al. 2019 ) . However, RL methods typically require long training times and substantial computational resources, such as large-scale GPU clusters, to achieve effective control. Recent advances in differentiable simulation have enabled alternative policy optimization methods, known as analytical policy gradients (APG) ( Freeman et al. 2021 ; Xu et al. 2021 ; Schwarke et al. 2024 ) . These methods achieve much higher computational efficiency by replacing zeroth-order gradient estimates with first-order gradients. However, extending APG methods to visual control remains challenging: obtaining high-quality differentiable rendering is non-trivial, and computing Jacobians over pixel inputs is both memory- and computationally inefficient. Consequently, existing approaches either train separate differentiable renderers ( Wiedemann et al. 2023 ; Liu et al. 2024 ) or rely on low-dimensional visual features, which demand considerable engineering effort ( Heeg et al. 2024 ; Luo et al. 2024 ) . In this work, we propose Decoupled Visual-Based Analytical Policy Gradient (D.Va), a novel method for learning visual policies using differentiable simulation. The core idea is to decouple visual observations from the computation graph, eliminating the need to differentiate through the rendering process. We find that this decoupling not only improves memory and computational efficiency by avoiding Jacobian computations over pixel space, but also normalizes the policy gradient, making visual policy learning more stable. We further provide a formal analysis of our new computation graph, demonstrating that the proposed decoupling policy gradient can be interpreted as a form of policy distillation from open-loop trajectory optimization. This reveals a fundamental connection between open-loop trajectory optimization and closed-loop policy learning. Finally, we benchmark a diverse set of visual policy learning methods using a GPU-accelerated simulation platform that supports parallelized physics and rendering, providing a robust and scalable testbed for evaluation. Our comparisons include the proposed D.Va, two model-free RL algorithms ( Laskin et al. 2020a ; Yarats et al. 2021 ) , the model-based RL method DreamerV3 ( Hafner et al. 2023 ) , an analytical policy gradient method with differentiable rendering, and state-to-visual distillation ( Mu et al. 2025 ) —a two-stage framework that first trains a state-based policy and then distills it into a visual policy through imitation learning. Experiments highlight D.Va’s superior computational efficiency across a wide range of control tasks. In summary, our contributions are: (a) Proposing D.Va, a computationally efficient method for visual policy learning; (b) Benchmarking diverse visual policy learning approaches on state-of-the-art simulation platforms; (c) Providing an analysis of analytical policy gradients and highlighting the new opportunities for integrating policy learning with trajectory optimization techniques. Figure 1: Fast training of humanoid running policy from pixel input. Our method learns a stable running gait in 4 hours on a single RTX 4080 GPU. 2 Background In this section, we formally define the policy optimization problem and introduce key concepts in analytical policy gradient methods. 2.1 Problem formulation We consider the dynamical system \mathbf{s}_{t+1}=f(\mathbf{s}_{t},\mathbf{a}_{t}) , where \mathbf{s}_{t}\in\mathcal{S} denotes the state and \mathbf{a}_{t}\in\mathcal{A} the action. The dynamics function f:\mathcal{S}\times\mathcal{A}\rightarrow\mathcal{S} is assumed to be fully differentiable with respect to both state and action. Let \mathbf{o}_{t}=g(\mathbf{s}_{t})\in\mathcal{O} denote the observation, where g:\mathcal{S}\rightarrow\mathcal{O} is the sensor model. Throughout this paper, we assume \mathbf{s}_{t} is a low-dimensional internal representation of the robot (e.g., joint states), while \mathbf{o}_{t} represents observation derived from \mathbf{s}_{t} (e.g., images). Consider the trajectory \tau=\{\mathbf{s}_{0},\mathbf{a}_{0},\mathbf{s}_{1},\mathbf{a}_{1},\dots\mathbf{s}_{T},\mathbf{a}_{T}\} , which is a sequence of state-action pairs with horizon T . The total return is defined as \mathcal{J}(\tau)=\sum_{i=0}^{T}\gamma^{t}R(\mathbf{s}_{t},\mathbf{a}_{t}) , where \gamma\in(0,1) is the discount factor, and R:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R} is the reward function. We denotes the discounted temporal reward as r_{t}=\gamma^{t}R(\mathbf{s}_{t},\mathbf{a}_{t}) . A feedback policy \pi(\cdot|\mathbf{o}_{t},\boldsymbol{\theta}):\mathcal{O}\times\boldsymbol{\Theta}\rightarrow\Delta(\mathcal{A}) is a family of conditional probability distributions that maps an observation to a probability distribution over actions. Typically, this distribution is modeled as a Gaussian, allowing the action to be expressed via the reparameterization trick: \mathbf{a}_{t}=\boldsymbol{\mu}(\mathbf{o}_{t},\boldsymbol{\theta})+\boldsymbol{\sigma}(\mathbf{o}_{t},\boldsymbol{\theta})\odot\boldsymbol{\epsilon}_{t} where \boldsymbol{\mu}:\mathcal{O}\times\Theta\rightarrow\mathcal{A} , \boldsymbol{\sigma}:\mathcal{O}\times\Theta\rightarrow\mathcal{A} represent the mean and standard deviation respectively, and \boldsymbol{\epsilon}_{t}\sim\mathcal{N}(\mathbf{0},\mathbf{I}) is injected noise. Given an initial condition \mathbf{s}_{0} , and a sequence of injected noises \mathcal{E}=\{\boldsymbol{\epsilon}_{0},\boldsymbol{\epsilon}_{1},\dots\boldsymbol{\epsilon}_{T}\} , a trajectory \tau can be generated by rolling out from policy under dynamics f and sensor model g , which we explicitly denoted as \tau(\mathbf{s}_{0},\boldsymbol{\theta},\mathcal{E}) and the corresponding return as \mathcal{J}\big(\tau(\mathbf{s}_{0},\boldsymbol{\theta},\mathcal{E})\big) . For notational simplicity, we omit the explicit trajectory \tau and write the return directly as \mathcal{J}(\mathbf{s}_{0},\boldsymbol{\theta},\mathcal{E}) . The expected return for a given policy is defined as \mathcal{V}(\boldsymbol{\theta})=\mathbb{E}_{\mathbf{s}_{0}\sim\rho_{0}}\mathbb{E}_{\boldsymbol{\epsilon}_{t}\overset{\text{i.i.d.}}{\sim}\mathcal{N}(\mathbf{0},\mathbf{I})}\mathcal{J}(\mathbf{s}_{0},\boldsymbol{\theta},\mathcal{E}) , where \rho_{0} is initial distribution. The goal of policy optimization is to find policy parameters \boldsymbol{\theta} maximizing the expected return. 2.2 Analytical policy gradient Here, we provide background on analytical policy gradient (APG) methods. These methods compute the policy gradient as \nabla_{\boldsymbol{\theta}}\mathcal{V}=\mathbb{E}_{\mathbf{s}_{0}\sim\rho_{0}}\mathbb{E}_{\boldsymbol{\epsilon}_{t}\overset{\text{i.i.d.}}{\sim}\mathcal{N}(\mathbf{0},\mathbf{I})}\nabla_{\boldsymbol{\theta}}\mathcal{J}(\mathbf{s}_{0},\boldsymbol{\theta},\mathcal{E})=\mathbb{E}_{\mathbf{s}_{0}\sim\rho_{0}}\mathbb{E}_{\boldsymbol{\epsilon}_{t}\overset{\text{i.i.d.}}{\sim}\mathcal{N}(\mathbf{0},\mathbf{I})}\Big[\sum_{t=0}^{T}\nabla_{\boldsymbol{\theta}}r_{t}\Big], (1) where the gradient of each term in the sum is given by \displaystyle\nabla_{\theta}r_{t} \displaystyle=\frac{\partial r_{t}}{\partial\textbf{a}_{t}}\frac{d\textbf{a}_{t}}{d\boldsymbol{\theta}}+\frac{\partial r_{t}}{\partial\textbf{s}_{t}}\frac{d\textbf{s}_{t}}{d\boldsymbol{\theta}},\ t=0,\dots,T \displaystyle\frac{d\textbf{a}_{t}}{d\boldsymbol{\theta}} \displaystyle=\frac{\partial\textbf{a}_{t}}{\partial\boldsymbol{\theta}}+\frac{\partial\textbf{a}_{t}}{\partial\textbf{o}_{t}}\frac{d\textbf{o}_{t}}{d\textbf{s}_{t}}\frac{d\textbf{s}_{t}}{d\boldsymbol{\theta}},\ t=0,\dots,T \displaystyle\frac{d\textbf{s}_{t}}{d\boldsymbol{\theta}} \displaystyle=\frac{\partial\mathbf{s}_{t}}{\partial\mathbf{s}_{t-1}}\frac{d\mathbf{s}_{t-1}}{d\boldsymbol{\theta}}+\frac{\partial\mathbf{s}_{t}}{\partial\textbf{a}_{t-1}}\frac{d\textbf{a}_{t-1}}{d\boldsymbol{\theta}},\ t=1\dots T\ \text{and}\ \frac{d\mathbf{s}_{0}}{d\boldsymbol{\theta}}=\textbf{0}. (2) Here, \frac{d}{d} denotes the total derivative, and \frac{\partial}{\partial} represents the partial derivative, both expressed in matrix form as Jacobians. The expectation can then be estimated via empirical sum. As these methods estimate the first-order policy gradient by backpropagation through trajectories, they are also named as first-order policy gradients (FoPG) or backpropagation through time (BPTT). Short-horizon actor critic (SHAC) The trajectory gradient, i.e., \nabla_{\boldsymbol{\theta}}\mathcal{J}(\mathbf{s}_{0},\boldsymbol{\theta},\mathcal{E}) , can quickly become intractable as the horizon T increases due to exploding gradients by multiplying a series of matrices. The exploding gradient of an individual trajectory leads to the high-variance empirical estimate of the policy gradient ( Metz et al. 2021 ) , as well as an empirical bias problem ( Suh et al. 2022 ) . Fortunately, these problems are largely solved by the Short-Horizon Actor-Critic (SHAC)

Read the complete text