DiffCoTune: Differentiable Co-Tuning for Cross-domain Robot Control

Lokesh Krishna, Sheng Cheng, Junheng Li, Naira Hovakimyan, Quan Nguyen · 2025

Read DiffCoTune: Differentiable Co-Tuning for Cross-domain Robot Control by Lokesh Krishna, Sheng Cheng, Junheng Li, Naira Hovakimyan, Quan Nguyen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-24068.txt.

Introduction

Lokesh Krishna and four co-authors present DiffCoTune, a method that retunes both a robot simulator and its controller together using gradients, in this 2025 arXiv paper. Controllers trained in simulation usually stumble in the real world, where models are simplified or just wrong. Hand tuning fixes the gap slowly and unevenly.

Working with a simulator that supports gradient flow, the method collects a few real trials and alternately adjusts the simulator's physics and the controller's settings. The pair converge toward deployment-ready behavior within a handful of trials. Tests ranged from a cart-pole up to walking quadrupeds and bipeds, across model-based and learned controllers. Performance rose in every target domain tried.

Catalog record

CREATORLokesh Krishna, Sheng Cheng, Junheng Li, Naira Hovakimyan, Quan Nguyen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-24068

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DiffCoTune: Differentiable Co-Tuning for Cross-domain Robot Control Lokesh Krishna, Sheng Cheng, Junheng Li, Naira Hovakimyan, Quan Nguyen Abstract The deployment of robot controllers is hindered by modeling discrepancies due to necessary simplifications for computational tractability or inaccuracies in data-generating simulators. Such discrepancies typically require ad-hoc tuning to meet the desired performance, thereby ensuring successful transfer to a target domain. We propose a framework for automated, gradient-based tuning to enhance performance in the deployment domain by leveraging differentiable simulators. Our method collects rollouts in an iterative manner to co-tune the simulator and controller parameters, enabling systematic transfer within a few trials in the deployment domain. Specifically, we formulate multi-step objectives for tuning and employ alternating optimization to effectively adapt the controller to the deployment domain. The scalability of our framework is demonstrated by co-tuning model-based and learning-based controllers of arbitrary complexity for tasks ranging from low-dimensional cart-pole stabilization to high-dimensional quadruped and biped tracking, showing performance improvements across different deployment domains. I Introduction Meeting a desired performance while transferring robot controllers from a synthesis domain , like a simulator or dynamics model, to a deployment domain , like the real system, remains a ubiquitous challenge across synthesis paradigms due to uncertainties from unmodeled dynamics, noise, and perturbations. Model-based control synthesis offers well-defined techniques to address uncertainty, namely robust or stochastic control [ 1 ] , adaptive control [ 2 ] , and system identification (sysId) [ 3 ] . In principle, robust or stochastic control designs consider the worst case or expectation over some distribution of the uncertainty; adaptive control dynamically adjusts parameters based on an estimation of uncertainty; and system identification aims to tune the synthesis domain’s parameters to best reflect the deployment domain using a physics- or first-principle-based model. Scaling these distinct notions, model-free learning-based control has conceptually equivalent techniques, namely domain randomization [ 4 ] , domain adaptation [ 5 ] , and sysId using deep neural networks (DNNs) [ 6 ] . Domain randomization assumes an uncertainty distribution like stochastic control, while domain adaptation estimates invariant features or parameters like adaptive control. Each method, when applied individually, has specific limitations: robust control and domain randomization can yield conservative performance under a poor choice of uncertainty bounds [ 7 , 8 ] , adaptive control and domain adaptation require estimateable uncertainty from the observable states [ 9 ] . To this end, given a reasonable robot model or simulator a combination of these methods has been employed to achieve empirical success, such as robust adaptive control [ 10 , 11 ] , domain adaptation with randomization [ 5 , 12 ] , and domain randomization with DNN-based sysId [ 6 ] . Fig. 1 : Overview of the proposed automated co-tuning approach for cross-domain transfer. Accompanying video results at : link In a similar vein, we focus on the intersection of domain adaptation and system identification for the automated transfer of robot controllers. Starting with a nominal controller designed in the synthesis domain, we aim to adapt it for the deployment domain by simultaneously calibrating model parameters in an intermediate tuning domain. Thus, we formulate the cross-domain transfer as a co-tuning problem and develop a scalable technique for solving it across different systems and controllers. To this end, we propose to study system identification and controller adaptation as a coupled problem, with the sole focus being performance improvement in the deployment domain. Given a nominal differentiable controller, we leverage a differentiable simulator to co-tune the parameters of the controller and the simulator, utilizing a handful of rollouts in the deployment domain. Thus, our proposed formulation eases the demand on a global optimum (as usually required in sysId or control synthesis) by requiring neither an accurate recreation of the target system nor the synthesis of a controller from scratch. Instead, we only seek the adaptation of the nominal controller to perform better in the deployment domain. Such a perspective aims to tune existing empirically compelling model-based [ 13 ] and model-free [ 14 ] controllers, utilizing gradient-based updates for searching an improved local solution within the neighborhood of the nominal controller. Recent approaches employ on-policy reinforcement learning (RL) to learn residual policies compensating for unmodelled dynamics, followed by fine-tuning the nominal policy in the augmented simulator [ 15 , 16 ] . Such multi-stage optimization for tuning relies on reward shaping and has only been shown on specific systems with learnt controllers. In contrast, we aim to develop a generic tuning framework without the need to tune its own set of hyperparameters. Our proposed iterative approach is similar in principle to [ 17 ] alternating between gradient-based optimization to refine the differentiable simulator from the rollouts and model-free Deep RL for learning a policy in the updated simulator from scratch. While [ 17 ] relies on a carefully engineered choice of low-level gait controller specific to the quadrupedal locomotion, we focus on pure gradient-based co-tuning of a nominal controller on systems of arbitrary complexity without task-specific design requirements. The main contribution of this work is twofold: 1. We present co-tuning techniques to simultaneously tune a nominal controller with a differentiable simulator for an automated and systematic cross-domain transfer, improving performance with a few trials ( <5 ) in the deployment domain. 2. Experimental evaluation spans systems from low-dimensional cart-pole stabilization to high-dimensional quadruped and bipedal tracking. The proposed approach improves the performance of diverse controllers such as LQR, PD, MPC, and DNN policies, successfully transferring to different simulators and hardware. II Related work Controller auto-tuning: Previous auto-tuning methods can be categorized into model-free [ 18 , 19 , 20 , 21 , 22 , 23 , 24 ] and model-based [ 25 , 26 , 27 , 28 , 29 ] . Both approaches iteratively pick the parameter candidate for evaluation that is likely to improve the performance over the previous trials. Model-free auto-tuning methods apply a data-driven model to capture the relationship between a parameter choice and the performance metric. Representative approaches include Markov chain Monte Carlo [ 22 ] , policy search [ 23 ] , and Bayesian optimization [ 18 , 19 , 20 , 21 , 30 , 24 ] . Such approaches do not rely on the knowledge of the model and controller and are compatible with physical systems’ data owing to their data-driven nature. However, the absence of physics or first-principle model knowledge (offering a low-dimensional compact representation to the system) is remedied by data-driven models (DNN or Gaussian processes) that often yield higher-dimensional parameter space to characterize the system. In particular, Bayesian optimization is hardly scalable to tuning in high-dimensional ( > 20) parameter spaces [ 19 ] . Model-based auto-tuning methods leverage the knowledge of the system model and a model-based controller, often applying gradient descent so that the performance can improve based on the local gradient information [ 25 , 26 , 27 , 28 , 29 ] . However, model-based auto-tuning suffers from biased gradients due to model mismatch that originated from uncertainties or time-varying system configurations, which is one cause of the sim-to-real gap from the perspective of controller tuning. Previous work [ 27 ] has applied the \mathcal{L}_{1} adaptive control [ 31 ] to enforce the system

Read the complete text