Domain Randomization is Sample Efficient for Linear Quadratic Control
Tesshu Fujinami, Bruce D. Lee, Nikolai Matni, George J. Pappas · 2025
Read Domain Randomization is Sample Efficient for Linear Quadratic Control by Tesshu Fujinami, Bruce D. Lee, Nikolai Matni, George J. Pappas (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-12310.txt.
Introduction
Tesshu Fujinami, Bruce D. Lee, Nikolai Matni, and George J. Pappas study how sample-efficient domain randomization is for a classic control problem. Domain randomization trains a controller by averaging performance over a spread of model settings. The method works well in robotics practice, but its theory has lagged. The test bed here is the linear quadratic regulator, a benchmark for learning control laws.
The team proves that, with a well-chosen sampling spread, domain randomization reaches the best possible rate at which extra cost decays. It matches the benchmark method that knows the system exactly. Robust control, by contrast, can overshoot on caution, yet it does better when data is scarce because it can steady an uncertain system from rough estimates. The authors add a gradient-based algorithm for the randomization method and test it numerically. The results point to open questions about wider classes of systems.
Catalog record
| CREATOR | Tesshu Fujinami, Bruce D. Lee, Nikolai Matni, George J. Pappas |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 121 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-12310
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Domain Randomization is Sample Efficient for Linear Quadratic Control Tesshu Fujinami, Bruce D. Lee, Nikolai Matni, George J. Pappas Abstract We study the sample efficiency of domain randomization and robust control for the benchmark problem of learning the linear quadratic regulator (LQR). Domain randomization, which synthesizes controllers by minimizing average performance over a distribution of model parameters, has achieved empirical success in robotics, but its theoretical properties remain poorly understood. We establish that with an appropriately chosen sampling distribution, domain randomization achieves the optimal asymptotic rate of decay in the excess cost, matching certainty equivalence. We further demonstrate that robust control, while potentially overly conservative, exhibits superior performance in the low-data regime due to its ability to stabilize uncertain systems with coarse parameter estimates. We propose a gradient-based algorithm for domain randomization that performs well in numerical experiments, which enables us to validate the trends predicted by our analysis. These results provide insights into the use of domain randomization in learning-enabled control, and highlight several open questions about its application to broader classes of systems. 1 Introduction Figure 1: Illustration of the sample efficinecy of various synthesis methods. The use of learned world models to synthesize controllers via policy optimization is becoming increasingly prevalent in reinforcement learning ( Wu et al., 2023 ; Matsuo et al., 2022 ) . The performance of the resulting controller depends heavily upon the synthesis procedure. Simple approaches that do not account for uncertainty, known as certainty equivalence, can overfit to errors in the learned model. Robust control approaches can tolerate some error in the learned model, but may be overly conservative and computationally demanding. Consequently, domain randomization has emerged as a dominant paradigm in robotics for enabling transfer of policies optimized in simulation on a learned or physics-based simulator to the real world by randomizing system parameters during policy optimization ( Tobin et al., 2017 ; Akkaya et al., 2019 ) . Despite the empirical success of domain randomization, it remains poorly understood how to select the randomization distribution, and how much of a discrepancy between the learned model and the real world can be tolerated. We seek to address these issues by restricting attention to the benchmark problem of learning the linear quadratic regulator (LQR). This problem consists of collecting experimental interaction data from a linear dynamical system, and using this data to synthesize a controller that optimizes a control objective. The linear dynamical system is described by \displaystyle X_{t+1}=A(\theta^{\star})X_{t}+B(\theta^{\star})U_{t}+W_{t}\textrm{ for }t=1,\dots,T, (1) where X_{t}\in\mathbb{R}^{d_{\mathsf{X}}} is the system state, U_{t}\in\mathbb{R}^{d_{\mathsf{U}}} is the control input, W_{t}\in\mathbb{R}^{d_{\mathsf{X}}} is i.i.d. mean zero Gaussian noise, and \theta^{\star}\in\mathbb{R}^{d_{\theta}} is an unknown parameter. The goal is to design a controller K to minimize the objective C(K,\theta^{\star}) , defined as \displaystyle C(K,\theta)\triangleq\limsup_{T\to\infty}\E_{\theta}^{K}\mathopen{}\left[\frac{1}{T}\sum_{t=1}^{T}\mathopen{}\left(\left\|X_{t}\right\|_{Q}^{2}+\left\|U_{t}\right\|_{R}^{2}\right)\mathclose{}\right]\mathclose{}, (2) for Q and R positive definite weight matrices. The subscript on the expectation denotes that the states evolves according to X_{t+1}=A(\theta)X_{t}+B(\theta)U_{t}+W_{t} , and the superscript denotes that the inputs are selected according to the linear feedback U_{t}=KX_{t} . This benchmark problem has been used to study the sample efficiency of certainty equivalence ( Mania et al., 2019 ) and robust control ( Dean et al., 2020 ) by quantifying how many experiments from the system ( 1 ) are sufficient to achieve some level of control performance. In this work, we study the sample efficiency of domain randomization, which chooses a control policy as \displaystyle K_{\mathsf{DR}}(\mathcal{D})=\argmin_{K}\E_{\theta\sim\mathcal{D}}\mathopen{}\left[C(K,\theta)\right]\mathclose{}, (3) for a sampling distribution \mathcal{D} determined using the dataset of experiments collected from ( 1 ). 1.1 Contributions Our contribution is to study the sample efficiency of domain randomization and robust control to establish the relationships visualized as a conceptual diagram in Figure 1 . In particular, we achieve the following: • Sample Effiency of Domain Randomization: We prove that with an appropriately chosen sampling distribution \mathcal{D} , the domain randomization procedure of ( 3 ) achieves the optimal asymptotic rate of decay with the number of samples, thereby matching the performance of certainty equivalence in the large sample regime. We further conjecture that the burn-in time for DR lies between that of RC and CE. We leave proving this to future work, but verify this conjecture numerically. • Sample Effiency of Robust Control: We prove the tightest known bound on robust control, improving the asymptotic rate of decay with the number of samples N from 1/\sqrt{N} to 1/N . The upper bounds indicate a gap between the asymptotic rate of decay for robust control, and the rate of decay for domain randomization and certainty equivalence in terms of system-theoretic quantities. We conjecture that this gap is fundamental, due to the conservative nature of robust control. However, we establish that robust control can achieve a smaller burn-in time relative to certainty equivalence, due to its ability to stabilize the system with a coarse estimate. • Algorithm for Domain Randomization: We propose an algorithm to solve ( 3 ) which proves effective in numerical experiments. This enables verification of the trends predicted in the aforementioned results, and aligns with the conceptual diagram in Figure 1 . While the focus of this work is restricted to linear systems, the proposed algorithm can in principle extend to general nonlinear systems, whereas similar extensions for robust control face computational challenges. By providing this characterization, our work demonstrates the potential of domain randomization in learning-enabled control, and partially explains the empirical success that it has achieved in robotics applications. We therefore conclude by highlighting several interesting open questions regarding the use of domain randomization for learning-enabled control. 1.2 Related Work Domain Randomization Domain randomization, introduced by Tobin et al. (2017) , is widely used for sim-to-real transfer . By randomizing simulator parameters during training, it aims to produce policies robust to simulator variations, thereby enabling transfer to the real-world. This approach has been applied in areas like autonomous racing ( Loquercio et al., 2019 ) and robotic control ( Peng et al., 2018 ; Akkaya et al., 2019 ) . However, its success depends heavily on selecting an effective sampling strategy ( Mehta et al., 2020 ) , which is often challenging. While previous work has explored generalization of domain randomization in discrete Markov Decision Processes ( Chen et al., 2021 ; Zhong et al., 2019 ; Jiang, 2018 ) , formalizing generalization for continuous control remains an open problem, which we address in this work. Identification and Control The linear quadratic regulator problem has become a key benchmark for evaluating reinforcement learning in continuous control ( Abbasi-Yadkori and Szepesvári, 2011 ; Recht, 2019 ) . The offline setting has been extensively studied: Dean et al. (2020) analyzed the sample efficiency of robust control, while Mania et al. (2019) ; Wagenmaker et al. (2021) ; Lee et al. (2023) showed that certainty equivalence is asymptotically instance-optimal, achieving the best possible sample efficiency with respect to system-theoretic quantities. Extensions to smooth nonlinear systems were made by Wagenmaker et al. (2024) ; Lee et al. (2024) . However, certainty equivalence can perform poorly with limited data. Alternative Bayesian approaches ( von Rohr et al., 2022 ; Chiuso et al., 2023 ) can mitigate such limitations. We therefore show that such uncertainty-aware synthesis methods can match the asymptotic efficiency of certainty equivalence while achieving better performance in low-data