On Globally Optimal Stochastic Policy Gradient Methods for Domain Randomized LQR Synthesis
Alex Nguyen-Le, Nikolai Matni · 2026
Read On Globally Optimal Stochastic Policy Gradient Methods for Domain Randomized LQR Synthesis by Alex Nguyen-Le, Nikolai Matni (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-14197.txt.
Introduction
Nguyen-Le and Matni study stochastic policy gradient methods for domain randomized control design, using the linear-quadratic regulator as the test case. Domain randomization trains on varied fake systems to shrink the sim-to-real gap. They show that fresh system samples at each gradient step lead to global optima under the right step settings. The resampled approach also gives controllers with lower final spread than fixed-sample training. Since sampling is cheap, they argue it should be the default.
Catalog record
| CREATOR | Alex Nguyen-Le, Nikolai Matni |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 68989 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.14197
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
On Globally Optimal Stochastic Policy Gradient Methods for Domain Randomized LQR Synthesis Alex Nguyen-Le, Nikolai Matni Abstract Domain randomization is a simple, effective, and flexible scheme for obtaining robust feedback policies aimed at reducing the sim-to-real gap due to model mismatch. While domain randomization methods have yielded impressive demonstrations in the robotics-learning literature, general and theoretically motivated principles for designing optimization schemes that effectively leverage the randomization are largely unexplored. We address this gap by considering a stochastic policy gradient descent method for the domain randomized linear-quadratic regulator synthesis problem, a situation simple enough to provide theoretical guarantees. In particular, we demonstrate that stochastic gradients obtained by repeatedly sampling new systems at each gradient step converge to global optima with appropriate hyperparameters choices, and yield better controllers with lower variability in the final controllers when compared to approaches that do not resample. Sampling is often a quick and cheap operation, so computing policy gradients with newly sampled systems at each iteration is preferable to evaluating gradients on a fixed set of systems. I Introduction Domain randomization (DR) has gained wide adoption in reinforcement learning and robot learning for enabling sim-to-real transfer [ 1 ] . By harnessing massive GPU parallelism, DR trains controllers across randomized environments to robustify controllers against model uncertainty by minimizing expected cost over a sampled uncertainty distribution. In contrast, control theoretic approaches typically handle uncertainty via robust control, designing for the worst case over the uncertainty set. While the robust control methods come with theoretical guarantees that are largely absent in DR, robust control methods frequently suffer from over-conservatism due to their pessimistic formulation. DR trades these theoretical guarantees for straightforward implementation and effective use of GPU computing resources, which are major contributor to DR’s practical success. An intermediate formulation between DR and robust control, introduced by Fujinami et al. [ 2 , 3 ] , is the domain‑randomized linear‑quadratic regulator (DR‑LQR), which retains DR’s ease of implementation while adding stability guarantees. The only missing benefit of DR in Fujinami et al. [ 2 , 3 ] is the effective utilization of available compute resources, which we address in our work. While Fujinami et al. can scale their policy gradient method with available GPU compute, their approach is inherently limited by the surrogate cost function employed instead of the DR cost function, which we directly optimize with stochastic gradient descent (SGD) methods. I-A Related Work: LQR Policy Gradient Methods The seminal work in applying policy gradient methods to LQR synthesis is Fazel et al. [ 4 ] , which established that the policy gradient satisfies a gradient-domination condition [ 5 ] that can find global optimal solutions to (LQR) synthesis problem despite nonconvexity in the cost-function and feasible set. This has been greatly refined and extended over the past decade; many other optimal control synthesis problems also support globally convergent policy optimization, notably H_{2}/H_{\infty} and H_{\infty} control synthesis [ 6 ] , Markov jump linear system control synthesis [ 7 ] , and even to the reinforcement learning settings [ 8 ] . Despite the impact of Fazel et al. [ 4 ] in the control-literature, many extensions to LQR-synthesis problems possess convex-optimization formulations so policy-gradient methods are not the only techniques that can produce optimal controllers [ 9 ] . A notable exception is that of the domain randomized LQR (DR-LQR) problem [ 2 , 3 ] where Fujinami et al. [ 3 ] show that gradient descent applied to a surrogate sample-average cost converges to a nearly optimal controller. In contrast, we show that an optimal controller can be found by applying stochastic gradient descent directly to the DR-LQR objective. I-B Related Work: Domain Randomization Domain randomization, popularized in the reinforcement learning and robotics literature [ 1 , 10 ] , randomizes simulator parameters towards reducing the sim-to-real gap. While the parameters chosen for randomization are highly task dependent, tasks employing dynamics simulations typically vary physical constants like masses, geometries, or scaling factors [ 10 , 11 , 12 ] . Similarly, when vision is an important component for solving the task at hand, randomization is typically employed over features like textures, lighting, or field-of-view of the vision system [ 1 ] [ 13 ] . This randomization is typically employed by sampling new parameters at each training episode, as opposed to fixed at the outset as in predecessors to DR, like the scenario-approach [ 14 , 15 ] from robust control or ensemble randomization techniques from robot learning [ 16 ] [ 17 ] . This ability to sample and roll out randomized environments in parallel is a key factor in the success of domain randomization: we show how this capability can be leveraged to efficiently compute an optimal DR-LQR controller. I-C Main Contributions • Stochastic Policy Gradient Convergence: We establish the first proof of linear convergence of minibatched SGD to a \varepsilon -suboptimal solution of the DR-LQR problem. We further characterize sufficient conditions on the amount of samples a minibatch should containas a function of this suboptimality level. • Computational Advantage: We empirically show that, for a fixed computational budget, solving the DR-LQR problem via SGD yields a policy that outperforms a controller obtained from the surrogate sample-average formulation of [ 3 ] . II Problem Description Consider a fully observed linear dynamical system, with state x_{t}\in\mathbb{R}^{n_{x}} , input u_{t}\in\mathbb{R}^{n_{u}} , and noise w_{t}\in\mathbb{R}^{n_{x}} . The system evolves as x_{t+1}=Ax_{t}+Bu_{t}+w_{t} where w_{t}\sim\mathcal{N}(0,\Sigma_{w}) and we assume that \Sigma_{w}\succeq I . For fixed system parameters, \theta:=[A,B] the infinite-horizon stochastic LQR cost is given by, \displaystyle J(K,\theta) \displaystyle=\limsup_{T\rightarrow\infty}\mathbb{E}_{w}^{K}\left[\frac{1}{T}\sum_{t=0}^{T}\begin{bmatrix}x_{t}\\ u_{t}\end{bmatrix}\begin{bmatrix}Q&S\\ S^{\top}&R\end{bmatrix}\begin{bmatrix}x_{t}\\ u_{t}\end{bmatrix}\right] \displaystyle=\tr(P(K,\theta)\Sigma_{w}) (1) where P(K,\theta) is the cost-to-go matrix and satisfies [ 18 ] , P(K,\theta)=\dlyap((A\!-\!BK)^{\top},Q\!-\!SK\!-\!S^{\top}K^{\top}\!+\!K^{\top}RK)\\ and, \dlyap(A-BK,X)=\sum_{\ell=0}^{\infty}(A-BK)^{\ell}X(A-BK)^{\ell\top}. Further, the superscript K indicates that the expectation is taken under the closed-loop policy u_{t}=-Kx_{t} with respect to the noise process w_{t} . We also assume that the quadratic cost parameter \mathcal{Q}\coloneqq\begin{bmatrix}Q&S\\ S^{\top}&R\end{bmatrix} satisfies \mathcal{Q}\succeq I . We model the system parameters \theta as random variables supported on a convex and compact set \Theta with density p_{\Theta} , and define the domain randomized linear-quadratic-regulator ( DR-LQR ) synthesis problem as seeking a feedback gain K which minimizes the expecation of the LQR cost ( 1 ) with respect to the system parameters: \displaystyle\begin{array}[]{cl}\underset{K}{\text{minimize}}&\mathbb{E}_{\theta}[J(K,\theta)]\\ \text{subject to:}&\displaystyle\sup_{\theta\in\Theta}J(K,\theta)<\infty\end{array} (DR-LQR) We define J_{DR}(K):=\mathbb{E}_{\theta}[J(K,\theta)] and drop the expecation subscript, as all expectations will be with respect to \theta , i.e., J_{DR}(K)=\mathbb{E}[J(K,\theta)] . Solving problem ( DR-LQR ) is challenging because both the objective and constraint set is nonconvex, and neither the objective nor its gradient can be exactly computed. Nevertheless, we show that we can solve ( DR-LQR ) via the minibatched SGD scheme in Algorithm 1 . In § III , we first show that despite the nonconvexity of the underlying problem, gradient descent converges to a global optimum as the objective function enjoys a gradient dominance property. In § IV , we then show that stochastic gradient descent implemented with approximate gradients estimated via minibatch sampling also converge to a globally optimal solution, albeit at a degraded rate. Algorithm 1 DR Minibatched Stochastic Gradient Descent 1: Input: System Probability Distribution p_{\Theta}(\cdot) , Minibatch Size M , Step-Size, \eta , Number of Gradient Steps t , Starting Point K_{0} 2: Output: