Policy Gradient Method for LQG Control via Input–Output - History Representation: Convergence to 𝒪 ⁡ ( ϵ ) \mathcal{O}(\epsilon) - Stationary Points Journal: Systems & Control Letters

Tomonori Sadamoto, Takashi Tanaka · 2025

Read Policy Gradient Method for LQG Control via Input–Output - History Representation: Convergence to 𝒪 ⁡ ( ϵ ) \mathcal{O}(\epsilon) - Stationary Points Journal: Systems & Control Letters by Tomonori Sadamoto, Takashi Tanaka (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-19141.txt.

Introduction

Tomonori Sadamoto and Takashi Tanaka study policy gradient methods. The setting is linear quadratic Gaussian control. A controller must act from noisy, partial readings. The authors rewrite the closed-loop system. They use a history of past inputs and outputs. A dynamic controller then looks like a simple static gain.

That rewrite lets them add a small noise term. The term secures what gradient methods need. They then prove convergence to near-stationary points. Control engineers gain firmer ground for tuning by gradient search. Such tuning is common but had thin theory behind it.

Catalog record

CREATORTomonori Sadamoto, Takashi Tanaka
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE85 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-19141

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Policy Gradient Method for LQG Control via Input–Output - History Representation: Convergence to 𝒪 ⁡ ( ϵ ) \mathcal{O}(\epsilon) - Stationary Points Journal: Systems & Control Letters Tomonori Sadamoto, Takashi Tanaka Abstract We study the policy gradient method (PGM) for the linear quadratic Gaussian (LQG) dynamic output - feedback control problem using an input–output - history (IOH) representation of the closed - loop system. First, we show that any dynamic output - feedback controller is equivalent to a static partial - state feedback gain for a new system representation characterized by a finite-length IOH. Leveraging this equivalence, we reformulate the search for an optimal dynamic output feedback controller as an optimization problem over the corresponding partial-state feedback gain. Next, we introduce a relaxed version of the IOH-based LQG problem by incorporating a small process noise with covariance ϵ ​ I \epsilon I into the new system to ensure coerciveness, a key condition for establishing gradient - based convergence guarantees. Consequently, we show that a vanilla PGM for the relaxed problem converges to an 𝒪 ⁡ ( ϵ ) \mathcal{O}(\epsilon) - stationary point, i.e., K ¯ \overline{K} satisfying ‖ ∇ J ​ ( K ¯ ) ‖ F ≤ 𝒪 ⁡ ( ϵ ) \|\nabla J(\overline{K})\|_{F}\leq\mathcal{O}(\epsilon) , where J J denotes the original LQG cost. Numerical experiments empirically indicate convergence to the vicinity of the globally optimal LQG controller. 1 Introduction The linear quadratic Gaussian (LQG) control problem is one of the most fundamental problems in modern control theory [ 1 , 2 ] . Given the explicit knowledge of the system model and the cost function, it is widely recognized that the optimal policy is given by a dynamic output feedback law and can be synthesized by solving two Riccati equations [ 3 ] . Motivated by many practical situations where the system model is unknown, there has been increased research interest in model-free approaches where the policy is learned from the system’s input-output behavior. To achieve this goal, so-called indirect approaches first attempt to estimate the system model [ 4 , 5 ] from the input-output data, and then apply a classical model-based synthesis. Alternatively, direct approaches attempt to optimize the policy incrementally using data by certain update rules such as gradient methods. While the direct approaches are natural given the popularity of the policy gradient methods (PGMs) in the learning theory community, recent studies have revealed prominent challenges in the PGM-based LQG synthesis. These challenges are primarily due to the complex optimization landscape underlying the LQG control problems. For example, recent works [ 6 , 7 , 8 ] have shown that optimization of the system matrices of dynamic output-feedback controllers has many stationary points. To make matters worse, the optimization problem is not coercive [ 8 ] ; therefore, it is not even guaranteed that PGMs will converge to one of these numerous stationary points. Consequently, most studies have been limited to analyzing the optimization landscape, while no theoretical analysis has been conducted on the convergence of PGMs. In this study, we consider a particular parameterization of dynamic controllers and propose a vanilla PGM with a theoretical guarantee of convergence to \mathcal{O}(\epsilon) - stationary point, i.e., \overline{K} satisfying \|\nabla J(\overline{K})\|_{F}\leq\mathcal{O}(\epsilon) , where J denotes the original LQG cost and \epsilon\ll 1 is a design parameter. While our analysis assumes that the plant model is known in advance, the fact that the PGM converges to \mathcal{O}(\epsilon) - stationary points offers valuable insights into the convergence analysis of model-free PGM-based LQG synthesis. The contributions of this paper are as follows: First, we show that designing a dynamic output feedback controller without a feed-through term for a partially observable system affected by process and measurement noise is equivalent to designing a static partial-state feedback gain (which we refer to as the IOH gain hereafter) for a new system whose internal state consists of a finite-length history of the input-output data and noise. Hereafter, we refer to the new system as history dynamics . We show how a dynamic output feedback controller can be written as an equivalent IOH gain and vice versa . This equivalence enables us to search for \mathcal{O}(\epsilon) -stationary points of the LQG problem via the search for the IOH gain. Second, we propose a vanilla PGM with a theoretical guarantee of convergence to \mathcal{O}(\epsilon) -stationary points of the LQG problem. The LQG cost as a function of the IOH gain is still not coercive in general. To address this issue, we introduce a relaxed version of the LQG problem, where a small process noise with covariance \epsilon I is added to the history dynamics. This relaxation ensures that the cost function becomes coercive over the set of stabilizing IOH gains and smooth over a sublevel set. Furthermore, we show that \overline{K} , a stationary point of the relaxed LQG cost, satisfies \|\nabla J(\overline{K})\|_{F}\leq\mathcal{O}(\epsilon) where J is the original LQG cost function. As a result, we establish that a vanilla PGM for the relaxed problem can find an \mathcal{O}(\epsilon) -stationary point of the original LQG cost. While this theoretical guarantee is limited to the convergence to \mathcal{O}(\epsilon) -stationary points, our numerical study suggests that the PGM converges to the optimal LQG controller if \epsilon is sufficiently small. Moreover, even when the controller dimension is restricted to be less than that of the system, we show that the PGM can find a better reduced-order controller than a conventional controller reduction method. Related Works Convergence of the PGM for discrete-time linear quadratic regulator (LQR) problems was studied by [ 9 ] . Despite the non-convex nature of the problem, they proved that the PGM converges to the globally optimal solution because the problem has benign properties such as smoothness, coerciveness, and gradient dominance. This result has been extended to PGMs for continuous-time systems [ 10 , 11 ] , zero-sum games [ 12 ] , and distributed controls [ 13 ] . While these results are limited to state feedback synthesis, extensions to static output-feedback synthesis have been considered [ 14 , 15 , 16 , 17 ] . Unlike the state-feedback case, the set of stabilizing controllers is typically disconnected and has many stationary points, making the convergence analysis more complex. Furthermore, even if a globally optimal solution is found, the achievable control performance is inherently more limited than that of dynamic output-feedback controllers. These challenges have motivated the line of research on dynamic output-feedback synthesis. While the classical model-based methods for the LQG dynamic output feedback synthesis (e.g., via Riccati equations) are well-established, PGM-based approaches present notable challenges. The existing approaches can be organized into three categories in terms of the parameterization methods of dynamic controllers. The first category uses the state space model (the system matrices) to parametrize dynamic controllers [ 6 , 7 , 8 , 18 ] . While this approach is naturally inspired by the classical model-based synthesis, its optimization landscapes have turned out to be extremely complex, in part due to the inherent symmetry caused by similarity transformations. This approach enjoys neither gradient dominance nor coerciveness; thus, convergence to at least stationary points is not guaranteed. The second approach is based on a convex reparameterization of the controller [ 10 , 19

Read the complete text