A Theoretical Justification for Asymmetric Actor-Critic Algorithms

Gaspard Lambrechts, Damien Ernst, Aditya Mahajan · 2025

Read A Theoretical Justification for Asymmetric Actor-Critic Algorithms by Gaspard Lambrechts, Damien Ernst, Aditya Mahajan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-19116.txt.

Introduction

Gaspard Lambrechts, Damien Ernst, and Aditya Mahajan supply in a 2025 paper a theory base for asymmetric actor critic learning. Agents that see only part of their world often train with extra state facts in hand. That head start speeds learning, but why it helps lacked a sharp proof.

Their proof adapts a finite time convergence study to linear function approximators in this setting. The bound shows the informed critic wipes out error terms born of state aliasing. Aliasing is the blur where unlike true states look the same to the agent. Removing those terms explains the speed edge seen in practice. The result firms up a widely used design pattern in the field.

Catalog record

CREATORGaspard Lambrechts, Damien Ernst, Aditya Mahajan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE124 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-19116

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Theoretical Justification for Asymmetric Actor-Critic Algorithms Gaspard Lambrechts, Damien Ernst, Aditya Mahajan Abstract In reinforcement learning for partially observable environments, many successful algorithms have been developed within the asymmetric learning paradigm. This paradigm leverages additional state information available at training time for faster learning. Although the proposed learning objectives are usually theoretically sound, these methods still lack a precise theoretical justification for their potential benefits. We propose such a justification for asymmetric actor-critic algorithms with linear function approximators by adapting a finite-time convergence analysis to this setting. The resulting finite-time bound reveals that the asymmetric critic eliminates error terms arising from aliasing in the agent state. † † affiliationnotice: Equal contribution 1 Introduction Reinforcement learning (RL) is an appealing framework for solving decision making problems, notably because it makes very few assumptions about the problem at hand. In its purest form, the promise of an RL algorithm is to learn an optimal behavior from interaction with an environment whose dynamics are unknown. More formally, an RL algorithm aims to learn a policy – which is defined as a mapping from observations to actions – from interaction samples, in order to maximize a reward signal. While RL has obtained empirical successes for a plethora of challenging problems ranging from games to robotics ( Mnih et al., 2015 ; Schrittwieser et al., 2020 ; Levine et al., 2015 ; Akkaya et al., 2019 ) , most of these achievements have assumed full state observability. A more realistic assumption is partial state observability, where only a partial observation of the state of the environment is available for taking actions. In this setting, the optimal action generally depends on the complete history of past observations and actions. Traditional RL approaches have thus been adapted by considering history-dependent policies, usually with a recurrent neural network to process histories ( Bakker, 2001 ; Wierstra et al., 2007 ; Hausknecht & Stone, 2015 ; Heess et al., 2015 ; Zhang et al., 2016 ; Zhu et al., 2017 ) . Given the difficulty of learning effective history-dependent policies, various auxiliary representation learning objectives have been proposed to compress the history into useful representations ( Igl et al., 2018 ; Buesing et al., 2018 ; Guo et al., 2018 ; Gregor et al., 2019 ; Han et al., 2019 ; Guo et al., 2020 ; Lee et al., 2020 ; Subramanian et al., 2022 ; Ni et al., 2024 ) . Such methods usually seek to learn history representations that encode the belief, defined as the posterior distributions over the states given the history, which is a sufficient statistic of the history for optimal control. While these methods are theoretically able to learn optimal history-dependent policies, they usually learn solely from the partial state observations, which can be restrictive. Indeed, assuming the same partial observability at training time and execution time can be too pessimistic for many environments, notably for those that are simulated. This motivated the asymmetric learning paradigm, where additional state information available at training time is leveraged during the process of learning a history-dependent policy. Although the optimal policies obtained by asymmetric learning are theoretically equivalent to those learned by symmetric learning, the promise of asymmetric learning is to improve the convergence speed. Early approaches proposed to imitate a privileged policy conditioned on the state ( Choudhury et al., 2018 ) , or to use an asymmetric critic conditioned on the state ( Pinto et al., 2018 ) . These heuristic methods initially lacked a theoretical framework, and a recent line of work has focused on proposing theoretically grounded asymmetric learning objectives. First, imitation learning of a privileged policy was known to be suboptimal, and it was addressed by constraining the privileged policy so that its imitation results in an optimal policy for the partially observable environment ( Warrington et al., 2021 ) . Similarly, asymmetric actor-critic approaches were proven to provide biased gradients, and an unbiased actor-critic approach was proposed by introducing the history-state value function ( Baisero & Amato, 2022 ) . In model-based RL, several works proposed world model objectives that are proved to provide sufficient statistics of the history, by leveraging the state ( Avalos et al., 2024 ) or arbitrary state information ( Lambrechts et al., 2024 ) . Finally, asymmetric representation learning approaches were proposed to learn sufficient statistics from state samples ( Wang et al., 2023 ; Sinha & Mahajan, 2023 ) . It is worth noting that many recent successful applications of RL have greatly benefited from asymmetric learning, usually through an asymmetric critic ( Degrave et al., 2022 ; Kaufmann et al., 2023 ; Vasco et al., 2024 ) . Despite these methods being theoretically grounded, in the sense that policies satisfying these objectives are optimal policies, they still lack a theoretical justification for their potential benefit. In particular, there is no theoretical justification for the improved convergence speed of asymmetric learning. In this work, we propose such a justification for an asymmetric actor-critic algorithm, using agent-state policies and linear function approximators. Agent-state policies rely on an internal state, which is updated recurrently based on successive actions and observations, from which the next action is selected. This agent state can introduce aliasing, a phenomenon in which an agent state may correspond to two different beliefs. Our argument relies on the comparaison of two analogous finite-time bounds: one for a symmetric natural actor-critic algorithm ( Cayci et al., 2024 ) , and its adaptation to the asymmetric setting that we derive in this paper. This comparison reveals that asymmetric learning eliminates error terms arising from aliasing in the agent state in symmetric learning. These aliasing terms are given by the difference between the true belief (i.e., the posterior distribution over the states given the history) and the approximate belief (i.e., the posterior distribution over the states given the agent state). This suggests that asymmetric learning may be particularly useful when aliasing is high. A recent related work proposed a model-based asymmetric actor-critic algorithm relying on belief approximation, and proved its sample efficiency ( Cai et al., 2024 ) . It also considered agent-state policies, and studied the finite-time performance by providing a probably approximately correct (PAC) bound, instead of an expectation bound as here. While the algorithm was restricted to finite horizon and discrete spaces, notably for implementing count-based exploration strategies, it tackled the online exploration setting and its performance bound did not present a concentrability coefficient. This related analysis thus provides a promising framework for future works in a more challenging setting. However, it did not study the existing asymmetric actor-critic algorithm, and did not provides a direct comparison with symmetric learning. In contrast, we focus on providing comparable bounds for the existing model-free asymmetric actor-critic algorithm and its symmetric counterpart. In Section 2 , we formalize the environments, policies, and Q-functions that are considered. In Section 3 , we introduce the asymmetric and symmetric actor-critic algorithms that are studied. In Section 4 , we provide the finite-time bounds for the asymmetric and symmetric actor-critic algorithms. Finally, in Section 5 , we conclude by summarizing the contributions and providing avenues for future works. 2 Background In Subsection 2.1 , we introduce the decision processes and

Read the complete text