Functional Critic Modeling for Provably Convergent Off-Policy Actor-Critic

Qinxun Bai, Yuxuan Han, Wei Xu, Zhengyuan Zhou · 2025

Read Functional Critic Modeling for Provably Convergent Off-Policy Actor-Critic by Qinxun Bai, Yuxuan Han, Wei Xu, Zhengyuan Zhou (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-22964.txt.

Introduction

Actor-critic methods learn well off-policy, but the policy under review keeps shifting. Qinxun Bai and 3 co-authors argue that functional critics are a must, not a luxury. A functional critic takes a code for the policy as input, not just states and actions. Past tries with such critics lagged behind standard methods. The team find two reasons they matter.

First, functional critics calm the clash known as the deadly triad plus moving target. The team give an algorithm that provably settles, with fewer harsh assumptions than before. Second, such critics capture policy-linked uncertainty that exploration tools miss. A small neural version matched top methods on control tests.

Catalog record

CREATORQinxun Bai, Yuxuan Han, Wei Xu, Zhengyuan Zhou
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE60 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-22964

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Functional Critic Modeling for Provably Convergent Off-Policy Actor-Critic Qinxun Bai, Yuxuan Han, Wei Xu, Zhengyuan Zhou Abstract Off-policy reinforcement learning (RL) with function approximation offers an effective way to improve sample efficiency by reusing past experience. Within this setting, the actor–critic (AC) framework has achieved strong empirical success. However, both the critic and actor learning is challenging for the off-policy AC methods: first of all, in addition to the classic “deadly triad” instability of off-policy evaluation, it also suffers from a “moving target” problem, where the policy being evaluated changes continually; secondly, actor learning becomes less efficient due to the difficulty of estimating the exact off-policy policy gradient. The first challenge essentially reduces the problem to repeatedly performing off-policy evaluation for changing policies. For the second challenge, the off-policy policy gradient theorem requires a complex and often impractical algorithm to estimate an additional emphasis critic, which is typically neglected in practice, thereby reducing to the on-policy policy gradient as an approximation. In this work, we introduce a novel concept of functional critic modeling, which leads to a new AC framework that addresses both challenges for actor-critic learning under the deadly triad setting. We provide a theoretical analysis in the linear function setting, establishing the provable convergence of our framework, which, to the best of our knowledge, is the first convergent off-policy target-based AC algorithm. From a practical perspective, we further propose a carefully designed neural network architecture for the functional critic modeling and demonstrate its effectiveness through preliminary experiments on widely-used RL tasks from the DeepMind Control Benchmark. 1 1 footnotetext: These authors contributed equally to this work. 1 1 footnotetext: Horizon Robotics. {qinxun.bai, wei.xu}@horizon.auto 2 2 footnotetext: Stern School of Business, New York University. {yh6061, zzhou}@stern.nyu.edu 1 Introduction Off-policy AC. Reinforcement learning (RL) has proven to be a powerful tool for sequential decision-making. In this work, we focus on off-policy RL under the actor-critic (AC) framework. In contrast to on-policy methods, off-policy methods enjoy improved sample efficiency by reusing past data collected by a behavior policy ( White 2015 ; Sutton et al. 2011 ; Lin 1992 ; Mnih et al. 2015 ; Schaul et al. 2016 ) . The actor-critic framework ( Degris et al. 2012 ; Maei 2018 ; Silver et al. 2014 ; Lillicrap et al. 2015 ; Wang et al. 2016 ; Gu et al. 2017 ; Konda 2002 ) is arguably one of the most successful frameworks for policy-based control, which is favored in applications with continuous and/or high-dimensional action spaces, such as robotics and large language models. Challenges. Despite the empirical success of off-policy AC, this framework still faces several challenges—both in practice and in theory. One widely-studied difficulty lies in the critic update step, which conducts off-policy evaluation for the target policy. In contrast to the on-policy setting, applying standard evaluation techniques such as temporal-difference (TD) learning with function approximation under the off-policy setting can lead to instability—a phenomenon known as the “deadly triad” problem ( Baird et al. 1995 ; Tsitsiklis and Van Roy 1996 ; Kolter 2011 ; Sutton and Barto 2018 ) . Although there are convergent off-policy evaluation methods under the deadly triad, such as gradient TD methods ( Sutton et al. 2009 ; Qian and Zhang 2025 ; Yu 2017 ; Maei et al. 2009 ) , emphatic TD methods ( Sutton et al. 2016 ; Yu 2015 ) , and target critic-based methods ( Mnih et al. 2015 ; Lee and He 2019 ; Zhang et al. 2021 ; Chen et al. 2023 ) , provable convergence within the off-policy AC framework remains largely unresolved, due to two additional challenges in critic learning and actor learning respectively. First, under the AC framework, the evaluated policy is constantly changing, as the actor is updated continuously—leading to a moving target problem for the off-policy evaluation/critic learning. In fact, even for on-policy AC, existing convergence guarantee depends on a two-timescale framework ( Konda 2002 ) , where the policy has to be updated much more slowly than the updates driving the critic’s convergence. Second, the off-policy policy gradient ( Imani et al. 2018 ) is more complex than its on-policy counterpart, as it involves not only value estimation but also correction terms, known as the emphatic term, to account for distribution mismatch. These corrections introduce additional sample-based estimation steps to critic learning, which is vulnerable to the same instability issues as the policy-evaluation step ( Zhang et al. 2020 ) . State-of-the-art. To address the instability issue and additional challenges of off-policy actor–critic (AC), Zhang et al. 2020 —to our knowledge the only prior work with a convergence guarantee—combines two types of provably stable gradient-based evaluation methods, known as gradient TD (GTD) ( Sutton et al. 2009 ) and Emphatic TD (ETD) ( Sutton et al. 2016 ) within a two-timescale framework. While this ensures convergence, it substantially slows policy improvement, since most samples are consumed by critic updates rather than policy updates. Moreover, the GTD and ETD-based algorithm is complicated, requiring estimation of several additional critic-like quantities that introduce extra computational and numerical burdens as well as provable instability concerns under practical settings ( Manek and Kolter 2022 ) . In practice, most successful off-policy RL algorithms ( Fujimoto et al. 2018 ; Haarnoja et al. 2018 ; Lillicrap et al. 2015 ; Ball et al. 2023 ) use target critics to stabilize TD learning, as this approach has been empirically shown to be easier to tune than gradient TD methods. However, because of the two additional challenges noted above in off-policy AC, none of the target-based methods is provably convergent under function approximation. In particular, since off-policy policy gradients require extra sample-based estimation for the emphatic term—and the corresponding target-based estimation techniques remain unclear—target-based AC methods often fall back to on-policy gradient formulas, avoiding algorithmic complications but resulting in less efficient policy improvement and lacking the convergence guarantee of exact off-policy gradients. Moreover, to address the moving-policy issue, state-of-the-art empirical methods ( Chen et al. 2021 ; Ball et al. 2023 ) typically employ a high update-to-data (UTD) ratio, which mirrors the two-timescale design but likewise reduces sample efficiency. Our solution. To overcome the challenges faced by existing off-policy AC methods, we introduce a new actor-critic framework with functional critics that eliminates both the need for slow-rate policy updates and the need for correction terms in exact off-policy gradients. The key idea is to model the critic as a functional that maps the policy space to policy values, for given state or state-action pair. For policy evaluation, this allows the critic to generalize to new, changing policies without needing to restart the evaluation process after every actor update. For policy improvement, exact off-policy gradients can be directly computed from the learned functional critic, without correction terms and additional estimation loops. In learning functional critics, we adopt a functional TD learning scheme that leverages the existence of the Bellman equation for each changing policy. To support our framework with theoretical insights, we analyze it in a setting under linear functional approximation —an adaptation of the standard linear value function approximation widely-used by existing theoretical work, to the functional setting (see § 3.2 ). This simplified model enables us

Read the complete text