Pessimistic Auxiliary Policy for Offline Reinforcement Learning

Fan Zhang Baoru Huang, Xin Zhang · 2026

Read Pessimistic Auxiliary Policy for Offline Reinforcement Learning by Fan Zhang Baoru Huang, Xin Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-23974.txt.

Description

Fan Zhang and two co-authors present a pessimistic auxiliary policy for offline reinforcement learning, where agents learn only from pre-collected data. The 2026 paper notes that offline learning inevitably samples actions outside the dataset, whose value estimates carry errors that compound into large overestimation.

The auxiliary policy samples reliable actions by maximizing a lower confidence bound on the action-value function. Near the learned policy it favors actions that are both high-value and low-uncertainty, steering learning away from tempting but poorly supported actions. With fewer approximation errors entering from sampled actions, error accumulation eases. Tests on offline benchmarks show that the pessimistic auxiliary strategy improves the efficacy of other offline methods when added to them, offering a general, cautious sampling fix.

Catalog record

CREATORFan Zhang Baoru Huang, Xin Zhang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-23974

Browse the full library · About the Tower of Babel