Expected Return Causes Outcome-Level Mode Collapse in Reinforcement Learning and How to Fix It with Inverse Probability Scaling

Abhijeet Sinha, Sundari Elango, Dinabo Liu · 2026

Read Expected Return Causes Outcome-Level Mode Collapse in Reinforcement Learning and How to Fix It with Inverse Probability Scaling by Abhijeet Sinha, Sundari Elango, Dinabo Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-21669.txt.

Description

Abhijeet Sinha, Sundari Elango, and Dinabo Liu explain a collapse that plagues reinforcement learning, and offer a fix. Many tasks have several end states of similar quality. The goal is often to keep a diverse set of good outcomes alive, not to crown one winner.

Standard training maximizes expected return, and policies trained that way shrink onto a few outcomes. The shrinkage is usually blamed on weak exploration or regularization. The authors trace it to the expected-return objective itself. Their inverse probability scaling counteracts the bias. Keeping diverse good outcomes matters in design and planning, where the single best answer is rarely the whole story.

Catalog record

CREATORAbhijeet Sinha, Sundari Elango, Dinabo Liu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-21669

Browse the full library · About the Tower of Babel