Expected Return Causes Outcome-Level Mode Collapse in Reinforcement Learning and How to Fix It with Inverse Probability Scaling
Abhijeet Sinha, Sundari Elango, Dinabo Liu · 2026
Read Expected Return Causes Outcome-Level Mode Collapse in Reinforcement Learning and How to Fix It with Inverse Probability Scaling by Abhijeet Sinha, Sundari Elango, Dinabo Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-21669.txt.
Description
Abhijeet Sinha, Sundari Elango, and Dinabo Liu explain a collapse that plagues reinforcement learning, and offer a fix. Many tasks have several end states of similar quality. The goal is often to keep a diverse set of good outcomes alive, not to crown one winner.
Standard training maximizes expected return, and policies trained that way shrink onto a few outcomes. The shrinkage is usually blamed on weak exploration or regularization. The authors trace it to the expected-return objective itself. Their inverse probability scaling counteracts the bias. Keeping diverse good outcomes matters in design and planning, where the single best answer is rarely the whole story.
Catalog record
| CREATOR | Abhijeet Sinha, Sundari Elango, Dinabo Liu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-21669