Stationary Reweighting Yields Local Convergence of Soft Fitted Q-Iteration
Lars van der Laan, Nathan Kallus · 2025
Read Stationary Reweighting Yields Local Convergence of Soft Fitted Q-Iteration by Lars van der Laan, Nathan Kallus (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-23927.txt.
Description
Lars van der Laan and Nathan Kallus work on fitted Q-iteration, a workhorse method for learning control policies from logged data in offline reinforcement learning. The method's stability is usually argued through a demanding assumption called Bellman completeness, which real function classes often fail. The authors ask whether a softened version of the method can be trusted without it.
Their key observation is local: near the best soft policy, the update behaves like ordinary policy evaluation, which shrinks errors when states are weighted by how often that policy visits them. Reweighting the fitting step by those visit ratios restores the shrinking property, and the team proves local convergence with finite data. A temperature schedule then carries the result further, starting hot, where convergence is global, and cooling step by step toward the target. Under a margin condition, a final switch to the hard method reaches the unregularized optimum.
Catalog record
| CREATOR | Lars van der Laan, Nathan Kallus |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 126 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-23927