Improving and Accelerating Offline RL in Large Discrete Action Spaces with Structured Policy Initialization
Matthew Landers, Taylor W. Killian, Thomas Hartvigsen, Afsaneh Doryab · 2026
Read Improving and Accelerating Offline RL in Large Discrete Action Spaces with Structured Policy Initialization by Matthew Landers, Taylor W. Killian, Thomas Hartvigsen, Afsaneh Doryab (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-04441.txt.
Description
Matthew Landers and three co-authors help offline reinforcement learning handle action spaces too big to search, in a 2026 arXiv paper. When an action is a combination of many sub-choices, the joint possibilities explode; assuming the pieces are independent yields incoherent actions, while learning structure and control together is slow and unstable. Their framework, SPIN, separates the two jobs into stages.
First, an action structure model is pretrained to learn the shape of valid action combinations, and frozen. Lightweight policy heads then learn control on top of that fixed representation, using only logged data, as offline learning requires. On hard discrete control benchmarks drawn from the DM Control suite. SPIN improved average returns by up to roughly forty percent over the best prior methods and reached convergence more than twelve times faster. Evidence, the authors say, that structure should be learned before control, not during.
Catalog record
| CREATOR | Matthew Landers, Taylor W. Killian, Thomas Hartvigsen, Afsaneh Doryab |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 54 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-04441