Improving and Accelerating Offline RL in Large Discrete Action Spaces with Structured Policy Initialization

Matthew Landers, Taylor W. Killian, Thomas Hartvigsen, Afsaneh Doryab · 2026

Read Improving and Accelerating Offline RL in Large Discrete Action Spaces with Structured Policy Initialization by Matthew Landers, Taylor W. Killian, Thomas Hartvigsen, Afsaneh Doryab (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-04441.txt.

Description

Matthew Landers and three co-authors help offline reinforcement learning handle action spaces too big to search, in a 2026 arXiv paper. When an action is a combination of many sub-choices, the joint possibilities explode; assuming the pieces are independent yields incoherent actions, while learning structure and control together is slow and unstable. Their framework, SPIN, separates the two jobs into stages.

First, an action structure model is pretrained to learn the shape of valid action combinations, and frozen. Lightweight policy heads then learn control on top of that fixed representation, using only logged data, as offline learning requires. On hard discrete control benchmarks drawn from the DM Control suite. SPIN improved average returns by up to roughly forty percent over the best prior methods and reached convergence more than twelve times faster. Evidence, the authors say, that structure should be learned before control, not during.

Catalog record

CREATORMatthew Landers, Taylor W. Killian, Thomas Hartvigsen, Afsaneh Doryab
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-04441

Browse the full library · About the Tower of Babel