Planning without Search: Refining Frontier LLMs with Offline Goal-Conditioned RL
Joey Hong, Anca Dragan, Sergey Levine · 2025
Read Planning without Search: Refining Frontier LLMs with Offline Goal-Conditioned RL by Joey Hong, Anca Dragan, Sergey Levine (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-18098.txt.
Description
Joey Hong and co-authors give large language models a sense of consequence without retraining the giant. Their 2025 paper targets interactive tasks like negotiation, persuasion, and tool use, where success depends on long-horizon planning. Reinforcement learning could teach planning, but training frontier models that way is costly and often impossible through closed APIs.
Their approach trains small goal-conditioned value functions instead: predictors of how a task will unfold after a candidate reasoning step. At decision time the language model consults these critics to weigh possible futures, good and bad. Because the critics judge reasoning steps rather than full actions, they stay light and portable. Across tool use, social deduction, and dialogue tasks, the method beat both reinforcement learning fine-tuning and elaborate prompting, while remaining efficient. The pattern generalizes beyond this paper: keep the frontier model frozen, and learn the judgment around it.
Catalog record
| CREATOR | Joey Hong, Anca Dragan, Sergey Levine |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-18098