Adapting Generalist Robot Policies with Semantic Reinforcement Learning

Jagdeep Singh Bhatia, Andrew Wagenmaker, William Chen, Sergey Levine · 2026

Read Adapting Generalist Robot Policies with Semantic Reinforcement Learning by Jagdeep Singh Bhatia, Andrew Wagenmaker, William Chen, Sergey Levine (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-31958.txt.

Introduction

Bhatia and three colleagues propose learning robot tasks in prompt space rather than action space. Generalist robot policies hold broad skill repertoires from pretraining. Standard reinforcement learning tunes raw actions, which only works when the base policy already starts near a good action distribution. Long-horizon tasks outside the pretraining mix break that assumption.

Semantic Action Reinforcement Learning treats the policy's language instructions as the action space. Adjusting prompts elicits skills the policy already owns, so exploration stays structured and meaningful. Grounding each prompt in the behavior it actually induces lets skills compose into solutions beyond zero-shot reach. Across real-world settings and simulated benchmarks, SARL adapts vision-language-action behavior to complex tasks and beats existing deployment-improvement methods.

Catalog record

CREATORJagdeep Singh Bhatia, Andrew Wagenmaker, William Chen, Sergey Levine
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE91 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.31958

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Adapting Generalist Robot Policies with Semantic Reinforcement Learning Jagdeep Singh Bhatia, Andrew Wagenmaker, William Chen, Sergey Levine Abstract Generalist robot policies learn a diverse repertoire of behaviors from large-scale pretraining. In principle, this makes them excellent priors for downstream adaptation via reinforcement learning (RL). In practice, however, standard RL methods leveraging this prior optimize directly over robot actions, requiring the base policy’s action distribution to be close to that of a performant policy from the start. This assumption breaks down for complex or long-horizon tasks that fall outside the pretraining distribution. Our key insight is that, for sufficiently expressive generalist policies, language prompts are an effective alternative space for learning to solve such tasks: modulating language inputs elicits skills already within the policy’s repertoire, which can be composed to solve tasks beyond its zero-shot capabilities. We propose Semantic Action Reinforcement Learning (SARL), which learns to optimize this prompt space through online interaction, treating the generalist policy as a controllable skill prior. Importantly, leveraging pretrained skills rather than learning new ones from scratch yields structured, semantically meaningful exploration and highly efficient online improvement, and learning to modulate prompts through experience grounds them in induced real-world behaviors for robust task-solving. Across real-world settings and simulated benchmarks, we show SARL unlocks fundamentally new capabilities—adapting VLA behavior to solve complex, long-horizon tasks—and significantly outperforms existing approaches for improving robot behavior in deployment. Keywords: Language Steering, Real-world RL, Robot Foundation Model 1 Introduction Vision-language-action (VLA) models can learn broad skill repertoires from pretraining [ 5 , 9 , 38 , 62 ] , providing powerful priors over plausible robot behaviors. However, leveraging VLAs to solve new tasks remains challenging. While we could directly prompt models to carry out new tasks of interest, this strategy requires models to not only have the necessary skills in their repertoire, but to also deploy them correctly. This is especially a challenge for complex and long-horizon tasks — models must not only break down the semantics of such long-horizon tasks into executable, atomic behaviors, but also ground these behaviors in skills the robot can perform successfully. As such, directly prompting VLAs to carry out novel complex tasks typically fails. How can we better leverage the prior knowledge in general-purpose robot policies to learn to solve these tasks effectively? Our key insight is that learning to modulate a VLA’s prompts through reinforcement learning (RL) is an effective way to probe VLA priors to efficiently learn to solve new tasks. In particular, modulating language inputs elicits skills already within a VLA’s repertoire, yielding constrained, high-quality exploration. By grounding each prompt in the behavior it induces through real-world interaction, these skills can be composed to solve tasks beyond a VLA’s zero-shot capabilities. Unlike prior robotic RL approaches that operate at the robot action level, we propose learning at the semantic level, enabling a more effective use of the behavioral priors encoded in VLAs. For example, to train an omelet-making robot, we can simply learn semantic language commands to steer the VLA: “turn on the stove”, “open the fridge”, “grab the egg”, and so on, rather than needing to learn the precise low-level robot actions required to execute each of these skills. Reusing pre-trained skills rather than discovering new ones from scratch presents a potentially much easier learning problem, since the search space can be naturally constrained to task-appropriate behaviors using the semantics of the task, but still allows for improvement over the base policy. For VLAs with sufficiently broad language-inducible skill sets, language-based steering produces exactly the kind of exploration RL demands: highly expressive yet always grounded in reasonable actions, enabling efficient learning. Our proposed approach, Semantic Action Reinforcement Learning (SARL), instantiates this capability through an online RL loop that treats the VLA’s language instructions as an action space . SARL uses interactions with the world to learn which language prompts will actually lead the VLA to complete the task of interest. By treating the VLA’s language prompt as an “action,” we can lift the learning problem from low-level robot action space to semantic language space. By grounding prompts in real-world experience, we can efficiently refine which of them actually lead to success. Our core contribution is an RL method that learns over the language inputs into a VLA, enabling substantially more structured exploration and faster learning. Our results—both in simulation and on real-world robots—highlight that SARL unlocks fundamentally new capabilities, enabling us to efficiently learn to solve complex, long-horizon tasks that are infeasible for 1) existing RL approaches, 2) zero-shot VLA deployment, and 3) VLAs deployed with high-level language steering (e.g., from a VLM with in-context learning). Finally, we investigate why SARL learns effectively, and find that it is not only because SARL decomposes complex goals into achievable skills, but that SARL also learns a grounded mapping between semantic actions and the physical behaviors they induce. Figure 1: Steering VLA priors at deployment to solve complex, long-horizon tasks is challenging. To overcome this, we propose optimizing VLA language inputs via RL. The resulting semantic action space probes skills already encoded in VLAs from pretraining, yielding expressive exploration and efficient adaptation. However, not all induced behaviors make task progress: by learning to decompose complex tasks into grounded skills, our method identifies effective task-solving prompts and enables real-robot adaptation in under 100 episodes. 2 Related Work RL improvement of VLAs. There has been significant progress in developing general-purpose language-conditioned robot policies, such as vision-language-action models (VLAs) [ 11 , 22 , 74 , 38 , 9 , 8 , 73 , 47 , 62 , 83 ] . While enabling effective task-solving in settings similar to their training data, VLAs often struggle to adapt to new tasks. To address this, a variety of approaches for RL improvement of VLAs have been proposed. These include training a residual policy [ 34 , 4 , 81 , 35 , 17 , 79 , 80 , 70 ] , steering the VLA’s denoising process (when the VLA is a flow or diffusion model) [ 75 ] , augmenting the VLA with “advantage conditioning” [ 3 ] , and others [ 50 , 56 , 14 , 28 , 24 , 48 , 46 ] . However, these works all run RL at the level of robot actions , typically fixing the VLA’s task prompt. In contrast, we run RL over the language prompt itself , which we show unlocks novel task-solving abilities, enabling VLAs to generalize beyond single-task proficiency to solve novel long-horizon tasks. Our work is also related to hierarchical RL and skill learning [ 15 , 40 , 60 , 64 , 55 , 20 , 51 , 2 , 69 , 61 , 58 , 78 ] , where a high-level policy learns to direct a low-level policy to maximize reward. While conceptually similar, unlike these works we focus on the setting where the “low-level policy” is a VLA and the interface between both policies is language. Steering robots with language. Many works consider how a language-conditioned policy’s prompt should be chosen for successful robot task-solving. Early works in this vein explore using LLMs or VLMs to generate useful instructions for solving real-world tasks [ 29 , 82 , 31 , 18 ] ,

Read the complete text