GROW: Aligning GRPO with State-Action Modeling for Open-World VLM Agents
Xiongbin Wu, Zhihao Luo, Shanzhe Lei, Lechao Zhang, Xuhong Wang, JIE YANG, Zhonglong Zheng, Yuanjie Zheng, Xin Tan, Wei Liu · 2026
Read GROW: Aligning GRPO with State-Action Modeling for Open-World VLM Agents by Xiongbin Wu, Zhihao Luo, Shanzhe Lei, Lechao Zhang, Xuhong Wang, JIE YANG, Zhonglong Zheng, Yuanjie Zheng, Xin Tan, Wei Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-20246.txt.
Introduction
Xiongbin Wu and nine co-workers present GROW, a framework for open-world vision-language agents. Vision-language agents show promise on open-world tasks, but supervised fine-tuning with demos dominates, while group relative policy optimization needs full trajectories, giving long noisy context. GROW splits trajectories into state-action samples and compares advantages across samples. An analysis shows grouped samples keep the relative optimization signal under simplifying assumptions, despite different local states. Tests on over 800 Minecraft tasks show top performance.
Catalog record
| CREATOR | Xiongbin Wu, Zhihao Luo, Shanzhe Lei, Lechao Zhang, Xuhong Wang, JIE YANG, Zhonglong Zheng, Yuanjie Zheng, Xin Tan, Wei Liu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48617 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.20246
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
GROW: Aligning GRPO with State-Action Modeling for Open-World VLM Agents Xiongbin Wu, Zhihao Luo, Shanzhe Lei, Lechao Zhang, Xuhong Wang, JIE YANG, Zhonglong Zheng, Yuanjie Zheng, Xin Tan, Wei Liu Abstract Recently, vision-language model (VLM) agents have shown promising progress in open-world tasks, where successful task completion often requires multiple turns of visual perception and action execution. However, existing methods still rely primarily on Supervised Fine-Tuning (SFT) with expert demonstrations, while the advanced reinforcement learning (RL) algorithm, specifically Group Relative Policy Optimization (GRPO), has not been effectively employed for multi-turn RL in these tasks because standard GRPO requires full trajectories as training samples which leads to excessively long context and noise. To address this issue, we propose GROW, a RL framework for open-world VLM agents that decomposes collected trajectories into state-action samples, and computes advantages between these samples rather than treating a full trajectory as a single entity. We further provide a surrogate analysis indicating that, even though the grouped samples are conditioned on different local states rather than an identical prompt context, the objective can preserve the core relative policy optimization signal of GRPO under simplifying assumptions. Experiments on more than 800 Minecraft tasks show that our method achieves state-of-the-art (SOTA) performance, demonstrating the effectiveness of our proposed RL framework for open-world VLM agents. 1 Introduction VLM agents have become increasingly capable in open-world environments [ 20 , 15 , 9 , 29 ] . In these domains, an agent must repeatedly interpret visual states, choose actions, and adapt its behavior through multi-turn interaction with the environment. Most recent efforts improve such agents through SFT on expert demonstrations [ 16 , 23 ] , enabling them to imitate task-relevant perception-action behaviors from curated demonstrations. However, SFT relies on large amounts of high-quality expert data, whose collection is often expensive and difficult to scale. Moreover, prior studies [ 19 , 25 , 27 ] have shown that SFT alone can lag behind RL-trained VLM agents in performance. These limitations motivate the need for advanced RL methods that can effectively train open-world VLM agents through interaction. Recent researchs on VLM [ 13 , 8 , 7 , 27 , 25 , 19 ] which use GRPO [ 17 ] as their RL algorithms have shown the effectiveness of GRPO in improving foundation-model policies through group-wise relative optimization. By comparing multiple sampled outputs within the same group, GRPO constructs relative advantages without training an additional value model, making it especially suitable for large-scale VLM optimization where value estimation can be costly and unstable. These strengths make GRPO a natural algorithmic basis for refining VLM agents through environment interaction. However, directly transferring GRPO to open-world tasks is nontrivial. Standard GRPO compute advantages across trajectories conditioned on the same prompts and optimize these full trajectories. As shown in Figure 1 , it can introduce excessively long context and also include too much noise in the context when full trajectories are used to predict actions in open-world tasks. Figure 1 : Context length in the trajectory increases with the number of interaction steps between the VLM agent and the environment, often exceeding the maximum token length as interactions accumulate. To address this issue, we propose G R PO for o pen- w orld VLM agents (GROW), a RL framework that adapting GRPO to open-world tasks where trajectories are often too long and agents make decisions often based on short-horizon states. GROW first applies trajectory decomposition to collected rollouts trajectories and then compute relative advantages among the state-action samples within the same rollout groups. However, this reformulation also introduces a theoretical issue that does not arise in standard GRPO. After trajectory decomposition, grouped samples are no longer different responses to the same prompt. At this, we provide a surrogate analysis under simplifying assumptions shows that the proposed objective can still provide an effective relative policy optimization signal. Our main contributions are summarized as follows: • We propose GROW, a RL framework for open-world VLM agents. The framework combines cold-start training on state-action samples with GRPO-based policy refinement through trajectory decomposition. • We provide a surrogate analysis suggesting that, under reasonable tractability approximations, relative policy optimization remains effective in GROW even when grouped samples are conditioned on different local states rather than an identical prompt context. • We instantiate and test the GROW mainly in Minecraft [ 12 , 9 , 2 ] . Across more than 800 Minecraft tasks, ranging from embodied spatial navigation and precise GUI manipulation to highly dynamic combat scenarios, our method achieves SOTA performance demonstrating that GROW establishes a new SOTA in both success rate and execution efficiency. Notably, our framework exhibits strong generalization to unseen tasks and fosters sophisticated behavioral skills, such as active target reacquisition and distractor-robust GUI operation , proving its effectiveness in learning reusable interaction strategies rather than merely memorizing trajectories. 2 Related Work 2.1 Agents in the Open World Open-world tasks provide an important testbed for developing general-purpose agents because they require agents to perceive visual observations, reason about dynamic environments, and execute actions over extended interaction sequences. Minecraft is a representative example due to its high degree of freedom and broad task diversity, spanning embodied control, resource manipulation, and precise graphical user interface interactions. Following the seminal work of VPT [ 3 ] which leverages large-scale expert demonstrations and explores RL for fine-tuning, subsequent studies have advanced agent learning in several directions. Building upon VPT [ 3 ] , STEVE-1 [ 11 ] is trained for text-to-behavior generation in Minecraft, enabling users to use text instructions to control agents for completing short-horizon, open-ended tasks relying on raw pixels and low-level controls. ROCKET-1 [ 5 ] introduces visual-temporal context prompting to connect high-level VLM reasoning with low-level policy execution for spatially grounded interaction. ROCKET-3 [ 4 ] further improves exploration in unseen environments through RL with cross-view reasoning. More recently, research has increasingly focused on VLM agents. For example, JARVIS-VLA [ 9 ] adopts staged training to improve task completion in Minecraft, while similar imitation-based VLM agents have also shown effectiveness in other game environments such as Genshin Impact [ 20 ] and Steam games [ 15 ] . Despite this progress, efficient RL methods for VLM agents remain underexplored, as most existing approaches still depend heavily on imitation learning. Our work addresses this gap by providing a scalable RL framework for training open-world VLM agents. 2.2 GRPO for Multi-Turn VLM Agents GRPO has been widely explored for RL in tasks requiring many rounds of interaction. AgentGym-RL [ 24 ] studies multi-turn RL for large language model agents and improves long-horizon decision making with a curriculum over interaction length. InquireMobile [ 1 ] and ColorAgent [ 10 ] extend this paradigm to settings where the agent must request authorization before taking actions or incorporate human instructions during execution. AGENTRL [ 26 ] further investigates system-level scheduling and resource allocation for GRPO-based multi-turn training in order to improve training efficiency. However, although these works substantially broaden the study of GRPO in multi-turn settings, they mainly retain a trajectory-level or dialogue-level formulation, where each optimization sample may contain an increasingly long interaction history. When directly applied to open-world tasks, such full-trajectory samples can introduce substantial irrelevant noise and lead