VLM Q-Learning: Aligning Vision-Language Models for Interactive Decision-Making
Jake Grigsby, Yuke Zhu, Michael Ryoo, Juan Carlos Niebles · 2025
Read VLM Q-Learning: Aligning Vision-Language Models for Interactive Decision-Making by Jake Grigsby, Yuke Zhu, Michael Ryoo, Juan Carlos Niebles (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-03181.txt.
Introduction
Jake Grigsby and three co-authors train vision-language models to act as agents through reinforcement learning, in a 2025 paper. Agents must follow strict output formats and act step by step in interactive settings. Open vision models lag behind text models on those skills. Standard tuning needs expert demonstrations, which are costly.
Their method starts offline and moves online, learning from failed attempts as well as good ones. It keeps the stability of the standard tuning workflow. Only the model's own errors, or those of larger models, supply the lessons. Two open vision models were tested across three agent domains. Those domains need multi-modal input and strict action formats. The technique let agents self-improve from low-quality data sets. The authors present it as a practical route to capable visual agents.
Catalog record
| CREATOR | Jake Grigsby, Yuke Zhu, Michael Ryoo, Juan Carlos Niebles |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-03181
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
VLM Q-Learning: Aligning Vision-Language Models for Interactive Decision-Making Jake Grigsby, Yuke Zhu, Michael Ryoo, Juan Carlos Niebles Abstract Recent research looks to harness the general knowledge and reasoning of large language models (LLMs) into agents that accomplish user-specified goals in interactive environments. Vision-language models (VLMs) extend LLMs to multi-modal data and provide agents with the visual reasoning necessary for new applications in areas such as computer automation. However, agent tasks emphasize skills where accessible open-weight VLMs lag behind their LLM equivalents. For example, VLMs are less capable of following an environment’s strict output syntax requirements and are more focused on open-ended question answering. Overcoming these limitations requires supervised fine-tuning (SFT) on task-specific expert demonstrations. Our work approaches these challenges from an offline-to-online reinforcement learning (RL) perspective. RL lets us fine-tune VLMs to agent tasks while learning from the unsuccessful decisions of our own model or more capable (larger) models. We explore an off-policy RL solution that retains the stability and simplicity of the widely used SFT workflow while allowing our agent to self-improve and learn from low-quality datasets. We demonstrate this technique with two open-weight VLMs across three multi-modal agent domains. 1 Introduction Diverse training datasets give large language models (LLMs) the ability to generate knowledgeable text across a wide range of topics. LLM pre-training produces a model that outputs the most likely continuation of its input text according to a distribution of training examples sourced from across the web. However, there are many applications where we would like to direct our model to accomplish a specific objective. In these tasks, our model acts as an agent in an interactive environment and makes a sequence of decisions to maximize an evaluation metric. Because the model was not trained to optimize this objective, the best decision-making strategy may be an unlikely output. Ongoing research looks to fix this alignment problem and repurpose generative models for decision-making. LLM-agent techniques are largely enabled by the post-training process, where the model is fine-tuned on curated datasets of chat or assistant-like behavior. A natural first step is to continue training with supervised fine-tuning (SFT) on task-specific datasets. However, SFT cannot outperform the best decision-making strategy in its training data and relies on collecting successful demonstrations. LLMs’ ability to adapt to new tasks based on examples in their input ( Brown et al. 2020 ) allows this realignment to be done without further training: agent behavior can be generated by prompting an LLM with demonstrations. Additional prompting techniques improve long-term planning by eliciting self-reflection and reasoning ( Yao et al. 2022b ; Wei et al. 2022b ) . Vision-language models (VLMs) extend LLMs to multi-modal inputs. VLMs are trained by using smaller datasets of interleaved text and image data to mix visual features into an existing LLM ( Liu et al. 2024b ; Liu et al. 2024a ) . The high-level problem of realigning VLMs with decision-making objectives is similar to that of LLM agents. However, VLM research is still in its early stages, and current models lag behind their LLM equivalents in two critical areas: 1) agent action syntax and 2) long-context prompting. Agent tasks often do not involve freeform dialogue and instead require outputs with a specific syntax that the environment can interpret. The ability to plan and act in this more restricted output space is closely related to the tool-use (or function-calling) capabilities of LLMs ( Qu et al. 2024 ) . Perhaps a more long-term issue is that environment interactions can stretch over many model outputs, making it impractical to include complete in-context examples and align the VLM via prompting. SFT can realign a base model with agent tasks, but it is only effective when we can demonstrate a strategy that meets our performance expectations. Figure 1: VLM-Environment Interaction. We format text and image data from the environment into an input prompt for the VLM. The models’s text reply is parsed into a concrete action executed by the environment to produce a reward and new inputs for the VLM. Our work approaches the challenges of fine-tuning VLMs for agent tasks from a reinforcement learning (RL) perspective (Figure 1 ). RL solves the agent alignment issue by explicitly training our model to maximize success in its environment. Decision-making is often formalized as an RL problem, but RL methods have a reputation for adding complexity; simpler prompting and SFT techniques are effective for LLM-agent tasks and are more widely used in recent work. However, we note that a category of offline RL techniques ( Levine et al. 2020 ) that filter demonstration datasets according to a learned value model ( Wang et al. 2020 ) have important advantages in the emerging area of VLM agent training. Like most offline RL algorithms, these filtered supervised fine-tuning (FSFT) methods let our agent outperform its demonstrations. Unlike other offline techniques, FSFT is effectively identical to SFT when the dataset is of high quality and lets us seamlessly collect new data when our agent needs further improvement. This lets FSFT serve as a low-downside, high-upside replacement for SFT in VLM agent domains where SFT is widely used. Our work explores the challenges of applying VLMs to agent tasks and demonstrates the strengths of off-policy RL applied to two open-weight models across three domains. 2 Background Vision-Language Models. VLMs extend the capabilities of LLMs to visual data by taking text and images as input and producing text outputs. They process multi-modal inputs by projecting the features of a vision encoder into the embedding space of an LLM — allowing the LLM to attend to the features of both modalities. While there have been many VLM models released in recent months, our work will focus on publicly available (open-weight) models that are small enough to be fine-tuned with accessible GPU resources. Table 1 introduces three models used in our experiments: PaliGemma ( Beyer et al. 2024 ) , xGen-MM ( Xue et al. 2024 ) , and MoonDream2 ( Korrapati 2024 ) . MoonDream2 PaliGemma xGen-MM Parameters 1.9B 2.9B 4.6B Feature Dim. 2048 2048 3072 Vocabulary 51,200 257,216 32,015 Table 1: VLMs in Our Experiments. . Foundation Models as Agents. We focus on agent applications where a model is evaluated based on its ability to make decisions that achieve a specific outcome. Examples include: controlling an operating system with text commands ( Wu et al. 2024 ) , writing code to answer data analysis questions ( Hu et al. 2024 ) , navigating graphical interfaces, ( Gao et al. 2023 ; Tao et al. 2023 ; Xie et al. 2023 ) , ordering items from a web store ( Yao et al. 2022a ; Tao et al. 2023 ) , carrying out administrative tasks in business software ( Drouin et al. 2024 ) , making PowerPoint slides based on a specific topic ( Guo et al. 2023 ) , and playing games ( Chen et al. 2024a ) , among many others. These tasks are characterized by multi-turn interactions , where the model iteratively reacts to changes in the environment until it is successful. Success is measured by explicit task-specific objectives and not by the likelihood of the model’s response or subjective human preferences. This creates a fundamental mismatch between