Vision Language Models Cannot Plan, but Can They Formalize?

Muyu He, Yuxi Zheng, Yuchen Liu, Bill Cai, Jiani Huang, Feng Liu, Ziyang Li, Li Zhang, {harry.zhang}@drexel.edu · 2025

Read Vision Language Models Cannot Plan, but Can They Formalize? by Muyu He, Yuxi Zheng, Yuchen Liu, Bill Cai, Jiani Huang, Feng Liu, Ziyang Li, Li Zhang, {harry.zhang}@drexel.edu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-21576.txt.

Introduction

Long horizon planning defeats vision language models that must emit action sequences directly. Muyu He and 9 co-authors test a different role for those models in a 2025 paper. In text settings, language models translate problems into a formal planning language for a solver. That formalizer role improved long horizon results. Multimodal versions remain scarce and often rely on gross simplifications.

Five pipelines for vision language model formalization are presented. They tackle one shot, open vocabulary, multimodal formalization into PDDL, the Planning Domain Definition Language. Past work used predefined object vocabularies or overly similar examples. Evaluation runs on an existing benchmark. The suite tests whether models that cannot plan can still formalize problems for verifiable solvers.

Catalog record

CREATORMuyu He, Yuxi Zheng, Yuchen Liu, Bill Cai, Jiani Huang, Feng Liu, Ziyang Li, Li Zhang, {harry.zhang}@drexel.edu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-21576

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Vision Language Models Cannot Plan, but Can They Formalize? Muyu He, Yuxi Zheng, Yuchen Liu, Bill Cai, Jiani Huang, Feng Liu, Ziyang Li, Li Zhang, {harry.zhang}@drexel.edu Abstract The advancement of vision language models (VLMs) has empowered embodied agents to accomplish simple multimodal planning tasks, but not long-horizon ones requiring long sequences of actions. In text-only simulations, long-horizon planning has seen significant improvement brought by repositioning the role of LLMs. Instead of directly generating action sequences, LLMs translate the planning domain and problem into a formal planning language like the Planning Domain Definition Language (PDDL), which can call a formal solver to derive the plan in a verifiable manner. In multimodal environments, research on vlm-as-formalizer remains scarce, usually involving gross simplifications such as predefined object vocabulary or overly similar few-shot examples. In this work, we present a suite of five vlm-as-formalizer pipelines that tackle one-shot, open-vocabulary, and multimodal PDDL formalization. We evaluate those on an existing benchmark while presenting another two that for the first time account for planning with authentic, multi-view, and low-quality images. We conclude that vlm-as-formalizer greatly outperforms end-to-end plan generation. We reveal the bottleneck to be vision rather than language, as VLMs often fail to capture an exhaustive set of necessary object relations. While generating intermediate, textual representations such as captions or scene graphs partially compensate for the performance, their inconsistent gain leaves headroom for future research directions on multimodal planning formalization. 1 1 1 Code and data are at https://github.com/RiddleHe/vllm_as_formalizer 1 Introduction Embodied planning has seen impressive advances in the last few years, especially with the rise of vision language models (VLM) and vision language action models (VLA). The standard setup involves giving the model interleaved vision-language inputs, such as images or videos and a natural language instruction, and expecting the model to predict a sequence of actions that logically form a plan to achieve the specified goal. While either VLM or VLA models may directly output high-level or low-level actions, their performance is limited in long-horizontal planning with little to no interpretability ( Liu et al. 2023 ; Yang et al. 2025b ) . A mainstream alternative is a hierarchical pipelining approach that involves multiple models for detecting objects, extracting relations, predicting task-level actions, and translating to motion-level actions ( Yenamandra et al. 2023 ; Wang et al. 2024a ) . While modular, such approach requires training for each modules in specific domains, thus lacking few-shot generalization abilities in an open-vocabulary setting. Large language models (LLM), the basis of VLM and VLA, have similarly driven great progress in textual planning with two leading paradigms under few-shot settings. Given a textual description of the environment and the goal, LLM-as-planner directly generates the actions ( Wei et al. 2025 ) . In addition to mixed performance, this approach offers little interpretability and verifiability. Alternatively, LLM-as-formalizer instead generates a formal language like the Planning Domain Definition Language (PDDL) which can be input into a symbolic solver to derive a plan deterministically ( Tantakoun et al. 2025 ) . Powered by pre-trained LLM’s strong in-context learning skills and code generation ability, this approach has shown promising performance while offering some formal guarantee that is crucial for high-stakes domains. Despite the emerging success of LLM-as-formalizer, application to VLM has been understudied in multimodal planning environments, with some incomplete attempts leveraging non-visual cues ( Li et al. 2025 ; Kwon et al. 2025 ) or close-vocabulary, few-shot examples ( Herzog et al. 2025b ) . As a result, such explorations are limited to particular tasks and lack generality ( Jenamani et al. 2025 ) . We advance vlm-as-formalizer as an effective paradigm on long-horizon, one-shot, open-vocabulary, visual-language planning tasks. We are the first to systematically evaluate its strengths and weaknesses. To do so, we consider 2 VLMs and design 5 vlm-as-formalizer pipelines including generating a detailed caption or scene graph as an intermediate step to PDDL formulation, compared with one VLMs-as-planner baseline. We evaluate each method over three vision-based datasets that drastically differ in difficulty and realism. On top of one existing visual Blocksworld dataset by Shirai et al. 2024b that is small-scale, simulated, and fully observable via one single image, we propose a novel challenge dataset, Blocksworld-Real , based on real images captured by sensors of physical robots. For each planning problem, we provide multiple photos from ego-centric viewpoints, challenging the model to identify and track objects across perspectives. These photos also closely resemble real-world visual conditions by including occlusion, motion blur, discoloration, and noisy background. We derive a similar multi-view planning dataset from ALFRED ( Shridhar et al. 2020a ) , where the planning problem is described by multiple rendered images. Our results position vlm-as-formalizer as a much stronger and more generalizable paradigm than end-to-end planning for long-horizon, visual-language planning. Even so, we attribute the still significant headroom to the VLMs’ major weakness in visual detection rather than code generation, failing to capture an exhaustive set of objects and relations. While intermediate representations such as captions or scene graphs help, their inconsistent gain suggests future efforts. 2 Problem Formulation 2.1 Formal Definition Formally, we define the Vision-PDDL-Planning task in line with established literature of STRIPS ( Fikes & Nilsson 1971 ) as follows (Figure 1 ). The input consists of a triplet (V,I,\mathcal{D}) , where: 1. V={v_{1},\ldots,v_{n}} is a sequence of n images, with each image v_{i} representing (possibly partial) observations of the initial environment; 2. I is a natural language instruction specifying the goal; 3. \mathcal{D} is a PDDL domain file, which formally defines the planning environment. The domain file \mathcal{D} provides the type system for entities, a set of relational predicates R , and a set of parameterized actions A . Each action is specified by its preconditions and effects, both expressed as conjunctive logical formulas over the predicates. The final objective of the Vision-PDDL-Planning task is to generate a plan L=[a_{1}(\bar{e}_{1}),\ldots,a_{m}(\bar{e}_{m})] , where each a_{i}\in A is an action schema defined in the domain \mathcal{D} , and each \bar{e}_{i} is a tuple of grounded entities corresponding to the parameters of a_{i} . The plan L , when executed in the environment represented by the images V , must achieve the goal specified by the instruction I . While one could imagine an end-to-end model that maps the input triplet (V,I,\mathcal{D}) directly to a plan L , in this work we focus on a modular paradigm: VLM-as-Formalizer . Specifically, this involves first generating a PDDL problem file \mathcal{P} , which encodes the initial state and goal extracted from V and I , and then employing a PDDL solver to compute a valid plan. At a high level, the problem file \mathcal{P} is defined by three main components: objects ( E ), the initial state ( s_{0} ), and the goal state ( \psi{\texttt{goal}} ): 1. Objects: E={e_{1},\ldots,e_{n}} is the set of named entities present in the environment. Each object is an instance of an entity type defined in \mathcal{D} . 2. Initial State: s_{0}\in\mathcal{S} is a set of relational facts of the form r(\bar{e}) , where r\in R is a relational predicate defined in \mathcal{D} ; \bar{e} is a tuple of entities from E that participate in relation r . 3. Goal State: \psi_{\texttt{goal}}:\mathcal{S}\rightarrow\mathbb{B} is a boolean

Read the complete text