GaLa: Hypergraph-Guided Visual Language Models for Procedural Planning

Kun Wang, Yiming Li, Mingcheng Qu, Aqiang Zhang, Guang Yang, Tonghua Su · 2026

Read GaLa: Hypergraph-Guided Visual Language Models for Procedural Planning by Kun Wang, Yiming Li, Mingcheng Qu, Aqiang Zhang, Guang Yang, Tonghua Su (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-17241.txt.

Introduction

Wang and five colleagues present GaLa, a vision-language system for step-by-step planning in robots. It models object items as nodes in a graph, with region-level links grouping objects by traits and use-based meaning, showing hidden links and the ranking of use-based regions. A TriView HyperGraph Encoder keeps meaning match across node, area, and node-area link views via contrast learning, feeding graph meaning into downstream vision-language thinking. On ActPlan1K and ALFRED, GaLa beats methods on task success rate, LCS, and planning scores.

Catalog record

CREATORKun Wang, Yiming Li, Mingcheng Qu, Aqiang Zhang, Guang Yang, Tonghua Su
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30992
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.17241

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

GaLa: Hypergraph-Guided Visual Language Models for Procedural Planning Kun Wang, Yiming Li, Mingcheng Qu, Aqiang Zhang, Guang Yang, Tonghua Su Abstract Implicit spatial relations and deep semantic structures encoded in object attributes are crucial for procedural planning in embodied AI systems. However, existing approaches often over-rely on the reasoning capabilities of vision language models (VLMs) themselves, while overlooking the rich structured semantic information that can be mined from multimodal inputs. As a result, models struggle to effectively understand functional spatial relationships in complex scenes. To fully exploit implicit spatial relations and deep semantic structures in multimodal data, we propose GaLa, a vision–language framework for multimodal procedural planning. GaLa introduces a hypergraph-based representation, where object instances in the image are modeled as nodes, and region-level hyperedges are constructed by aggregating objects according to their attributes and functional semantics. This design explicitly captures implicit semantic relations among objects as well as the hierarchical organization of functional regions. Furthermore, we design a Tri-View HyperGraph Encoder that enforces semantic consistency across the node view, area view, and node–area association view via contrastive learning, enabling hypergraph semantics to be more effectively injected into downstream VLM reasoning. Extensive experiments on the ActPlan-1K and ALFRED benchmarks demonstrate that GaLa significantly outperforms existing methods in terms of execution success rate, LCS, and planning correctness. 1 Introduction In recent years, research in procedural planning has significantly advanced the capability of embodied agents to execute complex instructions by understanding multimodal information 38 ; 15 ; 36 . Procedural planning refers to the process of progressively decomposing high-level linguistic commands into a sequence of precise actions required to accomplish a task 39 , ultimately enabling embodied agents to perform task planning and execution in real-world environments 11 . During this process, textual instructions enable the agent to effectively understand the required task 49 , while visual inputs allow the agent to perceive its surroundings with precision 20 . Through multimodal fusion 47 , the embodied agent is ultimately able to execute the specified instructions. Figure 1: On the left, when the hypergraph is introduced, the deep semantic information contained in the visual data of the room is extracted, enabling the execution of the procedural planning process. In the middle, the abstraction process of building the hypergraph is shown. On the right, without the introduction of the hypergraph, the procedural planning results in a logical deadlock. Recent research in procedural planning can be categorized into two main streams based on the modality they primarily rely on: Large Language Models (LLMs) 10 ; 32 ; 17 and Vision-Language Models (VLMs) 24 ; 48 ; 46 . VLM-based methods, by contrast, integrate visual observations with language input, enabling improved spatial perception and grounding. This capability allows VLMs 45 to generate action plans that better respect physical constraints in the environment. Despite this advantage, existing VLM-based approaches predominantly operate on explicit spatial representations, such as bounding boxes or object coordinates, and focus primarily on image-level semantic cues. As a result, they remain limited in capturing deeper semantic structures that are implicitly embedded in visual scenes. In particular, two critical limitations emerge. First, implicit spatial relations conveyed through object attributes—such as "can-be-placed-on", "is-supported-by" are often overlooked, as they are not explicitly encoded in geometric representations. Second, hierarchical semantics arising from groups of objects forming functional regions (e.g., "dining area", "kitchen area") are rarely modeled in a structured manner. Consequently, the agent’s understanding of the scene is reduced to a collection of isolated objects, which can lead to suboptimal or even degenerate planning behaviors, such as repetitive or oscillatory actions in ambiguous contexts (as illustrated in Figure 1 ). This phenomenon raises a critical question: Are current approaches overly reliant on the explicit spatial reasoning capabilities of VLMs, while neglecting the opportunity to mine deeper semantic structures from images? To address this challenge, we propose GaLa, a framework that introduces a hypergraph-based representation as an intermediate semantic abstraction between vision and language. Rather than treating spatial reasoning as purely geometric, GaLa models the scene as a hypergraph that explicitly captures both object-level semantic attributes and area-level functional organization. Specifically, GaLa constructs a semantic hypergraph in which nodes correspond to individual object instances, while hyperedges represent functional or spatial regions inferred from object attributes (e.g., "dining area", "kitchen area"). By aggregating object-level semantics into structured area-level representations, the hypergraph elevates implicit spatial relations—originally latent in textual and semantic cues—into an explicit, reasoning-friendly form. To effectively integrate this structured knowledge into downstream reasoning, we further introduce a "Tri-View HyperGraph Encoder", which employs contrastive learning to enforce semantic consistency across three complementary views: object-level (node-view), region-level (area-view), and object–region association (all-view). This tri-view objective encourages the model to learn robust representations that preserve both fine-grained object semantics and higher-order relational structure, enabling more coherent and spatially grounded procedural planning. Through this design, GaLa provides VLMs with richer semantic context and mitigates common planning failures arising from ambiguous or under-structured scene representations. Experimental validation on the ActPlan-1k benchmark dataset shows that the GaLa model achieves significant performance improvements. The results indicate that, thanks to the explicit modeling of implicit spatial relations and object cluster semantics via the hypergraph, our framework can guide the VLM to generate action sequences that are more spatially accurate and semantically consistent, demonstrating superior performance in complex procedural planning tasks. Our key contributions are summarized as follows: •We propose GaLa, the first framework that introduces graph-theoretic structural semantic information into a VLM-based procedural planning architecture. •We design a HyperGraph Semantic Encoder that models visual information to construct a hypergraph enriched with semantic information, thereby enhancing the structural semantics inherent in the image. •We introduce a Tri-View HyperGraph Encoder that employs a contrastive learning approach to ensure that the hypergraph information is fully preserved and effectively transmitted into the VLM. Figure 2: We present the model pipeline for GaLa. In Step 1 , we initially build the hypergraph semantic information. In Step 2 , we optimize the hypergraph information for CoT using the semantic information from Step 1 and the Tri-View HyperGraph. In Step 3 , we train the constructed knowledge, enabling the model to predict the next action more accurately. Figure 3: The detailed architecture of Step 2 is illustrated in Figure 2. The upper part depicts the process of Hypergraph Semantic Construction, while the lower-left part shows the Tri-View HyperGraph module, and the lower-right part presents the resulting HyperGraph Knowledges. The entire pipeline is organized as a chain-of-thought (CoT) process, which progressively refines and optimizes the HyperGraph Knowledges. 2 Related Work 2.1 Vision-Language Models (VLMs) VLMs jointly model visual and linguistic information and are widely used in multimodal tasks such as image captioning 6 , visual question answering 2 ; 19 , and image–text retrieval 22 ; 4 ; 26 . Early contrastive methods, including CLIP 28 and ALIGN 13 , established large-scale image–text alignment. Modern VLMs integrate visual features into LLM pipelines with powerful backbones (GPT-style, LLaMA-family and Qwen-series 44 ) and visual instruction tuning. Representative methods include Flamingo 1 , BLIP-2 16 , and LLaVA 21 , using different mechanisms to inject visual information. Despite strong performance, most VLMs treat images as unstructured tokens, limiting explicit modeling of object relations and higher-order structure, motivating

Read the complete text