Thinking with Patterns: Breaking the Perceptual Bottleneck in Visual Planning via Pattern Induction

Yichang Jian, Boyuan Xiao, Zhenyuan Huang, Yifei Peng, Yao-Xiang Ding · 2026

Read Thinking with Patterns: Breaking the Perceptual Bottleneck in Visual Planning via Pattern Induction by Yichang Jian, Boyuan Xiao, Zhenyuan Huang, Yifei Peng, Yao-Xiang Ding (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-16848.txt.

Introduction

Jian and colleagues tackle visual planning with vision-language models, where raw visual input exceeds one-step perception. Following image-thinking, they treat planning as building and reflecting a world model from local evidence. A training-free strategy lets models solve tasks beyond initial reach, but many image-thinking steps raise compute cost. Pattern Inference lets models spot patterns in new tasks and infer world-model structure. Pattern Induction discovers and optimizes such patterns as experts. Tests on FrozenLake, Crafter, and CubeBench show balanced accuracy and efficiency.

Catalog record

CREATORYichang Jian, Boyuan Xiao, Zhenyuan Huang, Yifei Peng, Yao-Xiang Ding
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE63992
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.16848

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Thinking with Patterns: Breaking the Perceptual Bottleneck in Visual Planning via Pattern Induction Yichang Jian, Boyuan Xiao, Zhenyuan Huang, Yifei Peng, Yao-Xiang Ding Abstract Planning from raw visual input remains a significant challenge for current Vision-Language Models (VLMs), when the complexity of input is beyond their one-step perception capability. Motivated by recent advances in Thinking with Images (TWI), a reasonable solution is to decompose the perception process into simpler steps by iteratively acquiring and incorporating local visual evidence. However, even though current VLMs are well-trained in general TWI ability, their perceptual bottleneck in the planning domain remains. To tackle this challenge, we formulate TWI as a tool to gradually build and reflect an accurate internal world model. We find that the resulting training-free planning strategy enables VLMs to solve tasks that are far beyond their initial capabilities, at the cost that too many TWI operations would significantly increase the computational overhead. To further improve efficiency, we propose Pattern Inference , a novel TWI strategy enabling VLMs to actively recognize known visual patterns in the new tasks and directly infer local world model structures. To obtain these patterns, we propose Pattern Induction , an online inductive learning strategy treating visual patterns as composite and reusable experts, which are autonomously discovered and optimized from experience. Experimental evaluations in FrozenLake , Crafter and CubeBench domains show that our approaches achieve a desirable balance between accuracy and efficiency. Project Page: https://future-item.github.io/PI-TWI . 1 Introduction Due to remarkable reasoning abilities, Vision-Language Models (VLMs) have shown promising potentials in visual planning tasks ( Driess et al. 2023 ; Mei et al. 2024 ; Feng et al. 2025 ; Zhao et al. 2025 ) . However, we observe that current VLMs could still struggle in planning domains with raw visual inputs, especially those with complicated relationships among a large number of target objects to be discovered and understood. In our experiments, we show that current VLMs can fail in challenging planning tasks even when provided with ground-truth planners. This suggests a major bottleneck beyond planning ability: The failure essentially lies in the fact that the complexity of input is beyond the capability of VLMs to perceive at once. To tackle this perceptual bottleneck, we turn to recent advances in Thinking with Images (TWI) ( OpenAI 2025 ; Su et al. 2025b ) . This emerging paradigm is characterized by models leveraging visual information as intermediate steps in their thought process by generating intermediate visual cues in the original input, or even predicting purely with visual reasoning traces ( Luo et al. 2025 ; Wu et al. 2025 ; Li et al. 2025a ; Fu et al. 2025 ; Duan et al. 2025 ; Xu et al. 2026 ) . The benefit of TWI could include various aspects, while one is particularly related to our problem, such that TWI can be utilized to break the visual perception process of VLMs into simpler steps ( Liu et al. 2024 ) . The key idea lies in iteratively simplifying the visual scenes during reasoning via image modification operations, making the VLMs attend to key aspects related to the current reasoning step. Through this strategy, the one-step perceptual bottleneck is significantly alleviated without directly enhancing the native visual perception ability of VLMs. It is natural to think that the similar idea can also be utilized in planning. Unfortunately, we observe that even though current VLMs are well-trained in general TWI ability, their perceptual obstacle in the planning domain remains (See Table 1 ). To further explore potential solutions, we propose a novel formulation of TWI specially designed for planning tasks. In this formulation, we assume that VLMs are already armed with ground-truth planners in each task domain, functioning only as a tool to gradually build and reflect an accurate internal world model for running the planners. Specifically, we propose a training-free planning strategy that functions as follows: At the beginning, VLM is asked to build a noisy world model from global visual input. During the planning process, VLMs utilize TWI operators to iteratively attend to local parts of the visual input and conducts reflection on the correctness of the local world model structures. We find that given unlimited budget of TWI operations, VLMs are able to solve tasks that are far beyond their initial capabilities. On the other hand, we also recognize a fundamental potential issue: Too many TWI operations would significantly increase the computational overhead, which is measured by token usage in this paper. To further improve efficiency, we propose Pattern Inference , a novel TWI strategy centered on the concept of visual patterns . A visual pattern represents a specific regularity for a local part of the visual input that (1) appears frequently among all inputs from one domain; (2) has a critical impact on the planning results. Assuming that VLMs keep a well-constructed library of such reusable and composite patterns, during planning, they can utilize the matching operation to actively recognize known visual patterns in all tasks in the domain and directly infer local world model structures. This operation has much lower cost than the naive structure reflection TWI operation, hence can be utilized to effectively address the previous concern on efficiency. To build such pattern libraries, we propose Pattern Induction , an inductive learning strategy inspired by recent advances in neuro-symbolic inductive learning ( Shindo et al. 2023 ; Shindo et al. 2024 ; Piriyakulkij et al. 2025 ) . Based on the mixture of experts (MoE) ( Jacobs et al. 1991 ) and self-supervised learning by random masking ( He et al. 2022 ) frameworks, the method establishes an online learning strategy that continuously proposes new patterns by VLMs, reweights them using masked past trajectories, and exploits them in pattern inference to improve the efficiency of TWI for planning, allowing visual patterns to be autonomously discovered, optimized, and utilized from experience. To summarize, we frame our paper as a proof-of-concept work, targeting at the justification of the following arguments: (1) Formulating TWI as a tool to gradually build and reflect an accurate internal world model could enable VLMs to effectively break their perceptual limitations in planning; (2) Visual patterns could be utilized as reusable and composite tools to significantly reduce the cost of building the internal world model; (3) Critical visual patterns could be autonomously constructed by VLMs based on the inductive online learning framework. We pack our approaches as Pattern-Induced Thinking with Images ( PI-TWI ) and evaluate it under three challenging visual planning domains, FrozenLake , Crafter , and CubeBench . The results show that our approaches achieve a desirable balance between accuracy and efficiency, empirically justifying the above arguments. 2 Related Work 2.1 Planning with VLMs VLM-based planning has evolved from grounding language goals in visual observations to increasingly structured forms of long-horizon reasoning. Early embodied multimodal models, such as PaLM-E ( Driess et al. 2023 ) , show that visual observations and language instructions can be integrated to support high-level planning for sequential robotic manipulation. Later approaches introduce closed-loop visual feedback, enabling planners such as ReplanVLM ( Mei et al. 2024 ) to detect execution failures and revise plans accordingly. More recent methods further move toward world-model-style reasoning: Reflective Planning ( Feng

Read the complete text