Thinking with Programming Vision: Towards a Unified View for Thinking with Images
Zirun Guo 1,2 Minjie Hong 1 Feng Zhang 2 Kai Jia 2 Tao Jin 1 1 Zhejiang University 2 ByteDance, BandAI · 2025
Read Thinking with Programming Vision: Towards a Unified View for Thinking with Images by Zirun Guo 1,2 Minjie Hong 1 Feng Zhang 2 Kai Jia 2 Tao Jin 1 1 Zhejiang University 2 ByteDance, BandAI (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-03746.txt.
Introduction
Multimodal models that think with images can call tools to inspect a picture while they reason. Zirun Guo and colleagues at Zhejiang University and ByteDance show in a 2025 preprint that even leading models stumble on rotated or corrupted images. Tool sets offered to such models also tend to be narrow.
Their framework, CodeVision, replaces a fixed tool registry with code itself: the model writes programs that can invoke any image operation. Training begins with supervised fine-tuning on curated examples of multi-step tool use and recovery from errors. Reinforcement learning follows, guided by a dense process reward that favours strategic, efficient tool calls. The authors built fresh training sets and a harder benchmark stressing orientation changes and multi-tool problems. On two Qwen model families, the method improved performance and produced flexible tool chaining and self-correction from runtime feedback.
Catalog record
| CREATOR | Zirun Guo 1,2 Minjie Hong 1 Feng Zhang 2 Kai Jia 2 Tao Jin 1 1 Zhejiang University 2 ByteDance, BandAI |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-03746
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Thinking with Programming Vision: Towards a Unified View for Thinking with Images Zirun Guo 1,2 Minjie Hong 1 Feng Zhang 2 Kai Jia 2 Tao Jin 1 1 Zhejiang University 2 ByteDance, BandAI Abstract Multimodal large language models (MLLMs) that “think with images” can interactively use tools to reason about visual inputs, but current approaches often rely on a narrow set of tools with limited real-world necessity and scalability. In this work, we first reveal a critical and previously overlooked weakness: even state-of-the-art MLLMs are surprisingly brittle, showing significant performance degradation on images with simple orientation changes or natural corruptions, underscoring the need for more robust tool-based reasoning. To address this, we propose CodeVision , a flexible and scalable “code-as-tool” framework where the model generates code as a universal interface to invoke any image operation, moving beyond fixed tool registries. We train our model using a two-stage methodology, beginning with Supervised Fine-Tuning (SFT) on a high-quality dataset curated for complex, multi-turn tool composition and error recovery, followed by Reinforcement Learning (RL) with a novel and dense process reward function to encourage strategic and efficient tool use. To facilitate this research, we construct new SFT and RL datasets and introduce a challenging new benchmark suite designed to rigorously evaluate robustness to orientation changes and multi-tool reasoning. Experiments on Qwen2.5-VL and Qwen3-VL series show that our approach significantly improves model performance and fosters emergent capabilities such as flexible tool composition, efficient chained execution, and robust error recovery from runtime feedback. \project https://github.com/ByteDance-BandAI/CodeVision 1 Introduction Thinking with images ( OpenAI, 2025c ) equips multimodal large language models (MLLMs) with interactive, tool-augmented reasoning over visual inputs. Rather than passively describing an image, the model actively manipulates it using tools ( e.g., “zoom in”, OCR) to acquire the evidence needed for reliable reasoning. Despite rapid progress, current research still faces three limitations: \circled 1 The necessity of tools : Current methods mainly emphasize the “crop” tool, which zooms in on regions of interest for clearer observation, and evaluate tool ability on benchmarks such as V* ( Wu and Xie, 2024 ) and HRBench ( Wang et al., 2025b ) . Yet the benefit is marginal: using tools often yields only 2–5% accuracy gains, and reinforcement learning (RL) without tools can match those results. This suggests that the potential—and necessity—of tools is not being fully exercised by existing tasks. \circled 2 Flexibility and scalability : Methods frequently require manually specifying tool names and arguments ( Su et al., 2025b ) , which is brittle and not scalable. Even renaming a tool ( e.g., crop \rightarrow zoomin ) can necessitate retraining, hindering generalization to new tools and argument schemas. \circled 3 Multi-turn, multi-tool use : Many systems support only a single tool or a few tools within a single turn. While several works ( Lai et al., 2025 ; Zhang et al., 2025 ) explore multi-turn settings, they largely focus on repeated cropping rather than composing different tools across multiple turns, which is what real-world tasks often require. Figure 1 : Diagnostic results on image orientation identification. Based on these observations, we propose a new framework to address all three challenges. To create scenarios where tools are genuinely required, we examine ubiquitous real-world perturbations: incorrect image orientation due to landscape/portrait capture and mirrored selfies that confound text recognition. Compared with cropping, restoring the canonical orientation is often strictly more necessary for downstream recognition and reasoning. We conduct a simple diagnostic: starting from 200 images from various domains, we uniformly apply one of five transformations—rotation by 90/180/270 degrees, horizontal flip, or vertical flip—and ask models to identify the transformation (a five-way multiple-choice question). As illustrated in Figure 1 , even state-of-the-art models like GPT-5 ( OpenAI, 2025b ) and Gemini2.5 Pro ( Comanici et al., 2025 ) perform poorly, whereas humans achieve 100% accuracy easily. Remarkably, as summarized in Table 1 , simple rotation/flip operations can reduce model performance by up to 80%, revealing substantial brittleness of current MLLMs and underscoring the necessity of using tools for these real-world corruptions. Figure 2 : Three advantages of CodeVision we observe in the training and inference stage. To address the second challenge, inspired by OpenAI o3 ( OpenAI, 2025c ) , we treat code itself as a unified tool: the model writes code that invokes whatever image operations are needed. This eliminates hand-crafted tool name/argument specifications and dramatically improves generalization: the model is no longer restricted to a fixed registry but can call an effectively unbounded set of tools through code. While the paradigm of using code as a tool has been explored previously ( Chen et al., 2023 ; Zhang et al., 2025 ) , we unlock three notable advantages through our principled RL training and rigorous dataset construction (see Figure 2 ): (1) Emergence of new tools : the model calls tools that never appeared in the RL training data to solve novel problems; (2) Efficiency : the model chains multiple tools within a single execution; (3) Robustness : the model leverages runtime error messages and outputs to revise code, improving failure recovery and out-of-distribution generalization. To target the third challenge, we design datasets and a benchmark that require composing multiple tools across multiple turns, and we introduce dense process rewards that encourage effective tool selection and steady progress throughout the dialogue. This setting better reflects realistic problem solving and promotes the development of models that plan, adapt, and utilize different tools across steps. In summary, our contributions are threefold: • We identify a critical brittleness in state-of-the-art MLLMs and propose a flexible “code-as-tool” framework that treats code as a universal interface for tool invocation, enhancing scalability and generalization. • We construct high-quality SFT and RL datasets focused on multi-turn, multi-tool composition and error handling, and introduce three new benchmarks covering both single- and multi-tool scenarios to rigorously evaluate model robustness and complex tool use. • Our experiments demonstrate that our approach significantly improves model performance on these challenging benchmarks and validates the benefits of the code-as-tool paradigm, which enables emergent tool use, efficient tool chaining, and robust error recovery. 2 Related Work Thinking with Images. OpenAI’s o3 ( OpenAI, 2025c ) popularized the idea of thinking with images , where MLLMs actively operate on images via tools. Most follow-ups focus on the crop/zoom tool ( Zheng et al., 2025b ; Lai et al., 2025 ; Su et al., 2025a ; Fan et al., 2025 ) , in part because grounding-rich pretraining eases integration ( Bai et al., 2025b ) . However, cropping is often unnecessary or the benefits are hard to observe for many tasks because similar performance can be achieved without it. Meanwhile, broader tool suites ( e.g., line drawing, OCR, segmentation) ( Su et al., 2025b ; Zhang et al., 2025 ; Liu et al., 2025 ) have been investigated, but their effectiveness has not been sufficiently validated because they are not evaluated on tasks that truly require tools. Tool Integration. Equipping large models with tools ( Comanici et al., 2025 ; OpenAI, 2025c ; OpenAI, 2025b ) mitigates intrinsic limitations by enabling external capabilities such as search, code execution, and generative models. For instance, LLM-I ( Guo et al., 2025b ) leverages tools such