TIR-Bench: A Comprehensive Benchmark for Agentic Thinking-with-Images Reasoning
Ming Li, Jike Zhong, Shitian Zhao, Haoquan Zhang, Shaoheng Lin, Yuxiang Lai, Chen Wei, Konstantinos Psounis, Kaipeng Zhang · 2025
Read TIR-Bench: A Comprehensive Benchmark for Agentic Thinking-with-Images Reasoning by Ming Li, Jike Zhong, Shitian Zhao, Haoquan Zhang, Shaoheng Lin, Yuxiang Lai, Chen Wei, Konstantinos Psounis, Kaipeng Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-01833.txt.
Introduction
Ming Li and eight colleagues benchmark a new skill in artificial intelligence: thinking with images. Here a model actively creates and manipulates pictures as part of its reasoning. It crops, transforms, and inspects them with tools mid-thought, as OpenAI's o3 does. Older benchmarks barely test this; the common Visual Search test asks only for localization and cropping.
TIR-Bench spans 13 diverse tasks, each demanding novel tool use for image processing inside the reasoning chain. The authors evaluated 22 multimodal language models, open and proprietary, including some with explicit tool-use training. Every model found the benchmark hard, and the results indicate that strong scores require genuine thinking-with-images ability rather than static visual skill. A pilot study comparing direct fine-tuning with agentic fine-tuning rounds out the report.
Catalog record
| CREATOR | Ming Li, Jike Zhong, Shitian Zhao, Haoquan Zhang, Shaoheng Lin, Yuxiang Lai, Chen Wei, Konstantinos Psounis, Kaipeng Zhang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-01833
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
TIR-Bench: A Comprehensive Benchmark for Agentic Thinking-with-Images Reasoning Ming Li, Jike Zhong, Shitian Zhao, Haoquan Zhang, Shaoheng Lin, Yuxiang Lai, Chen Wei, Konstantinos Psounis, Kaipeng Zhang Abstract The frontier of visual reasoning is shifting toward models like OpenAI o3, which can intelligently create and operate tools to transform images for problem-solving, also known as thinking- with -images in chain-of-thought. Yet existing benchmarks fail to fully capture this advanced capability. Even Visual Search, the most common benchmark for current thinking- with -images methods, tests only basic operations such as localization and cropping, offering little insight into more complex, dynamic, and tool-dependent reasoning. We introduce TIR-Bench , a comprehensive benchmark for evaluating agentic thinking-with-images across 13 diverse tasks, each requiring novel tool use for image processing and manipulation in chain-of-thought. We evaluate 22 multimodal large language models (MLLMs), from leading open-sourced and proprietary models to those with explicit tool-use augmentation. Results show that TIR-Bench is universally challenging, and strong performance requires genuine thinking-with-images capabilities. Finally, we present a pilot study comparing direct versus agentic fine-tuning. Implementation Code Benchmark Data 1 Introduction The reasoning abilities of recent multimodal large language models (MLLMs) ( 5 ; 3 ) have advanced significantly, driven in large part by reasoning techniques such as chain-of-thought (CoT) ( 33 ) . By decomposing reasoning of complex visual questions into a series of textual steps, MLLMs are able to achieve improved performance. While promising, these techniques are confined to the textual domain, conducting their reasoning solely through language while treating the visual information as a static, unalterable input ( 26 ) . To effectively process visual information, thinking-with-images has been proposed ( 22 ; 26 ; 4 ) . This approach enables a model to generate new visual information by actively manipulating input images with tools. For example, when faced with a complex visual problem, OpenAI’s o3 model ( 22 ) first writes code to create an image-processing tool, then executes it to modify the image ( e.g. , cropping, flipping, or rotating). The transformed visual data then informs the next stage of its linguistic reasoning. To assess the agentic thinking-with-images capabilities of MLLMs, existing benchmarks ( 34 ; 30 ) have largely centered on visual search and tasks requiring the analysis of high-resolution images. These evaluations primarily validate a model’s ability to accurately localize and crop specific regions within an image for better capturing detailed information to answer corresponding questions. However, these assessments tend to focus narrowly on the visual search capabilities of agentic MLLMs, leaving a broader spectrum of thinking-with-images abilities unevaluated. Therefore, there is an urgent need for a benchmark that incorporates a diverse range of tasks requiring sophisticated tool use to properly assess integrated multimodal reasoning. In this paper, we introduce TIR-Bench, a comprehensive benchmark designed to evaluate the diverse thinking-with-images capabilities of agentic MLLMs. Unlike previous benchmarks focusing solely on visual search problems, TIR-Bench incorporates a diverse set of 13 tasks that require a wide range of tool-based interactions, such as zooming, rotating, increasing image contrast, adding auxiliary lines, and others to assess a model’s tool integrated reasoning capabilities. The design of each task is predicated on the human intuition that solving it requires actively manipulating the visual input, rather than relying on static observation alone. For example, in TIR-Bench, a math problem might require the model to draw auxiliary lines or a coordinate system to find a solution, while a jigsaw puzzle task demands that it segment and then reassemble the image pieces. Consequently, TIR-Bench enables a more holistic evaluation of a model’s thinking-with-images abilities, assessing a spectrum of skills not limited to visual search. Figure 1: Overview of performance of open models (left), proprietary models (middle), and agentic tool-using models (right). SR: Symbolic Reasoning, WS: Word Search, LL-VQA: Low-Light VQA, IR: Instrument Reasoning, SD: Spot Difference, JG: Jigsaw Game, VS: Visual Search, RG: Rotation Game, Pro.: Proportion VQA. o3-TU: o3-tool-using, i.e., o3 with code interpreter. Using TIR-Bench, we conduct a comprehensive performance evaluation of 22 leading MLLMs across three categories: open-source models, proprietary models, and tool-using agents. The overall experimental results, illustrated in Figure 1 , reveal that TIR-Bench is a challenging benchmark for thinking-with-images abilities, as the best performance achieved is only 46%. Moreover, traditional non-agentic models perform poorly on TIR-Bench, with the best-performing model Gemini-2.5-pro reaching an accuracy of merely 28.9%. These findings highlight the importance of the thinking-with-images ability for this benchmark, as models equipped with tool-use capabilities, such as o3, o4-mini, and PyVision ( 41 ) , achieve much higher performance than other models. Lastly, we assess the function-calling proficiency of various MLLMs and conduct a pilot study contrasting direct supervised fine-tuning (SFT) with an agentic SFT approach on rotated image OCR task. Many recent MLLMs are equipped with function-calling capabilities. Our evaluation on rotation game task of TIR-Bench measures a model’s proficiency in accurately executing the correct tool parameters as part of its reasoning chain. Results show that recent models like o3 perform well, whereas earlier models such as GPT-4o perform significantly worse. The pilot study on the rotated image OCR task compares two training methodologies and examines whether end-to-end SFT can achieve strong performance across different data scales for tasks involving image operations. Our findings indicate that the agentic SFT on full problem-solving trajectories with generated images is significantly more effective than direct SFT. This implies that agentic fine-tuning enables the emergence of more complex and robust problem-solving behaviors, allowing models to tackle multi-step tasks that are intractable with direct fine-tuning alone. 2 Related Works 2.1 Multimodal Benchmarks As the capabilities of Multimodal Large Language Models (MLLMs) evolve rapidly, a variety of benchmarks have been proposed to evaluate their performance, identify limitations, and guide future improvements ( 17 ; 38 ; 24 ; 29 ; 32 ; 11 ; 18 ; 12 ) . These benchmarks are typically either specialized for specific domains ( 32 ; 11 ; 18 ; 12 ) or designed to be versatile and cover a broad range of tasks ( 10 ; 16 ; 37 ; 31 ) . MLLMs often use CoT reasoning on these benchmarks, though solving them only requires static image information. More recently, benchmarks such as V* Bench and HR-Bench have been introduced to evaluate agentic capabilities, specifically for visual search in high-resolution images ( 34 ; 30 ) . While these benchmarks advance agentic evaluation by requiring models to programmatically crop high-resolution images, their focus is narrowly confined to visual search. Consequently, a broader spectrum of thinking-with-images abilities, such as rotation and drawing, remains largely underexplored. 2.2 Think with Images In previous works, Visual Sketchpad ( 4 ) , CoGCoM ( 23 ) , DeepEyes ( 42 ) , Pixel Reasoner ( 25 ) , OpenThinkIMG ( 26 ) , Chain-of-Focus ( 39 ) , Mini-o3 ( 8 ) and REVPT ( 43 ) generate and executes tool calling in a predefined visual-specific toolset. In the intermediate reasoning steps, processed images are reinjected to the context, resulting in a multi-modal rational. However, these methods rely on a fixed collection of external visual parsers—such as detection models (e.g., GroundingDINO ( 15 ) ) and segmentation models (e.g., SAM ( 6 ) )—which constrains their generality across diverse vision