VisTIRA: Closing the Image–Text Modality Gap in Visual Math Reasoning via Structured Tool Integration

Saeed Khaki, Ashudeep Singh, Nima Safaei, Kamal Ginotra · 2026

Read VisTIRA: Closing the Image–Text Modality Gap in Visual Math Reasoning via Structured Tool Integration by Saeed Khaki, Ashudeep Singh, Nima Safaei, Kamal Ginotra (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-14440.txt.

Introduction

Saeed Khaki, Ashudeep Singh, Nima Safaei, and Kamal Ginotra study a gap in AI math skills in this 2026 paper. Vision-language models lag behind text-only models on math reasoning. The gap shows when the same problem is given as an image, not as text. The team names this the modality gap and measures it.

They trace the gap to compounded failures. Models misread dense formulas, layout, and mixed symbolic and diagram context. Their answer is VisTIRA, which closes the gap through structured tool integration. Tools let the model work on the problem in steps. Readers get both a diagnosis of a clear weakness and a tested style of fix.

Catalog record

CREATORSaeed Khaki, Ashudeep Singh, Nima Safaei, Kamal Ginotra
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-14440

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

VisTIRA: Closing the Image–Text Modality Gap in Visual Math Reasoning via Structured Tool Integration Saeed Khaki, Ashudeep Singh, Nima Safaei, Kamal Ginotra Abstract Vision-language models (VLMs) lag behind text-only language models on mathematical reasoning when the same problems are presented as images rather than text. We empirically characterize this as a modality gap : the same question in text form yields markedly higher accuracy than its visually typeset counterpart, due to compounded failures in reading dense formulas, layout, and mixed symbolic–diagrammatic context. First, we introduce VisTIRA (Vision and Tool-Integrated Reasoning Agent), a tool-integrated reasoning framework that enables structured problem solving by iteratively decomposing a given math problem (as an image) into natural language rationales and executable Python steps to determine the final answer. Second, we build a framework to measure and improve visual math reasoning: a LaTeX-based pipeline that converts chain-of-thought math corpora (e.g., NuminaMath) into challenging image counterparts, and a large set of synthetic tool-use trajectories derived from a real-world, homework-style image dataset (called SnapAsk) for fine-tuning VLMs. Our experiments show that tool-integrated supervision improves image-based reasoning, and OCR grounding can further narrow the gap for smaller models, although its benefit diminishes at scale. These findings highlight that modality gap severity inversely correlates with model size, and that structured reasoning and OCR-based grounding are complementary strategies for advancing visual mathematical reasoning. January 2026 1 Introduction Vision-language models (VLMs) have achieved strong performance on generic multimodal tasks, including document understanding and visual question answering (e.g., DocVQA and chart- or plot-based QA benchmarks) ( 17 ; 16 ; 18 ) . However, when mathematical problems are presented as images that mix dense symbolic expressions, multi-line equations, diagrams, plots, and embedded textual instructions, current VLMs still trail their text-only large language model counterparts. This discrepancy manifests as a persistent image-text modality gap : the same math question, when rendered visually rather than provided as plaintext, yields significantly lower accuracy, as shown by recent multimodal math benchmarks ( 15 ; 35 ; 38 ; 45 ; 36 ) and analyses that formalize modality and capability mismatches in VLMs ( 43 ; 27 ) . Even frontier systems show degradation on visual math, whereas mid-sized open models suffer large drops ( 15 ; 35 ) . Accurate mathematical reasoning in the visual setting requires two coupled competencies: (i) faithful visual parsing of layout-level structure, such as fraction baselines, integral bounds, superscripts and subscripts, piecewise braces, and diagram annotations, and (ii) robust symbolic and quantitative reasoning ( 34 ; 47 ; 5 ) . Small misreads cascade; for example, an incorrectly transcribed exponent or reversed limit alters subsequent algebra, leading to compounding errors until the final answer diverges. Similar cascades have been documented in visual numeracy tasks, where mis-parsed axes or legends propagate into incorrect computations ( 16 ; 18 ) . Recent advances in textual mathematical reasoning have leveraged chain-of-thought (CoT) prompting ( 40 ) , program-of-thought (PoT) execution, and tool integration (e.g., symbolic solvers and Python libraries) ( 9 ) , yielding substantial gains on math and symbolic tasks ( 31 ; 6 ) . In contrast, visual mathematical reasoning has received less attention. Existing VQA and OCR-style datasets rarely contain stepwise executable trajectories paired with images and typically provide only final answers or short natural language rationales ( 15 ; 16 ; 18 ) . Bridging this gap requires frameworks and data that align visual perception with iterative computation. We introduce VisTIRA (Vision and Tool-Integrated Reasoning Agent), a framework that equips a VLM to iteratively solve image-based math problems by interleaving natural language rationales with executable Python programs ( 32 ) . At each step, the model proposes a rationale and a code snippet; the code is executed externally, its output is appended to the evolving trajectory, and the model decides whether to stop after producing a boxed final answer or to continue refining. This closed-loop design reduces hallucinated arithmetic by deferring exact manipulation to a symbolic engine and by providing verifiable intermediate signals, consistent with evidence that program execution mitigates reasoning errors ( 6 ; 11 ) and with broader VLM hallucination mitigation via verification-style signals ( 46 ; 41 ; 22 ; 26 ) . To supervise VisTIRA, we construct a large corpus of high-quality tool-integrated trajectories from real-world homework-style images (SnapAsk), generated by prompting strong teacher models and filtered for internal consistency ( 7 ) . Our data construction strategy is informed by prior trajectory-supervised tool-augmented math corpora in the text domain ( 11 ; 44 ) , but extends them to the visual domain with explicit layout-aware parsing and executable traces. Separately, we develop a LaTeX-based text-to-image rendering pipeline that converts existing text-only chain-of-thought math problems, such as NuminaMath, into visually typeset images, enabling controlled modality gap evaluation ( 14 ; 30 ) . Using this pipeline, we generate and release 360k rendered NuminaMath images to advance open research in mathematical visual reasoning. We also investigate OCR grounding as a complementary lever for mitigating the modality gap. Applying a state-of-the-art OCR system, such as DeepSeek-OCR ( 39 ) , to extract textual content from math images and then inputting it alongside the original image markedly improves accuracy for smaller VLMs ( 28 ) , indicating that explicit text extraction can compensate for weaker visual encoders, in line with findings from document VQA ( 17 ) and recent OCR advances ( 39 ) . For larger models, however, raw OCR concatenation can introduce redundancy or noise, revealing a scale-dependent trade-off ( 2 ) consistent with observations in large VLM hallucination analyses ( 46 ; 41 ) . Comprehensive experiments on the real-world SnapAsk and rendered NuminaMath benchmarks show that supervised fine-tuning on VisTIRA trajectories yields measurable gains over instruction-only baselines ( 11 ; 29 ) . They also show that the modality gap remains substantial, particularly for smaller models, but can be partially mitigated via tool-integrated reasoning and OCR-based grounding ( 15 ; 35 ; 38 ; 45 ) . Contributions. We summarize our main contributions: • Framework: We propose VisTIRA, an iterative tool-integrated vision language reasoning framework that decomposes visual math problems into rationale, code, and execution loops until solution ( 6 ; 10 ; 32 ) . • Trajectory Supervision: We construct a large corpus of verified rationale, code, and output trajectories from real-world homework-style images (SnapAsk), enabling supervised fine-tuning of mathematical VLM agents; our approach builds on trajectory-supervised tool use while extending to the visual setting ( 11 ; 44 ) . • Evaluation Pipeline: We introduce a LaTeX-based rendering pipeline to convert text-only CoT math corpora such as NuminaMath into paired image modalities for controlled modality gap analysis ( 14 ; 30 ) . • Open Data Release: We release 360k rendered NuminaMath images to support open research in visual mathematical reasoning ( 14 ) . • Benchmark Recommendation: We release a 5k NuminaMath image test set with accompanying DeepSeek-OCR textual transcriptions and recommend it as a standardized benchmark for assessing mathematical reasoning in VLMs ( 14 ; 39 ) . • OCR Grounding Study: We analyze the impact of OCR text extraction (DeepSeek-OCR), showing substantial gains for weaker models and scale-dependent diminishing returns ( 17 ; 39 ; 46 ; 2 ) . •

Read the complete text