A 2 R 2 A^{2}R^{2} : A dvancing Img2LaTeX Conversion via Visual R easoning with A ttention-Guided R efinement
Zhecheng Li, Guoxian Song, Yiwei Wang, Zhen Xiong, Junsong Yuan, Yujun Cai · 2025
Read A 2 R 2 A^{2}R^{2} : A dvancing Img2LaTeX Conversion via Visual R easoning with A ttention-Guided R efinement by Zhecheng Li, Guoxian Song, Yiwei Wang, Zhen Xiong, Junsong Yuan, Yujun Cai (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-20890.txt.
Introduction
Zhecheng Li and colleagues improve the conversion of photographed math expressions into LaTeX code, a chore every scientist knows. Vision-language models handle broad visual tasks well, but they trip on fine print such as subscripts and superscripts, producing code that compiles into the wrong formula. Writing in twenty twenty-five, the team's framework makes the model reason visually in rounds. Attention maps show which symbols it is looking at, and an attention-guided refinement loop lets it re-check regions and correct itself step by step. More rounds kept helping in their tests.
To measure progress honestly, the authors built a hard test set of over a thousand curated examples designed to stress exactly these weak points. Gains appeared on both text-level and visual-level metrics across models. The work targets document digitization pipelines where one wrong exponent can silently change a formula's meaning.
Catalog record
| CREATOR | Zhecheng Li, Guoxian Song, Yiwei Wang, Zhen Xiong, Junsong Yuan, Yujun Cai |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-20890
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A 2 R 2 A^{2}R^{2} : A dvancing Img2LaTeX Conversion via Visual R easoning with A ttention-Guided R efinement Zhecheng Li, Guoxian Song, Yiwei Wang, Zhen Xiong, Junsong Yuan, Yujun Cai Abstract Img2LaTeX is a practically significant task that involves converting mathematical expressions or tabular data from images into LaTeX code. In recent years, vision-language models (VLMs) have demonstrated strong performance across a variety of visual understanding tasks, owing to their generalization capabilities. While some studies have explored the use of VLMs for the Img2LaTeX task, their performance often falls short of expectations. Empirically, VLMs sometimes struggle with fine-grained visual elements, leading to inaccurate LaTeX predictions. To address this challenge, we propose A 2 R 2 A^{2}R^{2} : A dvancing Img2LaTeX Conversion via Visual R easoning with A ttention-Guided R efinement, a framework that effectively integrates attention localization and iterative refinement within a visual reasoning framework, enabling VLMs to perform self-correction and progressively improve prediction quality. For effective evaluation, we introduce a new dataset, Img2LaTex-Hard-1K , consisting of 1,100 carefully curated and challenging examples designed to rigorously evaluate the capabilities of VLMs within this task domain. Extensive experimental results demonstrate that: (1) A 2 R 2 A^{2}R^{2} significantly improves model performance across six evaluation metrics spanning both textual and visual levels, consistently outperforming other baseline methods; (2) Increasing the number of inference rounds yields notable performance gains, underscoring the potential of A 2 R 2 A^{2}R^{2} in test-time scaling scenarios; (3) Ablation studies and human evaluations validate the practical effectiveness of our approach, as well as the strong synergy among its core components during inference. 1 Introduction In contemporary applications, users increasingly engage with chat agents and research articles where mathematical expressions and structured content are often required in LaTeX format. This demand necessitates models capable of converting screenshots or images into their corresponding LaTeX source code. Existing approaches predominantly leverage convolutional neural networks (CNNs) or vision transformer (ViT) based architectures, which are fine-tuned on large-scale datasets tailored to this task ( Jiang et al. 2025 ; Wang et al. 2019b ; Dosovitskiy et al. 2021 ; Wang & Liu 2021 ; Wang et al. 2019a ) . However, these models typically depend heavily on extensive training data and lack the ability to perform human-like reasoning and self-correction when encountering mismatches or errors. Recently, vision-language models (VLMs) have exhibited strong potential in multimodal understanding, particularly in tasks involving image-text interactions ( Zhang et al. 2024b ; Du et al. 2022 ; Ghosh et al. 2024 ; Caffagni et al. 2024 ; Zhang et al. 2024a ; Yin et al. 2024 ) . With the increasing availability of such models, VLMs are emerging as promising candidates for tackling the Img2LaTeX task. Nonetheless, prior research evaluating VLMs on this task indicates that their performance falls short of expectations, leaving substantial room for improvement ( Roberts et al. 2024 ) . One possible explanation is the direct evaluation of VLMs, which may overlook the potential benefits of leveraging their visual reasoning capabilities. Figure 1: A simple illustration of how our method solves the Img2LaTeX task. While direct inference produces an incorrect answer, our method incorporates multiple steps in the inference process, using iterative refinement with test-time scaling progressively improves the output and leads to the correct answer. To address the aforementioned limitations, we curate a more challenging subset from the im2latex-100k dataset ( Deng et al. 2017 ) , selecting approximately 1,100 difficult examples through a combination of metric-based filtering and evaluations from multi-modal large language models (MLLMs). This new dataset, Img2LaTex-Hard-1K , is specifically designed to stress-test current VLMs. Inspired by the conception of visual reasoning ( Tan et al. 2025 ; Bi et al. 2025 ; Chen et al. 2024a ; Zhang et al. 2025 ; OpenAI 2025 ; Google 2024 ; Xu et al. 2025 ; Wang et al. 2025 ) , we propose a novel training-free plug-in framework, A^{2}R^{2} : A dvancing Img2LaTeX Conversion via Visual R easoning with A ttention-Guided R efinement. Our method enhances VLM performance on the Img2LaTeX task through attention-based localization and iterative feedback refinement. As illustrated in Figure 1 , A^{2}R^{2} comprises multiple core stages: (1) Generation – the model produces an initial LaTeX hypothesis from the input image; (2) Rendering and Comparison – the predicted LaTeX is rendered into an image and compared against the original to identify visual discrepancies, followed by feedback elicitation; (3) Attention Localization and Feedback Verification – attention mechanisms guide the model to focus on the mismatched regions and assess the reliability of the feedback; (4) Refinement – the LaTeX output is revised based on the verified feedback, and the process loops back to stage (2), enabling iterative visual reasoning and test-time enhancement. Extensive experiments validate the superior effectiveness and adaptability of our approach compared to various baseline methods. In summary, our key contributions are as follows: (1) We introduce A^{2}R^{2} , a novel visual reasoning framework that integrates attention-based localization and iterative self-refinement to enhance VLM performance on the Img2LaTeX task, all within a training-free paradigm. (2) Through extensive experiments, we demonstrate that A^{2}R^{2} significantly outperforms all evaluated baselines. Moreover, our findings reveal that increased inference steps generally correlate with improved performance, aligning with the principles of test-time scaling. (3) Ablation studies and human evaluations further corroborate the practical advantages of our approach, highlighting the effective interplay among its core components during inference. (4) We construct Img2LaTex-Hard-1K , a dataset containing 1,100 challenging samples, specifically designed to benchmark current VLMs, which have become more powerful than previous generations. 2 Related Works 2.1 Vision Language Models Vision-language models (VLMs) aim to unify visual perception and linguistic understanding. Their development is primarily driven by the emergence of Transformer and contrastive learning techniques, which align visual and textual representations within a shared semantic space ( Vaswani et al. 2023 ; Chen et al. 2020 ; Radford et al. 2021 ) . VLMs are explicitly designed to process and integrate multimodal inputs, enabling strong performance across a range of tasks such as image captioning, visual question answering, etc ( Liu et al. 2023 ; OpenAI et al. 2024b ; Caffagni et al. 2024 ; Zhang et al. 2024a ; Yin et al. 2024 ; Li et al. 2023 ; Bordes et al. 2024 ; Li et al. 2022 ; Alayrac et al. 2022 ) . Recent advances in scaling strategies, both in model capacity and data volume, along with the availability of large-scale multimodal datasets, have substantially improved generalization and task-specific accuracy ( Bai et al. 2025 ; Team et al. 2024 ; Li et al. 2024 ; Schuhmann et al. 2022 ; Team et al. 2025 ; Chen et al. 2024b ; Chen et al. 2025 ; Zhang et al. 2024c ) . State-of-the-art models now exhibit robust performance across diverse benchmarks, underscoring the rapid evolution and growing potential of VLMs in tackling complex multimodal challenges. 2.2 Visual Reasoning With the emergence of the test-time scaling paradigm, researchers begin adopting training methods such as supervised fine-tuning (SFT) and group relative policy optimization (GRPO) to improve the reasoning capabilities of large language models (LLMs) ( Shao et al. 2024b ; Yeo et al. 2025 ; Muennighoff et al.