ARIAL: An Agentic Framework for Document VQA with Precise Answer Localization
Ahmad Mohammadshirazi, Pinaki Prasad Guha Neogi, Dheeraj Kulshrestha, Rajiv Ramnath · 2025
Read ARIAL: An Agentic Framework for Document VQA with Precise Answer Localization by Ahmad Mohammadshirazi, Pinaki Prasad Guha Neogi, Dheeraj Kulshrestha, Rajiv Ramnath (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-18192.txt.
Introduction
Ahmad Mohammadshirazi, Pinaki Prasad Guha Neogi, Dheeraj Kulshrestha, and Rajiv Ramnath present ARIAL, a system for answering questions about document images. The system must give the right text answer and also point to where that answer sits on the page. That second step matters when the result will be checked by a person.
ARIAL splits the task into small steps run by separate tools. One tool reads the text, one picks the useful context, one drafts the answer, and one links the answer to a box on the page. Tests on four standard sets show strong answers with clear traces of each step. A reader can see both the result and how it was found.
Catalog record
| CREATOR | Ahmad Mohammadshirazi, Pinaki Prasad Guha Neogi, Dheeraj Kulshrestha, Rajiv Ramnath |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-18192
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ARIAL: An Agentic Framework for Document VQA with Precise Answer Localization Ahmad Mohammadshirazi, Pinaki Prasad Guha Neogi, Dheeraj Kulshrestha, Rajiv Ramnath Abstract Document Visual Question Answering (VQA) requires models to not only extract accurate textual answers but also precisely localize them within document images—a capability critical for interpretability in high-stakes applications. However, existing systems achieve strong textual accuracy while producing unreliable spatial grounding, or sacrifice performance for interpretability. We present ARIAL ( A gentic R easoning for I nterpretable A nswer L ocalization), a modular framework that orchestrates specialized tools through an LLM-based planning agent to achieve both precise answer extraction and reliable spatial grounding. ARIAL decomposes Document VQA into structured subtasks: OCR-based text extraction with TrOCR, retrieval-augmented context selection using semantic search, answer generation via fine-tuned Gemma 3-27B, and explicit bounding-box localization through text-to-region alignment. This modular architecture produces transparent reasoning traces, enabling tool-level auditability and independent component optimization. We evaluate ARIAL on four benchmarks—DocVQA, FUNSD, CORD, and SROIE—using both textual accuracy (ANLS) and spatial precision (mAP@IoU 0.50:0.95). ARIAL achieves SoTA results across all datasets: 88.7 ANLS and 50.1 mAP on DocVQA, 90.0 ANLS and 50.3 mAP on FUNSD, 85.5 ANLS and 60.2 mAP on CORD, and 93.1 ANLS on SROIE, surpassing the previous best method (DLaVA) by +2.8 ANLS and +3.9 mAP on DocVQA. Our work demonstrates how agentic orchestration of specialized tools can simultaneously improve performance and interpretability, providing a pathway toward trustworthy, explainable document AI systems. Code is available at: https://github.com/ahmad-shirazi/ARIAL 1 Introduction Figure 1 : Overview of the ARIAL agentic workflow for Document VQA. The system consists of three modular stages: (1) Input Processing, where an OCR module extracts text segments and bounding boxes from a document image; (2) Agentic Reasoning Pipeline, where the planner agent coordinates task execution—retrieving relevant text, invoking QA or computation, and triggering spatial grounding; and (3) Output Generation, where the final answer and its bounding box are produced. The reasoning loop enables iterative refinement based on confidence, supporting flexible and context-aware decision-making. Document Visual Question Answering (VQA) requires reasoning over both textual content and visual layout in scanned or digitally rendered documents. Models must not only read and understand diverse formats—forms, receipts, reports—but also locate where answers appear within the document structure. While recent models such as LayoutLMv3 [ 14 ] , LayoutLLM [ 29 ] , and DocLayLLM [ 25 ] have improved textual accuracy by combining language with layout features, they often treat localization as a secondary task. Consequently, they may generate plausible answers without clearly identifying their source in the document, making verification difficult. Standard metrics like ANLS [ 40 ] capture string similarity but fail to reflect spatial correctness, prompting a shift towards combined evaluations that include IoU for grounding precision. DLaVA [ 33 ] introduced answer localization by integrating bounding-box prediction within a large multimodal transformer. However, its monolithic design can be computationally intensive and may struggle with fine-grained details in dense or handwritten layouts. We propose ARIAL (Agentic Reasoning for Interpretable Answer Localization), a modular document VQA framework built around an agentic planning model. Rather than using a single large model, ARIAL delegates subtasks—OCR, layout analysis, retrieval, reasoning, and grounding—to specialized modules orchestrated by a central agent. This agent, implemented with LLaMA 4 Scout [ 31 ] , dynamically selects tools and composes multi-step reasoning chains, enabling accurate and interpretable answers with precise spatial grounding. Our key contributions are: 1. Agentic Document QA : We introduce an agent-based document VQA system that decomposes queries into tool calls for OCR, retrieval, and grounding. The modular design enables tool reuse, error tracing, and flexible adaptation across document types. 2. Precise Answer Localization : ARIAL produces both answer text and corresponding bounding boxes by aligning answers to OCR-detected spans and contextual cues, ensuring visual traceability. 3. Retrieval-Augmented Reasoning : ARIAL incorporates retrieval-augmented generation [ 19 ] to focus on relevant text segments, enhancing both reasoning accuracy and efficiency for long or noisy documents. 4. SoTA Results : On four benchmarks—DocVQA [ 30 ] , FUNSD [ 17 ] , CORD [ 34 ] , and SROIE [ 15 ] —ARIAL achieves new best results in both ANLS and mAP@IoU, reaching 88.7 ANLS and 50.1 mAP on DocVQA. ARIAL demonstrates how LLMs can be effectively constrained through modular tool orchestration, where each answer is locked to specific pixel coordinates and traceable through interpretable reasoning chains. This addresses fundamental challenges in developing trustworthy, location-aware AI systems for document understanding. The remainder of this paper is organized as follows: Section 2 reviews related work in document VQA and agentic AI. Section 3 details ARIAL’s architecture and modules. Section 4 outlines datasets and evaluation protocols. Results and analysis appear in Section 5 , followed by discussion in Section 6 and conclusions in Section 7 . Figure 2 : Illustrative examples of visual information extraction on receipt images from the CORD dataset [ 34 ] . Each colored annotation corresponds to its extracted answer, highlighted by a matching colored bounding box. 2 Related Work 2.1 Document VQA and Layout-Aware Models Early document QA systems treated the task as text-only reading comprehension by applying OCR and feeding results into standard NLP models [ 32 ] . However, such approaches ignored document structure, prompting the development of layout-aware models. LayoutLM [ 39 ] , LayoutLMv3 [ 14 ] , DocFormer [ 3 ] , and StrucTexT [ 22 ] embed both text and spatial coordinates to model document layouts more effectively, achieving strong performance on datasets like DocVQA through unified transformer architectures. Nevertheless, most models output only answer text and treat localization as auxiliary prediction or post-hoc mapping. Methods like TILT [ 35 ] and Donut [ 18 ] explore end-to-end generation—Donut bypasses explicit OCR—but lack transparent mechanisms for spatial grounding. As highlighted by DLaVA [ 33 ] , the inability to visualize answer provenance limits model interpretability and hinders error analysis in high-trust domains. 2.2 Multimodal LLMs for Documents Multimodal large language models (MLLMs) such as GPT-4o [ 16 ] , Gemini 2.5 Pro [ 10 ] , and LLaVA 1.5 [ 27 ] extend VQA capabilities by jointly modeling vision and language. These systems answer questions directly from document images using prompt-based interfaces but often function as black boxes, lacking explicit reasoning steps and failing to highlight the visual basis of their answers. Their reliance on global visual understanding can lead to errors in fine-grained text recognition and spatial disambiguation [ 5 ] . Recent domain-specific adaptations like LayoutLLM [ 29 ] augment prompts with structured spatial cues to guide model focus. DLaVA [ 33 ] combines detected text with bounding box metadata or constructed text images, enabling prediction of both answer and spatial location. While DLaVA improves interpretability, it relies on a large, end-to-end multimodal backbone. Our method adopts a modular agentic design enabling more transparent and controllable reasoning while retaining compatibility with any OCR or LLM module. 2.3 Agent-Based and Modular Reasoning Agentic frameworks have emerged as powerful alternatives to monolithic models for complex tasks [ 12 ] . Systems like HuggingGPT [ 36 ] use a central language model to coordinate multiple tools for multi-step reasoning. Multi-agent paradigms have been explored for general VQA [ 41 ] , where