DMAP: Human-Aligned Structural Document Map for Multimodal Document Understanding Conference: Proceedings of the ACM Web Conference 2026; April 13–17, 2026; Dubai, United Arab Emirates Proceedings of the ACM Web Conference 2026 (WWW ’26), April 13–17, 2026, Dubai, United Arab Emirates DOI: 10.1145/3
ShunLiang Fu, Yanxin Zhang, Yixin Xiang, Xiaoyu Du, Jinhui Tang · 2026
Read DMAP: Human-Aligned Structural Document Map for Multimodal Document Understanding Conference: Proceedings of the ACM Web Conference 2026; April 13–17, 2026; Dubai, United Arab Emirates Proceedings of the ACM Web Conference 2026 (WWW ’26), April 13–17, 2026, Dubai, United Arab Emirates DOI: 10.1145/3 by ShunLiang Fu, Yanxin Zhang, Yixin Xiang, Xiaoyu Du, Jinhui Tang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-18203.txt.
Introduction
ShunLiang Fu, Yanxin Zhang, Yixin Xiang, Xiaoyu Du, and Jinhui Tang build document maps that match how people see a page's structure. Their 2026 preprint presents DMAP for multimodal document understanding. Most question-answering systems treat a document as a bag of loose text chunks. That flat view drops the document's hierarchy and its links. Sections, figure-text ties, and cross-references all vanish, and answers suffer.
DMAP keeps a structural map of the document, aligned with human reading. Retrieval can then follow the document's own logic and layout. Questions that hinge on a figure or a section gain the context they need. The work serves anyone who asks machines to read reports, papers, and manuals.
Catalog record
| CREATOR | ShunLiang Fu, Yanxin Zhang, Yixin Xiang, Xiaoyu Du, Jinhui Tang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-18203
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DMAP: Human-Aligned Structural Document Map for Multimodal Document Understanding Conference: Proceedings of the ACM Web Conference 2026; April 13–17, 2026; Dubai, United Arab Emirates Proceedings of the ACM Web Conference 2026 (WWW ’26), April 13–17, 2026, Dubai, United Arab Emirates DOI: 10.1145/3774904.3792229 ISBN: 979-8-4007-2307-0/2026/04 CCS: Computing methodologies Natural language generation ShunLiang Fu, Yanxin Zhang, Yixin Xiang, Xiaoyu Du, Jinhui Tang Abstract . Existing multimodal document question-answering (QA) systems predominantly rely on flat semantic retrieval, representing documents as a set of disconnected text chunks and largely neglecting their intrinsic hierarchical and relational structures. Such flattening disrupts logical and spatial dependencies—such as section organization, figure-text correspondence, and cross-reference relations—that humans naturally exploit for comprehension. To address this limitation, we introduce a document-level structural D ocument MAP (DMAP), which explicitly encodes both hierarchical organization and inter-element relationships within multimodal documents. Specifically, we design a Structured-Semantic Understanding Agent to construct DMAP by organizing textual content together with figures, tables, charts, etc. into a human-aligned hierarchical schema that captures both semantic and layout dependencies. Building upon this representation, a Reflective Reasoning Agent performs structure-aware and evidence-driven reasoning, dynamically assessing the sufficiency of retrieved context and iteratively refining answers through targeted interactions with DMAP. Extensive experiments on MMDocQA benchmarks demonstrate that DMAP yields document-specific structural representations aligned with human interpretive patterns, substantially enhancing retrieval precision, reasoning consistency, and multimodal comprehension over conventional RAG-based approaches. Code is available at https://github.com/Forlorin/DMAP † † cc-license: by 1. Introduction Multimodal document understanding aims to extract and reason over textual and visual information—such as figures, tables, charts, and text—within documents ( 15 ; 21 ; 27 ; 8 ) . This task is challenging for two main reasons: a) raw documents are complex and heterogeneous, making it difficult to organize and interpret the information, and b) modeling the relationships among different elements to support reasoning is non-trivial. Documents are designed for human consumption, containing hierarchical cues such as titles, sections, paragraphs, and cross-page references, as well as intra-page visual-text alignments. Efficient understanding requires preserving these structural cues while enabling reasoning over both content and layout. Figure 1. The utilities of document knowledge. A common solution for document question answering is Retrieval-Augmented Generation (RAG), where documents are split into several chunks and mapped into a high-dimensional semantic vector space ( 5 ; 26 ; 7 ; 38 ; 24 ) . Queries are then matched to chunks based on vector similarity. While effective in capturing local semantic relevance, such methods destroy the document’s inherent structure. Flattening causes the loss of hierarchical and relational cues, leading to broken causal relationships, inconsistent parallel reasoning, and failures in handling referential expressions ( e.g. , ‘Table X’ or ‘Page X’) ( 40 ; 37 ; 43 ; 49 ) . Recent approaches, such as MDocAgent ( 15 ) , attempt to leverage LLMs or LVLMs to refine retrieval outputs with the whole page layout, but they still neglect the natural structural organization of documents—an aspect so significant that humans intuitively exploit it to locate, contextualize, and reason about information. As illustrated in Figure 1 , the conventional manner is to build a chunk-based index by fragmenting both textual and visual pages into isolated chunks and indexing them semantically, thereby losing the structural integrity of the original document. This flattening disrupts the hierarchical and relational dependencies among document elements—for example, the logical progression between paragraphs or the alignment between a figure and its corresponding caption. Such relations are central to human understanding but are entirely absent from traditional chunk-based retrieval. The inability to preserve structure often leads to incoherent reasoning paths, missing causal or comparative cues, and incorrect scope during retrieval. To achieve human-aligned document understanding, it is therefore crucial to restore and exploit the inherent organizational structure of documents. To bridge this gap, we introduce a human-aligned structural D ocument MAP (DMAP), a document-level representation that explicitly encodes hierarchical organization and inter-element relationships. As shown in Figure 1 , DMAP serves as a unified knowledge map that captures both the semantic and spatial dependencies among textual, tabular, and visual components, thereby preserving the intrinsic structure of the document. To construct DMAP for specific documents, we design a Structured-Semantic Understanding Agent (SSUA), which analyzes the document’s multimodal content and organizes textual segments, figures, tables, and charts into a human-intuitive hierarchical layout. This process transforms unstructured document inputs into a well-formed DMAP, where relationships such as section containment, figure-text alignment, and cross-page linkage are explicitly modeled. In order to validate the effectiveness of DMAP, we further devise a Reflective Reasoning Agent (RRA) that incorporates DMAP for the multimodal question answering task. Unlike conventional RAG-based generators that directly rely on retrieved chunks, RRA leverages DMAP as a structured knowledge base: it heuristically evaluates whether the current evidence is sufficient for accurate reasoning, and—when necessary—actively queries DMAP to locate supplementary information. This iterative reflection process mimics how humans verify and refine their understanding, ensuring that the generated answers are both semantically grounded and structurally consistent with the document. We evaluate DMAP on MMDocQA (multi-modal document question answering) benchmarks, demonstrating that it effectively preserves structural and relational cues and consistently improves multimodal question answering performance. Our contributions are three-fold: (1) We identify the limitations of existing structure-agnostic RAG methods and reveal the necessity of explicitly modeling document structures for faithful multimodal understanding. (2) We propose DMAP, a human-aligned hierarchical representation that integrates textual, tabular, and visual information, and design SSUA (Structured-Semantic Understanding Agent) to automatically construct DMAP from multimodal documents. (3) We further design RRA (Reasoning and Response Agent), which leverages DMAP to perform structure-aware retrieval and reasoning, producing coherent and reasoning-consistent answers. 2. Related Work LLMs for Document Understanding. With the emergence of large language models (LLMs), document understanding has made rapid progress. However, early LLMs and their multimodal counterparts (MLLMs) offered limited context window lengths. As a result, document understanding systems built on these models primarily targeted single-page or short documents, limiting comprehensive, fine-grained understanding of long documents ( 32 ; 41 ; 36 ) . Recently, with the rapid advancements in LLM capabilities, numerous models have been introduced that support substantially longer context windows, such as Qwen-VL ( 2 ) and GPT-4o ( 34 ) . This progress has made it feasible to directly process multi-page documents as input ( 25 ) . At the same time, some studies have focused on developing multimodal models tailored specifically for DocQA, targeting further extensions of supported context length and improved comprehension of multimodal information ( 42 ; 17 ; 19 ) . Nevertheless, these approaches still face limitations. Although the maximum context length of LLMs has been significantly extended, processing long documents remains challenging due to constraints in context window size and computational resources. Moreover, fine-grained and key pieces of information in a document can still be easily overlooked when embedded in overly long contexts. Therefore, an important research challenge lies in enabling DocQA systems to both retrieve sufficient contextual information and utilize it effectively for accurate and detailed comprehension. Retrieval-Augmented Generation. Retrieval-Augmented Generation (RAG) is an alternative approach to improving the performance of large language models, complementing fine-tuning and prompt engineering ( 13 ; 22 ) . By integrating retrieval with generation, RAG enables models to leverage information from