LED Benchmark: Diagnosing Structural Layout Errors for Document Layout Analysis
Inbum Heo, Taewook Hwang, Jeesu Jung, Sangkeun Jung · 2025
Read LED Benchmark: Diagnosing Structural Layout Errors for Document Layout Analysis by Inbum Heo, Taewook Hwang, Jeesu Jung, Sangkeun Jung (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-23295.txt.
Introduction
Inbum Heo and co-authors build a benchmark that catches structural mistakes in document layout analysis, in a 2025 paper. Modern models, including multimodal ones, detect page regions well. They still merge, split, or drop regions in damaging ways. Overlap scores like IoU barely notice such errors. A page can score high while its structure is broken.
The LED benchmark defines eight standard error types. Three tasks test detection, classification, and per-element labeling of errors. Structural robustness becomes measurable for the first time. Document AI feeds archives, forms, and accessibility tools. Developers gain a diagnostic rather than a single score. The work pushes evaluation toward what users actually suffer.
Catalog record
| CREATOR | Inbum Heo, Taewook Hwang, Jeesu Jung, Sangkeun Jung |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-23295
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LED Benchmark: Diagnosing Structural Layout Errors for Document Layout Analysis Inbum Heo, Taewook Hwang, Jeesu Jung, Sangkeun Jung Abstract Recent advancements in Document Layout Analysis through Large Language Models and Multimodal Models have significantly improved layout detection. However, despite these improvements, challenges remain in addressing critical structural errors, such as region merging, splitting, and missing content. Conventional evaluation metrics like IoU and mAP, which focus primarily on spatial overlap, are insufficient for detecting these errors. To address this limitation, we propose Layout Error Detection (LED) , a novel benchmark designed to evaluate the structural robustness of document layout predictions. LED defines eight standardized error types, and formulates three complementary tasks: error existence detection, error type classification, and element-wise error type classification. Furthermore, we construct LED-Dataset , a synthetic dataset generated by injecting realistic structural errors based on empirical distributions from DLA models. Experimental results across a range of LMMs reveal that LED effectively differentiates structural understanding capabilities, exposing modality biases and performance trade-offs not visible through traditional metrics. Introduction The recent advancements in Large Language Models (LLMs) and Large Multimodal Models (LMMs) have significantly improved the overall performance of Document AI systems. As a result, document images are increasingly being utilized in applications such as academic paper retrieval and document-level question answering. These document understanding tasks require simultaneous comprehension of both visual layout and logical structure. To enable such capabilities, a critical preprocessing step— Document Layout Analysis (DLA)—is essential ( Binmakhashen and Mahmoud 2019 ) . DLA partitions a document page into meaningful units such as text blocks, tables, and figures, directly influencing the accuracy of downstream tasks including OCR, information extraction, and question answering. Despite rapid progress in object detection models and vision-language models (VLMs), DLA outputs still suffer from various types of errors ( Vesalainen, Tolonen, and Ruotsalainen 2024 ) . Beyond typical localization errors (e.g., slight misalignments in bounding boxes), a more severe issue lies in structural errors , where semantically distinct regions are incorrectly merged or split. These structural errors substantially hinder document understanding performance. However, conventional metrics such as IoU and mAP are insufficient to detect or interpret such structural issues, as they mainly capture spatial overlaps without reflecting the logical consistency of layout predictions. To address this gap, we introduce a new evaluation task— Layout Error Detection (LED) —that systematically diagnoses structural errors in document layout predictions. The LED benchmark consists of the following components: 1. Definition of DLA-specific structural errors: We define eight types of structural errors commonly observed in DLA outputs (e.g., Missing , Merge , Split , Hallucination , etc.). Each error type is accompanied by criteria and quantitative thresholds, enabling a standardized diagnosis of layout failures. This taxonomy provides a unified framework to evaluate the structural reasoning capabilities of LMMs. 2. Task formulation: The LED benchmark includes three task variants: binary detection of error presence, classification of error type, and box-level error localization. These tasks collectively measure how well a model understands document structure and detects layout inconsistencies. We apply the LED benchmark to multiple state-of-the-art LMMs to assess their robustness in document error detection. 3. Synthetic dataset construction—LED-Dataset: We develop a synthetic benchmark dataset, LED-Dataset, by injecting structural errors into model predictions based on real DLA error distributions. Our injection algorithm generates realistic and diverse erroneous layouts, reflecting the frequency and characteristics of actual DLA model failures. This dataset serves as a controlled environment for rigorous evaluation of layout understanding. Using the LED benchmark, we evaluate a range of LMMs with varying input modalities—text-centric, image-centric, and fully multimodal. Our results reveal that LED is sensitive to structural comprehension and highlights significant performance differences depending on the model architecture and input modality. Notably, models that prioritize textual inputs tend to outperform those relying heavily on visual features in detecting layout errors, suggesting a modality bias in current vision-language models. Related Work Document Layout Analysis (DLA) aims to segment and categorize visual elements in document images into meaningful structural units. Traditional approaches have primarily relied on object detection models, leveraging large-scale layout datasets such as PubLayNet ( Zhong, Tang, and Yepes 2019 ) and DocLayNet ( Pfitzmann et al. 2022 ) . Detection backbones like YOLO ( Wang et al. 2024 ) and Deformable-DETR have been widely used, and more recently, vision-language models (VLMs) such as LayoutLM ( Xu et al. 2020 ) have extended DLA into the multimodal domain. In the broader field of document understanding ( Cui et al. 2021 ) , evaluation typically focuses on sub-tasks such as OCR accuracy, document-level question answering, information extraction, and layout structure prediction. These tasks are often benchmarked using object detection metrics such as Intersection-over-Union (IoU) ( Everingham et al. 2010 ) and mean Average Precision (mAP) ( Lin et al. 2014 ) . Some post-processing techniques have been proposed to mitigate layout errors, including region merging/splitting algorithms and rule-based heuristics. However, such methods rarely address the systematic classification or diagnosis of structural errors. Despite growing interest in understanding document structure ( Li et al. 2021 ) , there has been little attention paid to diagnosing layout errors themselves. To the best of our knowledge, a limited number of benchmarks or datasets have been proposed to explicitly address structural layout failures. In response, we introduce the LED-Dataset , which incorporates formally defined structural errors into the evaluation process. By injecting well-categorized synthetic errors, this dataset enables fine-grained interpretation of model predictions and provides a structured foundation for analyzing the limitations of current DLA systems. Error in Document Layout Analysis This section provides a systematic taxonomy of the recurring error types observed in DLA outputs. We detail the characteristics, diagnostic criteria, and algorithmic procedures for injecting each error type. Our proposed error categorization framework extends and unifies prior notions of layout errors while minimizing overlaps and ambiguities through clearly defined, rule-based criteria. Error Definition To enable fine-grained diagnosis of layout prediction failures, we define eight distinct types of structural errors based on empirical observations from existing DLA models. Each error type is formally described with structural patterns, discriminative rules, and injection conditions to ensure clear identification and quantitative evaluation. With the exception of misclassification, all other error types— Missing , Hallucination , Size , Split , Merge , Overlap , and Duplicate —are defined as mutually exclusive structural phenomena. Each error is independently detectable and designed to appear at most once per target object. This strict independence is crucial for interpretability and accurate diagnosis, ensuring that the boundaries between error types remain unambiguous and non-overlapping. • Missing (False Negative): A ground truth box B_{\text{gt}} exists, but there is no predicted box B_{\text{pred}} satisfying: \text{IoU}(B_{\text{gt}},B_{\text{pred}})\geq 0.1 • Hallucination (False Positive): A predicted box B_{\text{pred}} exists, but there is no ground truth box B_{\text{gt}} satisfying: \text{IoU}(B_{\text{gt}},B_{\text{pred}})\geq 0.1 • Size Error: The predicted and ground truth boxes have similar centers, but their area ratio falls outside the acceptable range: \frac{\text{area}(B_{\text{pred}})}{\text{area}(B_{\text{gt}})}\notin[0.6,\ 1.4] • Split: A single ground truth box B_{\text{gt}} is fragmented across multiple predicted boxes \{B_{\text{pred}}^{(i)}\}_{i=1}^{n} (n\geq 2) , such that each predicted box insufficiently overlaps with B_{\text{gt}} , but collectively they cover a significant portion: \forall i,\ \text{IoU}(B_{\text{gt}},B_{\text{pred}}^{(i)})<0.5,\quad\sum_{i=1}^{n}\text{IoU}(B_{\text{gt}},B_{\text{pred}}^{(i)})\geq 0.5 • Merge: Two or more semantically distinct ground truth boxes \{B_{\text{gt}}^{(1)},B_{\text{gt}}^{(2)},\dots\} are erroneously merged into a