ShredBench: Evaluating the Semantic Reasoning Capabilities of Multimodal LLMs in Document Reconstruction
Zichun Guo, Yuling Shi, Wenhao Zeng, Chao Hu, Haotian Lin, Terry Yue Zhuo, Jiawei Chen, Xiaodong Gu, Wenping Ma · 2026
Read ShredBench: Evaluating the Semantic Reasoning Capabilities of Multimodal LLMs in Document Reconstruction by Zichun Guo, Yuling Shi, Wenhao Zeng, Chao Hu, Haotian Lin, Terry Yue Zhuo, Jiawei Chen, Xiaodong Gu, Wenping Ma (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-23813.txt.
Introduction
Guo, Shi, Zeng, Hu, Lin, Zhuo, Chen, Gu, and Ma present ShredBench, a benchmark for piecing texts back from shreds. An auto pipeline renders shredded texts from Markdown. It spans four cases and three shred levels, using fresh sources to avoid training-data leaks. Top models handle intact texts well but falter sharply as shredding rises. The gap exposes weak fine-grained cross-modal links.
Catalog record
| CREATOR | Zichun Guo, Yuling Shi, Wenhao Zeng, Chao Hu, Haotian Lin, Terry Yue Zhuo, Jiawei Chen, Xiaodong Gu, Wenping Ma |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41661 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.23813
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ShredBench: Evaluating the Semantic Reasoning Capabilities of Multimodal LLMs in Document Reconstruction Zichun Guo, Yuling Shi, Wenhao Zeng, Chao Hu, Haotian Lin, Terry Yue Zhuo, Jiawei Chen, Xiaodong Gu, Wenping Ma Abstract Multimodal Large Language Models (MLLMs) have achieved remarkable performance in Visually Rich Document Understanding (VRDU) tasks, but their capabilities are mainly evaluated on pristine, well-structured document images. We consider content restoration from shredded fragments, a challenging VRDU setting that requires integrating visual pattern recognition with semantic reasoning under significant content discontinuities. To facilitate systematic evaluation of complex VRDU tasks, we introduce ShredBench , a benchmark supported by an automated generation pipeline that renders fragmented documents directly from Markdown. The proposed pipeline ensures evaluation validity by allowing the flexible integration of latest or unseen textual sources to prevent training data contamination. ShredBench assesses four scenarios (English, Chinese, Code, Table) with three fragmentation granularities (8, 12, 16 pieces). Empirical evaluations on state-of-the-art MLLMs reveal a significant performance gap: The method is effective on intact documents; however, once the document is shredded, restoration becomes a significant challenge, with NED dropping sharply as fragmentation increases. Our findings highlight that current MLLMs lack the fine-grained cross-modal reasoning required to bridge visual discontinuities, identifying a critical gap in robust VRDU research 1 1 1 Code and dataset are available at https://github.com/ythere-y/ShredBench . . † † footnotetext: † Equal contribution. ✉ Corresponding author. 1 Introduction Figure 1: Evaluation results on ShredBench across 6 dimensions (Metric: ROUGE-L). Our proposed benchmark reveals significant gaps in current MLLMs’ capabilities on fragmented documents. The advance in Multimodal Large Language Models (MLLMs), such as GPT-5 35 and Gemini 3 Pro 12 , has revolutionized the field of Visually Rich Document Understanding (VRDU) 59 ; 55 ; 53 ; 52 . By projecting visual features into a shared semantic space with textual representations, these models have almost achieved human expert performance on tasks ranging from standard Optical Character Recognition (OCR) 18 ; 30 to complex information extraction (CIE) from well-formatted documents 16 ; 60 ; 46 . However, real-world document processing often encounters inputs that are far from ideal, where documents may be occluded, damaged, or physically torn. Although recent high-resolution MLLMs 54 ; 23 ; 3 ; 4 attempt to mitigate visual noise and enhance fine-grained perception, the specific challenge of reconstructing physically fragmented information remains underexplored. While recent benchmarks have begun to address robustness against image corruptions 40 or super-long context retrieval 7 , the challenge of reconstructing physically fragmented information remains underexplored. While humans can rely on strong language priors and world knowledge 50 ; 41 to mentally piece together fragmented information, the extent to which MLLMs possess this capability remains an open question. In this paper, we explore shredded content restoration at the intersection of vision and NLP. Unlike traditional jigsaw puzzles based on edge matching, this task demands profound semantic reasoning 63 . For instance, connecting “The algorithm optimiz-” with “-es the loss function” relies less on ambiguous visual cuts than on syntactic expectation. Consequently, this task serves as a rigorous probe for evaluating whether MLLMs can leverage internal language priors to maintain coherence across visual discontinuities. To systematically evaluate this, we propose ShredBench , a benchmark characterized by three key dimensions: (1) Multi-Granularity Complexity. We partition images into 8, 12, and 16 fragments. This hierarchy enables the analysis of how visual entropy correlates with performance degradation. (2) Diverse Scenarios. Comprising 756 documents, our dataset spans English and Chinese text, source code (strict syntax), and tables (complex 2D structure). Tables and code are notably difficult, requiring models to restore rigid indentation and alignment—a challenge even for specialized models 64 . (3) Extensive Experiments. We evaluate state-of-the-art proprietary and open-source MLLMs. Using standard textual metrics, we establish the first quantitative baselines to facilitate future research. We employ NED, TEDS, BLEU, and ROUGE-L as our primary evaluation metrics and conduct extensive experiments across 14 representative MLLMs, including both leading proprietary and open-source models. The results are sobering: While models exhibit high proficiency on intact documents, their performance collapses under fragmentation. In the hardest setting (16 fragments), the average NED reaches a high of 0.73, even the most advanced models failing to identify correct reading orders or hallucinating non-existent bridging text 13 ; 22 . Our study reveals that current MLLMs struggle to effectively align visual positional embeddings with semantic continuity, often treating fragments as independent entities rather than parts of a cohesive whole. Our contributions are summarized as follows. First, we introduce ShredBench , the first benchmark specifically designed to stress-test the semantic reasoning capabilities of MLLMs via shredded content restoration. Second, we design an automated pipeline for generating shredded document benchmarks with adjustable granularity. This enables the synthesis of diverse samples covering English and Chinese text, source code, and tables, thereby presenting a comprehensive range of semantic and structural challenges. Third, we conduct a comprehensive evaluation of various MLLMs, revealing significant limitations in their ability to handle visual structural noise and maintain coherence in both textual semantics and 2D spatial layouts. 2 Related Work 2.1 Benchmarking Multimodal Reasoning Recent MLLM benchmarks have expanded beyond visual perception to evaluate complex reasoning 15 ; 9 ; 21 . Representative works include MMBench 26 and SEED-Bench 20 for general and generative comprehension, alongside domain-specific benchmarks like MathVista 29 that target mathematical and logical deduction. However, these benchmarks largely focus on coherent and clean inputs, leaving models’ ability to reason under structurally disordered or fragmented data underexplored. In contrast, ShredBench is specifically designed to evaluate semantic reconstruction in the presence of structural disruption, providing a rigorous assessment of long-context coherence under disordered inputs. Benchmark Domain Modality Deformation Reasoning Type Granularity Capabilities OCR Reconst. Document Parsing Benchmarks OmniDocBench 36 Document Text, Table, Formula / Structural Parsing / ✓ ✗ WildDoc 51 Scene Doc Text, Chart Shadow, Blur, Warp Robust Perception / ✓ ✗ DocPTBench 10 Photo Doc Text Geom. Warp Parsing & Trans. / ✓ ✗ Visual Jigsaw & Reconstruction Benchmarks Jigsaw-Puzzles 31 Natural Img Visual Pixels Grid Crop (2D) Spatial Arrangement Grid (2x2 to 5x5) ✗ ✓ RePAIR 49 Artifacts 3D Geometry Erosion, Fragments Geometric Matching / ✗ ✓ Proposed Benchmark ShredBench (Ours) Hybrid Text, Table, Code 3D Shredding Semantic Bridging Voronoi (8, 12, 16 pcs) ✓ ✓ Table 1: Comparison of ShredBench with representative benchmarks. Domain: target data domain. Modality: input data types. Deformation: visual or physical distortion applied to inputs. Reasoning Type: core cognitive ability evaluated. Granularity: fragment or subunit size/layout. Capabilities: evaluated capabilities, including OCR and implicit reconstruction reasoning. 2.2 Document Parsing and Understanding The field has evolved from modular OCR to end-to-end MLLMs capable of holistic parsing and understanding. In document parsing , models like Nougat 2 reconstruct papers into markup, while TextMonkey 27 and Vary 57 handle dense text and layout reconstruction. For document understanding , proprietary models such as GPT-5 35 and Gemini 3 Pro 12 show strong zero-shot reasoning, while open-source models like LLaVA 25 , Qwen-VL 1 , and InternVL 5 focus on high-resolution processing and reducing hallucinations. Comprehensive benchmarks support these tasks: OmniDoc 36 and HierText 28 target multi-task reconstruction and dense text perception, DocVQA 33 and ChartQA 32 ; 6 assess information extraction and logical reasoning, and WildDoc 51 evaluates MLLMs