Can MLLMs "Read" What is Missing?

Jindi Guo, Chaozheng Huang, Xi Fang · 2026

Read Can MLLMs "Read" What is Missing? by Jindi Guo, Chaozheng Huang, Xi Fang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-21277.txt.

Introduction

Jindi Guo, Chaozheng Huang, and Xi Fang test whether multimodal language models can rebuild masked text from visual context alone, without explicit questions. Their MMTR-Bench spans 2,771 samples across languages and target lengths, covering documents and webpages in single- and multi-page form. A level-aware scoring protocol handles the diversity. Representative models struggle, most at sentence- and paragraph-level reconstruction, showing the benchmark probes layout understanding, visual grounding, and knowledge integration rather than instruction following.

Catalog record

CREATORJindi Guo, Chaozheng Huang, Xi Fang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE68190
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.21277

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Can MLLMs "Read" What is Missing? Jindi Guo, Chaozheng Huang, Xi Fang Abstract We introduce MMTR-Bench, a benchmark designed to evaluate the intrinsic ability of Multimodal Large Language Models (MLLMs) to reconstruct masked text directly from visual context. Unlike conventional question-answering tasks, MMTR-Bench eliminates explicit prompts, requiring models to recover masked text from single- or multi-page inputs across real-world domains such as documents and webpages. This design isolates the reconstruction task from instruction-following abilities, enabling a direct assessment of a model’s layout understanding, visual grounding, and knowledge integration. MMTR-Bench comprises 2,771 test samples spanning multiple languages and varying target lengths. To account for this diversity, we propose a level-aware evaluation protocol. Experiments on representative MLLMs show that the benchmark poses a significant challenge, especially for sentence- and paragraph-level reconstruction. The homepage is available at MMTR-Bench . 1 Introduction Figure 1: Overall performance of representative models on MMTR-Bench. Models from the same provider share the same color. Strong closed-source models achieve the best results, while smaller open-source vision-language models remain clearly behind. In recent years, multimodal large language models (MLLMs) have made significant progress in understanding documents, charts, and webpages. However, most existing benchmarks still rely heavily on explicit question answering (QA). In these tests, models are given an image along with a clear question that tells them exactly what to look for. While the QA format is useful, it does not fully reflect how models process real-world visual data. In practical scenarios—such as reading papers, analyzing multi-page reports, or parsing complex webpages—inputs do not come with guiding prompts. Instead, information is naturally distributed across layouts, tables, figures, and cross-page text. To truly understand this content, an MLLM must identify structural gaps and recover missing information by combining the surrounding visual context with its own world knowledge. Although masking is widely used for training, we still lack a benchmark to test this native recovery ability. To fill this gap, we introduce MMTR-Bench (Multimodal Masked Text Reconstruction Benchmark). Instead of asking explicit questions, we give models masked single- or multi-page inputs. The task is to recover the hidden text using the remaining visual and structural context—such as titles, charts, table layouts, and cross-page clues—or by integrating this context with world knowledge. MMTR-Bench includes 2,771 test samples across multiple languages. It covers diverse sources, ranging from academic documents and webpage screenshots to natural scene text. The masked targets also vary in length, from short strings (like years or numbers) to full sentences and explanatory paragraphs. Since targets of different lengths need different evaluation criteria, we design a level-aware evaluation pipeline . We divide the samples into four levels. For short targets, we focus on exact matching. For longer ones, we measure semantic similarity and factual consistency. To ensure high-quality scoring for complex targets, we also introduce an LLM-based factuality gate. This approach brings the automated metrics much closer to human judgment. Finally, we evaluate several representative closed-source and open-source models,as illustrated in Figure 1 . Our results show that MMTR-Bench is still highly challenging. Stronger closed-source models achieve the best results, but smaller open-source vision-language models clearly lag behind. Overall, there is still plenty of room for improvement, especially for sentence- and paragraph-level recovery under native visual inputs. Our main contributions are as follows: • We introduce MMTR-Bench , a new benchmark designed to evaluate native multimodal perception and reasoning through masked visual context recovery, moving away from explicit question-based guidance. • We build a diverse test set of 2,771 samples. It covers single- and multi-page inputs, multiple languages, and various real-world sources. • We propose a level-aware evaluation pipeline , integrating lexical matching, semantic similarity, and an LLM-based factuality gate, to fairly judge recovery targets of different lengths and difficulties. 2 Related work Existing multimodal benchmarks have substantially improved document, webpage, and chart understanding, but most of them are still framed as question answering. Representative examples include DocVQA [ 10 ] for document image question answering, InfographicVQA [ 11 ] for infographic understanding, ChartQA [ 9 ] for chart reasoning, and WebSRC [ 1 ] for webpage reading comprehension over screenshots and HTML structure. More recent benchmarks such as DUDE [ 13 ] , MMLongBench-Doc [ 8 ] , LongDocURL [ 3 ] , and M-LongDoc [ 2 ] further extend evaluation to multi-page or long-context settings, showing that current models still struggle when evidence is distributed across pages and layout regions. In addition, WorldVQA [ 15 ] focuses on measuring atomic visual world knowledge in MLLMs, emphasizing whether models can correctly ground and recognize real-world entities rather than perform only task-local reasoning. This perspective is also relevant to our setting, since some masked targets in MMTR-Bench cannot be recovered solely from local string matching and instead require implicit world knowledge together with surrounding visual and structural context. However, unlike WorldVQA [ 15 ] and most prior benchmarks, our work is centered on reconstructing masked content directly from multimodal context rather than answering explicit questions. Masking and reconstruction have also been widely used in document and webpage modeling, but mostly as training objectives rather than standalone evaluation tasks. LayoutLMv3 [ 6 ] uses unified text and image masking for document pre-training, UDOP [ 12 ] combines vision, text, and layout modeling with reconstruction-style objectives, and Pix2Struct [ 7 ] treats masked screenshot parsing as a pretraining signal for visually situated language understanding. These works suggest that reconstruction-based learning is useful for structured multimodal inputs, but they do not directly provide a benchmark centered on masked contextual reconstruction. In terms of evaluation, prior work on generative tasks often combines lexical metrics and semantic metrics, and recent LLM-based evaluators further indicate that no single metric is sufficient for all answer lengths and granularities [ 4 ] . This motivates our level-aware evaluation design for MMTR-Bench. To make the position of our benchmark clearer, Table 1 summarizes the main differences between MMTR-Bench and several representative benchmarks. While existing work has substantially improved evaluation across documents, charts, webpages, and world knowledge, most are still built around explicit question answering. In contrast, MMTR-Bench abandons explicit questions, requiring models to recover masked targets from surrounding visual, structural, and semantic context. Furthermore, unlike benchmarks such as WorldVQA that focus primarily on visual world knowledge, MMTR-Bench integrates this as just one component of a broader contextual reconstruction framework, uniquely combining multi-page inputs, diverse source coverage, long-context samples, and a level-aware evaluation protocol. Table 1: Comparison with related benchmarks. MMTR-Bench differs from prior work mainly in task format and evaluation design: it focuses on masked context reconstruction rather than explicit question answering, while also supporting broad visual-text sources, world-knowledge-dependent cases, and level-aware evaluation. Benchmark Explicit QA Multi-page / multi-page Web / Chart / Scene Coverage Masked Reconstruction Long-context / Long-form World Knowledge Level-aware Evaluation DocVQA [ 10 ] ✓ – – – – – – ChartQA [ 9 ] ✓ – ✓ – – – – WebSRC [ 1 ] ✓ – ✓ – – – – DUDE [ 13 ] ✓ ✓ – – – – – MMLongBench-Doc [ 8 ] ✓ ✓ – – ✓ – – LongDocURL [ 3 ] ✓ ✓ – – ✓ – – M-LongDoc [ 2 ] ✓ ✓ – – ✓ – –

Read the complete text