VLM-SlideEval: Evaluating VLMs on Structured Comprehension and Perturbation Sensitivity in PPT

Hyeonsu B. Kang, Emily Bao, Anjan Goswami · 2025

Read VLM-SlideEval: Evaluating VLMs on Structured Comprehension and Perturbation Sensitivity in PPT by Hyeonsu B. Kang, Emily Bao, Anjan Goswami (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-22045.txt.

Introduction

Hyeonsu Kang, Emily Bao, and Anjan Goswami grade vision-language models on an everyday artifact they are increasingly asked to judge: presentation slides. As models take on critic roles in automated pipelines, slide understanding matters, yet it has rarely been tested on its own terms.

Their framework, VLM-SlideEval, probes three levels of skill. Models must extract slide elements to match ground truth pulled from real deck files, stay consistent when geometry, style, or wording is deliberately perturbed. Recover the story order of a deck whose slides have been shuffled. On public decks, the models fell short at pixel-accurate extraction and wobbled noticeably under controlled changes, while doing better at the broad comprehension tasks. The results map where slide-savvy evaluation still needs work.

Catalog record

CREATORHyeonsu B. Kang, Emily Bao, Anjan Goswami
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-22045

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

VLM-SlideEval: Evaluating VLMs on Structured Comprehension and Perturbation Sensitivity in PPT Hyeonsu B. Kang, Emily Bao, Anjan Goswami Abstract Vision-language models (VLMs) are increasingly used to evaluate multimodal content, including presentation slides, yet their slide-specific understanding remains underexplored despite their growing role as critics in agentic, model-forward pipelines. We introduce VLM-SlideEval , an evaluation framework that probes VLMs along three axes: (1) element-level extraction from slide images aligned to ground truth; (2) robustness to controlled perturbations in geometry, style, and text; and (3) higher-level comprehension, such as recovering a deck’s narrative order from shuffled slides. Using publicly available decks from Zenodo 1 1 1 https://zenodo.org ; HF viewer: https://huggingface.co/datasets/Forceless/Zenodo10K/viewer/default/pptx , we standardize ground-truth element metadata from PowerPoint XML and live renderings into a unified, verifiable schema. Empirically, VLMs underperform on pixel-accurate extraction and show non-trivial agreement, fidelity, and consistency under controlled perturbations, while performing better on single-slide content understanding; however, they do not reliably capture narrative structure across slides. These results highlight the limits of current VLMs for slide evaluation and motivate calibrated, critic-in-the-loop evaluators that drive iterative refinement and selection in agentic pipelines. \workshoptitle Evaluating the Evolving LLM Lifecycle - Benchmarks, Emergent Abilities, and Scaling 1 Introduction Presentation slides are a primary vehicle for conveying structured ideas across domains ranging from education to scientific communication to corporate decision-making. Automatic evaluation of slide quality and content understanding is an emerging and pronounced need, particularly in light of advances in agentic, model-forward slide generation [ 1 , 2 ] . While prior work on document analysis has focused on optical character recognition (OCR) [ 3 , 4 , 5 ] and XML-based parsing [ 6 ] , these approaches are brittle when slides are only available as rendered images, and are limited to low-level layout information without reasoning about higher-level semantics. In contrast, vision-language models (VLMs) promise a unified mechanism for parsing slide content directly from images while also supporting tasks that require semantic or narrative comprehension. Despite the promise, it remains unclear to what extent current VLMs truly comprehend presentation slides. On one hand, VLMs may struggle with precise pixel-level tasks such as identifying bounding boxes, font attributes, or alignment, since they may not have been directly trained on raw presentation rendering pipelines or large scale OCR data of slide presentations. On the other hand, VLMs may excel at higher-level understanding, such as identifying the role of slide elements ( e.g., title, subtitle, body text), inferring content hierarchy, or reasoning over narrative flow in a deck. Understanding these trade-offs is crucial for designing reliable and scalable evaluation pipelines that utilize VLMs. We introduce VLM-SlideEval as a first-class critic in agentic, model-forward pipelines and systematically probe VLM slide comprehension. Our contributions are threefold. First, we curate a diverse dataset of PowerPoint decks and extract ground-truth geometry, style, and text via a pipeline combining PowerPoint XML with rasterized renders. Second, we design protocols for low-level fidelity and structured comprehension, including element-wise Hungarian alignment and refinement-relevant probes of judge reliability (variance, sensitivity) and robustness via controlled perturbations to geometry, style, and text. Third, we extend evaluation to deck-level narrative by asking VLMs to reorder shuffled slides, assessing coherence. Applying VLM-SlideEval, we surface clear limits and strengths. VLMs struggle with pixel-accurate extraction and show behavioral divergence under controlled perturbations, yet they competently extract structured content on single slides while remaining unreliable for deck-level narrative. These findings caution against over-reliance on current VLMs for fine-grained slide evaluation and motivate more calibrated critic-in-the-loop refinement and selection gates for agentic pipelines. Figure 1 : Evaluation Task Examples: Top: From an original slide (A), we parse a simplified schema JSON (Table 1 ) (B), reconstruct a normalized slide (C; blue dashed boxes show theme-embedded content omitted by the schema). A VLM predicts the schema from the re-rendered slide (D), and we score accuracy. Bottom: We subsample 100 decks, retain slides with \geq 3 visible elements (234 slides total), and apply perturbations to geometry, text, and style with severity s\in[0,1] (larger s means stronger changes; details in § 3 ). Perturbed slides are then used for VLM quality evaluation and sensitivity analyses (§ 4 ). 2 Related Work Calibrated VLM evaluators are increasingly critical in agentic, model-forward pipelines: they guide candidate selection, drive iterative refinement at inference time, and even supply reward signals for training. Recent work shows verifier-guided decoding that improves performance without weight updates [ 7 ] , generalist multimodal judges used both as LMM-as-a-Judge and as reward models [ 8 ] , actor-critic loops that critique and correct reasoning [ 9 ] , and refinement-centric benchmarks plus standardization frameworks that emphasize granular measurement [ 10 , 11 ] . Concurrently, Image2Struct benchmarks VLM image reconstruction on webpages, LaTex, and musical scores [ 12 ] . This motivates a slide-native, verifiable evaluator that produces actionable signals at pixel, element, and deck levels. Yet VLM evaluation remains challenging. Open-ended judging often relies on incomplete visual context and fuzzy rubrics, yielding inconsistent scores [ 13 ] , while models hallucinate and make perceptual errors in visually grounded reasoning [ 14 ] . Under controlled manipulations and counterfactuals, VLMs may inject priors unsupported by pixels and show limited sensitivity to fine-grained changes [ 15 , 16 ] . Robustness studies further find text corruptions especially damaging, lightweight adapters sometimes rivaling full fine-tuning, and broader axes (fairness, toxicity, multilinguality) underexplored [ 17 , 18 ] . Slide presentations sit within multimodal document understanding, where structured parsing underpins both comprehension and authoring. Prior work has explored language-driven manipulation of slide objects (not pixels) for faster, faithful editing [ 19 ] , OCR-free pretraining for screenshots and UI/text layouts that improves element-level parsing [ 20 ] , and automatic extraction of deck structure for role identification and accessibility [ 21 ] . In parallel, systems that generate slides from long-form documents highlight the need for scalable, slide-specific evaluation [ 2 , 22 ] . Unlike work that omits a slide-native evaluator, relies on QA proxies, or focuses robustness on charts/UIs, VLM-SlideEval provides a slide-specific framework that couples pixel-accurate alignment to PPT-native ground truth with slide-relevant perturbations and deck ordering, positioning the evaluator as a critic for agentic pipelines . 3 Method Data Source. We sample 100 English-dominant ( \geq\!70\% by langid [ 23 ] ) .pptx decks from Zenodo10K (legacy .ppt excluded), totaling 1,948 slides, with CC-BY 4.0 license (Summary statistics in Appendix A , Table 2 ). Ground Truth Element geometry , content , and style are extracted from PowerPoint XML and post-layout rendering. We parse static XML and then query the COM (Component Object Model) API after a layout pass to recover effective font metrics and tight text bounds (mitigating AutoFit and container/tight-box discrepancies). Elements are stored in a standardized schema with explicit units (Appendix A , Table 1 ). VLM Parsing & GT Matching. Slides are rasterized to PNG and sent with a fixed 960 \times 540px coordinate frame; we test five VLMs (via Azure) to return JSON validated against our schema (invalid JSON counts as a parse failure). Each slide is run ( N=3 ) times (low temperature), and metrics are reported per-run and pooled. Predictions are aligned to GT via Hungarian matching ( cf. [ 24 , 25

Read the complete text