PRISMM-Bench: A Benchmark of Peer-Review Grounded Multimodal Inconsistencies

Lukas Selch Yufang Hou M. Jehanzeb Mirza Sivan Doveh, James Glass Rogerio Feris Wei Lin, [0.5em] Johannes Kepler University Linz · 2025

Read PRISMM-Bench: A Benchmark of Peer-Review Grounded Multimodal Inconsistencies by Lukas Selch Yufang Hou M. Jehanzeb Mirza Sivan Doveh, James Glass Rogerio Feris Wei Lin, [0.5em] Johannes Kepler University Linz (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-16505.txt.

Introduction

Lukas Selch leads six co-authors in a 2025 test set for spotting flaws in science papers. Big models that read text and images now help with real research. Yet no one knew if they catch clashes among text, figures, tables, and maths. Their set, PRISMM-Bench, draws flaws flagged by real peer reviewers. It holds 384 such flaws from 353 papers.

Three tasks test if a model can find a flaw, fix it, and match flaw pairs. A strict answer format blocks cheap wins from answer patterns. The team tested 21 top models, both open and closed. Scores ran from about 28 to 54 percent, a low range. The gap shows how far models are from trusted science help.

Catalog record

CREATORLukas Selch Yufang Hou M. Jehanzeb Mirza Sivan Doveh, James Glass Rogerio Feris Wei Lin, [0.5em] Johannes Kepler University Linz
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE65 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-16505

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

PRISMM-Bench: A Benchmark of Peer-Review Grounded Multimodal Inconsistencies Lukas Selch Yufang Hou M. Jehanzeb Mirza Sivan Doveh, James Glass Rogerio Feris Wei Lin, [0.5em] Johannes Kepler University Linz Abstract Large Multimodal Models (LMMs) are increasingly applied to scientific research, yet it remains unclear whether they can reliably understand and reason over the multimodal complexity of papers. A central challenge lies in detecting and resolving inconsistencies across text, figures, tables, and equations, issues that are often subtle, domain-specific, and ultimately undermine clarity, reproducibility, and trust. Existing benchmarks overlook this issue, either isolating single modalities or relying on synthetic errors that fail to capture real-world complexity. We introduce PRISMM-Bench ( P eer- R eview-sourced I nconsistency S et for M ultimodal M odels), the first benchmark grounded in real reviewer-flagged inconsistencies in scientific papers. Through a multi-stage pipeline of review mining, LLM-assisted filtering and human verification, we curate 262 inconsistencies from 242 papers. Based on this set, we design three tasks, namely inconsistency identification, remedy and pair matching, which assess a model’s capacity to detect, correct, and reason over inconsistencies across different modalities. Furthermore, to address the notorious problem of choice-only shortcuts in multiple-choice evaluation, where models exploit answer patterns without truly understanding the question, we further introduce structured JSON-based answer representations that minimize linguistic biases by reducing reliance on superficial stylistic cues. We benchmark 21 leading LMMs, including large open-weight models (GLM-4.5V 106B, InternVL3 78B) and proprietary models (Gemini 2.5 Pro, GPT-5 with high reasoning). Results reveal strikingly low performance (26.1–54.2%), underscoring the challenge of multimodal scientific reasoning and motivating progress towards trustworthy scientific assistants. Figure 1: We collect reviewer-flagged inconsistencies in scientific papers and transform them into QA tasks that probe detection, correction, and reasoning over multimodal inconsistencies. 0 0 footnotetext: \dagger Correspondence: [email protected] 1 Introduction Recent advances in Large Multimodal Models (LMMs) have sparked growing interest in their potential to serve as intelligent assistants for scientific research ( Eger et al. 2025 ) , supporting tasks such as figure and chart interpretation ( Roberts et al. 2024 ; Tanasă & Oprea 2025 ; Wu et al. 2024 ) , paper summarization ( Tan et al. 2025 ; Saxena et al. 2025 ; Yu et al. 2025 ) , and error detection ( Yang et al. 2025b ; Miyai et al. 2024 ; Alsaif et al. 2024 ) . Yet, a central open question remains: can LMMs truly reason over the complex multimodal structure of scientific documents? A central challenge in this setting is detecting and resolving inconsistencies between text, figures, tables, or equations in scientific papers. These issues are often subtle, arising from copy-paste mistakes, outdated results, or inconsistent notation, and require domain knowledge to detect. Fig. 1 illustrates such a case, where the reward function is defined differently in the figure and the in-line text. Our analysis of ICLR 2025 submissions reveals that 17.0% contained at least one such inconsistency flagged by reviewers. These discrepancies undermine clarity, reproducibility, and ultimately scientific trust. Existing benchmarks, however, fall short of exposing this. Document QA datasets ( Mathew et al. 2021 ; Mathew et al. 2022 ; Zhu et al. 2022 ) or standalone scientific visual element tasks focusing on diagrams, charts, or tables ( Kafle et al. 2018 ; Masry et al. 2022 ; Cheng et al. 2022 ) miss the multimodal dependencies of scholarly works. Synthetic datasets ( Yan et al. 2025 ) generate artificial errors, but these are often obvious and unrepresentative of real-world complexity. However, constructing such a benchmark of authentic inconsistencies is challenging as these cases are rare, scattered, and labor-intensive to verify, often requiring domain expertise to identify and validate. To address this, we leverage a valuable but underutilized resource - open peer reviews , focusing on instances where reviewers flag mismatches across different modalities, such as between text, figures, tables, and equations, thereby providing a natural source of real-world, human-identified inconsistencies. In this paper, we introduce PRISMM-Bench, a P eer- R eview-sourced I nconsistency S et for M ultimodal M odels. Unlike synthetic datasets, PRISMM-Bench captures inconsistencies explicitly flagged by human reviewers in scientific papers on OpenReview. Through a multi-stage pipeline combining large-scale review mining, LLM-assisted filtering, and rigorous human verification, we curate a dataset of 262 inconsistencies across 242 papers submitted to ICLR 2025, spanning 13 categories of visual-textual and inter-visual mismatches. PRISMM-Bench provides a principled resource for evaluating and improving LMMs, grounded in the real challenges of understanding and verifying scientific papers. Grounded on these recent reviews, we minimize data contamination risk and demonstrate a pipeline with the potential to construct a continuously updated live benchmark. Building on this inconsistency set, we design a benchmark suite of three multiple-choice question (MCQ) tasks: 1) Inconsistency Identification - detect what the inconsistency is; 2) Inconsistency Remedy - determine how to fix the inconsistency; and 3) Inconsistency Pair Match - identify which two elements conflict. Together, these tasks form a tiered framework that evaluates not only a model’s ability to spot inconsistencies but also its capacity to propose remedy, and reason over relationships between different modality components. In MCQ evaluation, a key challenge is models’ tendency to rely on linguistic biases in answer choices. Prior work has shown that LLMs often exploit choice-only shortcuts, achieving non-trivial accuracy without reading the question ( Chandak et al. 2025 ; Turner & Kurzeja 2025 ; Balepur et al. 2024 ; Chizhov et al. 2025 ) , and similar effects appear in multimodal MCQs ( Chandak et al. 2025 ) . To address this, we propose a novel structured JSON-based answer representation that de-emphasizes stylistic cues and minimizes spurious correlations. Inspired by Das et al. 2014 and Banarescu et al. 2013 , our design converts free-form natural language into uniform structured representations that reduce model sensitivity to surface-level patterns. Our user study confirms that this approach suppresses linguistic shortcuts and better aligns models with human reasoning. We benchmark 21 state-of-the-art LMMs, spanning large open-weight models such as GLM-4.5V 106B ( Hong et al. 2025 ) and InternVL3 78B ( Zhu et al. 2025 ) , as well as leading proprietary models including Gemini 2.5 Pro ( Comanici et al. 2025 ) and GPT-5 ( OpenAI 2025 ) . Results show that while large open-weight models achieve around 40% accuracy, even the strongest proprietary models reach just 54.2%, underscoring difficulty of the benchmark and limitations of current LMMs. Our contributions are fourfold: (1) We propose a reviewer-sourced dataset of real multimodal inconsistencies in scientific papers, spanning diverse categories and grounded in peer review. (2) We construct a benchmark suite of three tasks probing detection, correction, and relational reasoning over these inconsistencies. (3) We are the first to propose JSON-based debiasing method for MCQ, converting free-form responses into uniform structured representations. (4) We evaluate 21 state-of-the-art LMMs, exposing their current limitations in detecting, understanding and correcting inconsistencies in scientific papers. The dataset and code for creating the benchmark and evaluating LMMs are publicly available. 2 Related Work Large Multimodal Models (LMMs). Large Multimodal Models (LMMs) pair a vision encoder with a large language model, enabling open-ended reasoning across tasks such as image captioning, VQA, document understanding, and chart interpretation. Early approaches like BLIP-2 ( Li et al. 2023 ) and InstructBLIP

Read the complete text