Format Matters: The Robustness of Multimodal LLMs in Reviewing Evidence from Tables and Charts

Xanh Ho, Yun-Ang Wu, Sunisth Kumar, Florian Boudin, Atsuhiro Takasu, Akiko Aizawa · 2025

Read Format Matters: The Robustness of Multimodal LLMs in Reviewing Evidence from Tables and Charts by Xanh Ho, Yun-Ang Wu, Sunisth Kumar, Florian Boudin, Atsuhiro Takasu, Akiko Aizawa (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-10075.txt.

Introduction

Xanh Ho, Yun-Ang Wu, Sunisth Kumar, Florian Boudin, Atsuhiro Takasu, and Akiko Aizawa ask whether the format of evidence changes how well multimodal models can check scientific claims. Results in papers appear as tables or charts, and reviewer-assistance tools must handle both. The team adapted two existing paper data sets with the annotations needed for a claim verification task in both formats.

Twelve multimodal models were tested, and all handled table evidence better than chart evidence. Human evaluators, by contrast, stayed strong across both formats, exposing a specifically machine weakness. Smaller models showed weak correlation between their table and chart scores, signaling limited transfer between formats. The authors urge future models to prioritize chart understanding for scientific reviewing support.

Catalog record

CREATORXanh Ho, Yun-Ang Wu, Sunisth Kumar, Florian Boudin, Atsuhiro Takasu, Akiko Aizawa
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-10075

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Format Matters: The Robustness of Multimodal LLMs in Reviewing Evidence from Tables and Charts Xanh Ho, Yun-Ang Wu, Sunisth Kumar, Florian Boudin, Atsuhiro Takasu, Akiko Aizawa Abstract With the growing number of submitted scientific papers, there is an increasing demand for systems that can assist reviewers in evaluating research claims. Experimental results are a core component of scientific work, often presented in varying formats such as tables or charts. Understanding how robust current multimodal large language models (multimodal LLMs) are at verifying scientific claims across different evidence formats remains an important and underexplored challenge. In this paper, we design and conduct a series of experiments to assess the ability of multimodal LLMs to verify scientific claims using both tables and charts as evidence. To enable this evaluation, we adapt two existing datasets of scientific papers by incorporating annotations and structures necessary for a multimodal claim verification task. Using this adapted dataset, we evaluate 12 multimodal LLMs and find that current models perform better with table-based evidence while struggling with chart-based evidence. We further conduct human evaluations and observe that humans maintain strong performance across both formats, unlike the models. Our analysis also reveals that smaller multimodal LLMs (under 8B) show weak correlation in performance between table-based and chart-based tasks, indicating limited cross-modal generalization. These findings highlight a critical gap in current models’ multimodal reasoning capabilities. We suggest that future multimodal LLMs should place greater emphasis on improving chart understanding to better support scientific claim verification. * * footnotetext: Research conducted during internship at NII, Japan. Code — https://github.com/Alab-NII/tables-vs-charts 1 Introduction Scientific claim verification requires models to determine whether a given claim is supported or not, based on the provided evidence. This evidence can take the form of text, tables, or charts. In recent years, several datasets have been introduced for this task, including SciFact ( 23 ) , HealthVer ( 20 ) , SciTab ( 15 ) , and MuSciClaims ( 12 ) . Depending on the dataset, the evidence may be textual (e.g., SciFact), tabular (e.g., SciTab), or visual (e.g., MuSciClaims). However, existing datasets typically represent evidence in only a single format (details in Section 2 ). While the recent SciVer dataset ( 24 ) incorporates multiple evidence modalities such as textual, tabular, and visual, it still presents tables in visual rather than text-based table format. Moreover, the different types of evidence in SciVer provide complementary information rather than expressing the same content across modalities. In the era of generative AI and large language models (LLMs), researchers can now produce research papers more efficiently with support from AI agents ( 19 ; 21 ) , leading to a significant increase in the number of submissions. As a result, automated peer review systems and tools that assist human reviewers are becoming increasingly important. A critical part of many research papers is the experimental results section, which presents key findings that support the main claims. These results are typically presented in tables or charts, depending on the author’s preference. Understanding how multimodal LLMs handle different evidence formats is essential, especially as they are integrated into review systems. If LLMs perform well on one format but poorly on another, this could lead to biased or incomplete evaluations. Therefore, assessing their robustness across formats is vital for building reliable and generalizable AI-assisted review systems. However, it remains unclear whether current multimodal LLMs can consistently verify claims across different evidence formats. We argue that an effective review system must evaluate claims accurately, regardless of how the supporting evidence is presented. Figure 1: An example of the claim verification task in our experiment, featuring both types of evidence: table and chart formats that represent the same information. It is noted that the original example is from the SciTab dataset. To ensure a fair comparison between table- and chart-evidence formats, we modified claims by replacing references to tables with figures (e.g., “Table 7” → “Figure 7”) to make them compatible with chart-format evidence. From a different task perspective, Chart2Code and Data2Chart are tasks that focus on converting between charts and the code used to generate them, or on transforming tabular data into visual charts. Several datasets have been developed for these tasks, such as MatPlotBench ( 31 ) , Plot2Code ( 28 ) , and ChartMimic ( 30 ) . These datasets typically provide paired examples of code and the corresponding chart, with an emphasis on the transformation between these two formats. While our research also involves both data formats, specifically charts and tables (assuming that tables can be derived from the underlying code), our focus is different. We treat these formats as distinct types of evidence for a downstream task, claim verification, rather than for format conversion. In this paper, we aim to investigate the robustness of current multimodal LLMs in performing claim verification across different evidence formats. Specifically, we consider both tables and charts as representations of the evidence data. We found that existing datasets do not fully meet the requirements of our task, as they typically lack aligned table and chart evidence that convey the same underlying information. To address this gap, we extend two existing datasets, namely SciTabAlign ( 9 ) and ChartMimic ( 30 ) , to construct a new dataset tailored to our claim verification setting. In particular, we curate instances that consist of a scientific claim, a table serving as evidence, and a corresponding chart that represents the same data as the table. This alignment enables a systematic evaluation of multimodal LLM performance in verifying claims using different, yet equivalent, evidence formats. Figure 1 illustrates an example of the claim verification task used in our experiments. Using the extended versions of these datasets, we conduct a series of experiments to assess model performance across various settings, including table-only input, chart-only input, and combined table–chart input. We also evaluate performance across different chart types: basic bar charts, symbol bar charts, line charts, and swapped charts. Our results from 12 multimodal LLMs show that current models struggle to verify claims when evidence is presented as charts but perform better when the same information is provided in tables, indicating a strong reliance on structured, text-like input. This discrepancy highlights a notable limitation in current multimodal LLMs’ ability to interpret visual data, even when the content is semantically equivalent. To better understand this gap, we asked human annotators to complete the same tasks. Unlike the models, humans performed consistently well across both formats, suggesting that the observed difficulty stems from model limitations rather than task ambiguity. These findings underscore the need for improved multimodal reasoning and better alignment between visual and textual representations in future model development. In summary, our main contributions are as follows. (1) We extend two existing datasets, SciTabAlign and ChartMimic, to create enhanced versions that support scientific claim verification using different evidence formats representing the same information. (2) We comprehensively evaluate 12 multimodal LLMs under three input settings: table-only, chart-only, and a combination of both. We also investigate performance across different chart types. (3) Our experimental results show that current multimodal LLMs struggle to process chart-based input, while performing better with text-based table input. Human performance on the same task confirms that people do not face similar difficulties with

Read the complete text