Rethinking Comprehensive Benchmark for Chart Understanding: A Perspective from Scientific Literature

Lingdong Shen, Qigqi, Kun Ding, Gaofeng Meng, Shiming Xiang · 2024

Read Rethinking Comprehensive Benchmark for Chart Understanding: A Perspective from Scientific Literature by Lingdong Shen, Qigqi, Kun Ding, Gaofeng Meng, Shiming Xiang (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-12150.txt.

Introduction

Lingdong Shen and four co-authors wrote this 2024 paper on how well AI models read charts in science papers. Charts in real papers can be dense, with flow charts, stacked plots, and diagrams side by side. Older tests used plain charts and plain prompts, so scores looked better than the skill behind them. The team set out to build a harder and fairer test.

Their new test, called SCI-CQA, draws on charts from top computer science venues over ten years. The set holds tens of thousands of charts, plus more than five thousand exam style questions. Some questions have fixed answers, and some are open ended. A cheap labeling pipeline helped build the set. Tests with added context show that nearby text often turns a hard chart question into one a model can answer.

Catalog record

CREATORLingdong Shen, Qigqi, Kun Ding, Gaofeng Meng, Shiming Xiang
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-12150

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Rethinking Comprehensive Benchmark for Chart Understanding: A Perspective from Scientific Literature Lingdong Shen, Qigqi, Kun Ding, Gaofeng Meng, Shiming Xiang Abstract Scientific Literature charts often contain complex visual elements, including multi-plot figures, flowcharts, structural diagrams and etc. Evaluating multimodal models using these authentic and intricate charts provides a more accurate assessment of their understanding abilities. However, existing benchmarks face limitations: a narrow range of chart types, overly simplistic template-based questions and visual elements, and inadequate evaluation methods. These shortcomings lead to inflated performance scores that fail to hold up when models encounter real-world scientific charts. To address these challenges, we introduce a new benchmark, Scientific Chart QA (SCI-CQA), which emphasizes flowcharts as a critical yet often overlooked category. To overcome the limitations of chart variety and simplistic visual elements, we curated a dataset of 202,760 image-text pairs from 15 top-tier computer science conferences papers over the past decade. After rigorous filtering, we refined this to 37,607 high-quality charts with contextual information. SCI-CQA also introduces a novel evaluation framework inspired by human exams, encompassing 5,629 carefully curated questions, both objective and open-ended. Additionally, we propose an efficient annotation pipeline that significantly reduces data annotation costs. Finally, we explore context-based chart understanding, highlighting the crucial role of contextual information in solving previously unanswerable questions. Our project will be released at https://github.com/lydonShen/SCI-CQA . Introduction Deep learning-based chart understanding focuses on tasks like chart question answering ( Lu et al. 2024 ; Kafle et al. 2018 ; Masry et al. 2024 ) and chart captioning ( Tang, Boggust, and Satyanarayan 2023 ; Kantharaj et al. 2022 ) . In recent years, the advent of multimodal models ( Liu et al. 2023b ; Bai et al. 2023a ) has revolutionized this field, as highlighted in ( Huang et al. 2024 ) . Foundational models have consistently set new performance records on chart understanding benchmarks. However, these improved results do not conclusively demonstrate that large models have achieved human-level capabilities in chart understanding. Figure 1: Comparison of SCI-CQA with previous work. The black box represents prior work, the green box highlights the SCI-CQA flowchart section, and the orange box represents the SCI-CQA data chart section. This comparison underscores SCI-CQA’s advancements in data diversity and question complexity. Existing research encounters three critical issues. First, the charts used are limited in variety, characterized by fixed styles and simple visual representations, while overlooking challenging but essential types like flowcharts. For example, some studies rely on synthetic data to generate charts ( Kafle et al. 2018 ; Kahou et al. 2018 ) , or use web-sourced raw data with automated programs to create charts ( Methani et al. 2020 ) (Left of Fig. 1 ). These generated charts have a significant complexity gap with scientific literature chart. Second, the template-based questions and overly simplistic chart visual elements allows models to bypass the actual chart input, either by directly answering questions or by selecting correct answers based on hallucinations. For example, FigureQA ( Kahou et al. 2018 ) uses 15 templates to generate question-answer pairs, making it easy for large multimodal models ( Achiam et al. 2023 ; Bai et al. 2023a ) to achieve high accuracy without truly understanding the chart. Third, evaluation methods tend to be either only including subjective questions or objective questions, with insufficient differentiation between types of evaluation. This lack prevents a precise assessment of the models’ specific strengths and weaknesses. For instance, ( Xu et al. 2023 ) uses ACC+ for objective evaluation, while ( Han et al. 2023 ; Wang et al. 2024 ) rely on GPT-4 scoring for open-ended assessment. To ensure a more thorough and equitable assessment of models’ chart understanding abilities, we have created the SCI-CQA benchmark, drawing from scientific literature. By utilizing the wealth of charts available in scientific papers, SCI-CQA addresses the issues identified in previous benchmarks. We collected 202,760 image-text pairs with context and captions from papers presented at 15 top-tier computer science conferences. Through a rigorous three-stage filtering process, we refined this dataset to 37,607 high-quality image-text pairs . These pairs offer a diverse range of chart types and styles, rich in visual elements, varied in distribution, and include multiple sub-charts (see Right of Fig. 1 ). Notably, we are the first to collect and annotate scientific flowcharts and evaluate them as a distinct category. To avoid inaccurate performance evaluations due to template-based questions, we drew inspiration from ( Wang et al. 2024 ) and proposed six types of base questions to broaden the scope of answers. By incorporating charts and textual context inputs, we generated questions that cannot be answered solely from the images. We then used GPT-4o to produce the corresponding question-answer pairs. The question-answer pairs were then manually curated to form a high-quality validation set. As shown in Fig. 1 , structure diagrams ( Shen et al. 2024 ) frequently cause leading multimodal models to make errors due to their complex visual elements and relationships. Additionally, real chart data—characterized by multiple subplots, image-chart combinations, and intricate data distributions—poses significant challenges to both open-source and proprietary models, resulting in suboptimal performance. Finally, we introduced an evaluation method inspired by human exams, incorporating multiple-choice, true/false, and open-ended questions. We compiled a question bank of 5,629 questions , all of which were reviewed by master’s and doctoral students specializing in computer science. Given that these data were sourced from the computer science domain, the review by highly educated professionals ensured both the diversity and accuracy of the questions. Additionally, we developed an automated annotation pipeline to cut the costs associated with proprietary model annotation. Using the contextual data provided by SCI-CQA, we also explored whether complex reasoning questions, which cannot be answered solely from images, can be effectively resolved with the assistance of contextual information. In summary, our contributions include rethinking a more comprehensive evaluation method for chart understanding from the perspective of scientific literature, thereby promoting advancements in the field of chart understanding. Related Work Chart understanding benchmark Previous work reveals room for improvement in data diversity, question complexity, and evaluation fairness. In terms of data and visual element diversity, previous studies such as ( Kafle et al. 2018 ; Kahou et al. 2018 ; Methani et al. 2020 ; Singh et al. 2024 ) rely on charts synthesized from real data, resulting in limited data diversity and simple visual elements. These charts often lack the complexity and variety found in actual scientific literature, which typically features a wide range of presentation styles, complex data distributions, and rich visual information (see the right side of Fig. 1 ). While ( Han et al. 2023 ) utilizes multimodal large models to generate code for creating charts, their approach includes only 4-8 chart types. ( Xia et al. 2024 ) expanded the chart types to 18 categories but still faced challenges with visual element diversity. Additionally, ( Wang et al. 2024 ) used real chart data to address these limitations but overlooked flowcharts, an essential chart type. SCI-CQA, inspired by ( Wang et al. 2024 ) , utilizes natural data charts for model evaluation and includes flowcharts (such as model architecture diagrams and algorithm flowcharts), which are rich in information and crucial to scientific literature. Natural data charts

Read the complete text