Judging the Judges: Can Large Vision-Language Models Fairly Evaluate Chart Comprehension and Reasoning?
Md Tahmid Rahman Laskar, Mohammed Saidul Islam, Ridwan Mahbub ,, Ahmed Masry, Mizanur Rahman, Md Amran Hossen Bhuiyan,, Mir Tafseer Nayeem, Shafiq Joty, Enamul Hoque , Jimmy Xiangji Huang · 2025
Read Judging the Judges: Can Large Vision-Language Models Fairly Evaluate Chart Comprehension and Reasoning? by Md Tahmid Rahman Laskar, Mohammed Saidul Islam, Ridwan Mahbub ,, Ahmed Masry, Mizanur Rahman, Md Amran Hossen Bhuiyan,, Mir Tafseer Nayeem, Shafiq Joty, Enamul Hoque , Jimmy Xiangji Huang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-08468.txt.
Introduction
Md Tahmid Rahman Laskar and colleagues audited vision-language models hired to grade other models' chart reading in a 2025 paper. Charts carry much of the world's data communication, and tasks like chart question answering are booming, but scoring model answers by hand is slow and costly. Using a large model as judge is tempting, though proprietary data, restricted access, and price have slowed industrial adoption, so the team tested thirteen open-source models under ten billion parameters as judges.
Both pairwise comparisons and pointwise scores were run across criteria like factual correctness, informativeness, and relevance. The analysis also probed each judge's format adherence, positional consistency, length bias, and instruction following. The result is a cost-conscious map of which cheap judges can be trusted, and on what.
Catalog record
| CREATOR | Md Tahmid Rahman Laskar, Mohammed Saidul Islam, Ridwan Mahbub ,, Ahmed Masry, Mizanur Rahman, Md Amran Hossen Bhuiyan,, Mir Tafseer Nayeem, Shafiq Joty, Enamul Hoque , Jimmy Xiangji Huang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-08468
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Judging the Judges: Can Large Vision-Language Models Fairly Evaluate Chart Comprehension and Reasoning? Md Tahmid Rahman Laskar, Mohammed Saidul Islam, Ridwan Mahbub ,, Ahmed Masry, Mizanur Rahman, Md Amran Hossen Bhuiyan,, Mir Tafseer Nayeem, Shafiq Joty, Enamul Hoque , Jimmy Xiangji Huang Abstract Charts are ubiquitous as they help people understand and reason with data. Recently, various downstream tasks, such as chart question answering, chart captioning, etc. have emerged. Large Vision-Language Models (LVLMs) show promise in tackling these tasks, but their qualitative evaluation is costly and time-consuming, limiting real-world deployment. While using LVLMs as judges to assess chart comprehension capabilities of other LVLMs could streamline evaluation processes, challenges like proprietary datasets, restricted access to powerful models, and evaluation costs hinder their adoption in industrial settings. To this end, we present a comprehensive evaluation of 13 open-source LVLMs as judges for diverse chart comprehension and reasoning tasks. We design both pairwise and pointwise evaluation tasks covering criteria like factual correctness, informativeness, and relevancy. Additionally, we analyze LVLM judges based on format adherence, positional consistency, length bias, and instruction-following. We focus on cost-effective LVLMs ( ≤ 9 \leq 9 B parameters) suitable for both research and commercial use, following a standardized evaluation protocol and rubric to measure the LVLM judge accuracy. Experimental results reveal notable variability: while some open LVLM judges achieve GPT-4-level evaluation performance (about 80% agreement with GPT-4 judgments), others struggle (below 10% agreement). Our findings highlight that state-of-the-art open-source LVLMs can serve as cost-effective automatic evaluators for chart-related tasks, though biases such as positional preference and length bias persist. Figure 1: An example evaluation of Gemini-1.0-Pro model generated L2/L3 caption in the VisText dataset by an LVLM judge: the LLaVA-Critic-7B model. 1 Introduction Understanding data visualizations—such as bar and line charts—requires multimodal reasoning, as it involves integrating visual encodings with textual and contextual information Hoque and Islam (2024) . Recent research has introduced various tasks (e.g., chart question answering, chart captioning, fact-checking with charts, etc.) to facilitate chart-based reasoning via natural language. These tasks demand the understanding of both the chart’s visual content (data values, trends, visual encodings) and accompanying text or instructions. Large Language Models (LLMs) have revolutionized NLP and vision-language tasks Zhao et al. (2023) , with growing interest in their use for chart comprehension and reasoning due to their strong multimodal capabilities. This progress can have a substantial impact on real-world industrial applications, where extracting insights from charts and graphs can drive critical business decisions Obeid and Hoque (2020) ; Masry et al. (2023) ; Meng et al. (2024) . However, evaluating LLM performance in chart understanding presents notable challenges Islam et al. (2024) . For instance, traditional text-based metrics like BLEU fail to capture the quality of open-ended explanatory answers and also require human-annotated references. While human evaluation can address this problem, it is time-consuming and resource-intensive. To address this, recent studies have proposed using LLMs themselves as automatic evaluators or judges Gu et al. (2024) ; Li et al. (2024b) . By employing LLMs to evaluate the chart comprehension abilities of other models (see Figure 1 for an example), the evaluation process can be streamlined, making the process more efficient and reproducible without human intervention. While this method accelerates development and reduces dependency on human annotations, its real-world adoption is hindered by privacy and scalability constraints. For example, organizations may be unwilling to share proprietary data with closed-source models from OpenAI, Google, or Anthropic. While closed-source models demonstrate impressive judging capabilities, their compatible open-source models are often large in size (e.g., 70B to 400B parameters). This requires high computing resources and usage costs. Therefore hinders real-world utilization. To this end, this paper aims to investigate whether open-source smaller LVLMs (e.g., less than 10B parameters) can effectively evaluate answers about charts—assessing correctness, relevance, and other qualities—similarly to a human or a powerful LLM like GPT-4 OpenAI et al. (2023) . For this purpose, we conduct one of the first comprehensive evaluations of open-source LVLMs as evaluators on various chart benchmarks, consisting of diverse tasks like chart captioning and question answering. We focus on open-source, smaller VLMs (up to 10B parameters) to simulate realistic deployment scenarios where cost-effective or private models are preferred over large closed models. By benchmarking these models against high-quality reference judgments generated by closed-source LLMs like GPT-4 or 70B open-source LLM-Judge like LLaVA-Critic Xiong et al. (2024) , we aim to uncover to what extent current open models can serve as reliable judges, and when they fail. Our major contributions to this paper are: 1. We establish an evaluation framework for chart comprehension using “LVLM-as-a-Judge” , with clear rubrics for pairwise and pointwise assessments over 100K judgments generated by GPT-4o and LLaVA-Critic-70B. Additionally, we introduce a new benchmark to assess the instruction-following abilities of LVLMs in chart-related tasks. 2. We evaluate a wide range of open-source multimodal LLMs as judges – 13 models ranging from 2B to 9B parameters – and analyze their performance against LLM-annotated (GPT-4 and LLaVA-Critic) and human-annotated reference judgments, across diverse chart benchmarks (OpenCQA and VisText) on answers generated by different LLMs to create challenging evaluation scenarios. 3. We provide an in-depth analysis of the judges’ strengths and weaknesses, revealing issues like position bias and length bias, and discuss which models achieve substantially higher agreement with reference judgments, and which ones fail. In addition, our code, judgment data, and our proposed instruction-following evaluation benchmark is released here: https://github.com/tahmedge/chart_lvlm_judge 2 Related Work Earlier efforts in chart question answering include synthetic datasets like FigureQA Kahou et al. (2017) and DVQA Kafle et al. (2018) , which generated templated QA pairs for simple charts but lacked real-world complexity. ChartQA Masry et al. (2022) addressed this gap with real-world charts and more complex questions, while OpenCQA Kantharaj et al. (2022) pushed further with open-ended, explanatory queries. Meanwhile, chart captioning has emerged as another avenue for summarizing chart content Shankar et al. (2022) ; Rahman et al. (2023) ; Tang et al. (2023) . Together, these datasets highlight the growing complexity of chart-based reasoning tasks and the need for more robust evaluation methods. While the rise of multimodal LLMs offers potential for chart-related tasks, general vision-language models often struggle with chart-specific elements like axis text and precise data points Islam et al. (2024) . To address this, specialized models such as ChartLLaMA Han et al. (2023) , ChartInstruct Masry et al. (2024) , ChartGemma Masry et al. (2025) , and TinyChart ( Zhang et al., 2024 ) have been developed, showing strong performance. However, evaluating these models is challenging, as many still depend on time-consuming human assessments for open-ended responses. While using LLMs to evaluate other LLMs has gained a lot of attention, early efforts focused primarily on text-only tasks like summarization Li et al. (2024b) ; Zheng et al. (2023) . For multimodal tasks, models such as Prometheus-VL Lee et al. (2024) and LLaVA-Critic Xiong et al. (2024) introduced smaller open-source vision-language models (as small as 7B) fine-tuned to serve as general-purpose multimodal evaluators. Our work aligns with this direction, leveraging LVLMs as judges. Although concurrent studies explore similar capabilities Chen et al. (2024) , they report that early LVLMs like LLaVA-1.5 struggle with text-rich visuals such as charts and diagrams Lee