Redundancy Principles for MLLMs Benchmarks

Zicheng Zhang, Xiangyu Zhao, Xinyu Fang, Chunyi Li, Xiaohong Liu, Xiongkuo Min, Haodong Duan, Kai Chen, Guangtao Zhai · 2025

Read Redundancy Principles for MLLMs Benchmarks by Zicheng Zhang, Xiangyu Zhao, Xinyu Fang, Chunyi Li, Xiaohong Liu, Xiongkuo Min, Haodong Duan, Kai Chen, Guangtao Zhai (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-13953.txt.

Introduction

Zicheng Zhang and co-authors analyze, in twenty twenty-five, redundancy in benchmarks for multimodal large language models. New benchmarks now appear by the hundreds each year, and many overlap. They study redundancy from three angles: overlap in the capabilities tested, excess in the number of test questions, and cross-benchmark overlap within a domain. Their analysis spans hundreds of models' scores across more than twenty benchmarks. The goal is to measure how much duplication exists in current evaluations.

From that evidence, they propose principles for building leaner, more effective benchmarks. They also offer strategies to refine existing ones and cut redundancy. Code is released publicly. The work speaks to anyone designing evaluations for fast-moving AI systems.

Catalog record

CREATORZicheng Zhang, Xiangyu Zhao, Xinyu Fang, Chunyi Li, Xiaohong Liu, Xiongkuo Min, Haodong Duan, Kai Chen, Guangtao Zhai
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-13953

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Redundancy Principles for MLLMs Benchmarks Zicheng Zhang, Xiangyu Zhao, Xinyu Fang, Chunyi Li, Xiaohong Liu, Xiongkuo Min, Haodong Duan, Kai Chen, Guangtao Zhai Abstract With the rapid iteration of Multi-modality Large Language Models (MLLMs) and the evolving demands of the field, the number of benchmarks produced annually has surged into the hundreds. The rapid growth has inevitably led to significant redundancy among benchmarks. Therefore, it is crucial to take a step back and critically assess the current state of redundancy and propose targeted principles for constructing effective MLLM benchmarks. In this paper, we focus on redundancy from three key perspectives: 1) Redundancy of benchmark capability dimensions, 2) Redundancy in the number of test questions, and 3) Cross-benchmark redundancy within specific domains. Through the comprehensive analysis over hundreds of MLLMs’ performance across more than 20 benchmarks, we aim to quantitatively measure the level of redundancy lies in existing MLLM evaluations, provide valuable insights to guide the future development of MLLM benchmarks, and offer strategies to refine and address redundancy issues effectively. Figure 1: Where Redundancy Exists? and Why Evaluate Redundancy? 1 Introduction Model Evaluation has always played a crucial role in the development of Multi-modal Large Language Models (MLLMs). Benchmarks serve not only as tools for assessing model accuracy but also as catalysts for driving innovation and improvements within the field. In the early stages, traditional model evaluation benchmarks such as GQA Hudson and Manning (2019) , VQA-V2 Antol et al. (2015) , VizWiz Bigham et al. (2010) , and TextVQA Singh et al. (2019) are characterized by relatively simple questions and answers, with responses often being a single word. This limits the depth of understanding and reasoning required from the models, making them less effective at evaluating the complex capabilities of modern MLLMs that are expected to handle more nuanced tasks. With the emergence of more powerful MLLMs Achiam et al. (2023) ; Team et al. (2023) ; Chen et al. (2024b) ; Wang et al. (2024b) ; Li et al. (2024a) ; Liu et al. (2024) , traditional evaluation frameworks have become inadequate to meet the flexible evaluation requirements. In response, a new generation of VQA benchmarks has arisen, such as MMBench Liu et al. (2025) , MMVet Yu et al. (2023a) , and MMMU Yue et al. (2024) . As MLLMs have rapidly iterated and evolved, their diverse capabilities across various domains have garnered increasing attention, which has led to the development of specialized benchmarks to evaluate MLLMs’ performance in specific areas, like Mathematics Task Lu et al. (2023) ; Zhang et al. (2025) ; Wang et al. (2024a) ; Zou et al. (2024) , Optical Character Recognition (OCR) Mishra et al. (2019) ; Liu et al. (2023a) ; Mathew et al. (2021) , Medical Field Hu et al. (2024) , Remote Sensing Li et al. (2024e) , Agents Yang et al. (2024) , GUIs Baechler et al. (2024) , and so on. The rapid proliferation of benchmarks has inevitably introduced significant redundancies, with overlapping capabilities being assessed and recurring questions appearing within and across benchmarks. Such redundancies create inefficiencies in model evaluation, repeatedly testing similar aspects of MLLM performance without contributing meaningful new insights. Additionally, this trend risks overemphasizing certain task types while neglecting others, potentially distorting research priorities. In this work, we address these challenges through a comprehensive and systematic exploration. 1.1 Identifying Redundancy Redundancy is an intrinsic and multifaceted issue in benchmarks, appearing in several key forms: • Redundancy across dimensions (intra-bench): Tasks within the same benchmark may evaluate overlapping capabilities of MLLMs, leading to repetitive assessments. • Redundancy among instances (intra-bench): Certain instances closely resemble others, providing minimal additional differentiation or insight for model evaluation. • Redundancy across benchmarks within specific domains: Benchmarks targeting specific domains often exhibit overlapping objectives or scopes, resulting in duplicated efforts across different evaluation sets. 1.2 Ideal Redundancy Principles Effective benchmarks should adhere to the following principles regarding redundancy: • Independence of dimensions: Ideal benchmarks should ensure that its dimensions are largely independent, minimizing overlap between them. However, some degree of redundancy may be inevitable when certain capabilities naturally require the interaction of multiple foundational skills, and redundancy should be carefully balanced to avoid excessive overlap. • Optimal instance count: A well-designed benchmark should strike a balance in the number of instances: neither too few nor too many, to ensure reliable and meaningful evaluations without introducing unnecessary redundancies. • Domain representativeness: A comprehensive benchmark targeted to a specific domain should represent the domain. This may involve purposeful overlap with other benchmarks within the same domain to reflect shared core capabilities. 1.3 Benifits of Evaluating Redundancy Evaluating and addressing redundancy offers several significant benefits, as shown in Fig. 1 : • Optimizing benchmark design: 1). Determines whether certain dimensions within a benchmark warrant separate assessments or can be consolidated; 2). Identifies the minimal and sufficient number of instances required for accurate evaluation; 3). Assesses the necessity of introducing new benchmarks within specific domains. • Enhancing efficiency in MLLM evaluation: 1). Determines whether a benchmark deviates from the domain’s distribution ; 2). Identifies the anchor benchmarks required to evaluate model performance within the domain. By systematically addressing redundancy, we can not only enhance the principles of benchmark design but also alleviate the resource demands of MLLM evaluation, creating a more streamlined and effective evaluation ecosystem. 2 Redundancy Framework We present a framework for evaluating redundancy among MLLM capabilities, defined as specific tasks within a benchmark. Our framework is grounded in the following prior assumption: When evaluating similar capabilities, the performance rankings of MLLMs should exhibit strong correlation. Conversely, significant differences in these rankings suggest the evaluated capabilities are relatively independent. Based on this principle, we propose the Performance Correlation Redundancy Framework , which quantifies redundancy by measuring the correlation of MLLM performance rankings. To ensure robustness and generalization capability, we leverage the comprehensive data from VLMEvalKit Duan et al. (2024) , which includes diverse benchmarks and performance results from more than 100 MLLMs. 2.1 Dimensions Redundancy Assume a benchmark consists of a set of dimensions, denoted as X=\{X_{1},X_{2},\ldots,X_{m}\} , where each X_{i} represents a specific dimension. Let N denote the number of MLLMs evaluated on these dimensions. For a given dimension X_{i} , we denote the ranking of the N MLLMs on this dimension as R_{i} . To quantify the redundancy of X_{i} , we compute the average rank correlation between R_{i} and the rankings R_{j} of all other dimensions X_{j} ( j\neq i ). Formally, the redundancy \rho(X_{i}) is defined as: \rho(X_{i})=\frac{1}{m-1}\sum_{\begin{subarray}{c}j=1\\ j\neq i\end{subarray}}^{m}\text{CORR}(R_{i},R_{j}), (1) where \text{CORR}(R_{i},R_{j}) is the correlation coefficient between the rankings R_{i} and R_{j} . • High \text{CORR}(R_{i},R_{j}) values can help identify potentially redundant dimension pairs. • \rho(X_{i}) represents the average redundancy level of dimension X_{i} , quantifying its overall overlap. By calculating the redundancy \rho(X_{i}) for all dimensions X_{i} in the benchmark and averaging these values, we can obtain the overall internal redundancy of the benchmark as well. Formally, the benchmark internal redundancy \rho_{BI} is defined as: \rho_{BI}=\frac{1}{m}\sum_{i=1}^{m}\rho(X_{i}), (2) where \rho(X_{i}) is the redundancy of the i -th dimension as previously defined. This metric reflects the average similarity among all dimensions within the benchmark. A lower \rho_{BI} suggests that the dimensions are relatively independent and diverse.

Read the complete text