Benchmark Designers Should “Train on the Test Set” to Expose Exploitable Non-Visual Shortcuts

Ellis Brown Jihan Yang Shusheng Yang Rob Fergus Saining Xie New York University · 2025

Read Benchmark Designers Should “Train on the Test Set” to Expose Exploitable Non-Visual Shortcuts by Ellis Brown Jihan Yang Shusheng Yang Rob Fergus Saining Xie New York University (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-04655.txt.

Introduction

Ellis Brown leads five authors urging benchmark designers to train on their own test set. Multimodal models can ace vision benchmarks without seeing the images at all. Language priors and skewed answer patterns hand out free points.

Blind tests show what a model already gets for free, not what it could learn. Their Test-set Stress-Test cross-validates a text-only model on the test set's own questions. It yields a benchmark score plus a bias score for each question. A random forest on crafted features adds a fast, readable audit. The method exposes shortcuts before a benchmark ships.

Catalog record

CREATOREllis Brown Jihan Yang Shusheng Yang Rob Fergus Saining Xie New York University
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-04655

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Benchmark Designers Should “Train on the Test Set” to Expose Exploitable Non-Visual Shortcuts Ellis Brown Jihan Yang Shusheng Yang Rob Fergus Saining Xie New York University Abstract Robust benchmarks are crucial for accurately evaluating Multimodal Large Language Models (MLLMs). However, we find that models can ace many multimodal benchmarks without strong visual understanding by exploiting biases, linguistic priors, and superficial patterns. This is particularly problematic for vision-centric benchmarks, which explicitly aim to require visual inputs to be solved. We introduce a diagnostic principle for robust benchmark design: if a benchmark can be gamed, it will be. Therefore, designers should proactively try to “game” their own benchmarks first as a key step in the development lifecycle—adopting rigorous diagnostic and debiasing procedures to systematically identify, quantify, and mitigate non-visual biases. We demonstrate that effective diagnosis of these issues must involve directly “training on the test set”—i.e., probing the specific test set being released for its intrinsic, exploitable patterns. To demonstrate an effective realization of this standard, we propose a systematic approach involving two core components: First, we diagnose benchmark susceptibility using a “Test-set Stress-Test” ( TsT ) methodology. The primary diagnostic tool involves fine-tuning a powerful Large Language Model (LLM) via k k -fold cross-validation on exclusively the non-visual, textual inputs of the test set to unveil shortcut performance and derive a quantitative, sample-level bias score, s ⁡ ( x ) s(x) . We complement this with a lightweight Random Forest-based diagnostic trained on hand-crafted features, enabling rapid auditing and interpretable bias analysis. Second, we debias benchmarks by systematically filtering samples identified as highly biased according to s ⁡ ( x ) s(x) using an “Iterative Bias Pruning” ( IBP ) procedure. Applying this framework to four prominent benchmarks—VSI-Bench, CV-Bench, MMMU, and VideoMME—we uncover substantial and pervasive non-visual biases. As a case study, we apply our full framework to create VSI-Bench-Debiased, demonstrating a marked reduction in non-visual solvability and a significantly wider vision-blind performance gap compared to the original. Project Page: https://cambrian-mllm.github.io 1 Introduction Figure 1 : The Evolving Landscape of Visual Understanding Benchmarks. As benchmarks evolved from controlled, narrow tasks to open-ended VQA, they gained expressivity but became vulnerable to non-visual shortcuts. Language-driven evaluation enables flexible querying but risks models exploiting linguistic patterns rather than visual understanding. Despite significant advances in visual understanding, evaluating progress remains challenging. From MNIST [ 17 ] to ImageNet [ 11 ] , from COCO [ 19 ] to VQA [ 4 ] , benchmarks have served as the backbone of progress in computer vision. As tasks grew more complex and models more capable, benchmarks evolved accordingly—shifting from clean, domain-specific datasets to open-ended, real-world evaluations ( Fig. 1 ). Today, with multimodal LLMs, we can ask models anything about an image—but this expressive power comes at a hidden cost: we’ve lost control over what’s being measured. The uncomfortable truth is that models can ace multimodal benchmarks without strong visual understanding. They exploit biases, linguistic priors, and superficial patterns. A model can answer a visual question without looking at the image. It can ground language without grounding perception, yet still achieve a high score. Although concerns about language bias were raised years ago in the VQA era (see Section 5.1 ), the rise of LLMs as both knowledge engines and sources of shortcuts makes it necessary to reexamine evaluation methodology and benchmark design with greater attention to nuance and complexity [ 30 ] . This observation motivates the central argument of this paper: multimodal benchmark designers should proactively stress-test their creations for exploitable non-visual shortcuts . While it has become a common sanity check, the “blind” test [ 30 ] —where a multimodal benchmark is evaluated with vision disabled —only reveals when vision is unnecessary. It offers no insight into why specific samples are exploitable or how to fix the non-visual shortcuts. We demonstrate that the most rigorous and useful stress test involves directly “training on the test set” —not to overfit, but to adversarially probe the test set for such intrinsic vulnerabilities. This approach elevates robust benchmark design from simple static dataset curation to an iterative and adversarial refinement process. This “train on the test set ” framing is critical. While training diagnostic models on held-out in-distribution data can reveal biases that generalize across a domain, such an approach may miss idiosyncratic vulnerabilities unique to the specific test set artifact , e.g., those arising from the sampling process, templated questions, or human filtering decisions. Our proposed methodology directly targets these test-set-specific shortcuts, helping to ensure that reported benchmark scores reflect genuine multimodal capabilities rather than the exploitation of unique statistical artifacts present in the evaluation instrument. To demonstrate an effective realization of these principles, we introduce systematic approaches for both diagnosing and mitigating non-visual shortcuts. We first diagnose benchmark susceptibility using a “Test-set Stress-Test” ( TsT ) methodology that applies k -fold cross-validation to train diagnostic models exclusively on non-visual test-set features, deriving both an overall exploitability measure and sample-level bias scores, s(x) . We realize this diagnostic through two complementary approaches: a powerful LLM-based variant (TsT-LLM) that LoRA-tunes a language model on question-only inputs to capture complex shortcuts, and a lightweight Random Forest variant (TsT-RF) trained on hand-crafted features that offers efficiency and interpretability for rapid auditing. We then debias benchmarks by systematically filtering samples identified as highly biased according to s(x) using an “Iterative Bias Pruning” ( IBP ) procedure. Applying our diagnostic framework to prominent vision-centric multimodal benchmarks such as VSI-Bench [ 31 ] and CV-Bench [ 30 ] , we uncover substantial non-visual biases, highlighting the prevalence of this issue ( Section 3 ). As a primary case study, we apply our framework to VSI-Bench, resulting in VSI-Bench-Debiased ( Section 4.2 ). Our experiments demonstrate that TsT effectively quantifies these shortcuts, and that IBP debiasing markedly reduces non-visual solvability, leading to a significantly wider performance gap between vision-enabled and “blind” MLLM configurations on VSI-Bench-Debiased compared to the original. In summary, our contributions are fourfold: (1) A principled stance on what constitutes an “exploitable” non-visual shortcut ( Section 2.1 ); (2) A generalizable TsT diagnostic framework realized through both a high-efficacy LLM-based variant and an interpretable Random Forest variant ( Section 3 ); (3) Empirical validation across four major multimodal benchmarks—VSI-Bench, CV-Bench, MMMU, and VideoMME—revealing pervasive non-visual shortcuts ( Section 3.4 ); and (4) The IBP mitigation methodology and its application to create VSI-Bench-Debiased, a demonstrably less compromised subset ( Section 4 ). In the remainder of this paper, we detail the challenge of non-visual shortcuts ( Section 2 ), present our diagnostic and mitigation methodologies ( Sections 3 and 4 ), discuss related work ( Section 5 ), and conclude with a call for more rigorous benchmark design practices ( Section 6 ). 2 The Challenge: Non-Visual Shortcuts Undermine Multimodal Evaluation In multimodal evaluation, non-visual shortcuts occur when questions can be solved without utilizing the visual input. This can happen when MLLMs exploit world knowledge acquired during linguistic pretraining or leverage statistical correlations within the question-answer pairs themselves. The prevalence of such shortcuts leads to inflated performance metrics, misrepresents true visual understanding capabilities, and can misguide research by rewarding pattern matching over genuine multimodal reasoning. In this section, we first establish a

Read the complete text