AVA-Bench: A tomic V isual A bility Benchmark for Vision Foundation Models
Zheda Mai, Arpita Chowdhury, Zihe Wang, Sooyoung Jeon, Lemeng Wang, Jiacheng Hou, Jihyung Kil, Wei-Lun Chao, The Ohio State University, Adobe Research https://zheda-mai.github.io/AVA-Bench/ · 2025
Read AVA-Bench: A tomic V isual A bility Benchmark for Vision Foundation Models by Zheda Mai, Arpita Chowdhury, Zihe Wang, Sooyoung Jeon, Lemeng Wang, Jiacheng Hou, Jihyung Kil, Wei-Lun Chao, The Ohio State University, Adobe Research https://zheda-mai.github.io/AVA-Bench/ (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-09082.txt.
Introduction
The authors present AVA-Bench, a test that splits visual skill into fourteen basic abilities for checking vision models. Common tests pair a vision model with a language model and ask mixed questions, so one weak skill can hide among many. Each ability here, like judging depth or finding objects, is tested on its own with matched training and test data.
Runs on top models reveal distinct fingerprints of strengths and gaps. A small language model gave nearly the same rankings as one fourteen times larger, at an eighth of the computing cost. The team hopes the bench makes model choice a matter of design, not guesswork.
Catalog record
| CREATOR | Zheda Mai, Arpita Chowdhury, Zihe Wang, Sooyoung Jeon, Lemeng Wang, Jiacheng Hou, Jihyung Kil, Wei-Lun Chao, The Ohio State University, Adobe Research https://zheda-mai.github.io/AVA-Bench/ |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 73 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-09082
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
AVA-Bench: A tomic V isual A bility Benchmark for Vision Foundation Models Zheda Mai, Arpita Chowdhury, Zihe Wang, Sooyoung Jeon, Lemeng Wang, Jiacheng Hou, Jihyung Kil, Wei-Lun Chao, The Ohio State University, Adobe Research https://zheda-mai.github.io/AVA-Bench/ Abstract The rise of vision foundation models (VFMs) calls for systematic evaluation. A common approach pairs VFMs with large language models (LLMs) as general-purpose heads, followed by evaluation on broad Visual Question Answering (VQA) benchmarks. However, this protocol has two key blind spots: (i) the instruction tuning data may not align with VQA test distributions, meaning a wrong prediction can stem from such data mismatch rather than a VFM’ visual shortcomings; (ii) VQA benchmarks often require multiple visual abilities, making it hard to tell whether errors stem from lacking all required abilities or just a single critical one. To address these gaps, we introduce AVA-Bench , the first benchmark that explicitly disentangles 14 Atomic Visual Abilities (AVAs)—foundational skills like localization, depth estimation, and spatial understanding that collectively support complex visual reasoning tasks. By decoupling AVAs and matching training and test distributions within each, AVA-Bench pinpoints exactly where a VFM excels or falters. Applying AVA-Bench to leading VFMs thus reveals distinctive “ability fingerprints,” turning VFM selection from educated guesswork into principled engineering. Notably, we find that a 0.5B LLM yields similar VFM rankings as a 7B LLM while cutting GPU hours by 8 × \times , enabling more efficient evaluation. By offering a comprehensive and transparent benchmark, we hope AVA-Bench lays the foundation for the next generation of VFMs. Figure 1 : Vision foundation models (VFMs) trained with different data and objectives are evaluated on the proposed AVA-Bench to assess their strengths and limitations across atomic visual abilities (AVAs). 1 Introduction Figure 2 : Visual Question Answering (VQA) often requires multiple atomic visual abilities (AVAs) to answer a question. As such, when a model makes an incorrect prediction, it can be difficult to determine whether it stems from a failure to capture all required AVAs or just a single critical one. Vision Foundation Models (VFMs), pre-trained on large and diverse datasets, have become central to AI by providing transferable features for a wide range of downstream tasks [ 40 , 2 , 13 , 5 ] . The variety of pre-training objectives and supervision signals has led to a proliferation of specialized VFMs—such as DINOv2 [ 65 ] , CLIP [ 67 ] , and SAM [ 43 ] —each excelling in distinct visual capabilities while often exhibiting interesting emergent properties [ 24 , 6 , 64 ] . Consequently, establishing a systematic and effective evaluation protocol for VFMs has become increasingly crucial. Existing evaluation protocols can generally be categorized into two groups. The first focuses on task-specific capabilities, typically attaching tailored heads to VFMs, followed by fine-tuning and evaluation on dedicated datasets such as ImageNet for classification [ 30 , 57 ] and COCO for detection or segmentation [ 79 , 3 ] . To better capture the diverse and complex perception challenges of the real world, recent studies advocate a more generic approach that leverages large language models (LLMs) as general-purpose heads, evaluating VFMs on broad Visual Question Answering (VQA) benchmarks [ 54 , 110 , 12 ] . While increasingly adopted, this generic protocol suffers from two potential blind spots: (i) discrepancies between the instruction tuning data and test VQA data can cause performance drops due to data mismatch rather than genuine visual limitations in VFMs; and (ii) existing VQA benchmarks typically require multiple visual abilities simultaneously, making it difficult to determine whether a failure arises from the absence of multiple abilities or merely a single critical one. To address these challenges, we introduce AVA-Bench , the first VFM evaluation benchmark explicitly designed to disentangle A tomic V isual A bilities ( AVAs )—the fundamental visual capabilities that combine to solve complex visual reasoning tasks. For instance, answering typical VQA questions shown in Figure 2 necessitates integrating several AVAs. Specifically, AVA-Bench evaluates VFMs across 14 carefully identified AVAs (see Figure 3 ), including localization , counting , spatial reasoning , orientation , absolute and relative depth estimation , and recognition of textures , colors , objects , actions , emotions , optical characters (OCR) , and scenes . Each AVA comes with distribution-matched train and test splits and is probed in isolation , eliminating the two aforementioned ambiguities. This enables AVA-Bench to pinpoint exactly where a VFM excels or falters, providing a clear picture of its strengths and weaknesses. We systematically benchmark leading VFMs trained under diverse objectives and data (see Figure 1 ), covering language-supervised ( e.g. , SigLIP-1/2 [ 82 , 101 ] , CLIP [ 67 ] , InternVL-2.5 [ 9 ] ), multimodal autoregressive (AIMv2 [ 20 ] ), segmentation-supervised (SAM [ 43 ] ), depth-supervised (MiDas [ 68 ] ), contrastive self-supervised (DINOv2 [ 65 ] ), and agglomerative models (RADIO [ 70 ] ). We follow the standard protocol of adding an LLM on top, but fine-tune it separately for each AVA. Our extensive analyses reveal the following main findings: (1) SigLIP-1/2 and AIMv2 emerge as the most versatile VFMs, achieving the highest average rank across all AVAs— highlighting the critical role of language supervision in enhancing general visual capability ; (2) For vision-centric AVAs such as localization, absolute depth estimation, and orientation, the SSL-based DINOv2 performs comparably or better than language-supervised counterparts; (3) Conversely, language-centric tasks such as OCR strongly favor language-supervised VFMs; (4) Last but not least, we observe that VFMs universally excel at low- to mid-level AVAs ( e.g. , texture, relative depth estimation, object recognition), regardless of their training objectives—suggesting that VQA failures typically stem from deficiencies in specific critical AVAs rather than broad visual incompetence. These insights turn VFM selection from educated guesswork into principled engineering, enabling practitioners to choose (or ensemble) VFMs based on the specific AVA strengths required by their downstream tasks. Alongside our main study, we identify a more resource-efficient evaluation strategy. Existing LLM-based VFM evaluations typically rely on heavyweight models such as Vicuna-1.5(7B/13B) [ 80 , 35 ] , aiming for high absolute accuracy but incurring significant computational overhead. However, when the goal is to compare VFMs, we advocate prioritizing relative performance over absolute accuracy. We demonstrate that a lightweight 0.5B LLM preserves reliable VFM rankings while reducing evaluation costs by 8 \times , making large-scale analysis substantially more practical. We also contextualize our findings within related work [ 24 , 80 ] to offer a holistic understanding of existing VFMs and highlight directions for future research (see section 6 ). Contributions. Our key contributions are three-fold: • We identify critical blind spots in existing evaluation protocols and introduce AVA-Bench , a systematic, diagnostic, and comprehensive VFM evaluation benchmark covering 14 atomic visual abilities (AVAs), clearly highlighting VFMs’ fundamental strengths and weaknesses. • We conduct a detailed evaluation and insightful analysis of diverse leading VFMs, deriving actionable guidance for VFM selection in downstream applications such as customized MLLMs. • We release a resource-efficient evaluation protocol along with an open-source codebase to facilitate the development of the next generation of accountable and versatile VFMs. 2 Vision Foundation Models (VFMs) and Evaluation Figure 3 : AVA-Bench consists of 14 Atomic