Beyond Accuracy: Metrics that Uncover What Makes a ‘Good’ Visual Descriptor
Ethan Lin, Linxi Zhao, Atharva Sehgal, Jennifer J. Sun, Cornell University · 2025
Read Beyond Accuracy: Metrics that Uncover What Makes a ‘Good’ Visual Descriptor by Ethan Lin, Linxi Zhao, Atharva Sehgal, Jennifer J. Sun, Cornell University (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-03542.txt.
Introduction
Words can stand in for pictures when a vision model sorts images, but which words deserve the job? Ethan Lin and co-authors argue that accuracy alone cannot say. A label set might score well for the wrong reasons, leaning on quirks of the model's training data rather than clean meaning. They grade descriptor sets on two deeper axes: how rich a representation space the words form, and how they relate to what the model saw in training.
Two new alignment-based scores, one global and one built on image-text similarity, expose differences that accuracy hides across generation methods, from one-shot prompts to iteratively refined lists. The metrics give descriptor designers a diagnostic, not just a verdict.
Catalog record
| CREATOR | Ethan Lin, Linxi Zhao, Atharva Sehgal, Jennifer J. Sun, Cornell University |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 24 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-03542
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Beyond Accuracy: Metrics that Uncover What Makes a ‘Good’ Visual Descriptor Ethan Lin, Linxi Zhao, Atharva Sehgal, Jennifer J. Sun, Cornell University Abstract Text-based visual descriptors—ranging from simple class names to more descriptive phrases—are widely used in visual concept discovery and image classification with vision-language models (VLMs). Their effectiveness, however, depends on a complex interplay of factors, including semantic clarity, presence in the VLM’s pre-training data, and how well the descriptors serve as a meaningful representation space. In this work, we systematically analyze descriptor quality along two key dimensions: (1) representational capacity, and (2) relationship with VLM pre-training data. We evaluate a spectrum of descriptor generation methods, from zero-shot LLM-generated prompts to iteratively refined descriptors. Motivated by ideas from representation alignment and language understanding, we introduce two alignment-based metrics—Global Alignment and CLIP Similarity—that move beyond accuracy. These metrics shed light on how different descriptor generation strategies interact with foundation model properties, offering new ways to study descriptor effectiveness beyond accuracy evaluations. † † footnotetext: * Correspondence to: [email protected] . 2 2 footnotetext: Code available at https://github.com/ethan-y-lin/beyond_accuracy . 1 Introduction Classification-by-description frameworks extend zero-shot classification by pairing each class with more informative text descriptors instead of a simple label. These descriptors—ranging from class names to full sentences—serve as queries to a vision-language model (VLM), which classifies images based on image similarity to each descriptor in a shared embedding space. Such methods have demonstrated improved classification performance and offer greater interpretability by making the model’s reasoning process more transparent. Figure 1 : Overview: Achieving high accuracy does not guarantee that a set of descriptors is “good”. Other factors like interpretability may suffer. Global alignment and CLIP similarity can serve as new metrics for evaluating and understanding different sets of visual descriptors. While these approaches have demonstrated success in image classification, a fundamental question remains: What constitutes a “good” textual visual descriptor? Descriptors can be obtained in many ways—from using class names directly Radford et al. 2021b , to leveraging large language models (LLMs) for zero-shot descriptions ( Menon and Vondrick 2022a ) , to employing sophisticated iterative refinement algorithms Sehgal et al. 2025 . Descriptor quality is typically evaluated based on downstream classification accuracy. While informative, this metric offers limited insight into why certain descriptors succeed or fail, and provides little guidance for descriptor discovery ( Bhalla et al. 2024 ) or refinement beyond evolutionary search ( Chiquier et al. 2024 ) or task-specific feedback loops ( Sehgal et al. 2025 ) . Moreover, higher accuracy does not necessarily imply more meaningful descriptors—prior work has shown that appending random words or characters to class names can sometimes improve performance Roth et al. 2023 . Clearly, there is need for more principled methods to assess descriptor quality beyond accuracy. In this work, we propose a novel approach to assessing descriptor quality by probing the relationship between textual descriptors and the underlying VLM. We focus on two key aspects: (1) the representational capacity of a set of descriptors—that is, how well they capture and reconstruct image semantics when used as a projection space; and (2) the relationship between a set of descriptors and CLIP’s pre-training data distribution. To study these factors systematically, we analyze a diverse range of text descriptor sets—spanning class names, zero-shot LLM-generated descriptions, and iteratively refined prompts—using two simple but informative metrics (Figure 1 ). Motivated by recent work on representational alignment ( Huh et al. 2024 ) , we introduce a global alignment metric, Dino align , which measures the alignment between a vision model’s image embedding space and the representation space induced by a set of descriptors using CLIP ( Radford et al. 2021a ) . We also propose Clip sim , inspired by findings on language model familiarity with training data ( Kandpal et al. 2023 ) , which captures how descriptors relate to the CLIP training distribution. Together, these metrics move beyond accuracy to offer a deeper evaluation of text-based visual descriptor quality. Contributions. This work makes these contributions: • We propose two novel metrics — global representation alignment and CLIP similarity — to probe text descriptor quality beyond downstream accuracy. • We analyze how these metrics correlate with performance across fine-grained classification datasets, and present preliminary analyses that showcase how these metrics characterize the effectiveness of strong text-based visual descriptors. 2 Related Work Concept Learning with VLMs. Despite rapid improvements in large-scale visual pretraining ( Radford et al. 2021a ; Jia et al. 2021 ; Li et al. 2021 ; Mu et al. 2022 ; Yao et al. 2022 ; Yu et al. 2022 ; Alayrac et al. 2022 ; Liu et al. 2024b ; Liu et al. 2024a ) , vision-language models (VLMs) still struggle with even moderately complex perceptual questions ( Tong et al. 2024 ; Kamath et al. 2023 ; Alhamoud et al. 2025 ) . Concept learning has emerged as a promising direction, leveraging zero-shot queries to large language models (LLMs) to decompose complex perceptual reasoning tasks into logical combinations of simpler perceptual subtasks – tasks that off-the-shelf VLMs can reliably solve ( Menon and Vondrick 2022b ; Yan et al. 2023b ; Yang et al. 2023 ; Roth et al. 2023 ) . These logical combinations are typically expressed as learnable linear combinations over an overcomplete set of concepts, sampled from an LLM. Prior work improves this initial concept set either by subsampling or by sampling new concepts using task-specific visual feedback from an LLM ( Sehgal et al. 2025 ) . The quality of a set of concepts is quantitatively evaluated using downstream image classification performance. We complement prior work by analyzing the utility of a concept set through three orthogonal lenses: downstream concept accuracy, representational alignment, and alignment with VLM pre-training data. Representational Alignment with VLMs. Representational alignment refers to the degree of similarity between the internal representations learned by different models ( Sucholutsky et al. 2024 ) . In the context of VLMs, research has studied both explicit training methods to maximize alignment ( Jia et al. 2021 ; Radford et al. 2021a ) and the emergence of implicit alignment during training ( Huh et al. 2024 ; Tjandrasuwita et al. 2025 ) . In this work, we provide the first study of such alignment metrics applied to representations induced by different sets of descriptors. We show how these alignment measures can serve as indicators of the downstream utility and meaningfulness of a descriptor set. 3 Approach Figure 2 : Overview of our descriptor evaluation metrics. ( Left ) Dino align measures how well a set of descriptors preserves class structure by aligning the descriptor-induced similarity matrix with that of DINO image features. ( Right ) Clip sim assesses how well descriptors align with visual content by retrieving related image-text pairs from CLIP’s pre-training dataset and averaging their similarity scores. 3.1 Global Alignment Metric Classification by description using VLMs can be viewed as a semantic projection Yan et al. 2023a , where images are mapped to similarity scores over a set of textual descriptors. This forms a new representation space, with each dimension corresponding to a specific concept. We assess the quality of the descriptor-induced space using Dino align , a zero-shot metric that measures the correlation between