Self-Evolving Visual Concept Library using Vision-Language Critics

Atharva Sehgal Patrick Yuan Ziniu Hu Yisong Yue Jennifer J. Sun Swarat Chaudhuri · 2025

Read Self-Evolving Visual Concept Library using Vision-Language Critics by Atharva Sehgal Patrick Yuan Ziniu Hu Yisong Yue Jennifer J. Sun Swarat Chaudhuri (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-00185.txt.

Introduction

Atharva Sehgal, Patrick Yuan, Ziniu Hu, Yisong Yue, Jennifer Sun, and Swarat Chaudhuri teach a computer to build its own library of visual concepts. Their 2025 paper addresses visual recognition, where concept libraries, sets of named visual traits, help classifiers explain and decide. Hand-built libraries are slow to make, while purely language-generated ones include weak traits and ignore how concepts interact.

Their system, ESCHER, works like an editor with a critic. A vision-language model reviews the library's performance history and critiques it; the generator then revises concepts, accounting for interactions between them. No human labels are needed anywhere. Across zero-shot, few-shot, and fine-tuning tasks, the self-evolving library improved classification. The authors claim the first concept-library learning applied to real visual tasks.

Catalog record

CREATORAtharva Sehgal Patrick Yuan Ziniu Hu Yisong Yue Jennifer J. Sun Swarat Chaudhuri
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-00185

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Self-Evolving Visual Concept Library using Vision-Language Critics Atharva Sehgal Patrick Yuan Ziniu Hu Yisong Yue Jennifer J. Sun Swarat Chaudhuri Abstract We study the problem of building a visual concept library for visual recognition. Building effective visual concept libraries is challenging, as manual definition is labor-intensive, while relying solely on LLMs for concept generation can result in concepts that lack discriminative power or fail to account for the complex interactions between them. Our approach, Escher , takes a library learning perspective to iteratively discover and improve visual concepts. Escher uses a vision-language model (VLM) as a critic to iteratively refine the concept library, including accounting for interactions between concepts and how they affect downstream classifiers. By leveraging the in-context learning abilities of LLMs and the history of performance using various concepts, Escher dynamically improves its concept generation strategy based on the VLM critic’s feedback. Finally, Escher does not require any human annotations, and is thus an automated plug-and-play framework. We empirically demonstrate the ability of Escher to learn a concept library for zero-shot, few-shot, and fine-tuning visual classification tasks. This work represents, to our knowledge, the first application of concept library learning to real-world visual tasks. 1 Introduction Figure 1 : An overview of Escher . Prior work: concept-bottleneck visual recognition aims to leverage discriminative visual concepts to enable more accurate object classification. Ours: Escher is an approach for iteratively evolving a visual concept library using feedback from a VLM critic, to discover more effective visual concepts. How do humans recognize different visual categories? Consider the example in Figure 1 : while easily recognizable as a pastry, distinguishing between a “donut” and a “beignet” requires understanding visual concepts such circular shape with a hole for the donut, puffy texture, and the presence of powdered sugar specifically on the beignet. These visual concepts, including shape, texture, and the presence or absence of specific features, enable us to make distinctions between objects. Concept-bottleneck visual recognition [ 20 , 32 , 31 ] aims to leverage these discriminative visual concepts, to enable vision systems to more accurately recognize a wider range of classes. Here, we study general approaches for improving concept-bottleneck visual recognition systems by evolving visual concept libraries to find more effective concepts. Existing concept-bottleneck visual recognition systems typically leverage a Large Language Model (LLM) to generate a set of potential visual concepts relevant to the task, then use a Vision-Language Model (VLM) to make predictions from these concepts. This process potentially improves both interpretability and accuracy for classification [ 20 , 32 , 31 ] . However, existing methods face limitations: manually defined concepts are labor-intensive, and LLM-generated concepts can be inaccurate or fail to account for interactions between them. We need more effective methods to construct and refine visual concepts. One promising approach to improve visual concept learning is to leverage library learning [ 6 , 10 , 9 ] , which focuses on building a reusable collection of components. While library learning has shown success in domains that are naturally symbolically decomposable (e.g., equation learning [ 9 ] ), it is not well-explored for visual concept learning. Our key insight is that library learning complements visual concept learning by providing a structured and evolving repository of concepts that is more effective for visual recognition. To achieve this, we introduce Escher , a novel self-evolving framework to automatically discover and refine a library of visual concepts. Escher employs an iterative algorithm where a VLM acts as a critic, providing feedback on the effectiveness of concepts generated by an LLM. Specifically, the VLM evaluates the similarity between each image and its associated concepts compared to other images. This evaluation, captured in a contrastive score, serves as a feedback signal to guide the LLM in refining its generated concepts. Furthermore, Escher provides the history of concepts and feedbacks to the LLM, enabling the LLM to effectively learn from its past performance and improving its proposals over time. Through this iterative process, Escher produces a set of concepts that are both accurate and highly informative for the VLM, enabling it to make more effective predictions. Our approach offers several key advantages. First, it is broadly applicable and complements a range of existing concept-bottleneck visual recognition frameworks, including those designed for zero-shot, few-shot, and fine-tuned settings. This adaptability ensures that as LLMs, VLMs, and visual concept learning frameworks continue to evolve, Escher remains relevant and applicable to emerging techniques. Second, Escher requires no human annotations or labeled datasets, making it a plug-and-play solution for various visual recognition tasks. Finally, the iterative approach of Escher leverages the in-context learning capabilities of LLMs, allowing them to learn from their concept history and generate increasingly effective concept concepts. This iterative refinement process ensures that the concept library continuously adapts and improves, leading to more accurate and discriminative visual representations. To summarize, our contributions are: • We present Escher , a novel VLM- and LLM-based framework for self-evolving visual concept libraries. Our method does not require human-labeled data and can improve the quality of the learned concepts via an open-ended learning loop. • We develop an iterative concept refinement algorithm, leveraging the both the ability of VLMs to act as a critic and the ability of LLMs to incorporate history, to improve visual concepts based on past performance. • We demonstrate that Escher is complementary to a range of different state-of-the-art baselines, and our learned concept library improves performance across zero-shot, few-shot, and fine-tuned image classification settings. Figure 2 : ( Left ) Existing work on concept-bottleneck visual recognition, where a VLM scores a set of concepts to perform classification. The classification is based on the class with the maximum concept scores. ( Right ) Escher . ( 1 ) Escher follows previous work [ 20 ] in instantiating a set of concepts for each class using an LLM. ( 2 ) It initializes a concept-bottleneck model and collect the predictions for a classification dataset \mathcal{D}=\{(\mathbf{x}_{i},y_{i})\}_{i=1}^{N} (labels optional). ( 3 ) A concept similarity heuristic identifies frequently confused classes. ( 4 ) A history bank then stores relevant information to guide ( 5 ) the LLM sampling procedure for improved concepts that disambiguate these classes. The new concepts are integrated into the next iteration. 2 Related Work Vision-language models. VLMs have emerged as powerful tools for a wide range of visual tasks, from visual question answering to zero-shot image classification [ 24 , 2 , 19 ] . These models are typically trained on large datasets of image-text pairs, enabling them to reason about the relationship between these modalities. Some VLMs [ 11 , 24 , 33 ] are trained using contrastive learning objectives to learn embedding spaces aligned between image and text representations. These models have shown promising results for zero-shot and few-shot classification. While VLMs have shown remarkable capabilities, they have limitations that motivate exploring alternative approaches, such as concept-bottleneck models. One limitation is their difficulty in perceiving and reasoning about fine-grained visual concepts. For example, recent work has shown that VLMs can struggle to distinguish between subtle visual differences [ 27 ] . As such, despite recent advances, these models can still often fail with accurate visual recognition. The standard zero-shot method does not provide

Read the complete text