Open Ad-hoc Categorization with Contextualized Feature Learning

Zilin Wang, Sangwoo Mo, Stella X. Yu, Sima Behpour, Liu Ren · 2025

Read Open Ad-hoc Categorization with Contextualized Feature Learning by Zilin Wang, Sangwoo Mo, Stella X. Yu, Sima Behpour, Liu Ren (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-16202.txt.

Introduction

Zilin Wang and four co-authors teach AI to invent categories on the fly in an arXiv paper uploaded in 2025. People sort scenes by goal, not just by fixed labels: the same room can be grouped by activity, mood, or place. The authors study open ad-hoc categorization, where a system gets a few labeled examples plus many unlabeled images and must discover the hidden context.

Their model, OAK, adds a small set of learnable context tokens to a frozen CLIP vision-language model. Training mixes CLIP's image-text matching with a visual clustering objective borrowed from category discovery research. On Stanford and Clevr-4 data, OAK set state-of-the-art accuracy, reaching 87.4 percent on new categories in the Stanford Mood task. Its attention maps are readable, focusing on hands for actions, faces for mood, and backgrounds for location.

Catalog record

CREATORZilin Wang, Sangwoo Mo, Stella X. Yu, Sima Behpour, Liu Ren
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE64 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-16202

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Open Ad-hoc Categorization with Contextualized Feature Learning Zilin Wang, Sangwoo Mo, Stella X. Yu, Sima Behpour, Liu Ren Abstract Adaptive categorization of visual scenes is essential for AI agents to handle changing tasks. Unlike fixed common categories for plants or animals, ad-hoc categories are created dynamically to serve specific goals. We study open ad-hoc categorization : Given a few labeled exemplars and abundant unlabeled data, the goal is to discover the underlying context and to expand ad-hoc categories through semantic extension and visual clustering around it. Building on the insight that ad-hoc and common categories rely on similar perceptual mechanisms, we propose OAK, a simple model that introduces a small set of learnable context tokens at the input of a frozen CLIP and optimizes with both CLIP’s image-text alignment objective and GCD’s visual clustering objective. On Stanford and Clevr-4 datasets, OAK achieves state-of-the-art in accuracy and concept discovery across multiple categorizations, including 87.4% novel accuracy on Stanford Mood, surpassing CLIP and GCD by over 50%. Moreover, OAK produces interpretable saliency maps, focusing on hands for Action, faces for Mood, and backgrounds for Location, promoting transparency and trust while enabling adaptive and generalizable categorization. 1 Introduction Figure 1 : We study open ad-hoc categorization such as things to sell at a garage sale to achieve a specific goal ( selling unwanted items ). Given the context garage sale , labeled exemplars such as shoes , we need to recognize all items in the scene that can be sold at the garage sale , including novel ones. Supervised models like CLIP focus on 1) closed-world generalization , recognizing other shoes . 2) Novel semantic categories can be discovered by contextual expansion from shoes to hats . Unsupervised methods like GCD discover 3) novel visual clusters , identifying suitcases . Our work unifies these scenarios by discovering the latent context and expanding categories both semantically and visually around it. The concept of ad-hoc categories in cognitive science differs from common categories used for plants or animals ( 2 ) . For example, things to sell at a garage sale ( Fig. 1 ) may lack visual or semantic similarity yet are grouped together to achieve the goal of selling unwanted items . Unlike common categories , ad-hoc categories are less established in human memory, often lack clear labels, require explicit naming of exemplars, and depend strongly on context. Recognizing them relies on the same perceptual mechanisms as common categories but additionally requires contextualization to adapt to varying goals and situations ( 24 ; 49 ) . Translating this cognitive notion into machine perception motivates a novel problem setting we call open ad-hoc categorization , which learns to organize visual data under contexts that are not predefined but must be discovered from a few labeled exemplars and abundant unlabeled data. The context specifies the organizing principle (e.g., action or location ) and induces distinct category structures (e.g., drinking vs. reading , urban vs. natural ). This task has two objectives: to infer the latent context and to expand ad-hoc categories accordingly, both semantically by inferring new concept names and visually by identifying coherent clusters. Unlike open-world recognition, where the context is broad and unconstrained, open ad-hoc categorization operates under a specific contextual goal that tightly guides ad-hoc category formation. It leverages a few labeled exemplars to ground this context and reveal task-relevant structure within abundant unlabeled data. Figure 2 : Open ad-hoc categorization learns diverse categorization rules, dynamically adapting to varying user needs at hand. The same image could be recognized differently depending on the context, such as drinking for Action and residential for Location . We emphasize the ability to switch between multiple contexts in OAK. Specifically, given 1) a context defined by classes, 2) a few labeled images, and 3) a set of unlabeled images, OAK holistically reasons over labeled and unlabeled images, spanning both known and novel classes, to infer novel concepts and propagate labels across the entire dataset. We show the class names of labeled images in the color box and unlabeled images inside the parentheses, reflecting that the unlabeled class names are not available, only the images. The OAK setting introduces challenges beyond generalized category discovery (GCD), requiring adaptation to diverse ad-hoc categorization rules based on context. Fig. 1 illustrates three forms of discovery, each reflecting how context constrains category formation within unlabeled data grounded by labeled exemplars. 1) Closed-world generalization applies known concepts to new instances, such as CLIP 46 recognizing unseen shoes . 2) Novel semantic discovery extends familiar concepts to related ones, such as inferring hats from shoes . 3) Novel visual discovery uncovers new clusters that refine the context, such as GCD 57 identifying suitcases as a new category. Together, these levels trace a continuum from closed-world generalization to the emergence of new semantic and visual categories shaped by context and revealed through unlabeled data. We formulate the problem as predicting both known and novel concepts from unlabeled images using only a few labeled examples within a given context. Fig. 2 illustrates how the same set of images can be interpreted differently depending on the context and thus belong to distinct ad-hoc categories, such as Action , Location , or Mood . When the context is Action , images such as a child drinking in a bedroom are labeled drinking , and the task is to identify novel Action classes such as riding or climbing and categorize all unlabeled images. When the context shifts to Location , the same image is labeled residential area , and the task becomes inferring novel Location types such as sports field or store . The few-shot aspect is critical for practicality, enabling rapid adaptation to new contexts and scaling to many more without extensive human annotation. We build on two insights. 1) Recognizing ad-hoc categories relies on the same perceptual mechanisms as common categories but requires contextualized interpretation to adapt to varying goals. 2) Novel concepts can be discovered either semantically, by expanding contextual semantics, or visually, by clustering similar patterns. CLIP 46 , trained on internet-scale image–text pairs, captures both fundamental perceptual mechanisms and rich semantic knowledge of common categories, but it assumes a fixed and globally shared semantic space. In open ad-hoc categorization, however, the context is not predefined: The meaning of terms such as Action or Location is embodied in a few labeled exemplars and abundant unlabeled data. We therefore do not alter CLIP’s mechanisms but instead learn to capture the contextual semantics implicit in the data. To complement CLIP’s top-down, text-driven interpretation with bottom-up, data-driven discovery, we incorporate GCD 57 for visual clustering, enabling joint semantic and visual concept expansion under the discovered context. We propose O pen A d-hoc C ategorization with Contextualized Feature Learning (OAK for short, with K from Kategorisierung: Categorization in German), a simple model that represents the learned context through a small set of trainable context tokens introduced at the input of a frozen CLIP. These tokens condition CLIP’s image-text representations on the discovered context, grounding its semantic space without modifying its pretrained mechanisms. OAK is trained jointly with CLIP’s image-text alignment objective and GCD’s visual clustering objective, integrating top-down semantic alignment with bottom-up visual discovery. By switching only the

Read the complete text