TACO: Towards Task-Consistent Open-Vocabulary Adaptation in Video Recognition
Minghao Zhu, Xiao Lin, Mengxian Hu, Xun Zhou, Xiaoyan Qi, Chengju Liu, Qijun Chen · 2026
Read TACO: Towards Task-Consistent Open-Vocabulary Adaptation in Video Recognition by Minghao Zhu, Xiao Lin, Mengxian Hu, Xun Zhou, Xiaoyan Qi, Chengju Liu, Qijun Chen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-25478.txt.
Introduction
Adapting an image trained model like CLIP to recognise actions in video risks trading away its gift for naming things it has never seen. Minghao Zhu and colleagues propose TACO in 2026 to keep both sides of that bargain. Fine tuning usually warps the representation space in ways that only show up on unseen data. Their fix distils the relative geometry of the original space, penalising harmful shifts in how concepts sit relative to each other during training.
A light projection layer then holds the task specific changes, so the space used at test time stays closer to the pretrained one. Across cross dataset and base to novel benchmarks, the method set new state of the art results for open vocabulary video recognition. The authors frame the lesson broadly: match the training objective to how the model will actually be judged.
Catalog record
| CREATOR | Minghao Zhu, Xiao Lin, Mengxian Hu, Xun Zhou, Xiaoyan Qi, Chengju Liu, Qijun Chen |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.25478
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
TACO: Towards Task-Consistent Open-Vocabulary Adaptation in Video Recognition Minghao Zhu, Xiao Lin, Mengxian Hu, Xun Zhou, Xiaoyan Qi, Chengju Liu, Qijun Chen Abstract Adapting CLIP for open-vocabulary video recognition necessitates a delicate balance between newly acquired video knowledge and the pretrained generalization. While existing studies pursue this generalization-specialization trade-off with additional regularizations or constraints, we argue that they overlook the deviation of representations beyond the fine-tuning data distribution, resulting in suboptimal adaptation effects. We believe such deviation is inherited from the inconsistency between the fine-tuning and evaluation objectives, where model optimization is restricted to the known training distribution but evaluated on unseen ones. In this paper, we introduce TACO , a simple yet effective framework to mitigate the potential negative effects induced by this inconsistency. Our key insight is that adaptation should preserve OOD-relevant alignment beyond the training distribution. To this end, we propose Relative Structure Distillation , which regularizes the relative geometry of the representation space and suppresses harmful alignment shift during training. We further decouple the representation space from the optimization space with a lightweight specialization projection, allowing task-specific adaptation without directly overspecializing the representations used at test time. TACO establishes state-of-the-art performance on diverse benchmarks under cross-dataset and base-to-novel settings. Code will be released at https://github.com/ZMHH-H/TACO . 1 Introduction Open-vocabulary learning aims to identify novel visual concepts specified by language vocabularies unseen during training phase. This setting has been propelled by the rapid progress of large-scale Vision–Language Models (e.g., CLIP [ 37 ] , ALIGN [ 17 ] , Florence [ 50 ] ), which learn aligned multimodal representations from massive image-text pretraining. With their strong generalization to unseen visual concepts, these foundation models are highly practical in real-world applications and have been widely adapted to various downstream tasks [ 52 , 11 , 6 , 28 , 54 ] . Inspired by this progress, recent studies have explored adapting CLIP for general video recognition [ 33 , 38 , 55 ] , avoiding the substantial cost of collecting large-scale video-text data and pretraining from scratch. While CLIP provides a strong starting point for visual understanding, effective adaptation for open-vocabulary video recognition remains challenging - specifically in empowering models to accurately capture the temporal dynamics encoded in videos of unknown categories. The key challenge is to introduce video-specific knowledge while preserving the pretrained generalization, a problem commonly framed as the generalization/specialization trade-off. To strike this balance, recent studies (e.g., Open-VCLIP [ 42 ] , FROSTER [ 15 ] ) typically seek a middle ground between the two aspects by incorporating carefully designed regularizations during fine-tuning. Despite their promising results, we argue that these methods overlook representation variation beyond the training distribution, as their optimization objectives are strictly confined to the fine-tuning data distribution (i.e., ID space). Intuitively, without explicit constraints on the out-of-distribution (i.e. OOD) space, visual and text representations may drift in different directions, thereby disrupting their cross-modal alignment. Under the open-vocabulary setting, such alignment disruption can directly undermine the model’s generalization to videos from unseen categories. In practice, since the fine-tuning dataset is limited in both scale and diversity, this inconsistency between fine-tuning and evaluation objectives is prevalent in existing adaptation paradigms, as illustrated in Figure 1 . We revisit this inconsistency to better understand what must be preserved for open-vocabulary generalization. Our analysis suggests that OOD generalization still largely relies on CLIP’s original semantic space, whereas standard fine-tuning often induces representation drift and cross-modal alignment shift beyond the training distribution, leading to degraded generalization. Figure 1 : (a) Standard fine-tuning aims to align visual and text embeddings within the known training distribution (ID Space), but neglects the deviation in the out-of-distribution space. While (b) the open-vocabulary model is evaluated in the OOD space, (c) our fine-tuning objective can effectively maintain the relative structure across the entire embedding space and improve the generalization. Building upon these findings, we present TACO , a simple yet effective framework for task-consistent open-vocabulary video adaptation, and formulate a more explicit adaptation principle. Our key insight is that the adaptation process should properly regularize the entire representation space, thereby aligning more closely with the evaluation objective and curbing harmful alignment shift in OOD space. To this end, we propose Relative Structure Distillation , which ensures the consistent structure of the entire representation space throughout fine-tuning in a data-independent manner. Unlike existing distillation objectives that focus on matching within training distribution, our objective connects the ID space with a constructed OOD space, thereby effectively suppressing drift in OOD regions. We further show that simple random geometric anchors sampled on the CLIP hypersphere are sufficient to provide effective global structural references. Second, we propose a lightweight Specialization Projection that decouples the representation space from the optimization space, so that task-specific adaptation can occur in a disposable branch rather than directly overspecializing the representations used at test time. These two designs directly follow our analysis: RSD protects the global alignment structure beyond the training categories, while the specialization projection reduces the tendency of the cross-entropy objective to overfit the shared representation space. Overall, TACO is concise, scalable, and easily integrated into different adaptation pipelines, delivering significant improvements. Extensive experiments show that TACO consistently improves open-vocabulary generalization and establishes new state-of-the-art performance across multiple datasets and backbone scales. Our main contributions can be summarized as follows: • We introduce TACO , a simple yet effective framework for open-vocabulary video adaptation. TACO aims to mitigate the potential negative effects induced by the representation deviation beyond the training distribution, a factor not directly addressed in prior work. • We revisit the objective inconsistency in existing fine-tuning paradigms, analyze its impact on preserving pretrained generalization, and formulate a more explicit adaptation principle (§ 2 ). • We present Relative Structure Distillation, which effectively preserves generalization by keeping the consistent structure of OOD space without explicit OOD supervision. We further propose a generic representation-space decoupling scheme to mitigate overspecialization (§ 3 ). • TACO sets state-of-the-art performance on cross-dataset and base-to-novel settings across multiple benchmarks. Comprehensive ablations further demonstrate the effectiveness of our method (§ 4 ). 2 Analysis: Revisiting Open-Vocabulary Adaptation for Video 2.1 Preliminary: Adapting CLIP for Video Recognition To transfer the powerful generalization of the CLIP model [ 37 ] to the video domain, recent studies have fine-tuned its joint embedding space using video-text data and obtained promising results [ 33 , 38 , 55 ] . We briefly describe the standard fine-tuning paradigm in this context. Consider a CLIP-based video learner composed of a visual encoder f_{\theta_{v}}(\cdot) and a text encoder f_{\theta_{t}}(\cdot) , where \theta_{v} and \theta_{t} are the parameters of each encoder. Given a video V\in\mathbb{R}^{T\times H\times W\times 3} with T frames and the corresponding text description C embedded in a set of pre-defined templates (e.g., "A video of []" ), we extract the visual embedding \bm{v}\in\mathbb{R}^{D} , text embedding \bm{c}\in\mathbb{R}^{D} , and compute their similarity \operatorname{sim}(\bm{v},\bm{c}) as: \bm{v}=f_{\theta_{v}}(V),\kern 5.0pt\bm{c}=f_{\theta_{t}}(C),\kern 5.0pt\operatorname{sim}(\bm{v},\bm{c})=\frac{\langle\bm{v},\bm{c}\rangle}{\left\|\bm{v}\right\|\left\|\bm{c}\right\|}, (1) where D is the dimension of the joint embedding space. During fine-tuning, video-specific knowledge is injected by encouraging the video embedding to align with the matched category text embedding while separating it from other categories. Formally, the training objective can be written as: \mathcal{L}_{CE}=\mathbb{E}_{\scriptscriptstyle(V,C)\sim\mathcal{D}}[\,CE(\,\operatorname{sim}(\bm{v},\bm{W}),\,onehot(C)\,)\,], (2) where \mathcal{D} is the fine-tuning dataset,