ActAlign: Zero-Shot Fine-Grained Video Classification via Language-Guided Sequence Alignment

Amir Aghdam, Vincent Tao Hu · 2025

Read ActAlign: Zero-Shot Fine-Grained Video Classification via Language-Guided Sequence Alignment by Amir Aghdam, Vincent Tao Hu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-22967.txt.

Introduction

Amir Aghdam, Vincent Tao Hu, and Björn Ommer present ActAlign, a way to name very specific actions in video without any training examples. Sorting a windmill dunk from other dunks, sight unseen, strains normal video models. Image-language models read single frames well but miss how actions unfold over time.

For each candidate class, a language model writes the ordered steps of the action. ActAlign then matches that step list against the video's frames using dynamic time warping in a shared embedding space. No video training or fine-tuning is involved. On the ActionAtlas benchmark it reached about thirty percent accuracy, beating models with eight times more parameters. Human accuracy there is only about sixty-two percent.

Catalog record

CREATORAmir Aghdam, Vincent Tao Hu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-22967

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ActAlign: Zero-Shot Fine-Grained Video Classification via Language-Guided Sequence Alignment Amir Aghdam, Vincent Tao Hu Abstract We address the task of zero-shot fine-grained video classification, where no video examples or temporal annotations are available for unseen action classes. While contrastive vision–language models such as SigLIP demonstrate strong open-set recognition via mean-pooled image–text similarity, they fail to capture the temporal structure critical for distinguishing fine-grained activities. We introduce ActAlign , a zero-shot framework that formulates video classification as sequence alignment. For each class, a large language model generates an ordered sub-action sequence, which is aligned with video frames using Dynamic Time Warping (DTW) in a shared embedding space. Without any video–text supervision or fine-tuning, ActAlign achieves 30.5% accuracy on the extremely challenging ActionAtlas benchmark, where human accuracy is only 61.6% . ActAlign outperforms billion-parameter video–language models while using approximately 8 × \times less parameters. These results demonstrate that structured language priors, combined with classical alignment techniques, offer a scalable and general approach to unlocking the open-set recognition potential of vision–language models for fine-grained video understanding. Code — https://github.com/aghdamamir/ActAlign 1 Introduction Understanding fine-grained human activities in video—such as distinguishing a hook shot from a layup in basketball, or recognizing tactical formations in football—requires parsing subtle, temporally ordered visual cues across frames. These actions unfold in structured sequences of sub-events and are often nearly indistinguishable from one another in appearance. In contrast to general activities like swimming , which can often be inferred from a single frame showing a person in water, fine-grained recognition demands attention to temporally extended object interactions, spatial relations, and high-level intent. As such, models must not only understand what is present in a video but also when and how key sub-actions occur. This requires accurately aligning the temporal progression of sub-actions with each fine-grained activity to ensure a correct prediction, as shown in Figure 1 . At the same time, contrastive vision–language models such as CLIP ( Radford et al. 2021 ) and SigLIP ( Zhai et al. 2023 ) have demonstrated impressive open-set fine-grained recognition in static image domains by training on massive image–text pairs. These contrastive models enable zero-shot classification using natural language prompts and have been widely adopted for downstream recognition tasks. However, extending these capabilities to video understanding introduces new challenges and requires temporal modeling. Existing methods that adapt CLIP-style models to video recognition either average frame-level features ( Rasheed et al. 2023 ; Zohra et al. 2025 ) —ignoring temporal structure—or fine-tune on target datasets ( Wang, Xing, and Liu 2022 ; Ni et al. 2022 ; Kim et al. 2024 ; Wang et al. 2024a ) , sacrificing generalization and open-set recognition. In both cases, the fine-grained temporal semantics of actions are lost or diluted. Recent video–language architectures and instruction-tuned LLM-based systems such as Video-LLaMA ( Zhang, Li, and Bing 2023 ) , VideoChat ( KunChang Li 2023 ) mPLUG-Owl ( Ye et al. 2023 ) , Qwen2-VL ( Wang et al. 2024b ) , and DeepSeek-JanusPro ( Chen et al. 2025 ) enable open-ended, dialog-style video understanding through heavy instruction tuning, but they are not tailored for fine-grained video recognition. Meanwhile, textual grounding (image–text alignment) remains a central challenge in interactive video-language models, especially for open-set and fine-grained video recognition. Dynamic Time Warping (DTW) ( Vintsyuk 1968 ) , a classical algorithm for aligning temporally mismatched sequences, has seen renewed interest through differentiable variants ( Chang et al. 2019 ; Dogan et al. 2018 ) designed for supervised image–text temporal alignment. Yet, these methods rely on annotated transcripts or example support videos, making them impractical for zero-shot recognition. Likewise, approaches using part-level or attribute-level supervision ( Wu et al. 2023 ; Zhu et al. 2024 ) offer fine-grained cues but lack the ability to model the temporal structure between language-defined actions and visual content. Image-Text Embedding Space Figure 1: ActAlign significantly improves zero-shot fine-grained action recognition by modeling them as structured language sequences. By aligning sub-action descriptions with video frames (green vs. red paths), we achieve more accurate predictions without requiring any video-text training data. Figure 2: Our pipeline uses an LLM to generate structured sub-action sequences for each fine-grained candidate class in ActionAtlas ( Salehi et al. 2024 ) , forming SubActionAtlas . This structured representation enables alignment with video content for zero-shot recognition tasks. In this work, we introduce ActAlign , a novel framework that brings the open-set generalization power of image–text models to fine-grained video recognition through language-guided subaction alignment in a truly zero-shot setting . Rather than tuning a model for a specific domain or collapsing the video into a static representation, ActAlign operates in a training-free setting: for each unseen action class, we use a large language model (LLM) to generate a structured sequence of temporal sub-actions that semantically define the class. (see Figure 2 . Then, using the pretrained SigLIP model ( Zhai et al. 2023 ) to extract frame-wise visual and subaction features, we align the video sequence with the LLM-generated subaction script via Dynamic Time Warping (DTW).(see Figure 1 ) This allows us to compute a soft alignment score between different action classes that respects both content and temporal ordering, enabling fine-grained classification in a truly zero-shot manner. To rigorously evaluate our approach, we use ActionAtlas ( Salehi et al. 2024 ) , the most challenging fine-grained video recognition benchmark to our knowledge, with a human accuracy ceiling of 61.64%. It comprises sports footage paired with extremely fine-grained candidate tactics that demand close attention to subtle visual cues for correct classification. We construct SubActionAtlas , by decomposing each candidate action into a sequence of sub-actions via LLM prompting (Figure 2 ), providing structured templates for alignment. Our contributions are as follows: • We introduce a novel framework for fine-grained video recognition that models each action as a general, structured temporal sequence of sub-actions derived solely from action names— without access to videos or transcripts . • We propose ActAlign , a novel zero-shot framework that applies the open-set generalization strength of contrastive image–text models to the challenging task of fine-grained and open-set video classification , without requiring any video–text supervision. • We show that ActAlign consistently outperforms prior zero-shot and CLIP-based baselines, and even exceeds billion-parameter video–language models on challenging fine-grained video classification benchmarks. 2 Related Work 2.1 Contrastive Image–Language Models Contrastive vision–language models such as CLIP ( Radford et al. 2021 ) , SigLip ( Zhai et al. 2023 ) , and ALIGN ( Jia et al. 2021 ) learn joint image–text embeddings from large-scale image–caption pairs, enabling strong open-set recognition without task-specific training. These models serve as pretrained backbones for downstream tasks, including visual question answering and reasoning ( Li, Baldridge, and Hoi 2021 ; Li et al. 2022 ; Tsimpoukelli et al. 2021 ) , image captioning and generation ( Mokady, Hertz, and Bermano 2021 ; Wang et al. 2021b ) , and few-shot and zero-shot classification ( Zhou et al. 2021 ; Khattak et al. 2025 ) . Notably, SigLip ( Zhai et al. 2023 ) recently enhanced CLIP’s architecture and leveraged a larger training corpus to enhance in image–text matching ( Zhai et al. 2023 ) . However, such models lack any temporal structure for processing video inputs. This

Read the complete text