Adaptive Prototype Model for Attribute-based Multi-label Few-shot Action Recognition
Juefeng Xiao, Tianqi Xiang, Zhigang Tu* · 2025
Read Adaptive Prototype Model for Attribute-based Multi-label Few-shot Action Recognition by Juefeng Xiao, Tianqi Xiang, Zhigang Tu* (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-12582.txt.
Introduction
Juefeng Xiao, Tianqi Xiang, and Zhigang Tu propose a method to recognize human actions with many labels in a 2025 paper. Real action systems gain from attributes like scene and object, not just the action name. Yet one model that reads many attributes at once can lose accuracy.
Their model builds a prototype for each attribute, guided by text from the labels, and assigns attributes in a way that cuts training bias. Few examples per class make the task harder, so balance matters. They also built a new video set with rich attribute labels for testing. The work targets security and sports tools that read scenes broadly.
Catalog record
| CREATOR | Juefeng Xiao, Tianqi Xiang, Zhigang Tu* |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-12582
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Adaptive Prototype Model for Attribute-based Multi-label Few-shot Action Recognition Juefeng Xiao, Tianqi Xiang, Zhigang Tu* Abstract In real-world action recognition systems, incorporating more attributes helps achieve a more comprehensive understanding of human behavior. However, using a single model to simultaneously recognize multiple attributes can lead to a decrease in accuracy. In this work, we propose a novel method i.e. Adaptive Attribute Prototype Model (AAPM) for human action recognition, which captures rich action-relevant attribute information and strikes a balance between accuracy and robustness. Firstly, we introduce the Text-Constrain Module (TCM) to incorporate textual information from potential labels, and constrain the construction of different attributes prototype representations. In addition, we explore the Attribute Assignment Method (AAM) to address the issue of training bias and increase robustness during the training process. Furthermore, we construct a new video dataset with attribute-based multi-label called Multi-Kinetics for evaluation, which contains various attribute labels (e.g. action, scene, object, etc.) related to human behavior. Extensive experiments demonstrate that our AAPM achieves the state-of-the-art performance in both attribute-based multi-label few-shot action recognition and single-label few-shot action recognition. The project and dataset are available at an anonymous account https://github.com/theAAPM/AAPM Figure 1: The performance of different methods in constructing visual prototypes for action recognition when attributes increase. The results demonstrate that the performance of existing models will decrease as the number of attributes increases, while our AAPM overcomes the attribute-bias and maintains good performance. 1 Introduction Recognizing human action in videos is one of the most popular tasks in computer vision Carreira and Zisserman (2017) ; Lin et al. (2019) ; Herath et al. (2017) ; Kong and Fu (2022) . To achieve a better understanding of actions, more descriptive attributes* beyond basic action categories are needed. However, all the existing methods are designed for the specific task of action recognition. As a result, these models may perform well in recognizing actions but tend to introduce bias in feature extraction, such as overemphasizing certain action-related features and neglecting other important attributes, which can reduce their generalization ability to more complex tasks. Figure 1 illustrates this situation. However, this bias may not be harmful in tasks that focus on a single attribute but can limit the model’s performance across a wider range of attributes. We refer to this as attribute-bias. Attribute* : We define an attribute as a high-level category or abstract concept that encompasses a set of related subcategories, which share a common classification logic. For instance, ’action’ is an attribute that includes various human activities such as playing soccer, swimming, dancing, etc. These subcategories can be further refined into more specific attributes at different levels of granularity. For example, ’dancing’ can be considered an attribute that encompasses distinct dance styles like Latin dance, folk dance, ballet, etc. The VLP (Vision-Language Pretraining) models Chen et al. (2023) address the problem of attribute-bias through robust vision-language pre-training, which enhances the model’s robustness and generalization ability. CLIP Radford et al. (2021) aligns visual features with textual features by leveraging contrastive learning, enabling the model to be applied to the recognition of various attributes. The success of Ni et al. (2022) ; Wang et al. (2021) on action recognition demonstrates that the VLP model has made remarkable progress on the downstream video tasks. However, to make the model perform comparably to other models in action recognition, these downstream tasks involve fine-tuning the VLP encoder. This process sacrifices the model’s generalization ability, leading to bias towards specific tasks. Additionally, semantic drift refers to the model overfitting to the semantic understanding of the training data. This can happen during the fine-tuning of the text encoder, especially when the text volume is limited. On the other hand, few-shot learning Fei-Fei et al. (2006) addresses attribute-bias by constructing visual prototypes, enabling the model to generalize better with limited data. Based on this advantage, few-shot learning Vinyals et al. (2016) ; Snell et al. (2017) has been widely applied in action recognition Guo et al. (2020) . However, none of these few-shot action recognition methods has explored obtaining more action-related information, as they are constrained to the single attribute of action. An important reason is that when it comes to a multi-attribute way, the prototype representations may fall into confusion among various attributes, which will be discussed in detail in Section 3.1 . Meanwhile, multi-label classification tasks Tsoumakas and Katakis (2007) ; Chua et al. (2009) ; Chen et al. (2019) ; Xu (2023) annotate multiple labels to the data to provide more detailed information. However, the previous methods lack flexibility, as they require continuously adding new classification heads or discriminators and retraining the model to accommodate a broader range of categories. They mainly concentrate on discriminating whether the objects appear in the image/video without taking into account extracting the advanced information accurately that we are interested in. Therefore, the prior multi-label methods are not suitable for human action recognition. Based on the above observations, we exploit a method Adaptive Attribute Prototype Model (AAPM) for human action recognition, which can learn not only the human action but also other valuable action attribute information. Since AAPM leverages the textual information to provide guidance to construct different attributes of prototype representations, it enables the query data to recognize multiple attributes. To achieve this goal, we introduce Text-Constrain Module (TCM) to contribute robust prototype and exploit Attribute Assignment Method (AAM) for fair optimization. AAPM largely reduces the attribute-bias and enables the simultaneous recognition of multiple attributes. TCM leverages both video and text information to construct prototypes. We utilize the CLIP model to extract the textual feature and visual feature of video and label text, where the textual feature represents the classification logic and the visual feature represents the pattern to be recognized. However, relying solely on text features to construct support sets is insufficient, as videos contain richer temporal features that can better describe the entire motion process. The module firstly learns the adaptive attribute feature via a transformer block by the text feature, and then constrains the visual feature through the cross attention block to ensure that each prototype is constructed under the definite attribute to avoid confusion. However, the adaptive attribute feature in the latent space is hard to supervise. Besides, different attributes of a video may be relevant, leading to difficulty in extracting effective information by TCM. To overcome these problems, the AAM is adopted to assign stochastic irrelevant attributes to video data during the training process to make the optimization more robust. Additionally, by assigning more attributes, the text volume is much richer than previous works that only focus on action class. As a result, our Attribute Assignment module largely reduces the issue of semantic drift and increases the model’s generalizability. To validate the effectiveness of our model, we construct the Multi-Kinetics dataset, which is a multi-attribute labels dataset. Different from the prior ones Chua et al. (2009) , our dataset incorporates explicit classification logic among data, thus it provides a more structured and meaningful annotation framework. Meanwhile, this dataset introduces a new practical task called Attribute-based Multi-label Few-shot Action Recognition (AMFAR), which aims to recognize not only human action but also additional valuable information. Extensive experiments show that AAPM achieves the state-of-the-art performance on few-shot action recognition