CA3D: Convolutional-Attentional 3D Nets for Efficient Video Activity Recognition on the Edge
Gabriele Lagani, Fabrizio Falchi, Claudio Gennaro, Giuseppe Amato · 2025
Read CA3D: Convolutional-Attentional 3D Nets for Efficient Video Activity Recognition on the Edge by Gabriele Lagani, Fabrizio Falchi, Claudio Gennaro, Giuseppe Amato (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-19928.txt.
Introduction
Gabriele Lagani and three co-authors present CA3D, a 2025 network for recognizing activities in video on small devices. Video models usually demand heavy computing, which blocks smart home and health uses where cost and privacy matter. Their design mixes convolutional layers with an attention step that scales gently with input size.
A new quantization scheme trims the model's size further during both training and use. The network keeps strong learning ability at a reduced cost, the authors report. Tests on several public activity benchmarks show better accuracy than rival efficient models at a competitive computing cost. The work targets consumer and edge hardware. It aims to bring camera based activity sensing into homes without cloud dependence.
Catalog record
| CREATOR | Gabriele Lagani, Fabrizio Falchi, Claudio Gennaro, Giuseppe Amato |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-19928
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CA3D: Convolutional-Attentional 3D Nets for Efficient Video Activity Recognition on the Edge Gabriele Lagani, Fabrizio Falchi, Claudio Gennaro, Giuseppe Amato Abstract In this paper, we introduce a deep learning solution for video activity recognition that leverages an innovative combination of convolutional layers with a linear-complexity attention mechanism. Moreover, we introduce a novel quantization mechanism to further improve the efficiency of our model during both training and inference. Our model maintains a reduced computational cost, while preserving robust learning and generalization capabilities. Our approach addresses the issues related to the high computing requirements of current models, with the goal of achieving competitive accuracy on consumer and edge devices, enabling smart home and smart healthcare applications where efficiency and privacy issues are of concern. We experimentally validate our model on different established and publicly available video activity recognition benchmarks, improving accuracy over alternative models at a competitive computing cost. 1 Introduction State-of-the-art deep learning models for video processing have shown impressive results in modeling, interpreting, and performing inference on temporal sequences of video frames [ 44 , 48 , 15 , 14 , 37 , 5 , 1 , 33 , 36 ] . However, such models suffer from a number of issues related to computational efficiency, both at inference and training time, as well as energy costs [ 3 ] . In terms of computational efficiency, recent models are becoming more and more complex, making it hard to deploy them on commodity hardware, for consumer-oriented and privacy-oriented applications on the edge. Similarly, in terms of training efficiency, state-of-the-art models require large amounts of GPU hours (hence energy costs) to achieve good task performances [ 3 ] . Moreover, such models are generally pre-trained on large collections of data, and then fine-tuned on task-specific data, representing an additional cost in terms of data efficiency [ 29 ] . In video processing applications for environments such as smart home and smart healthcare, privacy represents a major concern, so that it is preferable to keep personal user data within the local environment. In this scenario, it is desirable to have deep learning models for video processing that are accurate enough for the task at hand, possibly leveraging user-specific data for online fine-tuning, while also maintaining a lightweight memory and computing footprint, so that applications can run effectively on edge devices and commodity hardware. Current state-of-the-art architectures are not well suited for the scenario described so far. On one hand, traditional convolutional network architectures are relatively efficient to run, and show good generalization performances from little amounts of data, thanks to the translation-invariance that is intrinsic in the convolutional layers [ 44 , 48 , 15 , 14 ] ; however, the performance that these models achieve, even when training on large collections of data, is overall suboptimal. On the other hand, more recent attention-based architectures [ 37 , 5 , 1 , 33 , 36 ] stemming from the successful Transformer model [ 45 ] , are able to achieve overall better performances, thanks to the capability of the attention layer to model global relationships between different parts of the input (such as frames that are far apart in time, while convolutions can only focus on local relationships); however, there are a number of drawbacks: 1) the attention computation is costly, 2) training such models to optimal performance requires huge data availability to achieve good generalization, and 3) the models are too complex for commodity hardware and edge devices. Figure 1 : Design of the Convolutional-Attentional 3D (CA3D) neural network for video processing, as a series of Convolutional-Attentional Spatio-Temporal (CAST) blocks. In CAST blocks, convolutional layers are alternated with attention layers, to take advantage of both types of processing. Convolutions are applied along the spatial dimensions, while attention aggregates global information from different frames along the temporal dimension. Processing of each layer is further enhanced with a deep column of residual stages. In order to overcome these challenges, we designed a novel deep learning block for video processing, named Convolutional-Attentional Spatio-Temporal (CAST) layer . The design of this block is carefully chosen in order to take the best advantages from both types of processing (convolutional and attention), while minimizing the impact on performance and training cost. In particular, we defined a novel linear-complexity attention block, based on local attention windows in the same spirit as [ 32 ] , to further reduce the cost of attention computation. We also designed a novel architecture, based on CAST layers, named Convolutional-Attentional 3D (CA3D) network (Fig. 1 ), together with a quantization mechanism that allows us to reduce the computational burden during both training and inference. In order to validate the proposed architecture, we performed experiments on three publicly available and widely used video activity recognition benchmarks, namely UCF101 [ 39 ] , HBDM51 [ 26 ] , and Kinetics400 [ 24 ] , comparing our method with several state-of-the-art models and under different quantization settings. By combining the three proposed ingredients – 1) combination of convolutional and attentional processing, 2) linear-complexity attention and, 3) our quantization mechanism – the overall model is able to run on commodity hardware, for both training and inference, while at the same time achieving performance competitive with state-of-the-art on video activity recognition benchmarks, when considering no additional input streams besides raw RGB frames, and no pre-training on large external datasets. This makes the resulting model suitable for real time video processing applications on the edge. In summary, our contributions are the following: • We design a novel Convolutional-Attentional 3D (CA3D) neural network model, based on the proposed Convolutional-Attentional Spatio-Temporal (CAST) layers, leveraging both convolutions and attention for efficient video activity recognition, with effective learning and generalization capabilities; • In order to guarantee efficient processing in attention blocks, we define a specific linear-complexity attention module based on local attention windows; • We introduce a novel quantization mechanism, that allows to further reduce the memory and computing footprint of our model, not only at inference time, but also during training. In the following, we provide a background on existing deep learning solutions for video processing, describing the limitations of current solutions (Section 2 ), and we provide the details of our design and our quantization strategy (Section 3 ). We provide an experimental validation of our design (Section 4 ), and finally we conclude with some remarks and possible future directions (Section 5 ). The code to reproduce our experiments is publicly available. 1 1 1 github.com/GabrieleLagani/ElderlyActivityRecognition 2 Related Work Early CNN architectures for video data extended the 2D convolutions used for image data to 3D convolutions, in order to process spatio-temporal video data, as in [ 22 , 23 ] . One such architecture was the C3D model [ 43 ] , using 3D convolutional blocks for video activity recognition. This model was further improved with the introduction of residual blocks, in the R3D architecture [ 44 ] . In the same contribution [ 44 ] , the authors proposed the R2+1D architecture, factorizing the spatio-temporal 3D convolutional kernel in a 2D spatial convolution, followed by a 1D temporal convolution, in order to achieve a more efficient processing without sacrificing accuracy. Other works explored architectural variations for improved performance and efficiency, such