SkillFormer: Unified Multi-View Video Understanding for Proficiency Estimation
Edoardo Bianchi, Antonio Liotta · 2025
Read SkillFormer: Unified Multi-View Video Understanding for Proficiency Estimation by Edoardo Bianchi, Antonio Liotta (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-08665.txt.
Introduction
Edoardo Bianchi and Antonio Liotta present SkillFormer, a model that rates human skill from video shot from several views, in a 2025 paper. Judging skill in complex tasks serves sports, rehab, and training. Views from the actor's eyes and from outside each hold part of the story.
Built on a video transformer, the model joins view features with cross-attention, gating, and self-calibration. Only a small subset of parameters is tuned, which cuts training cost. On a large public data set, it reached top accuracy in multi-view settings with far fewer parameters and training rounds than prior methods. The results confirm the value of joining views for fine skill judgment.
Catalog record
| CREATOR | Edoardo Bianchi, Antonio Liotta |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 31 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-08665
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SkillFormer: Unified Multi-View Video Understanding for Proficiency Estimation Edoardo Bianchi, Antonio Liotta Abstract Assessing human skill levels in complex activities is a challenging problem with applications in sports, rehabilitation, and training. In this work, we present SkillFormer, a parameter-efficient architecture for unified multi-view proficiency estimation from egocentric and exocentric videos. Building on the TimeSformer backbone, SkillFormer introduces a CrossViewFusion module that fuses view-specific features using multi-head cross-attention, learnable gating, and adaptive self-calibration. We leverage Low-Rank Adaptation to fine-tune only a small subset of parameters, significantly reducing training costs. In fact, when evaluated on the EgoExo4D dataset, SkillFormer achieves state-of-the-art accuracy in multi-view settings while demonstrating remarkable computational efficiency, using 4.5x fewer parameters and requiring 3.75x fewer training epochs than prior baselines. It excels in multiple structured tasks, confirming the value of multi-view integration for fine-grained skill assessment. SkillFormer † † email: [email protected] † † email: [email protected] † † affiliation: Free University of Bozen-Bolzano Bozen-Bolzano, IT 1 Introduction Automatically estimating human expertise from video is a key challenge in computer vision, especially in domains like sports, rehabilitation, and skill training. Yet, unlike action recognition, this task requires capturing subtle differences in motion quality and precision, which are often visible only through a combination of viewpoints. To address this, we propose SkillFormer, a lightweight and efficient architecture for multi-view skill assessment. SkillFormer processes synchronized egocentric and exocentric video streams using a TimeSformer backbone, enhanced by a novel CrossViewFusion module that leverages cross-attention and learnable gating. To enable efficient fine-tuning, it incorporates Low-Rank Adaptation (LoRA). This design achieves competitive overall accuracy while substantially improving computational efficiency through reduced trainable parameters and faster training convergence. This work introduces the following key contributions: • A CrossViewFusion module that fuses multiple views via multi-head cross-attention and learnable gating, allowing the model to focus on the most informative aspects of each view. • A self-calibration mechanism that aligns view-specific embeddings using learnable statistics. • A LoRA-based fine-tuning strategy applied to the TimeSformer backbone, enabling efficient adaptation with minimal overhead. Compared to state-of-the-art benchmarks, SkillFormer demonstrates strong overall classification accuracy in multi-view settings, which are the primary focus of our work. While our Ego-only configuration achieves comparable results to specialized single-view methods (45.9% vs 46.8%), the real strength of our approach emerges in multi-view scenarios. SkillFormer significantly improves accuracy in exocentric settings by 14% (from 40.6% to 46.3%) and in combined ego+exo setups by 16.4% (from 40.8% to 47.5%). The benefits are particularly pronounced in structured activities, where our multi-view approach achieves improvements for Basketball (77.88%), Cooking (60.53%), and Bouldering (33.52%) by effectively integrating complementary spatial and temporal features from synchronized perspectives. These substantial gains highlight the effectiveness of our integrated cross-attention fusion approach, particularly in scenarios where multiple viewpoints and fine-grained motion cues are crucial for expertise assessment. 2 Background and Related Works Action Quality Assessment (AQA) and Proficiency Estimation focus on evaluating the quality of human actions, with applications across sports, rehabilitation, and skill training [ Zhou et al.(2024)Zhou, Cai, Wang, Shum, and Liang ] . Traditional approaches relied on single-modality inputs, using handcrafted features or 3D CNNs like C3D and I3D to extract spatiotemporal representations [ Parmar and Morris(2019) , Carreira and Zisserman(2017) , Tran et al.(2015)Tran, Bourdev, Fergus, Torresani, and Paluri ] . More recent methods adopt transformer-based models such as TimeSformer, Video Swin, and ActionFormer to capture long-range dependencies and fine-grained motion dynamics [ Bertasius et al.(2021)Bertasius, Wang, and Torresani , Liu et al.(2021)Liu, Ning, Cao, Wei, Zhang, Lin, and Hu , Zhang et al.(2022)Zhang, Wu, and Li ] . Specialized architectures like GDLT decouple grade-aware features via decoder attention [ Xu et al.(2022)Xu, Zeng, and Zheng ] , while CoRe introduces contrastive regression for more precise skill scoring [ Yu et al.(2021)Yu, Rao, Zhao, Lu, and Zhou ] . Nonetheless, skill classification remains underexplored, particularly in multi-view settings. Multi-modal AQA improves robustness by integrating complementary signals—RGB, flow, audio, depth, or pose. Skating-Mixer [ Xia et al.(2022)Xia, Zhuge, Geng, Fan, Wei, He, and Zheng ] and PAMFN [ Zeng and Zheng(2024) ] incorporate audio-visual cues via staged fusion. Others combine RGB and depth for industrial tasks [ Bianchi and Lanz(2024) ] , or rely on skeletons to capture motion kinematics [ Bianchi and Lanz(2025) , Du et al.(2015)Du, Wang, and Wang , Duan et al.(2022)Duan, Zhao, Chen, Lin, and Dai ] . Datasets like EgoExoLearn [ Huang et al.(2024)Huang, Chen, Xu, Zhang, Yang, Pei, Zhang, Lu, Wang, Wang, and Qiao ] and EgoExo4D [ Grauman et al.(2024)Grauman, Westbury, Torresani, Kitani, Malik, Afouras, Ashutosh, Baiyya, Bansal, Boote, Byrne, Chavis, Chen, Cheng, Chu, Crane, Dasgupta, Dong, Escobar, Forigua, Gebreselasie, Haresh, Huang, Islam, Jain, Khirodkar, Kukreja, Liang, Liu, Majumder, Mao, Martin, Mavroudi, Nagarajan, Ragusa, Ramakrishnan, Seminara, Somayazulu, Song, Su, Xue, Zhang, Zhang, Castillo, Chen, Fu, Furuta, Gonzalez, Gupta, Hu, Huang, Huang, Khoo, Kumar, Kuo, Lakhavani, Liu, Luo, Luo, Meredith, Miller, Oguntola, Pan, Peng, Pramanick, Ramazanova, Ryan, Shan, Somasundaram, Song, Southerland, Tateno, Wang, Wang, Yagi, Yan, Yang, Yu, Zha, Zhao, Zhao, Zhu, Zhuo, Arbelaez, Bertasius, Damen, Engel, Farinella, Furnari, Ghanem, Hoffman, Jawahar, Newcombe, Park, Rehg, Sato, Savva, Shi, Shou, and Wray ] emphasize the importance of multi-perspective inputs, but existing fusion strategies often rely on heuristic or late fusion, limiting generalization. In EgoExo4D [ Grauman et al.(2024)Grauman, Westbury, Torresani, Kitani, Malik, Afouras, Ashutosh, Baiyya, Bansal, Boote, Byrne, Chavis, Chen, Cheng, Chu, Crane, Dasgupta, Dong, Escobar, Forigua, Gebreselasie, Haresh, Huang, Islam, Jain, Khirodkar, Kukreja, Liang, Liu, Majumder, Mao, Martin, Mavroudi, Nagarajan, Ragusa, Ramakrishnan, Seminara, Somayazulu, Song, Su, Xue, Zhang, Zhang, Castillo, Chen, Fu, Furuta, Gonzalez, Gupta, Hu, Huang, Huang, Khoo, Kumar, Kuo, Lakhavani, Liu, Luo, Luo, Meredith, Miller, Oguntola, Pan, Peng, Pramanick, Ramazanova, Ryan, Shan, Somasundaram, Song, Southerland, Tateno, Wang, Wang, Yagi, Yan, Yang, Yu, Zha, Zhao, Zhao, Zhu, Zhuo, Arbelaez, Bertasius, Damen, Engel, Farinella, Furnari, Ghanem, Hoffman, Jawahar, Newcombe, Park, Rehg, Sato, Savva, Shi, Shou, and Wray ] , baseline models are modality-specific—separate architectures are trained for first-person and third-person views, with predictions later combined via ensembling. These models also rely on heterogeneous pretraining datasets across modalities, making comparison and deployment under unified constraints more challenging. Recent work on efficient adaptation, such as LoRA [ Hu et al.(2021)Hu, Shen, Wallis, Allen-Zhu, Li, Wang, Wang, and Chen ] , enables fine-tuning with fewer trainable parameters. Knowledge distillation also supports low-resource adaptation for vision tasks [ Wang and Yoon(2022) , Meneghetti et al.(2025)Meneghetti, Bianchi, Demo, and Rozza ] . Our work advances the state-of-the-art by addressing key limitations in prior methods—such as reliance on modality-specific models, late fusion, and heavy pretraining. By introducing a unified architecture for multi-view skill classification, we simultaneously enhance classification accuracy and computational efficiency, enabling robust skill assessment under realistic resource constraints. Figure 1: Overview of the SkillFormer architecture. Multi-view video inputs (one egocentric and up to four exocentric) are processed through a shared TimeSformer backbone fine-tuned with LoRA. Features are fused using the CrossViewFusion module and passed to a classification head. 3 Proposed Methodology We present SkillFormer, a novel architecture for unified multi-view proficiency estimation. Our model builds upon the TimeSformer [ Bertasius et al.(2021)Bertasius, Wang, and Torresani ] framework, incorporating parameter-efficient fine-tuning through LoRA [ Hu et al.(2021)Hu, Shen, Wallis, Allen-Zhu, Li, Wang, Wang, and Chen ] and a specialized CrossViewFusion module for