Tri-Efficient Transfer Learning for Point Cloud Videos
Yiding Sun, Dongxu Zhang, Jihua Zhu †, Haozhe Cheng, Zhengqiao Li, Pengcheng Li, Chaowei Fang, Yonghao Dong, Lin Chen · 2026
Read Tri-Efficient Transfer Learning for Point Cloud Videos by Yiding Sun, Dongxu Zhang, Jihua Zhu †, Haozhe Cheng, Zhengqiao Li, Pengcheng Li, Chaowei Fang, Yonghao Dong, Lin Chen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-24175.txt.
Introduction
Sun and colleagues make fine-tuning of point cloud video models cheaper in three currencies at once: labelled data, trainable parameters, and memory. Point cloud videos record three-dimensional scenes over time. Adapting foundation models to them usually demands costly annotation and heavy fine-tuning. Their 2026 framework, PoinTriE, instead mines richer supervision from data that already exists.
For pretraining, fake motion paths are synthesised by rigidly transforming the clouds. Text descriptions and flat projections are derived from the raw data as extra training signals. A geometric-motion duality network learns through multimodal contrastive training, rotation prediction, and motion distribution matching. That yields dense self-supervision with no hand labels. During fine-tuning the pretrained backbone stays frozen. Only a light spatio-temporal side network, built with low-rank adapters, updates. Sun's team reports the mix excelling on all three efficiency axes, against existing parameter-efficient methods.
Catalog record
| CREATOR | Yiding Sun, Dongxu Zhang, Jihua Zhu †, Haozhe Cheng, Zhengqiao Li, Pengcheng Li, Chaowei Fang, Yonghao Dong, Lin Chen |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.24175
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Tri-Efficient Transfer Learning for Point Cloud Videos Yiding Sun, Dongxu Zhang, Jihua Zhu †, Haozhe Cheng, Zhengqiao Li, Pengcheng Li, Chaowei Fang, Yonghao Dong, Lin Chen Abstract While point cloud foundation models have significantly advanced point cloud video understanding, existing parameter-efficient fine-tuning (PEFT) methods still suffer from two critical limitations: prohibitive annotation costs for large-scale point cloud datasets and severe memory bottlenecks. In this paper, we aim to mine richer supervision signals from existing data rather than blindly scaling datasets. A further key principle is that the memory footprint of fine-tuning must be drastically reduced compared to full fine-tuning, which remains elusive for current PEFT techniques. Driven by these challenges, we identify three core desiderata: data-, parameter-, and memory efficiency, and present PoinTriE, a unified framework that excels along all three dimensions. For pre-training, pseudo-motion trajectories are synthesized via rigid transformations, paired with text corpora and 2D projections derived from raw point clouds. We then propose a Geometric-Motion Duality Network optimized via multimodal contrastive learning, rigid rotation prediction, and motion distribution divergence to produce dense self-supervision. During fine-tuning, we freeze the pretrained backbone and only update a lightweight Spatio-temporal Side Network built with LoRA units. Equipped with a gradient flow masking strategy, PoinTriE simultaneously reduces memory consumption and parameter overhead. Extensive experiments confirm that PoinTriE establishes new state-of-the-art results on action recognition and semantic segmentation tasks. 1 Introduction Fine-tuning pretrained point cloud foundation models (PFMs) has been proven remarkably effective in adapting to versatile scenarios, especially those in dynamic domains [ 3 , 4 ] . However, catastrophic forgetting and prior knowledge disruption occur frequently, particularly as model sizes continue to grow [ 36 , 23 ] . To overcome these challenges, Parameter-Efficient Fine-Tuning (PEFT) methods [ 59 , 67 , 51 ] , such as PointCSA [ 32 ] , have emerged as more promising alternatives, garnering significant attention. Instead of updating all model parameters directly, PointCSA inserts a learnable temporal adapter into the intertwined feature space of Transformer [ 50 ] blocks, bridging the gap between static and dynamic domains while keeping the PFM frozen. This method significantly reduces the number of parameters requiring fine-tuning, thereby reducing computational overhead. Follow-up research has further enhanced this pipeline by improving aspects such as adapter architecture [ 59 , 46 ] , local geometric interaction [ 25 ] , and backbone model scale [ 11 ] . Figure 1 : Three key limitations existing in point cloud video learning. L1 : PC and PCV data is two orders of magnitude smaller than images. L2 : Using task-specific dedicated models leads to severe parameter redundancy. L3 : Full fine-tuning often causes out-of-memory errors, and conventional PEFT methods still struggle to resolve this issue. To address these limitations, we propose PoinTriE, a unified framework that achieves strong performance along the three critical axes of efficiency. Despite these advancements, most of the prior art still relies on additive structures, i.e . , trainable adapter parameters within the pretrained model across various downstream tasks. However, as depicted in Fig. 1 , recent findings from NLP (Natural Language Processing) have revealed the redundancy of intermediate caches retained by each pretrained block during backpropagation in terms of GPU memory, underscoring the need to optimize the fine-tuning method to eliminate the substantial memory footprint [ 47 , 15 ] . On the other hand, unlike images and language [ 37 ] , collecting large-scale point cloud data requires enormous storage and computing resources [ 5 ] , let alone dynamic point cloud videos (PCV) [ 31 ] . This implies that indiscriminately using a fixed training and fine-tuning paradigm may not fully tap into the potential of PFMs, motivating us to efficiently adjust the existing paradigm to better accommodate each task. In this paper, we naturally consider data-, parameter-, and memory-efficiency as the ideal optimization objectives, which we refer to as tri-efficient transfer learning (TriETL). As the focus pivots towards the TriETL problem, several new research questions emerge. First, since the underlying nature of PCV learning is still underexplored, the definition of TriETL remains unclear. For instance, defining it as a single-stage process or a two-stage process leads to a critical consideration, namely whether the fine-tuning network has the privilege to intervene in the effectiveness of customized PFMs. This leads us to our first key question: (1) How can we appropriately and formally define TriETL? (§ 3.1 ). Second, once the definition of TriETL is established, a more essential question emerges. (2) How can we design an intuitive pipeline derived from this definition? (§ 3.2 , 3.3 ). To address these questions, we first theoretically analyze the TriETL problem in 3D vision. Our findings confirm the significance of TriETL and further reveal two key insights: rather than blindly expanding the scale of the dataset, it is better to carefully explore the intrinsic supervision within existing data and its variants. Moreover, the negative impact of intermediate cache occupancy on memory persists in the point cloud domain. Based on these insights, we propose PoinTriE, a novel two-stage PFM optimized for our objectives. Building on the definition, we utilize rigid body transformation to generate pseudo-motion trajectories, which comprise a series of variants derived from the same point cloud data. The Geometric-Motion Duality Network (GMD Net) first maps trajectory features and performs contrastive learning with their 2D projections and text corpora, while the subsequent dual-branch head assists in motion prior injection from the perspectives of regression and uncertainty. When it comes to fine-tuning, we freeze the backbone and adaptively learn the Spatio-temporal Side Network (STS Net) for each task in a ladder-side manner using a gradient flow masking strategy, thereby unlocking the full potential of PFM and low-rank linear transformation (LoRA) [ 21 ] units. Our main contributions are fourfold. • To the best of our knowledge, we are the first to investigate the TriETL problem in 3D vision and provide the corresponding theoretical proofs. • Building upon our insights into point cloud video learning, we propose a dedicated TriETL framework, termed PoinTriE, which can provide a baseline for further research from either a unified or an independent perspective. • We present the GMD Network for pretraining, which introduces pseudo-motion trajectories to augment under-utilized static point cloud data for multi-modal contrastive learning. Then we allocate LoRA units to STS blocks via the gradient flow masking strategy, resulting in significantly fewer trainable parameters and lower GPU memory usage. • Extensive experiments demonstrate that PoinTriE exhibits superior performance, e.g . , it attains 94.37\% average accuracy on MSR-Action3D and 84.11\% mIoU on Synthia 4D, with gains of 1.05pp and 0.95pp over baselines. 2 Related Work 2.1 Static&Dynamic Point Cloud Pretraining Within the domain of point cloud pretraining, self-supervised strategies are primarily divided into contrastive and generative frameworks. The contrastive framework aims to maximize agreement between different augmentations of the same point cloud while minimizing similarity across distinct samples. Initially proposed for 2D images [ 37 , 7 , 39 ] , it has been effectively extended to 3D data [ 18 , 60 , 61 , 52 ] and 4D data [ 42 , 64 , 16 , 40 ] . For instance, PointContrast [ 55 ] was an early milestone, followed by CPR