PointAlign: Feature-Level Alignment Regularization for 3D Vision-Language Models

Yuanhao Su, Shaofeng Zhang, Xiaosong Jia, Qi Fan · 2026

Read PointAlign: Feature-Level Alignment Regularization for 3D Vision-Language Models by Yuanhao Su, Shaofeng Zhang, Xiaosong Jia, Qi Fan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-00412.txt.

Introduction

Su and three colleagues improve three dimensional vision language models, which suffer because paired shape text data are scarce. Language only training wastes geometry held in middle layers. Their PointAlign method adds feature level supervision that keeps point cloud tokens aligned with visual tokens through a consistency loss, training only a small projector and adapter weights. On ModelNet40 and Objaverse tests it lifts classification and captioning scores, with the largest gain on hard open vocabulary Objaverse classification.

Catalog record

CREATORYuanhao Su, Shaofeng Zhang, Xiaosong Jia, Qi Fan
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38327
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.00412

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

PointAlign: Feature-Level Alignment Regularization for 3D Vision-Language Models Yuanhao Su, Shaofeng Zhang, Xiaosong Jia, Qi Fan Abstract The development of 3D Vision-Language Models (VLMs), crucial for applications in robotics, autonomous driving, and augmented reality, is severely constrained by the scarcity of paired 3D-text data. Existing methods rely solely on next-token prediction loss, using only language tokens for supervision. This results in inefficient utilization of limited 3D data and leads to a significant degradation and loss of valuable geometric information in intermediate representations. To address these limitations, we propose PointAlign, a novel feature-level alignment regularization method. PointAlign explicitly supervises intermediate point cloud tokens to preserve fine-grained 3D geometric-semantic information throughout the language modeling process. Specifically, we constrain the intermediate point cloud tokens within the LLM to align with visual input tokens via a consistency loss. By training only a lightweight alignment projector and LoRA adapters, PointAlign achieves explicit feature-level supervision with minimal computational overhead, effectively preventing geometric degradation. Extensive experiments on ModelNet40 and Objaverse datasets demonstrate that our method achieves 2.08 pp improvement on average for classification tasks, with a substantial 7.50 pp gain on the challenging open-vocabulary Objaverse classification task and 4.88 pp improvement on 3D object captioning evaluated by Qwen2-72B-Instruct, validating the effectiveness of PointAlign. Code is publicly available at https://github.com/yharoldsu0627/PointAlign . 1 Introduction While 2D vision-language models (VLMs) 22 ; 3 have achieved remarkable success through large-scale image-text datasets that enable effective cross-modal alignment 37 ; 32 , 3D vision-language models remain comparatively underdeveloped despite their importance for applications in robotics 55 ; 48 , autonomous driving 23 ; 26 , and augmented reality 8 ; 1 . A fundamental challenge in 3D VLM development is the scarcity of high-quality 3D-text paired data—unlike easily accessible 2D images 30 , 3D point clouds 41 require expensive acquisition, resulting in smaller datasets with often simplistic textual descriptions 34 . This data limitation poses a core research question: how to maximize knowledge extraction from scarce supervision for 3D understanding . Recent efforts such as PointLLM 42 , ShapeLLM 28 , and MiniGPT-3D 33 have made progress in addressing these issues. PointLLM and ShapeLLM achieve competitive performance via full-model fine-tuning at substantial computational cost, while MiniGPT-3D reduces costs by leveraging 2D VLM priors. However, these methods optimize through language modeling objectives, which only reward geometric features that directly facilitate next-token prediction. This creates a critical issue: structural cues valuable for spatial reasoning but orthogonal to immediate language tasks may be discarded during training, a degradation we confirm in our feature quality analysis (see Figure 3 ). Recent work on 2D vision-language models 19 ; 45 ; 21 has shown that without explicit visual supervision, representational quality degrades across network depth. In the context of point cloud understanding, this problem is amplified by limited 3D training data. Point cloud networks process information hierarchically: initial layers establish object-level context, intermediate layers resolve part-level geometry, and final layers produce language outputs 45 . Therefore, preserving structural information throughout the forward pass is critical for accurate 3D reasoning. To address this, we propose PointAlign, a novel feature-level alignment regularization method. Our method is built on a key insight: the representations from the initial stage already capture fine-grained geometric and semantic details, serving as an ideal internal supervision target. We directly align the LLM’s intermediate point cloud tokens with these high-quality, initial-stage features. This alignment is enforced by a consistency loss during the second training stage. This approach introduces explicit geometric guidance with minimal overhead, as we only update a lightweight alignment projector (only 8.39M parameters) and LoRA adapters while freezing all pre-trained modules. We conduct extensive experiments on several 3D vision-language benchmarks, including 3D classification and captioning. Results show that our alignment regularization improves performance across nearly all evaluated tasks , confirming that explicit feature-level supervision helps better preserve geometric structure (as validated by Figure 3 ) and exploit limited 3D data (as demonstrated in Figure 4 ). On average, PointAlign improves classification accuracy by 2.08 pp on ModelNet40 and Objaverse. Notably, it achieves a striking 7.50 pp gain on the challenging open-vocabulary Objaverse classification task. For 3D object captioning on Objaverse, evaluated by Qwen2-72B-Instruct, our method outperforms the baseline by a significant 4.88 pp, demonstrating strong generalization in open-domain 3D vision-language scenarios of the proposed PointAlign. 2 Related Work Large 3D Point Cloud-Language Models. The success of large language models in reasoning and generalization has motivated efforts to develop 3D point cloud-language understanding systems. Initial methods 43 ; 44 ; 49 ; 56 ; 13 ; 27 ; 25 ; 40 ; 9 ; 6 adopt an indirect approach by converting 3D shapes into multiple 2D projections, then leveraging pre-trained vision-language models for cross-modal alignment 51 ; 52 ; 50 . While computationally efficient, this rendering-based pipeline discards native geometric structure, hindering models from capturing spatial relationships inherent to 3D data. To address this limitation, recent work 42 ; 28 ; 29 ; 10 ; 33 directly processes point cloud tokens through learned projection modules that bridge the gap between 3D encoders and language models. Full model fine-tuning approaches like PointLLM 42 and ShapeLLM 28 achieve strong performance but at significant computational cost. For example, PointLLM-13B demands over 200 GPU-hours on high-end accelerators. MiniGPT-3D 33 mitigates this burden by incorporating 2D vision-language priors to facilitate 3D-text alignment, achieving comparable results with substantially reduced training resources. Beyond training efficiency, recent work has also explored inference acceleration for 3D MLLMs through visual token pruning techniques 14 . Representation Supervision in Multimodal LLMs. Recent studies on 2D vision-language models reveal that visual embeddings can lose fidelity across network depth when supervision signals are weak 19 ; 18 ; 20 . To address this issue, reconstruction-based approaches 11 ; 39 ; 38 ; 54 ; 53 have been proposed to supervise intermediate representations by recovering visual inputs, demonstrating effectiveness in retaining fine-grained details for 2D understanding tasks. However, 3D point cloud reasoning demands fundamentally different capabilities—models must capture structural relationships and geometric configurations that extend beyond low-level visual features 47 ; 36 . In the 3D point cloud domain, methods leveraging pre-trained 3D encoders 15 ; 21 provide specialized geometric supervision by distilling knowledge from foundation models trained on large-scale 3D point cloud data, though this introduces additional computational overhead during the training phase. 3 The Proposed PointAlign Preliminaries: MiniGPT-3D MiniGPT-3D 33 is an efficient 3D large language model that aligns 3D point clouds with LLMs by leveraging priors from 2D vision-language models. Given a point cloud \mathbf{P}\in\mathbb{R}^{n\times d} , where n denotes the number of points and d represents the feature dimension of each point, the point cloud encoder f_{pc}(\cdot) first encodes it into 3D features f_{pc}(\mathbf{P})\in\mathbb{R}^{m\times D} , where m is the length of the feature sequence and D is the feature dimension. Subsequently, an MLP projection layer f_{MLP}(\cdot) projects the point cloud features into the semantic space of Q-Former. Next, Q-Former 22 f_{QF}(\cdot) queries the point cloud features through learnable query vectors \mathbf{Q}\in\mathbb{R}^{o\times D_{1}} , yielding point cloud tokens \overline{\mathbf{Q}}=f_{QF}(f_{MLP}(f_{pc}(\mathbf{P})),\mathbf{Q})\in\mathbb{R}^{o\times D_{1}} , where o is the number of queries and D_{1} is the hidden dimension of Q-Former. Finally, the modality projector maps the Q-Former output to the input space of the LLM as \mathbf{T}_{pc}\in\mathbb{R}^{o\times C} ,

Read the complete text