Seeing 3D Through 2D Lenses: 3D Few-Shot Class-Incremental Learning via Cross-Modal Geometric Rectification

Tuo Xiang, Xuemiao Xu, Bangzhen Liu, Jinyi Li, Yong Li, Shengfeng He · 2025

Read Seeing 3D Through 2D Lenses: 3D Few-Shot Class-Incremental Learning via Cross-Modal Geometric Rectification by Tuo Xiang, Xuemiao Xu, Bangzhen Liu, Jinyi Li, Yong Li, Shengfeng He (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-14958.txt.

Introduction

Tuo Xiang and five co-authors teach machines to learn new 3D object classes from very few examples, in a 2025 computer vision paper. Three-dimensional recognition systems must keep growing in open-world use, but scarce examples per new class cause trouble. Projecting 3D shapes into 2D views to borrow knowledge from image models adds texture bias and blurs meaning.

Their framework, Cross-Modal Geometric Rectification, uses the layered spatial knowledge inside the CLIP image model to restore geometric fidelity. A structure-aware module aligns parts of the 3D shape with CLIP's intermediate spatial features through attention. A texture amplification module then synthesizes small but distinctive surface details. Together the parts fight unstable class prototypes and catastrophic forgetting, where old classes get erased as new ones arrive.

Catalog record

CREATORTuo Xiang, Xuemiao Xu, Bangzhen Liu, Jinyi Li, Yong Li, Shengfeng He
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-14958

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Seeing 3D Through 2D Lenses: 3D Few-Shot Class-Incremental Learning via Cross-Modal Geometric Rectification Tuo Xiang, Xuemiao Xu, Bangzhen Liu, Jinyi Li, Yong Li, Shengfeng He Abstract The rapid growth of 3D digital content necessitates expandable recognition systems for open-world scenarios. However, existing 3D class-incremental learning methods struggle under extreme data scarcity due to geometric misalignment and texture bias. While recent approaches integrate 3D data with 2D foundation models (e.g., CLIP), they suffer from semantic blurring caused by texture-biased projections and indiscriminate fusion of geometric-textural cues, leading to unstable decision prototypes and catastrophic forgetting. To address these issues, we propose Cross-Modal Geometric Rectification (CMGR), a framework that enhances 3D geometric fidelity by leveraging CLIP’s hierarchical spatial semantics. Specifically, we introduce a Structure-Aware Geometric Rectification module that hierarchically aligns 3D part structures with CLIP’s intermediate spatial priors through attention-driven geometric fusion. Additionally, a Texture Amplification Module synthesizes minimal yet discriminative textures to suppress noise and reinforce cross-modal consistency. To further stabilize incremental prototypes, we employ a Base-Novel Discriminator that isolates geometric variations. Extensive experiments demonstrate that our method significantly improves 3D few-shot class-incremental learning, achieving superior geometric coherence and robustness to texture bias across cross-domain and within-domain settings. 1 Introduction Figure 1: Direct alignment of point clouds with CLIP’s text features proves challenging due to the inherent domain gap. (a) While projecting depth maps to leverage 2D priors can bring point cloud and text features closer, discrepancies between the depth map and CLIP’s input space impede effective alignment. (b) In contrast, through 2D-3D rectification, point clouds are enriched with hierarchical 2D information derived from CLIP, facilitating a more precise alignment with text features. The proliferation of 3D digital content [ 7 , 35 , 30 , 33 , 12 , 43 , 20 , 38 , 23 ] has revolutionized environmental modeling in domains such as autonomous navigation and industrial robotics. However, existing 3D recognition systems [ 25 , 26 , 34 , 21 ] remain ill-equipped for open-world deployment, where novel object categories may dynamically emerge under extreme data scarcity, often leading to catastrophic degradation of performance. This limitation stems from a fundamental mismatch between static latent representations and the evolving geometric complexity of real-world 3D environments. Addressing this challenge necessitates progress in 3D Few-Shot Class-Incremental Learning (3DFSCIL), a setting in which models must incrementally integrate sparse geometric cues from limited point cloud data while preserving discriminative capabilities for previously encountered categories. Earlier 3DFSCIL approaches [ 11 ] employed low-dimensional embeddings derived from PCA or SVD to mitigate catastrophic forgetting. While effective in reducing overfitting on scarce incremental samples, they did so at the expense of geometric expressiveness, particularly in the context of structurally complex point clouds. More recent strategies [ 37 , 8 ] have leveraged 2D vision foundation models such as CLIP [ 28 ] by incorporating depth map rendering [ 37 ] or synthetic texture augmentation [ 8 ] to bridge modality gaps. However, these methods depend on single-step 3D-to-2D projections that extract only CLIP’s final-layer representations, discarding the rich spatial priors encoded in intermediate layers—priors that more naturally correspond to structural 3D hierarchies. This omission introduces semantic misalignment between novel 3D categories and their 2D projections (Figure 1 (a)), exacerbating semantic blurring due to the naive multimodal fusion mechanism. We posit that robust cross-modal alignment in 3DFSCIL hinges on a crucial yet underutilized insight: CLIP’s intermediate layers preserve spatial relationships (e.g., part adjacency and symmetry) that are inherently aligned with 3D geometric hierarchies. However, the exclusive focus on CLIP’s final layer of existing methods inadvertently forces depth maps into texture-biased semantics space and erodes geometric distinctiveness. Furthermore, conventional adapters indiscriminately fuse visual and geometric features, collapsing 3D representations into texture-dominated spaces. This dual misalignment induces geometric myopia, wherein incremental updates amplify texture-driven biases while suppressing structural coherence. To address these challenges, we propose cross-modal geometric rectification, which repurposes CLIP’s intermediate layers as “2D lenses” to enhance geometric fidelity. Our framework corrects three core misalignments. First, it mitigates semantic blurring by integrating 3D point cloud features with CLIP’s hierarchical spatial priors rather than relying on texture-biased final-layer representations. Second, it resolves modality mismatch through semantic-aware texture synthesis, aligning depth maps with CLIP’s pretrained space without corrupting structural integrity. Third, it stabilizes incremental learning by dynamically reconfiguring decision boundaries to isolate geometric variations from historical knowledge. Specifically, our approach consists of three key components: the Structure-Aware Geometric Rectification (SAGR) module, the Texture Amplification Module (TAM), and the Base-Novel Discriminator (BND). The SAGR module hierarchically integrates CLIP’s intermediate features with 3D structural embeddings through attention mechanisms that prioritize part-level coherence over texture cues. Meanwhile, the TAM learns minimal yet discriminative texture patterns to align depth maps with CLIP’s visual space, eliminating ambiguous texture extraction in geometric rectification. Finally, the BND isolates geometrically distinct regions for novel classes, ensuring stable knowledge retention across incremental phases. By transforming CLIP from a texture-driven projector into a geometry-refining lens, our framework sharpens 3D representations, enabling precise cross-modal alignment and improved incremental learning under extreme data scarcity. In summary, our contributions are threefold: • We propose Cross-Modal Geometric Rectification, a framework that leverages CLIP’s hierarchical 2D semantics to enhance 3D representations through structural fusion of intermediate spatial priors and 3D geometric embeddings. • We introduce a Structure-Aware Geometric Rectification module that hierarchically aligns 3D part structures with CLIP’s intermediate spatial priors via attention-driven geometric fusion, along with a Texture Amplification Module that synthesizes minimal yet discriminative textures to reinforce cross-modal consistency while suppressing noise. • Extensive experiments validate our method’s effectiveness in mitigating texture bias and preserving geometric coherence, achieving state-of-the-art performance in 3D few-shot class-incremental learning across both cross-domain and within-domain settings. 2 Related Work 2.1 Few-Shot Class Incremental Learning Early FSCIL methods focused on 2D images, graph-based classifier adaptation [ 32 ] was first proposed and followed by replay-based [ 18 , 22 , 2 , 36 ] and pseudo-based [ 44 , 24 , 40 , 47 ] methods. Attention mechanisms were further used in [ 40 , 45 , 10 , 42 ] . Subsequent 3D extensions like I3DOL [ 14 ] introduced geometric centroid alignment for point cloud incremental learning, while MicroShape [ 11 ] leveraged text embeddings to enhance few-shot feature transfer. Recent advances exploit vision-language models: FILP-3D [ 37 ] pioneers CLIP integration via depth map rendering, and C3PR [ 8 ] improves CLIP adaptation through viewpoint reprogramming. However, these methods rely on single-step 3D-to-2D projections that over-prioritize CLIP’s texture-biased final layers, neglecting intermediate spatial priors critical for geometric hierarchy alignment. This oversight perpetuates semantic ambiguity between modalities, particularly in disentangling structural coherence from texture noise during incremental updates. 2.2 Foundation Models Foundation models have redefined artificial intelligence through extensive pretraining on diverse datasets. BERT [ 13 ] has advanced natural language processing via large-scale masked language modeling, while GPT-3 [ 5 ] has demonstrated notable few-shot learning in language generation. In computer vision, CLIP [ 28 ] integrates visual and textual modalities through training on expansive image-text pairs, thereby driving progress in multimodal learning. More recently, Uni3D [ 46 ] introduced a unified, scalable framework for 3D pretraining, enabling large-scale 3D representation learning and extending the reach of foundation models to 3D tasks. These developments highlight the

Read the complete text