BlendCLIP: Bridging Synthetic and Real Domains for Zero-Shot 3D Object Classification with Multimodal Pretraining

Ajinkya Khoche, Gergő László Nagy, Maciej Wozniak, Thomas Gustafsson, Patric Jensfelt · 2025

Read BlendCLIP: Bridging Synthetic and Real Domains for Zero-Shot 3D Object Classification with Multimodal Pretraining by Ajinkya Khoche, Gergő László Nagy, Maciej Wozniak, Thomas Gustafsson, Patric Jensfelt (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-18244.txt.

Introduction

Ajinkya Khoche, with Gergő László Nagy and three co-authors, presents BlendCLIP in a 2025 paper on three-dimensional vision. The task is zero-shot classification, meaning naming objects the model never saw labeled examples of during training. Models trained on clean computer-made objects often fail on messy real LiDAR scans from cars.

BlendCLIP trains on both computer-made objects and real driving data together. The team built triples of point cloud, image, and text from real road scenes. A staged mixing plan starts with rich synthetic data, then slowly adds real scans. Adding even a tiny share of real samples sharply raised accuracy on road benchmarks.

Catalog record

CREATORAjinkya Khoche, Gergő László Nagy, Maciej Wozniak, Thomas Gustafsson, Patric Jensfelt
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-18244

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

BlendCLIP: Bridging Synthetic and Real Domains for Zero-Shot 3D Object Classification with Multimodal Pretraining Ajinkya Khoche, Gergő László Nagy, Maciej Wozniak, Thomas Gustafsson, Patric Jensfelt Abstract Zero-shot 3D object classification is crucial for real-world applications like autonomous driving, however it is often hindered by a significant domain gap between the synthetic data used for training and the sparse, noisy LiDAR scans encountered in the real-world. Current methods trained solely on synthetic data fail to generalize to outdoor scenes, while those trained only on real data lack the semantic diversity to recognize rare or unseen objects. We introduce BlendCLIP , a multimodal pretraining framework that bridges this synthetic-to-real gap by strategically combining the strengths of both domains. We first propose a pipeline to generate a large-scale dataset of object-level triplets—consisting of a point cloud, image, and text description—mined directly from real-world driving data and human annotated 3D boxes. Our core contribution is a curriculum-based data mixing strategy that first grounds the model in the semantically rich synthetic CAD data before progressively adapting it to the specific characteristics of real-world scans. Our experiments show that our approach is highly label-efficient: introducing as few as 1.5% real-world samples per batch into training boosts zero-shot accuracy on the nuScenes benchmark by 27% . Consequently, our final model achieves state-of-the-art performance on challenging outdoor datasets like nuScenes and TruckScenes, improving over the best prior method by 19.3% on nuScenes , while maintaining strong generalization on diverse synthetic benchmarks. Our findings demonstrate that effective domain adaptation, not full-scale real-world annotation, is the key to unlocking robust open-vocabulary 3D perception. Our code and dataset will be released upon acceptance on https://github.com/kesu1/BlendCLIP . † † footnotetext: These authors contributed equally to this work. 1 Introduction Figure 1 : Overview of BlendCLIP. We construct a large-scale set of outdoor LiDAR–image–text triplets from real-world driving data and introduce a curriculum-based data mixing strategy to integrate them with existing synthetic triplets. A 3D encoder is trained to align with frozen CLIP text and image embeddings, enabling robust zero-shot 3D classification across both synthetic and real-world domains. Outdoor 3D perception systems operate in complex, dynamic environments where the types of objects encountered are both diverse and unpredictable. Applications in autonomous driving and mobile robotics require models that can robustly recognize objects despite challenging conditions like sensor noise, occlusions, and the inherent sparsity of LiDAR scans. However, most existing 3D recognition pipelines rely on closed-set classification, where object categories are fixed in advance with full supervision for each class [ 25 , 26 , 27 , 42 ] . This assumption breaks in real-world deployments, where new categories frequently emerge, label coverage is incomplete, and retraining models for every change is infeasible. This reality motivates the critical need for zero-shot 3D object classification —the ability to assign meaningful semantic labels to 3D objects from unseen categories without retraining. In the 2D domain, this capability has been revolutionized by large-scale vision-language models (VLMs) like CLIP [ 28 ] , which align images and text in a shared embedding space. However, extending this paradigm to 3D point clouds remains a significant challenge. Early efforts have largely followed two paths, each with its own limitations. One line of work leverages large-scale synthetic CAD models to train 3D encoders aligned with VLM embeddings [ 41 , 21 ] . While these approaches benefit from clean, dense geometries and diverse semantic labels, they struggle to bridge the sim-to-real gap, where real data (coming from LiDAR) is sparse, incomplete, and suffers from a substantial domain gap. The other type of methods train exclusively on real-world outdoor data, often using automatically generated labels [ 14 , 43 ] . While tailored to the target domain, these models are constrained by the limited visual and semantic diversity of existing driving datasets, causing them to struggle with rare, long-tail object categories and perform poorly on broad zero-shot benchmarks. In this work, we argue that robust open-vocabulary 3D classification hinges on bridging this synthetic-to-real domain gap. We propose a new framework that synergistically combines the semantic richness of synthetic data with the real-world LiDAR data. To achieve this, we first construct a large-scale dataset of LiDAR–image–text triplets from real autonomous driving data ( Section 3.2 ). We then introduce a novel curriculum-based data mixing strategy that gradually introduces real-world LiDAR crops during training, allowing the model to adapt without forgetting the general knowledge learned from synthetic data( Section 3.3 ). We evaluate our method on three distinct datasets spanning synthetic, urban, and highway domains. Our experiments demonstrate state-of-the-art zero-shot classification performance, achieving up to a 19.3% improvement on real-world data when our curriculum strategy is applied, while maintaining competitive performance on synthetic data, demonstrating strong generalization across both domains. Notably, our model’s performance improves significantly with even a small fraction of real LiDAR samples ( Figure 3 ), suggesting that our approach can unlock strong generalization without requiring re-annotation of large-scale datasets. Our contributions are summarized as follows: • We introduce a pipeline for generating real-world object-level triplets (LiDAR, image, text) from large-scale autonomous driving data, and use it to construct a comprehensive triplet dataset. • We introduce a curriculum-based data mixing strategy that combines a semantically rich CAD object dataset with real LiDAR crops, enabling robust generalization across synthetic and real-world domains. • We evaluate our method across diverse scenarios and show significant gains on two challenging outdoor datasets (nuScenes and TruckScenes), achieving state-of-the-art zero-shot 3D classification performance. 2 Related Work 2.1 Multimodal Representation Learning Recent works aim to imbue 3D point-cloud encoders with the semantic richness of large-scale vision–language models, learning unified embeddings that support zero-shot classification and cross-modal retrieval of shapes. PointCLIP [ 44 ] projects point clouds into multi-view depth maps, encoding them with the off-the-shelf visual backbone of CLIP [ 28 ] . CLIP2Point [ 16 ] renders paired depth and RGB views from infinitely dense CAD objects for contrastive pre-training, directly mapping global CLIP representations to point clouds. ULIP [ 40 ] aligns 3D point-cloud encodings with frozen CLIP image and text embeddings, employing multimodal contrastive learning to enhance 3D understanding. Using large-scale vision-language priors through 3D-2D-text triplets, it significantly improves standard and zero-shot 3D classification. Follow-up works extend this by scaling up across multiple datasets, automating text caption generation [ 41 ] and introducing strategies to filter and enrich noisy user texts [ 21 ] . 2.2 Representation Learning for Outdoor Scenes A primary challenge in outdoor perception is learning semantically rich representations that can be effectively applied to downstream tasks. One line of research has focused on leveraging powerful, pre-trained 2D Vision-Language Models (VLMs). These methods generate rich map representations by directly lifting 2D VLM features to 3D, enabling queriable classification, segmentation, and visual navigation [ 18 , 4 , 12 ] . While effective, this direct mapping process can be computationally costly, limiting scalability. To address this, other works propose large-scale pseudo-labeling; for instance, SAL [ 24 ] presents a scalable method for generating per-point instance and CLIP pseudo-labels from 2D VLMs and distills this knowledge into a 3D model. SAL-4D [ 46 ] extends this by distilling video-based pseudo-labels into temporally consistent 4D annotations. Concurrently, methods like LidarCLIP [ 14 ] and CLIP

Read the complete text