TIGaussian : Disentangle Gaussians for Spatial-Awared Text-Image-3D Alignment
Jiarun Liu, Qifeng Chen, Yiru Zhao, Minghua Liu, Baorui Ma, Sheng Yang · 2026
Read TIGaussian : Disentangle Gaussians for Spatial-Awared Text-Image-3D Alignment by Jiarun Liu, Qifeng Chen, Yiru Zhao, Minghua Liu, Baorui Ma, Sheng Yang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-19247.txt.
Introduction
Jiarun Liu and five co-authors align text, images, and 3D shapes in one shared space. Vision language models have tied words to pictures with great success. Adding the 3D sense, through point clouds and 3D Gaussians, opens training for retrieval, zero shot sorting, and scene naming across all three.
Their method, TIGaussian, disentangles the Gaussians so the alignment stays spatially aware. Knowing where things are, not just what they are, is what 3D tasks demand. Aligned text, image, and 3D data would let a user search a scene with a sentence, a fresh power for design and robotics.
Catalog record
| CREATOR | Jiarun Liu, Qifeng Chen, Yiru Zhao, Minghua Liu, Baorui Ma, Sheng Yang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-19247
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
TIGaussian : Disentangle Gaussians for Spatial-Awared Text-Image-3D Alignment Jiarun Liu, Qifeng Chen, Yiru Zhao, Minghua Liu, Baorui Ma, Sheng Yang Abstract While visual-language models have profoundly linked features between texts and images, the incorporation of 3D modality data, such as point clouds and 3D Gaussians, further enables pretraining for 3D-related tasks, e.g., cross-modal retrieval, zero-shot classification, and scene recognition. As challenges remain in extracting 3D modal features and bridging the gap between different modalities, we propose TIGaussian , a framework that harnesses 3D Gaussian Splatting (3DGS) characteristics to strengthen cross-modality alignment through multi-branch 3DGS tokenizer and modality-specific 3D feature alignment strategies. Specifically, our multi-branch 3DGS tokenizer decouples the intrinsic properties of 3DGS structures into compact latent representations, enabling more generalizable feature extraction. To further bridge the modality gap, we develop a bidirectional cross-modal alignment strategies: a multi-view feature fusion mechanism that leverages diffusion priors to resolve perspective ambiguity in image-3D alignment, while a text-3D projection module adaptively maps 3D features to text embedding space for better text-3D alignment. Extensive experiments on various datasets demonstrate the state-of-the-art performance of TIGaussian in multiple tasks. Code repository: https://github.com/RUiN-jiarun/TIGaussian . Figure 1: TIGaussian enables 3D modal pretraining on several tasks, e.g., zero-shot classification, text-3D retrieval and image-3D retrieval. Left: Compared to prior 3D multi-modal alignment methods – Uni3D and UniGS, TIGaussian presents superior performance on multiple datasets – Objaverse(-LVIS) and ABO. Right: In challenging scenarios involving ambiguous or complex queries, TIGaussian demonstrates superior performance owing to its disentangled encoder architecture and specialized cross-modal alignment mechanism. We report the similarity score of each item in the figure. 1 Introduction Recent breakthroughs in vision-language pretraining ( 33 ; 40 ) have established text-image alignment as the foundation for multi-modal understanding. In recent years, significant advances have been made in expanding the field of 3D visual understanding through various 3D representations. Early works in 3D multi-modal alignment primarily relied on point clouds processed through 3D convolutions ( 52 ; 12 ; 46 ; 47 ) or 3D Vision Transformers ( 50 ) . Subsequently, works on meshes ( 39 ) and voxels ( 35 ) further explores the operations on organized 3D representations. More recently, UniGS ( 16 ) first leverages 3D Gaussian Splatting (3DGS) as an advanced 3D representation, demonstrating state-of-the-art text-image-3D alignment across multiple benchmarks. However, existing methods still facing challenges regarding the abstraction ability of 3D context, as well as the gaps between text-3D and image-3D modalities. Specifically, our analysis reveals that the current 3DGS-based multi-modal work UniGS, while groundbreaking, suffers from following limitations: (1) Entangled 3D encoding : All attributes of each Gaussian primitive are concatenated and encoded as a whole, insufficiently leveraged the distribution pattern and geometric significance of each attribute intrinsic relations between attributes; (2) Degraded 3D perception : Forced alignment from a single-view image fails to capture global context, causing a decrease in 3D perception ability. To overcome these limitations, we propose TIGaussian , a tri-modal alignment framework that bridges text, image, and 3DGS representations which outperforms existing feature alignment approaches. At its core, our method utilizes multi-branch 3DGS tokenizer to extract 3D contextual information, which reducing inter-attribute coupling, ultimately establishing a compact and effective latent representation for 3DGS. For image-3D alignment, we incorporate a diffusion-enhanced multi-view fusion strategy, which compensates for single-view limitations by leveraging implicit 3D awareness from pretrained diffusion priors. Meanwhile, for text-3D alignment, the encoded feature adapts the 3D latent space through a 3D-text projection module, thereby better matching text embedding structures. We conduct extensive experiments to validate the performance of TIGaussian on multiple tasks across the Objaverse ( 4 ) , ABO ( 3 ) and SUN RGBD ( 38 ) datasets. As presents in Fig. 1 , the results demonstrate the effectiveness of the proposed method, achieving superior performances on zero-shot classification, cross-modal retrieval, few-shot linear probing and open-world scene recognition. In summary, our contributions can be summarized as follows: • We propose TIGaussian , a framework for text-image-3DGS alignment, achieving state-of-the-art performance on various downstream tasks. • We design a multi-branch 3DGS tokenizer, with specialized Gaussian attribute modeling to enhance 3D feature abstraction and compression. • We propose a tri-modal alignment strategy through 3D-aware image feature fusion and 3D-text projection module, enabling robust alignment for both text and image modalities. 2 Related Work 2.1 3D Representation Learning. The development of 3D representations is an important factor in promoting research on 3D shape analysis and multi-modality tasks. Early works focused on volumetric grids ( 24 ) , which enabled structured 3D convolutions but incurred prohibitive computational costs for high-resolution scenes. With the development of neural networks, more works use point clouds as sparse geometric proxies, leveraging permutation-invariant operators like PointNet ( 28 ; 29 ) and other classical network architectures ( 2 ; 23 ) to extract localized features. In recent years, there have been many mesh-based feature extraction works in dense 3D representation ( 7 ; 10 ) , but the inherent complexity of mesh structures poses significant challenges for effective feature representation. A transformative leap arrived with implicit neural representations like DeepSDF ( 27 ) and NeRF ( 25 ) , which encode surfaces or radiance fields via coordinate-based MLPs. These methods achieved unprecedented reconstruction quality but suffered from slow optimization and rendering. The recent Gaussian Splatting ( 14 ; 11 ) further addresses these limitations by introducing explicit, differentiable primitives optimized for real-time photorealistic rendering. Some recent works exploit the hierarchical structures for scene organization and neural anchors representations ( 22 ; 34 ) for better feature extraction, or embedding the semantic or visual features into 3D Gaussian primitives ( 55 ; 36 ; 32 ) . These works in 3D representation learning in recent years have demonstrated the superiority of 3DGS in scene representation, understanding, and other aspects. 2.2 Multi-modal Pretraining in 3D Tasks. Text-2D Multi-modal Learning. Multi-modal pretraining via contrastive learning has revolutionized cross-modal understanding, with foundational frameworks like CLIP ( 33 ) and EVA-CLIP ( 40 ) demonstrating the power of aligning image-text pairs. BLIP series ( 18 ; 17 ) further enhance the text-image alignment capability through richer data and hybrid encoder-decoder structures. More recent works involve Large Language Models (LLMs) for visual-text understanding ( 41 ; 51 ; 5 ; 1 ) . Text-image-3D Tri-modal Learning. Recently, more works have extended to 3D fields. Early 3D adaptation strategies such as PointCLIP ( 52 ) and CLIP2Point ( 12 ) circumvented geometric complexity by rendering depth maps from point clouds, effectively projecting 3D data into 2D subspaces for compatibility with pretrained image encoders. Recent efforts like ULIP series ( 46 ; 47 ) and CLIP 2 ( 50 ) directly encode raw point clouds through 3D transformers, achieving improved robustness in real-world indoor/outdoor benchmarks. Other works such as TriCoLo ( 35 ) explore the performance of voxel representations in multi-modal alignment, but are limited to the scale of data and models. OpenShape ( 19 ) leverages Point-BERT ( 49 ) and a 3D convolution architecture to train a point cloud encoder for cross-modal understanding. The concurrent work Uni3D ( 54 ) utilizes the Vision Transformer architecture ( 6 ) and large amounts of point cloud data to achieve a model with 1B parameters large model, and realize