CLIP-GS: Unifying Vision-Language Representation with 3D Gaussian Splatting
Siyu Jiao, Haoye Dong, Yuyang Yin, Zequn Jie, Yinlong Qian, Yao Zhao Humphrey Shi Yunchao Wei · 2024
Read CLIP-GS: Unifying Vision-Language Representation with 3D Gaussian Splatting by Siyu Jiao, Haoye Dong, Yuyang Yin, Zequn Jie, Yinlong Qian, Yao Zhao Humphrey Shi Yunchao Wei (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-19142.txt.
Description
Siyu Jiao and seven co-authors present CLIP-GS in this 2024 paper, a framework that joins language and vision with 3D Gaussian Splatting. Splatting renders scenes from clouds of soft 3D blobs that carry texture, unlike sparse point clouds. The team built a tokenizer that turns those blobs into a sequence of tokens. Transformer layers, started from point cloud model weights, then encode the sequence into embeddings. Training aligns those embeddings with the image and text embeddings of CLIP.
An image voting loss steers the gradient so the alignment settles in the right direction. The team also built an efficient pipeline to mint matched triplets of splats, images, and text for training. The unified representation then serves many tasks. On retrieval, zero shot, and few shot classification tests, CLIP-GS beat point cloud based rivals.
Catalog record
| CREATOR | Siyu Jiao, Haoye Dong, Yuyang Yin, Zequn Jie, Yinlong Qian, Yao Zhao Humphrey Shi Yunchao Wei |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-19142