C3G: Learning Compact 3D Representations with 2K Gaussians

Honggyu An, Jaewoo Jung, Mungyeom Kim, Sunghwan Hong, Chaehyun Kim, Kazumi Fukuda, Minkyeong Jeon, Jisang Han, Takuya Narihira, Hyuna Ko, Junsu Kim, Yuki Mitsufuji, Seungryong Kim, [10pt] KAIST AI · 2025

Read C3G: Learning Compact 3D Representations with 2K Gaussians by Honggyu An, Jaewoo Jung, Mungyeom Kim, Sunghwan Hong, Chaehyun Kim, Kazumi Fukuda, Minkyeong Jeon, Jisang Han, Takuya Narihira, Hyuna Ko, Junsu Kim, Yuki Mitsufuji, Seungryong Kim, [10pt] KAIST AI (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-04021.txt.

Introduction

Rebuilding a three-dimensional scene from a handful of unposed photos usually takes millions of tiny Gaussian blobs, most of them redundant. Honggyu An and a large team propose C3G in a 2025 preprint, a feed-forward system placing only about two thousand Gaussians where they matter. The compactness cuts memory use and sharpens scene understanding at once.

Learnable tokens gather features from multiple views through self-attention and steer where Gaussians are generated, so each one carries information pooled across viewpoints. The same attention patterns then guide feature lifting, attaching rich descriptors to the sparse geometry. Experiments cover pose-free novel view synthesis, open-vocabulary segmentation, and view-invariant feature matching. C3G beat existing methods on memory efficiency and feature fidelity. Their message: a small, geometrically meaningful representation suffices for high-quality reconstruction.

Catalog record

CREATORHonggyu An, Jaewoo Jung, Mungyeom Kim, Sunghwan Hong, Chaehyun Kim, Kazumi Fukuda, Minkyeong Jeon, Jisang Han, Takuya Narihira, Hyuna Ko, Junsu Kim, Yuki Mitsufuji, Seungryong Kim, [10pt] KAIST AI
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE73 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-04021

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

C3G: Learning Compact 3D Representations with 2K Gaussians Honggyu An, Jaewoo Jung, Mungyeom Kim, Sunghwan Hong, Chaehyun Kim, Kazumi Fukuda, Minkyeong Jeon, Jisang Han, Takuya Narihira, Hyuna Ko, Junsu Kim, Yuki Mitsufuji, Seungryong Kim, [10pt] KAIST AI Abstract Reconstructing and understanding 3D scenes from unposed sparse views in a feed-forward manner remains as a challenging task in 3D computer vision. Recent approaches use per-pixel 3D Gaussian Splatting for reconstruction, followed by a 2D-to-3D feature lifting stage for scene understanding. However, they generate excessive redundant Gaussians, causing high memory overhead and sub-optimal multi-view feature aggregation, leading to degraded novel view synthesis and scene understanding performance. We propose C3G , a novel feed-forward framework that estimates compact 3D Gaussians only at essential spatial locations, minimizing redundancy while enabling effective feature lifting. We introduce learnable tokens that aggregate multi-view features through self-attention to guide Gaussian generation, ensuring each Gaussian integrates relevant visual features across views. We then exploit the learned attention patterns for Gaussian decoding to efficiently lift features. Extensive experiments on pose-free novel view synthesis, 3D open-vocabulary segmentation, and view-invariant feature aggregation demonstrate our approach’s effectiveness. Results show that a compact yet geometrically meaningful representation is sufficient for high-quality scene reconstruction and understanding, achieving superior memory efficiency and feature fidelity compared to existing methods. Figure 1 : Teaser. Our method learns compact 3D Gaussians from unposed multi-view images through a query-based Gaussian decoding pipeline. Compact representations enable efficient 2D-to-3D feature lifting for downstream applications, including 3D understanding, correspondence, and upsampling. Compared to prior works (LSM 14 and CF 3 33 ), Our C3G results in the fewest Gaussians—about 2K , which is roughly 65 \times fewer than LSM 14 —while achieving superior memory efficiency and novel view synthesis quality. † † ∗ Equal contributions. † † † Co-corresponding. 1 Introduction Obtaining 3D representations from unposed sparse multi-view images in a feed-forward manner remains a fundamental challenge in computer vision and graphics, with broad implications for robotics 54 ; 16 , scene understanding 15 , and novel view synthesis 71 . Recently, feed-forward 3D Gaussian splatting frameworks have gained considerable attention, demonstrating impressive performance in reconstruction and understanding 71 ; 24 ; 19 ; 14 ; 55 . However, these approaches predominantly rely on dense, per-pixel Gaussian predictions, which inherently leads to two critical issues: (1) it generates excessive redundant primitives that are often misaligned in 3D space (Fig. 2 ), and (2) it incurs substantial computational overhead when incorporating semantic features 55 ; 14 through 2D-to-3D feature lifting. Consequently, prior works compress rich semantic information 58 ; 45 ; 30 into lower-dimensional embeddings, resulting in information loss and sub-optimal scene understanding. This raises a fundamental question: do we need such pixel-aligned Gaussians to reconstruct and understand 3D scenes? As humans, we do not maintain pixel-perfect mental reconstructions of every surface to understand our surroundings. Instead, we form compact, semantically meaningful abstractions of identifying key objects, their rough spatial relationships, and overall scene structure 3 ; 56 . Drawing direct inspiration from human visual cognition, we propose a novel framework C3G for learning compact 3D Gaussians from unposed image observations in a feed-forward fashion. Similar to prior approaches 71 ; 24 , we first extract image features from visual encoders with rich geometric priors (e.g., VGGT 63 ). However, instead of learning to estimate per-pixel Gaussians directly from the extracted feature maps, we introduce a compact set of learnable query tokens that discover and decode essential 3D Gaussians. Specifically, we adopt a transformer architecture 62 where learnable query tokens and the image features are processed through multiple self-attention blocks. We decode the refined learnable query tokens as 3D Gaussians, where the query tokens learn to aggregate essential information across multiple views to faithfully represent the scene. Crucially, our framework requires no explicit supervision from ground-truth depths or scene decompositions. Despite training solely on photometric reconstruction, each token naturally learns to represent different regions, with each token attending to coherent spatial regions across views. This emergent behavior arises from the inherent structure of the task: to efficiently reconstruct novel views with a limited number of Gaussians, the model must learn to allocate Gaussians to meaningful regions. We show that after training, our model can estimate a compact set of 3D Gaussians, which enables efficient novel view synthesis while maintaining performance. In addition, the compact set of Gaussians enables 2D-to-3D feature lifting without compression, significantly improving 3D scene understanding tasks where the rich representation of the semantic features is critical. Figure 2 : Comparison of per-pixel and compact scene representations. (Left) : Existing per-pixel estimators 71 ; 24 predict one or multiple Gaussians per pixel, resulting in redundant Gaussians with misalignments across views. (Right) : Our method uses learnable Gaussian queries to discover and decode only compact 3D Gaussians at essential locations, achieving a compact representation with only 2K Gaussians and 4.1M memory while avoiding redundancy and achieving superior segmentation and novel view synthesis performance. Our framework also provides a novel solution to a key challenge in 2D-to-3D feature lifting: handling multi-view feature inconsistencies. While other methods 74 ; 33 ; 6 ; 41 require additional aggregation methods to handle inconsistent features across viewpoints, we observe that our model’s emergent property of tokens attending to spatially coherent regions across views can be directly leveraged for feature aggregation. Specifically, we propose a view-invariant feature decoder ( C3G- \mathcal{F} ) that reuses the attention maps from our learned Gaussian decoder ( C3G- \mathcal{G} ) while training only the value projections. This feature decoder can then take features from any visual encoder as input and decode multi-view aggregated features. By attaching these aggregated features to our estimated 3D Gaussians, we enable efficient novel view rendering with view-invariant features. We validate the effectiveness of our approach through extensive experiments. For novel view synthesis, despite using 65 \times fewer Gaussians (only 2K ) than per-pixel methods 71 ; 24 , we achieve competitive visual quality while enabling substantially faster rendering. More importantly, for 3D semantic understanding tasks, our compact Gaussians combined with the multi-view aggregated semantic features significantly outperform previous feed-forward approaches that attempt to lift view-inconsistent features through dense per-pixel Gaussians 14 . We further verify that our feature renderings can effectively replace previous feature upsamplers, while also outperforming previous view-invariant feature aggregation methods in multi-view correspondence evaluation. 2 Related work Learning compact 3D scene representations. Decomposing scenes into geometric primitives (e.g., meshes, polygons, superquadric primitives) has been extensively studied 61 ; 47 ; 46 , but these methods typically require 3D data (e.g., point clouds) as input. SuperDec 15 recently proposed feed-forward decomposition into superquadric primitives, but relies on pre-computed 3D point clouds and requires iterative application of an external 3D instance segmentation model to identify instances. The differentiable blocks world (DBW) 43 learns superquadric parameters directly from multi-view images through photometric optimization, but can only model up to 10 primitives and object-centric scenes. In contrast, we aim to learn an efficient solution that can estimate compact scene representations in a feed-forward manner, only given unposed multi-view images captured in real-world scenarios. Feed-forward 3D Gaussian splatting. Recently, 3D Gaussian Splatting (3DGS) 29 has gained significant attention for 3D reconstruction and novel view synthesis. However, it requires dense multi-view

Read the complete text