Adaptive Anchor Policies for Efficient 4D Gaussian Streaming
Ashim Dahal, Rabab Abdelfattah, Nick Rahimi · 2026
Read Adaptive Anchor Policies for Efficient 4D Gaussian Streaming by Ashim Dahal, Rabab Abdelfattah, Nick Rahimi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-17227.txt.
Introduction
Dahal, Abdelfattah, and Rahimi ask whether the choice of anchors matters in 4D Gaussian streaming. Anchor driven methods rebuild a dynamic scene each frame from a compact set of Gaussian anchors. The default picks anchors by farthest point sampling at a fixed budget. The authors hold the pipeline fixed and swap in random, uniform, and heuristic samplers, plus a learned policy. At deployment budgets, the sampler had no measurable effect on quality. A cheap random sampler at half the budget matched the default.
Catalog record
| CREATOR | Ashim Dahal, Rabab Abdelfattah, Nick Rahimi |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34907 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.17227
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Adaptive Anchor Policies for Efficient 4D Gaussian Streaming Ashim Dahal, Rabab Abdelfattah, Nick Rahimi Abstract Dynamic scene reconstruction with Gaussian Splatting has enabled efficient streaming for real-time rendering and free-viewpoint video. However, most pipelines rely on fixed anchor selection such as Farthest Point Sampling (FPS), typically using 8,192 anchors regardless of scene complexity, which over-allocates computation under strict budgets. We propose Efficient Gaussian Streaming (EGS), a plug-in, budget-aware anchor sampler that replaces FPS with a reinforcement-learned policy while keeping the Gaussian streaming reconstruction backbone unchanged. The policy jointly selects an anchor budget and a subset of informative anchors under discrete constraints, balancing reconstruction quality and runtime using spatial features of the Gaussian representation. We evaluate EGS in two settings: fast rendering, which prioritizes runtime efficiency, and high-quality refinement, which enables additional optimization. Experiments on dynamic multi-view datasets show consistent improvements in the quality–efficiency trade-off over FPS sampling. On unseen data, in fast rendering at 256 anchors ( 32 × 32\times fewer than 8,192), EGS improves PSNR by + 0.52 +0.52 – 0.61 0.61 dB while running 1.29 1.29 – 1.35 × 1.35\times faster than IGS@8192 (N3DV and MeetingRoom). In high-quality refinement, EGS remains competitive with the full-anchor baseline at substantially lower anchor budgets. Code and pretrained checkpoints will be released upon acceptance. 1 Introduction Reconstructing dynamic scenes from multi-view observations is a fundamental problem in computer vision, enabling applications such as free-viewpoint video (FVV), novel-view synthesis (NVS), immersive media, and telepresence. Achieving high visual fidelity while maintaining real-time performance remains challenging, particularly when large numbers of scene primitives must be processed during rendering and reconstruction. Neural radiance fields (NeRF) [ 29 ] have significantly advanced view synthesis by learning continuous volumetric representations of scenes. However, NeRF-based methods typically rely on computationally intensive volume rendering and per-scene optimization, which limits their efficiency for real-time applications. More recently, explicit scene representations have emerged as an efficient alternative. In particular, 3D Gaussian Splatting [ 13 ] represents scenes as collections of anisotropic Gaussians that can be efficiently rasterized, enabling high-quality rendering with substantially improved performance. Building on Gaussian representations, recent works have explored streaming-based reconstruction pipelines that progressively select subsets of Gaussian anchors to balance reconstruction quality and computational efficiency. In these pipelines, anchor selection plays a critical role in determining both rendering fidelity and runtime cost. However, existing approaches commonly rely on heuristic strategies such as Farthest Point Sampling (FPS) [ 36 ] , which allocate a fixed number of anchors regardless of scene complexity. While FPS promotes spatial coverage, it does not explicitly account for the varying importance of anchors across different scenes or viewpoints, potentially leading to inefficient allocation of computational resources. This observation motivates adaptive and content-aware anchor selection for Gaussian streaming pipelines. Instead of relying on fixed heuristic sampling, anchor selection can be formulated as a decision-making problem that determines which anchors are most informative for reconstruction under a given computational budget. Based on this idea, we develop Efficient Gaussian Streaming (EGS) , a framework that learns to select informative anchors while maintaining compatibility with existing Gaussian streaming architectures. Our contributions are summarized as follows: • We analyze the limitations of fixed heuristic anchor selection in Gaussian streaming pipelines and show that it can lead to inefficient allocation of computation under strict budgets. • We introduce Efficient Gaussian Streaming (EGS) , a plug-in, budget-aware sampler that adaptively selects both an anchor budget and a subset of informative anchors while keeping the streaming backbone unchanged. • We develop a two-stage learning scheme (SFT warm-start followed by RL Contextual Bandit optimization) with a reward that explicitly trades off sparsity, runtime, and reconstruction quality. • Extensive experiments on N3DV and MeetingRoom demonstrate consistent improvements in the quality–efficiency trade-off over FPS-based sampling. 2 Related Work Novel view synthesis and efficient scene representations. Novel view synthesis has been widely studied with implicit neural representations such as NeRF and dynamic variants that model time via deformation or conditioning [ 29 , 35 , 33 , 32 , 26 ] , supported by multi-view video benchmarks for free-viewpoint video [ 23 , 38 , 34 , 6 ] . Complementary approaches improve efficiency via factorized space–time fields [ 2 , 8 ] and explicit point-based representations; in particular, 3D Gaussian Splatting (3DGS) enables real-time rendering with differentiable rasterization and has become a strong foundation for interactive view synthesis [ 13 ] , with ongoing work improving efficiency and scalability [ 42 , 10 , 55 ] . Dynamic Gaussian splatting. Dynamic extensions of splatting model motion and time-varying appearance using explicit Gaussian primitives [ 49 , 25 , 7 , 52 ] . Recent work explores broader capture settings and additional priors, including multi-view consistency, physics, or hybrid representations [ 12 , 14 , 31 , 46 , 5 , 3 , 54 , 24 , 57 , 28 , 15 , 39 ] . These methods can achieve strong quality, but many remain offline or do not explicitly target strict streaming budgets. Streaming and online dynamic reconstruction. Streaming radiance fields have been studied via incremental learning and residual representations [ 22 , 44 ] . Gaussian streaming methods pursue low-latency updates either by training Gaussians on-the-fly [ 40 ] or by using generalizable feed-forward motion prediction driven by a compact set of anchors [ 51 , 20 , 37 ] . IGS is particularly relevant as it standardizes an anchor-driven streaming pipeline and motivates plug-in improvements to anchor selection under fixed budgets [ 51 ] . Anchor selection under budget constraints. Anchor-driven streaming critically depends on selecting a subset of Gaussians under strict compute and point budgets. Existing systems typically use hand-crafted heuristics—most notably farthest point sampling (FPS)—to promote spatial coverage [ 36 , 51 ] . While learned selection under resource constraints has been explored elsewhere [ 50 , 45 , 11 , 4 , 16 ] , applying it to anchor-based Gaussian streaming with joint budget+subset constraints remains underexplored. Our work targets budgeted anchor selection by learning a context-aware policy that directly optimizes the quality–runtime trade-off within the IGS streaming interface [ 51 ] . Learning policies for resource allocation. Reinforcement learning and contextual bandits provide principled tools for allocating limited computation based on observed context [ 41 , 21 , 17 ] . Policy-gradient methods offer a general mechanism for optimizing stochastic selection policies [ 48 ] . Attention-based set encoders further provide a natural architecture for scoring and selecting elements from unordered candidate sets [ 43 , 18 ] . EGS combines these perspectives to learn budgeted anchor selection for streaming 4DGS under strict fast-mode evaluation with same-budget comparisons. Figure 1 : Method overview of EGS. Our adaptive sampler selects a budgeted set of anchor Gaussians (stochastic during training; deterministic at inference) using point-MLP embeddings and a lightweight Transformer [ 43 , 18 ] , then feeds the selected anchors to the frozen IGS pipeline for anchor-graph construction and rasterization [ 51 ] . The training objective trades off sparsity, runtime, and PSNR against FPS-based teacher targets [ 36 , 9 ] . 3 Methodology We introduce Efficient Gaussian Streaming (EGS) as a learned plug-in sampler that selects anchors under an explicit budget while keeping the downstream IGS reconstruction and