ViSE: A Systematic Approach to Vision-Only Street-View Extrapolation
KaiyuanTan Yingying Shen Haiyang Sun Bing Wang, Guang Chen Hangjun Ye Xiaomi EV · 2025
Read ViSE: A Systematic Approach to Vision-Only Street-View Extrapolation by KaiyuanTan Yingying Shen Haiyang Sun Bing Wang, Guang Chen Hangjun Ye Xiaomi EV (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-18341.txt.
Introduction
Kaiyuan Tan, with Yingying Shen and four co-authors, reports ViSE in a 2025 technical report on street-view synthesis. The task extends a street scene beyond the car's original path, for closed-loop driving simulation. Closed-loop means the simulated car reacts and the scene updates in response. Current methods often warp or break images at new viewpoints.
ViSE runs in four stages, starting with a data-driven pseudo-LiDAR point cloud for initialization. LiDAR measures distance with laser pulses to map the surroundings. Later stages add road geometry, generated supervision for new views, and artifact cleanup. The system took first place in its workshop competition track.
Catalog record
| CREATOR | KaiyuanTan Yingying Shen Haiyang Sun Bing Wang, Guang Chen Hangjun Ye Xiaomi EV |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 18 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-18341
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ViSE: A Systematic Approach to Vision-Only Street-View Extrapolation KaiyuanTan Yingying Shen Haiyang Sun Bing Wang, Guang Chen Hangjun Ye Xiaomi EV Abstract Realistic view extrapolation is critical for closed-loop simulation in autonomous driving, yet it remains a significant challenge for current Novel View Synthesis (NVS) methods, which often produce distorted and inconsistent images beyond the original trajectory. This report presents our winning solution which ctook first place in the RealADSim Workshop NVS track at ICCV 2025. To address the core challenges of street view extrapolation, we introduce a comprehensive four-stage pipeline. First, we employ a data-driven initialization strategy to generate a robust pseudo-LiDAR point cloud, avoiding local minima. Second, we inject strong geometric priors by modeling the road surface with a novel dimension-reduced SDF termed 2D-SDF. Third, we leverage a generative prior to create pseudo ground truth for extrapolated viewpoints, providing auxilary supervision. Finally, a data-driven adaptation network removes time-specific artifacts. On the RealADSim-NVS benchmark, our method achieves a final score of 0.441, ranking first among all participants. † † footnotetext: \dagger Project leader 🖂 Corresponding Author 1 Introduction Validating autonomous driving algorithms in the real world faces high cost and significant safety risks, making high-fidelity simulation a critical enabling technology for advancing AD systems. Traditional game-engine-based simulators support closed-loop testing but suffer from a substantial domain gap and require labor-intensive, manual scene construction. Conversely, replay from real-world driving logs offers high visual fidelity but is limited to pre-recorded scenarios, unable to support interactive, closed-loop evaluation. Novel View Synthesis (NVS) presents a compelling pathway to bridge this gap, aiming to construct interactive, 4D driving environments directly from real-world captures. However, a critical obstacle remains: the inherent sparsity of driving logs means that most NVS methods excel at view interpolation but struggle severely with view extrapolation. Under extrapolated viewpoints, current techniques based on volumetric primitives like NeRF [ 7 ] or 3D Gaussian Splatting (3DGS) [ 4 ] often exhibit major geometric distortions and a collapse in texture realism. The RealADSim challenge was established to directly address this critical gap. It provides a set of multi-traversal driving logs with a quantitative metric designed for evaluating extrapolation performance. In this report, we introduce a holistic pipeline that achieves robust and geometrically consistent street view extrapolation. Our solution systematically tackles the core challenges through four key contributions: 1. A robust, LiDAR-free initialization strategy that generates a vision-based pseudo point cloud to provide a strong geometric prior and prevent overfitting. 2. A dimension-reduced 2D Signed Distance Function (2D-SDF) that enforces a locally planar prior for road surfaces, improving geometric consistency under large viewpoint changes. 3. An iterative pseudo-ground-truth framework that leverages a generative prior to provide supervision for unobserved, extrapolated viewpoints, effectively repairing artifacts. 4. A data-driven time-invariance adaptation network that removes transient, time-specific features, ensuring robust performance across logs captured under different conditions. On the RealADSim benchmark, our team achieves a final score of 0.441, ranking 1st among all participants. 2 Preliminary Figure 1 : Proposed Pipeline. We introduce a four-stage framework for consistent novel-view synthesis. The process begins with (1) generating a robust, vision-based point cloud for initialization. This enables (2) detailed 3D reconstruction, where a 2D-SDF provides strong geometric priors for the road. To address remaining artifacts in unobserved areas, we employ (3) an iterative pseudo-GT strategy using a generative prior. Finally, (4) a time-invariance adaptation network ensures the output is temporally invariant across different driving logs. 2.1 3D Gaussian Splatting 3D Gaussian Splatting (3DGS) [ 4 ] represents scenes using 3D Gaussian primitives. Each primitive is defined by a position \mathbf{\mu}\in\mathbb{R}^{3} , covariance matrix \mathbf{\Sigma}\in\mathbb{R}^{3\times 3} , and opacity \alpha\in\mathbb{R}^{+} : \mathcal{G}(\mathbf{x})=\alpha\exp\left(-\frac{1}{2}(\mathbf{x}-\mathbf{\mu})^{T}\mathbf{\Sigma}^{-1}(\mathbf{x}-\mathbf{\mu})\right) The covariance matrix is factorized into scaling and rotation components: \mathbf{\Sigma}=\mathbf{R}\mathbf{S}\mathbf{S}^{\top}\mathbf{R}^{\top} . For rendering, 3D Gaussians are transformed to camera coordinates via world-to-camera matrix \mathbf{W} and projected to 2D using local affine approximation \mathbf{J} : \mathbf{\Sigma}^{{}^{\prime}}=\mathbf{J}\mathbf{W}\mathbf{\Sigma}\mathbf{W}^{\top}\mathbf{J}^{\top} . By discarding the third dimension, we obtain 2D Gaussians \mathcal{G}^{2D} with covariance \mathbf{\Sigma}^{2D} . The final color is computed via front-to-back alpha blending: \mathbf{c}(\mathbf{x})=\sum^{K}_{k=1}\mathbf{c}_{k},\alpha_{k},\mathcal{G}^{2D}k(\mathbf{x})\prod{j=1}^{k-1}(1-\alpha_{j},\mathcal{G}^{2D}_{j}(\mathbf{x})) (1) where \mathbf{c}_{k} represents view-dependent color. This fully differentiable pipeline enables optimization through standard photometric losses. 2.2 Rendering Signed Distance Functions A common approach to representing a surface is through the zero-level set of a Signed Distance Function (SDF), where the function’s value indicates the distance to the nearest surface, with the sign denoting whether the point lies inside or outside the surface. NeuS [ 12 ] introduced a volume rendering technique for optimizing neural SDFs from posed images. Given a ray \mathbf{r} , with sampled points {\mathbf{p}(t_{i})}_{i=0}^{N} , the opacity \alpha_{i} at location \mathbf{p}(t_{i}) is computed as: \alpha_{i}=\max\left(\frac{\Phi_{s}(d(\mathbf{p}(t_{i})))-\Phi_{s}(d(\mathbf{p}(t_{i+1})))}{\Phi_{s}(d(\mathbf{p}(t_{i})))},0\right) (2) Where \Phi_{s}(x)=1/(1+\exp(-sx)) is the Sigmoid-based cumulative distribution function, with s controlling the sharpness of the transition and typically treated as a learnable parameter. Then, the color of each pixel is rendered using alpha-blending: \mathbf{C}(\mathbf{r})=\sum_{i=1}^{N}T_{i}\alpha_{i}\mathbf{c}_{i},\ \ T_{i}=\prod_{j=1}^{i-1}(1-\alpha_{j}) (3) 3 Method Problem Formulation Given a driving log comprising images {I_{i}}_{i=1}^{N} and their associated ego-vehicle poses {P_{i}}_{i=1}^{N} , our objective is to learn a 3D scene representation that maps a 6-DoF pose to an image, \mathcal{S}:SE(3)\rightarrow\mathbb{R}^{H\times W\times 3} . This work focuses on the challenging task of synthesizing novel views from extrapolated viewpoints beyond the original trajectory. We address four core challenges through a systematic, four-stage pipeline, illustrated in Figure 1 . 3.1 3D Scene Initialization without LiDAR The absence of LiDAR data poses a significant challenge for initializing 3D Gaussians. Conventional Structure-from-Motion (SfM) [ 10 ] points, often combined with random initialization, typically yield poor geometry due to the sparsity and limited viewing directions in driving logs. This frequently leads to convergence in a local minimum that fails under viewpoint extrapolation. To overcome this, we generate a vision-based pseudo-LiDAR point cloud for robust initialization. We adopt VGGT [ 11 ] for its favorable balance of quality and simplicity. As this method lacks absolute scale, we recover it by aligning the predicted camera poses with the ground-truth poses. The recovered scale is then multiplied to the predicted depth maps, which are subsequently unprojected and aggregated using the ground-truth poses to form a unified point cloud. Although this pseudo-LiDAR point cloud can be noisy, it provides critical geometric initialization that enables faster 3DGS convergence, recovers finer details, and prevents implausible geometric distortions during extrapolation. 3.2 Geometry-Aware 3D Scene Reconstruction 2D-SDF for Road Surface Representation Accurately modeling the road surface is paramount for driving simulation, as it contains critical visual elements like road signs and lane markings. However, this is challenging due to weak textures and large viewpoint changes in extrapolated scenarios. Methods based on 3DGS [ 4 ] and NeRF [ 7 ] often render severe distortions for the road surface under extrapolated trajectories. While several methods [ 20 , 5 , 19 ] explored constraining 3D Gaussians for a flat road surface, we consider an alternative representation. We introduce a novel 2D Signed Distance Function (2D-SDF) that incorporates the strong geometric prior that road surfaces are locally planar and exhibit globally smooth slope transitions . This allows us to compress the representation from a general 3D SDF to two efficient 2D fields defined over the horizontal plane, modeling surface height and local slope, respectively. Under the local planar assumption, the signed distance from a 3D point \mathbf{p} simplifies to: d(\mathbf{p})=|\cos\theta|\left(p_{z}-\mathcal{H}(p_{x},p_{y})\right),