Photorealistic Phantom Roads in Real Scenes: Disentangling 3D Hallucinations from Physical Geometry
Hoang Nguyen, Xiaohao Xu, Xiaonan Huang University of Michigan, Ann Arbor · 2025
Read Photorealistic Phantom Roads in Real Scenes: Disentangling 3D Hallucinations from Physical Geometry by Hoang Nguyen, Xiaohao Xu, Xiaonan Huang University of Michigan, Ann Arbor (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-15423.txt.
Introduction
Hoang Nguyen, Xiaohao Xu, and Xiaonan Huang study a safety flaw in depth models in a 2025 paper. Models that guess depth from one image can invent solid shapes where a flat surface only looks three dimensional. The team calls this failure the 3D Mirage, and sets out to measure it.
They build a benchmark of real world illusions, plus two scores that track false structure and unstable readings. They then train a Depth Anything V2 model with a focused self training method that fixes illusion areas while keeping other skill. The work pushes depth tests beyond pixel accuracy toward structure and context.
Catalog record
| CREATOR | Hoang Nguyen, Xiaohao Xu, Xiaonan Huang University of Michigan, Ann Arbor |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 55 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-15423
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Photorealistic Phantom Roads in Real Scenes: Disentangling 3D Hallucinations from Physical Geometry Hoang Nguyen, Xiaohao Xu, Xiaonan Huang University of Michigan, Ann Arbor Abstract Monocular depth foundation models achieve remarkable generalization by learning large-scale semantic priors, but this creates a critical vulnerability: they hallucinate illusory 3D structures from geometrically planar but perceptually ambiguous inputs. We term this failure the 3D Mirage . This paper introduces the first end-to-end framework to probe , quantify , and tame this unquantified safety risk. To probe , we present 3D-Mirage , the first benchmark of real-world illusions (e.g., street art) with precise planar-region annotations and context-restricted crops. To quantify , we propose a Laplacian-based evaluation framework with two metrics: the Deviation Composite Score (DCS) for spurious non-planarity and the Confusion Composite Score (CCS) for contextual instability. To tame this failure, we introduce Grounded Self-Distillation , a parameter-efficient strategy that surgically enforces planarity on illusion ROIs while using a frozen teacher to preserve background knowledge, thus avoiding catastrophic forgetting. Our work provides the essential tools to diagnose and mitigate this phenomenon, urging a necessary shift in MDE evaluation from pixel-wise accuracy to structural and contextual robustness. Our code and benchmark will be publicly available to foster this exciting research direction. 1 Introduction Figure 1 : The 3D Mirage: Hallucinations induced by Illusive Phantom Road Patterns. ( a ) A driving scene featuring a deceptive phantom road pattern (3D illusion). ( c ) With full global context, the depth foundation model 63 correctly identifies the road as planar. ( d- f) However, when the view is restricted to the local region, the model fails to disambiguate the texture from geometry. It hallucinates significant non-existent 3D obstacles ( f ) from the phantom pattern, illustrating a critical vulnerability in reliable 3D perception for autonomous driving scenarios. Enabling reliable perception and reconstruction of 3D scene is paramount for promising safe and robust visual intelligence 35 ; 36 ; 38 ; 62 ; 37 ; 34 ; 47 ; 46 and autonomous driving experience 39 . Driven by this necessity, Monocular Depth Estimation (MDE) has transitioned from a challenging academic problem to a core perception component in real-world systems. This rapid adoption is fueled by powerful foundation models such as Depth-Anything V2 63 , Zoe-Depth 4 , and MiDaS/DPT 48 ; 49 , which are trained on massive, diverse datasets. However, their remarkable zero-shot generalization obscures a critical and unexamined vulnerability: an over-reliance on large-scale statistical priors causes these models to trade geometric fidelity for semantic consistency, making them susceptible to perceptual ambiguity. In this work, we identify and analyze a critical failure mode we term the 3D Mirage . We find that SOTA depth foundation models fail in two common, safety-critical scenarios: 1) when presented with perceptually ambiguous 2D patterns, such as 3D street art, and 2) when operating under a restricted field-of-view (FOV) that removes broad contextual cues. The phenomenon is illustrated in Fig. 1 : given a full scene, a model correctly perceives a flat road. However, when the view is cropped to the same road section, which emulates a limited FOV or partial occlusion, the model hallucinates a significant, non-planar obstacle. This demonstrates a profound failure of contextual grounding, where the model’s depth prediction is not anchored in local geometric reality but is instead a fragile artifact of its large-scale training priors. This failure is not an isolated anecdote. We demonstrate that this vulnerability is systemic across the current generation of leading models. As shown in Fig. 2 , we subjected a wide range of architectures—from transformer-based (Depth-Anything V2 63 ) and diffusion-based (Marigold 28 ) to generative (DepthFM 20 ) and commercially-developed (Depth Pro 6 )—to these 3D mirage inputs. All models exhibited similar failures, unstably predicting spurious 3D structures from planar surfaces. This collective failure exposes a critical gap in how we evaluate these models. Standard metrics like Mean Absolute Error (MAE) and Root Mean Squared Error (RMSE) 58 are perceptually-blind to these structural failures. By averaging pixel-wise errors, they cannot differentiate between a slight, uniform mis-calibration and a massive, hallucinated obstacle. We posit that MDE evaluation must evolve to assess structural integrity and contextual stability , which are far more critical for real-world deployment than pure pixel accuracy. To address this, our work provides the first end-to-end framework to systematically probe , quantify , and tame 3D hallucinations. Our contributions are threefold: • We probe this vulnerability by introducing 3D-Mirage , the first benchmark of real-world images featuring 3D illusion patterns, complete with precise planar-region annotations and controlled, context-restricted crops. • We quantify these failures by proposing a novel Laplacian-based evaluation framework , introducing two metrics: the Deviation Composite Score (DCS) to measure spurious non-planarity (hallucination intensity) and the Confusion Composite Score (CCS) to measure contextual instability ( i.e. , the mirage effect). • We tame these hallucinations with a novel Grounded Self-Distillation strategy. By applying low-parameter adapters to the model’s encoder, we use our benchmark to enforce planarity on illusion ROIs while using the frozen teacher model to enforce alignment on stable background and border regions. This efficiently grounds the model, mitigating hallucinations without catastrophic forgetting of its core pre-trained knowledge. Ultimately, our contributions provide the essential tools—a targeted benchmark, perceptually-aware metrics, and an efficient mitigation strategy—to advance MDE from simple geometric accuracy to the structural and contextual robustness demanded by safety-critical applications. Figure 2 : Hallucinations across SOTA monocular depth models on images. Given an optical-illusion region or a view with restricted context, all tested monocular depth foundation models (DAv2 63 , Depth Pro 6 ), Marigold 28 , DepthFM 20 , ZoeDepth 4 , MiDaS 49 , predict spurious depth variation. 2 Related Works 2.1 2D and 3D Visual Hallucination Visual hallucination, predicting content not present in the input, is a known failure in 2D. This includes classifying nonsense images 44 , detecting objects in empty locations 27 ; 42 , or segmenting non-existent structures 30 ; 10 . Such failures, perilous in safety-critical domains 31 ; 68 , are linked to over-parameterization and models overly relying on context over evidence 29 ; 54 ; 53 ; 10 . In 3D, this problem is less studied but more complex. We define 3D hallucination as predicting depth variations on geometrically flat or smooth surfaces 43 . This is exacerbated by the ill-posed nature of MDE: the 3D-to-2D projection discards depth 51 , forcing networks to use learned priors to resolve ambiguity. This can yield multiple valid reconstructions 9 ; 5 or overfitting to dataset- or camera-specific biases like texture cues 12 . Consequently, most MDE literature has focused on geometric accuracy rather than characterizing these structural failures. 2.2 MDE Models and Benchmarks Monocular Depth Estimation (MDE) seeks to recover 3D structure from a single RGB image. Early methods evolved from supervised 15 to self-supervised using geometric constraints 69 ; 17 ; 18 . The field has recently shifted toward large foundation models like Depth Anything (DAv2) 64 ; 63 , ZoeDepth 4 , MiDaS/DPT 49 , and Marigold 28 , Depth Pro 6 , DepthFM 20 . Trained on broad data, these models achieve remarkable zero-shot generalization but rely heavily on statistical priors. This reliance enables them to fill in