RefRef: A Dataset and Benchmark for Reconstructing Refractive and Reflective Objects
Yue Yin, Enze Tao, Weijian Deng, Dylan Campbell · 2025
Read RefRef: A Dataset and Benchmark for Reconstructing Refractive and Reflective Objects by Yue Yin, Enze Tao, Weijian Deng, Dylan Campbell (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-05848.txt.
Introduction
Yue Yin and co-authors built RefRef in 2025, a data set for rebuilding clear and shiny objects in 3D. Standard rebuild tools presume light travels straight, so glass and mirrors break their math. Past data sets for such objects were narrow: thin glass, plain tint-free items, or backdrops set far away.
RefRef stages objects that bend and bounce light in rich ways, with ground truth to grade against. That lets each tool's failure modes show plainly rather than hide in a mixed score. The team also set a shared benchmark on the set. Better grades should push rebuild tools toward the glassy real world.
Catalog record
| CREATOR | Yue Yin, Enze Tao, Weijian Deng, Dylan Campbell |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 67 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-05848
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
RefRef: A Dataset and Benchmark for Reconstructing Refractive and Reflective Objects Yue Yin, Enze Tao, Weijian Deng, Dylan Campbell Abstract Modern 3D reconstruction and novel view synthesis approaches have demonstrated strong performance on scenes with opaque, non-refractive objects. However, most assume straight light paths and therefore cannot properly handle refractive and reflective materials. The lack of datasets specialized for these effects has impeded efforts to fairly and thoroughly evaluate performance and thereby make progress in this domain. Most existing datasets focus on opaque scenes, while those targeting refractive objects are often limited to thin glass with negligible ray bending, untinted single-material objects, or backgrounds treated as infinitely distant, leaving key failure modes under strong refraction untested. To address this gap, we introduce the RefRef dataset. It contains 150 synthetic and 60 real scenes spanning varying levels of geometric complexity and diverse background types that expose the limitations of existing methods. For the purpose of benchmarking, we also provide an oracle method that, given the object geometry and refractive indices, calculates accurate light paths for neural rendering, and a simple two-stage baseline that relaxes these assumptions. We evaluate these against state-of-the-art methods and show that the task is far from solved. (a) Opaque and Lambertian (b) Refractive and Reflective Bounded Unbounded (c) Synthetic Scenes Indoor Outdoor (d) Real-world Scenes GT Splatfacto TransparentGS NeuS Zip-NeRF MS-NeRF RayDef NU-NeRF R3F (e) An Unsolved Open Problem Figure 1 : Modeling refractive light transport for reconstruction and novel view synthesis is challenging, because while (a) opaque Lambertian objects allow modeling of light paths as linear, (b) refractive and reflective objects bend and branch light paths. While many methods address this issue in simplified settings, they are not evaluated on a comprehensive dataset that considers all of the factors that make this problem challenging. We present the RefRef dataset that contains (c) 150 synthetic scenes with diverse bounded and unbounded backgrounds, and (d) 60 real scenes from varied indoor and outdoor environments. Different levels of object complexity are considered, including convex, nonconvex, and multi-material objects, as well as tinted materials and multi-object configurations. Our benchmark reveals that the task is far from solved for refractive objects: not a single existing method can handle real scenes with multiple objects (e). Abstract 1 Introduction Refractive and reflective objects, such as glass and water, pose significant challenges for 3D reconstruction and novel view synthesis due to the complex behavior of light as it passes through or reflects off these materials. Accurate modeling of these optical phenomena is essential for precise 3D reconstruction and photorealistic rendering. However, most existing neural radiance approaches [ 37 , 3 , 12 , 39 ] and Gaussian splatting methods [ 28 , 23 , 65 ] struggle with refractive or reflective objects [ 17 ] . These methods typically assume that light travels in a straight path, which is valid for Lambertian objects, as illustrated in Fig. 1 , but fail in scenarios with complex optical behavior. A key obstacle hindering the development and evaluation of suitable methods is the lack of datasets that expose the full difficulty of reconstructing scenes with refractive and reflective objects. Standard 3D reconstruction datasets contain mostly opaque scenes, whereas datasets for refractive objects typically simplify many of the challenges: thin glass produces negligible ray bending; single-material objects omit interfaces between different refractive media; a limited range of object geometries prevents evaluation across diverse shapes; untinted materials omit wavelength-dependent attenuation; and infinitely distant backgrounds mean that the observed color only depends on the direction of the outgoing refracted ray, not its origin. These simplifications make the problem considerably easier and meet the baked-in assumptions of specialized methods. Even general-purpose methods perform well since limited view-dependent modeling is sufficient for such scenes. As a result, these datasets do not reveal how the methods perform under strong refraction, complex geometry, multiple materials, tinting, or diverse backgrounds. To address this dataset limitation, we introduce RefRef, a dataset and benchmark designed for the task of reconstructing scenes with complex refractive and reflective objects, as shown in Fig. 1 . RefRef consists of 150 synthetic scenes with 50 objects and 60 real scenes with 15 objects. Objects span three categories: convex single-material, nonconvex single-material, and nonconvex multi-material, with varied colors, opacities, and refractive indices. These categories allow us to evaluate methods across diverse geometric and material configurations with different levels of complexity and varied combinations of refraction, reflection, total internal reflection, and absorption. Each synthetic object is rendered against two bounded backgrounds and one unbounded background, while each real-world configuration is captured in two indoor and two outdoor environments with varied lighting and capture conditions. Bounded backgrounds contain finite-depth geometry near the refractive object, so the observed color depends on both the origin and direction of the outgoing refracted ray. Unbounded backgrounds use infinitely distant environment maps, so the color depends only on the ray direction, which simplifies the problem considerably. Together, these scenes provide a diverse set of challenges for evaluating 3D reconstruction methods under complex optical effects. For benchmarking, we also provide an oracle method that traces physically accurate light paths given the ground-truth object geometry and refractive indices, together with a simple two-stage baseline, R3F, that relaxes these requirements by estimating object geometry and properties. We evaluate existing state-of-the-art methods against these references, and show that existing approaches are largely unable to handle scenes with strongly refractive objects. Our contributions are: 1. a dataset of 150 synthetic and 60 real scenes with strongly refractive and reflective objects; 2. an oracle and a baseline that establish reference performance with and without ground-truth object geometry and refractive indices; and 3. a benchmark evaluating state-of-the-art methods on this dataset, revealing the limits of existing approaches at handling complex optical phenomena. 2 Related Work Table 1 : Comparison of datasets by counts of strongly refractive scenes |{\mathcal{S}}_{\mathrm{refr}}| and images |{\mathcal{I}}_{\mathrm{refr}}| , and the presence of refractive scenes with real data (Real), multi-material refractive objects (Multi), bounded backgrounds (Bounded), and tinted refractive materials (Tinted). ∗ Our estimates: the paper does not specify and the publicly released subset contains no strongly refractive scenes. Datasets |{\mathcal{S}}_{\mathrm{refr}}| |{\mathcal{I}}_{\mathrm{refr}}| Real Multi Bounded Tinted DTU [ 25 ] 0 0 ✗ ✗ ✗ ✗ T & T [ 29 ] 0 0 ✗ ✗ ✗ ✗ ShapeNet [ 11 ] 0 0 ✗ ✗ ✗ ✗ OmniObject3D [ 56 ] 0 0 ✗ ✗ ✗ ✗ Objaverse-XL [ 15 ] 0 0 ✗ ✗ ✗ ✗ Shiny [ 53 ] 2 348 ✓ ✓ ✓ ✓ OpenMaterial [ 14 ] 143 12.9k ✗ ✗ ✗ ✗ 3DReflecNet [ 33 ] 11k ∗ 655k ∗ ? ✓ ✗ ✗ RefRef (ours) 210 57k ✓ ✓ ✓ ✓ General 3D reconstruction datasets. As shown in Tab. 1 , several datasets have been widely adopted for 3D reconstruction tasks, each offering unique strengths for general scene understanding. The DTU dataset [ 25 ] provides multiview images and ground-truth geometry for object reconstruction. Tanks and Temples [ 29 ] provides a benchmark for 3D reconstruction with complex scenes captured using high-precision scanners, suitable for evaluating methods in real-world settings. Large-scale datasets such as Objaverse-XL [ 15 ] , ShapeNet [ 11 ] , and OmniObject3D