Real-time 3D Light-field Viewing with Eye-tracking on Conventional Displays

Trung Hieu PHAM, Chanh Minh TRAN, Eiji KAMIOKA, PHAN Xuan Tan · 2025

Read Real-time 3D Light-field Viewing with Eye-tracking on Conventional Displays by Trung Hieu PHAM, Chanh Minh TRAN, Eiji KAMIOKA, PHAN Xuan Tan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-16535.txt.

Introduction

Trung Hieu Pham and co-authors (2025) make a plain computer monitor show moving 3D, using a webcam and cheap red-cyan glasses. True 3D usually demands headsets or special screens, which shut out low-resource settings. Their system tracks the viewer's eyes with an ordinary camera. As the head moves, the software picks views from pre-captured light-field data. That data is a set of images of a scene from many angles. It blends the right pair into an anaglyph image, tinted for the two eyes.

The update happens in real time, so the scene shifts with the viewer like a window into depth. Everything runs on a standard CPU, no graphics card, and holds a steady thirty frames per second. That frugality is the point: classrooms, clinics, and small labs could add depth viewing with hardware they already own. The paper reports the pipeline and its measured performance.

Catalog record

CREATORTrung Hieu PHAM, Chanh Minh TRAN, Eiji KAMIOKA, PHAN Xuan Tan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE29 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-16535

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Real-time 3D Light-field Viewing with Eye-tracking on Conventional Displays Thanks: This work was supported by JSPS KAKENHI under Grant 24K20797. Trung Hieu PHAM, Chanh Minh TRAN, Eiji KAMIOKA, PHAN Xuan Tan Abstract Creating immersive 3D visual experiences typically requires expensive and specialized hardware such as VR headsets, autostereoscopic displays, or active shutter glasses. These constraints limit the accessibility and everyday use of 3D visualization technologies in resource-constrained settings. To address this, we propose a low-cost system that enables real-time 3D light-field viewing using only a standard 2D monitor, a conventional RGB webcam, and red-cyan anaglyph glasses. The system integrates real-time eye-tracking to dynamically adapt the displayed light-field image to the user’s head position with a lightweight rendering pipeline that selects and composites stereoscopic views from pre-captured light-field data. The resulting anaglyph image is updated in real-time, creating a more immersive and responsive 3D experience. The system operates entirely on CPU and maintains a stable frame rate of 30 FPS, confirming its feasibility on typical consumer-grade hardware. All of these highlight the potential of our approach as an accessible platform for interactive 3D applications in education, digital media, and beyond. 1 Introduction The ability to perceive and interact with three-dimensional (3D) imagery has significantly impacted a wide range of domains, from entertainment [ 13 , 29 ] to education [ 38 ] , and medical imaging [ 34 , 17 ] . Over the past several decades, advancements in display [ 41 ] and interaction technologies have brought immersive 3D experiences closer to mainstream use. These experiences typically rely on specialized hardware, such as head-mounted displays (HMDs) for virtual reality (VR), augmented reality (AR) headsets, or stereoscopic monitors using polarized or active-shutter glasses, to deliver distinct images to each eye and create a convincing illusion of depth. While such systems provide rich spatial cues and user immersion, they often entail high costs, intricate setup procedures, and ergonomic constraints, which limit their viability for everyday or large-scale deployment in informal environments such as homes or classrooms. To bridge the gap between high-quality immersive visualization and practical accessibility, simpler stereoscopic approaches [ 5 , 30 ] have emerged. Among these, anaglyph glasses, which use colored filters to deliver different images to each eye, stand out due to their extremely low cost and compatibility with standard display technology [ 7 ] . By requiring only an ordinary monitor and inexpensive paper or plastic glasses, anaglyph techniques allow basic 3D effects to be experienced without the need for specialized or proprietary hardware. However, this simplicity comes with significant trade-offs. Anaglyph rendering has long been associated with limited color fidelity, ghosting [ 40 ] , and a lack of dynamic interaction capabilities, such as head-coupled parallax, which impairs the realism of the 3D experience and reduces viewer comfort during extended use. Recent advances in real-time computer vision [ 35 , 1 , 8 ] , particularly in the domains of facial and eye-tracking using ordinary RGB cameras, open new possibilities for enhancing anaglyph-based systems. By leveraging these technologies, it becomes feasible to introduce dynamic perspective adjustment based on user position, restoring an essential depth cue known as motion parallax. When combined with pre-rendered or captured light-field imagery, this approach enables natural and immersive 3D experiences, even on basic display hardware. Light-field imaging [ 20 , 11 ] plays a central role in this work as a powerful method for capturing and rendering 3D content. Light-field imaging is an innovative technology that revolutionizes the process of capturing and perceiving images. It captures not only the intensity and color of light rays at different points in space but also their directional information. This enables the generation of multiple views from different angles, which is crucial for achieving viewpoint-dependent rendering and dynamic parallax. Furthermore, light-field content can be acquired easily by using many types of light-field cameras on the market. It alleviates the burden on creating a 3D content, making it easier for normal end-user. In this paper, we introduce a low-cost, eye-tracked light-field viewing system that provides interactive 3D visualization capabilities using standard 2D monitors and inexpensive anaglyph glasses. Our system relies on a single ordinary RGB camera to track the viewer’s eye position in real-time and adjust the displayed content to match the user’s viewpoint. Light-field image data is reprojected accordingly to provide coherent parallax and stereo cues that respond to head motion, resulting in a more immersive and convincing sense of depth. Notably, this system requires no special display panels, headsets, or high-end sensors, making it broadly accessible and easy to deploy. The proposed system integrates two key components: (1) robust, low-latency eye-tracking and real-time head-coupled perspective adjustment using computer vision techniques; and (2) a rendering pipeline that selects appropriate views from a light-field scene, optimized for anaglyph stereo display. Together, these elements form a practical solution for light-field content exploration, with potential applications in education, 3D data analysis, digital art, and beyond. In this paper, we have several contributions. First, we present the design and implementation of a functional prototype that achieves real-time performance on consumer-grade hardware using easily accessible components. Next, we demonstrate that our system provides natural perceived depth and parallax that reflect real-world perceptual effects. Finally, our system does not require specialized hardware, making it suitable for a broad deployment in settings such as classrooms, laboratories, and home environments. The rest of this paper is organized as follows. Section 2 reviews related work in 3D visualization technologies, light-field imaging and eye-tracking with Mediapipe [ 22 ] . Section 3 describes our system architecture, including hardware components, tracking algorithms, and the rendering pipeline. Section 4 presents implementation details, and experimental results. Section 5 discusses system limitations, potential applications, and directions for future work. Section 6 concludes with a summary of our contributions and their implications for accessible 3D content interaction. 2 Related Work 2.1 3D Visualization Technologies The development of 3D visualization technologies has produced a rich spectrum of systems that differ in terms of immersion, accessibility, and visual fidelity. These systems can generally be categorized into immersive head-mounted, glasses-free (autostereoscopic), and glasses-based stereoscopic solutions [ 9 , 10 ] . HMDs used in VR and AR provide fully immersive 3D experiences by enclosing the user’s field of view and tracking head movement. Devices like the Meta Quest 3 [ 25 ] , HTC Vive [ 16 ] , and Microsoft HoloLens [ 26 ] enable rich interactivity and spatial immersion by combining stereoscopic rendering with six degrees of freedom (6DoF) motion tracking. These systems provide excellent depth cues including motion parallax, and stereo disparity, but are often constrained by high cost, user discomfort, and content development overhead [ 24 , 19 ] . Autostereoscopic displays or glasses-free displays offer 3D visualization without requiring users to wear glasses. These systems include parallax barrier displays, lenticular lens screens, and more advanced volumetric and light-field displays. Notable examples include the Nintendo 3DS [ 27 ] , which uses a parallax barrier to present distinct images to each eye, and the Looking Glass series [ 21 ] of volumetric displays, which presents multiple views of a 3D scene simultaneously to support motion parallax and accommodate multiple viewers. However, Looking Glass displays are limited in both image

Read the complete text