Event Spectroscopy: Event-based Multispectral and Depth Sensing using Structured Light

Christian Geckeler, Niklas Neugebauer, Manasi Muglikar, Davide Scaramuzza, Stefano Mintchev · 2025

Read Event Spectroscopy: Event-based Multispectral and Depth Sensing using Structured Light by Christian Geckeler, Niklas Neugebauer, Manasi Muglikar, Davide Scaramuzza, Stefano Mintchev (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-06741.txt.

Introduction

Drones flying under forest canopies need depth and color vision that works in dim, tangled light. Christian Geckeler and four co-authors present a sensing system for that setting in a 2025 robotics paper. Ordinary cameras suffer delay, weak depth, and dependence on ambient light. Their event-based system uses structured light with a single sensor.

Projecting patterns at controlled wavelengths between red and near-infrared lets the sensor recover depth and spectral information together. Depth accuracy improved by up to sixty percent over commercial depth sensors in tests, and spectral readings matched a reference spectrometer and multispectral cameras. A portable red-green-blue version gathered real data in a rainforest hall, where adding depth to color raised accuracy at telling leaves from branches by more than thirty percent.

Catalog record

CREATORChristian Geckeler, Niklas Neugebauer, Manasi Muglikar, Davide Scaramuzza, Stefano Mintchev
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-06741

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Event Spectroscopy: Event-based Multispectral and Depth Sensing using Structured Light Christian Geckeler, Niklas Neugebauer, Manasi Muglikar, Davide Scaramuzza, Stefano Mintchev Abstract Uncrewed aerial vehicles (UAVs) are increasingly deployed in forest environments for tasks such as environmental monitoring and search and rescue, which require safe navigation through dense foliage and precise data collection. Traditional sensing approaches, including passive multispectral and RGB imaging, suffer from latency, poor depth resolution, and strong dependence on ambient light—especially under forest canopies. In this work, we present a novel event spectroscopy system that simultaneously enables high-resolution, low-latency depth reconstruction and multispectral imaging using a single sensor. Depth is reconstructed using structured light, and by modulating the wavelength of the projected structured light, our system captures spectral information in controlled bands between 650 650 nm and 850 850 nm. We demonstrate up to 60% improvement in RMSE over commercial depth sensors and validate the spectral accuracy against a reference spectrometer and commercial multispectral cameras, demonstrating comparable performance. A portable version limited to RGB (3 wavelengths) is used to collect real-world depth and spectral data from a Masoala Rainforest. We demonstrate the use of this prototype for color image reconstruction and material differentiation between leaves and branches using spectral and depth data. Our results show that adding depth (available at no extra effort with our setup) to material differentiation improves the accuracy by over 30% compared to color-only method. Our system, tested in both lab and real-world rainforest environments, shows strong performance in depth estimation, RGB reconstruction, and material differentiation—paving the way for lightweight, integrated, and robust UAV perception and data collection in complex natural environments. I Introduction Fig. 1: Event Spectroscopy : We propose an all-in-one, ambient-light independent solution for depth sensing, color image reconstruction and multispectral sensing. Usable for instance, for an uncrewed aerial vehicle (UAV) navigating in forest environments (bottom left). A) Our setup consisting of an event camera and a projector as an illumination source B) Generated depth of scene acquired using structured light C) Spectral image reconstructed using events D) Material segmentation of leaves using spectral and depth data. There is an increased demand for uncrewed aerial vehicles (UAVs) flying in forests [ 1 , 2 , 3 ] , both for environmental monitoring [ 4 , 5 , 6 ] , and for search and rescue applications [ 7 ] . Vision is essential for obstacle sensing and data collection to enable these applications. For instance, environmental monitoring tasks such as sensor deployment [ 5 , 8 ] or sample collection [ 4 , 6 , 9 ] require UAVs to fly in close proximity to tree branches and foliage, where fast-moving, thin structures must be detected under dynamic and often challenging lighting conditions. Flying in close proximity to trees makes interaction with foliage probable, and while it has been shown that UAVs can push aside flexible twigs and leaves while avoiding thicker branches [ 10 ] , proper sensing to differentiate between branches and foliage is needed [ 11 ] . This requires not only low latency and high resolution depth sensing of thin and fine structures, but also accurate differentiation between woody branches and foliage. Vision is also used for data collection tasks, including multispectral imagery for assessing tree health, physiological traits, and species identification [ 12 ] . Current methods typically use passive spectral sensors mounted on satellites, aircraft, or UAVs operating far above the forest canopy [ 13 ] . To generate more informative insights, higher spatial resolution data is required, which can be achieved by capturing images closer to or from within the canopy. However, these sensors are highly dependent on ambient illumination, and the pronounced variations in lighting intensity below the canopy make the use of standard multispectral cameras extremely challenging. Even setting aside this limitation, a UAV would currently still need to carry multiple conventional sensors; a high resolution depth camera for obstacle sensing, an RGB camera for scene video and context, and a multispectral camera to capture multispectral data. In this work, we propose a single event-based structured light solution which can simultaneously deliver high-resolution and low latency depth reconstruction with integrated multispectral sensing. The integrated system utilizes an event camera with structured light to reconstruct high-fidelity depth with low latency. By changing the wavelength of the light used for structured light we can not only reconstruct RGB images, but also perform multispectral sensing by projecting the desired wavelength - with lower latency than available multispectral sensors. First, we validate the quality of our depth reconstruction by demonstrating up to 60% average improvement in the RMSE over other commercial off-the-shelf depth sensors. Next, we demonstrate event-based multispectral sensing using wavelengths between 650 nm to 850 nm in a lab setting and compare our results to a commercial multispectral sensor and a ground truth spectrometer. Furthermore, we demonstrate RGB color image reconstruction using a portable version limited to RGB light projection, as well as material differentiation utilizing both the depth and spectral data on real-world data from a Masoala Rainforest. II Related Works Our solution provides event-based multispectral and depth sensing. This requires both spectral imaging as well as event-based luminance recovery and geometry estimation. In the following section, current multispectral sensing methodologies are showcased, including how our system relates to existing systems. To provide depth sensing and to relate event measurements to scene luminance, existing approaches to event-based luminance recovery and geometry estimation are examined, and our method presented. II-A Spectral Imaging Spectral imaging captures images in multiple wavelength bands, with established applications in precision agriculture, astronomy, and beyond. The resulting (X,Y,\lambda) data cube contains both spatial (X,Y) information and spectral information, \lambda , per wavelength. Data capture methods can be divided into two main groups: scanning and snapshot methods. Snapshot methods capture the full data cube during a single integration period of the sensor, whereas scanning methods achieve the same over multiple periods. Among scanning methods, tuned filters represent a popular option. These include a rotating filter wheel, an electro-mechanical Fabry-Ferot filter [ 14 , 15 ] , liquid-crystals [ 16 ] , or acousto-optic tunable filters [ 17 ] . Multiplexing techniques such as Fourier Transform Spectroscopy [ 18 ] and Computed Tomography Multi-Spectral-Imaging [ 19 ] allow for the reconstruction of multiple wavelengths from fewer images than the number of wavelengths at the expense of some artifacts. Classic snapshot methods make use of multiple sensors in the form of Bayer Pattern filter layouts, beam-splitters, or simply by using multiple full camera sensors. Recent developments in compressive sensing like CASSI [ 20 ] use a two-dimensional patterned grating to reconstruct all images of all wavelengths in a single shot. While the computational complexity required to decode the image presents a major bottleneck, improvements have made this method viable for high-resolution imaging in both spatial and spectral dimensions [ 20 , 21 , 22 ] . In this work, we employ an inverse variant of a turned filter. Instead of filtering the light entering the imaging sensor, we illuminate the scene with light of a specific wavelength band, and measure the resulting change in reflected light off the scene with respect to ambient illumination. II-B Event-based Luminance Recovery and Geometry Estimation Event-based luminance

Read the complete text