Target-depth sensing with metasurface-encoder integrated optoelectronic neural network Preprint: APS/123-QED

Shuo Wang, Deyu Zhu, Chenjie Xiong, Bin Hu, Chunqi Jin, Yu Wang, Chengjun Zou · 2026

Read Target-depth sensing with metasurface-encoder integrated optoelectronic neural network Preprint: APS/123-QED by Shuo Wang, Deyu Zhu, Chenjie Xiong, Bin Hu, Chunqi Jin, Yu Wang, Chengjun Zou (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-25160.txt.

Introduction

Wang and six colleagues propose a metasurface-encoder optoelectronic neural network for sensing target depth in three-dimensional scenes. Standard 3D sensing leans on heavy postprocessing of multi-view images or LiDAR point clouds, costing compute, power and time. Their design compresses 3D information into two-dimensional images by encoding depth with a double-helix point spread function from a metasurface. A conventional monocular camera captures the depth-encoded images. A lightweight shadow ResNet network then processes them, enabling real-time tracking with high classification and depth accuracy.

Catalog record

CREATORShuo Wang, Deyu Zhu, Chenjie Xiong, Bin Hu, Chunqi Jin, Yu Wang, Chengjun Zou
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41025
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.25160

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Target-depth sensing with metasurface-encoder integrated optoelectronic neural network Preprint: APS/123-QED Shuo Wang, Deyu Zhu, Chenjie Xiong, Bin Hu, Chunqi Jin, Yu Wang, Chengjun Zou Abstract Accurate and real-time sensing of targets in three-dimensional (3D) environments is essential for modern machine vision, underpinning emerging technologies such as autonomous systems, robotic manipulation, augmented reality, and intelligent surveillance. However, state-of-the-art 3D sensing approaches typically rely on complex postprocessing of multi-view images or LiDAR point clouds, resulting in considerable computational load, power consumption, and latency. To address these challenges, we propose a metasurface-encoder integrated optoelectronic neural network architecture that compresses 3D information into two-dimensional images by encoding depth using double-helix point spread function generated by a metasurface. The depth-encoded images are captured with a conventional monocular camera and subsequently processed by a lightweight shadow ResNet neural network. We experimentally validate the proposed architecture on the MNIST and Vehicle-Image datasets, achieving high accuracy simultaneously in target classification and depth estimation, thereby enabling real-time target tracking. The framework is readily extendable to other depth- or angle-encoding metasurfaces for multidimensional compression and detection. Our results demonstrate the effectiveness of the meta-optic-encoder/electronic-decoder paradigm in significantly reducing network complexity and computational burden while maintaining strong performance for smart vision sensory applications. I Introduction Accurate perception of three-dimensional (3D) environments is fundamental to modern machine vision, enabling the reconstruction of 3D scenes that underpin a broad range of emerging technologies, including robotics yu2023brain , autonomous navigation chen2023self , augmented reality krichenbauer2017augmented ; geroimenko2023augmented , and intelligent surveillance yan2019introduction . Over the past decades, substantial progress has been achieved in 3D sensing for depth recovery. Stereo vision estimates depth passively by triangulating corresponding features observed from multiple viewpoints saxena2007depth , while structured light infers depth by analyzing the deformation of projected spatial patterns on a scene fanello2016hyperdepth . Time-of-flight techniques provide pixel-level depth measurements by detecting the propagation delay or phase shift of reflected light, whereas light detection and ranging (LiDAR) determines distance by measuring the round-trip travel time or frequency shifts of scanned laser points liu2018tof . These techniques varies in terms of their measurement accuracy, application scenarios, and postprocessing complexity. Despite these advances, significant challenges persist as emerging technologies become increasingly interactive, integrated, and intelligent. For example, humanoid robots must perceive dynamic 3D scenes in real time while simultaneously performing dexterous tasks such as precision assembly. Autonomous vehicles require rapid reconstruction of their surrounding environments and corresponding control decisions within millisecond-scale latency. These demanding capabilities must be realized on resource-constrained platforms with continually tightening energy budgets. However, existing 3D sensing approaches typically rely on high-power active illumination and computationally intensive post-processing algorithms wang2021challenges , leading to substantial latency, elevated power consumption, and increased system complexity. These limitations are further exacerbated by the rapid integration of artificial intelligence (AI), which drives explosive growth in sensory data volume and inference complexity wang2021multi , while progress in Moore’s law continues to slow lundstrom2022moore . Consequently, there is an urgent need for 3D sensing technologies that provide not only accurate depth and object recognition, but also fast, low-latency, and energy-efficient perception in dynamic real-world environments. In conventional processing architectures, sensors such as cameras primarily perform data acquisition, while nearly all subsequent information processing is delegated to electronic hardware. By contrast, integrating free-space optical encoding or computation directly during data acquisition can significantly reduce processing complexity, thereby lowering energy consumption and system latency choi2025free ; froch2025computational . This so-called “pre-sensor” processing paradigm huang2024pre has recently attracted considerable attention. At the macroscopic scale, diffractive masks yan2019fourier ; shi2022loen , spatial light modulators (SLMs) and digital micromirror devices (DMDs) miscuglio2020massively ; zhou2021large ; yuan2023training have been widely employed to encode or modulate incident light fields, serving as key components for optical convolutional processing. Diffractive multilayer devices have even been trained to emulate electronic deep neural networks, enabling functionalities such as image classification lin2018all , noise-robust information encryption icsil2024all ; zhang2024memory , communication multiplexing fang2024orbital , and spectrometers wang2024opto ; cheong2024broadband . At the microscopic scale, metasurfaces have been engineered to perform a variety of specialized image-processing operations, including differentiation zhou2020flat ; wang2023metalens ; zhang2025momentum , correlation wang2022single , and phase-contrast imaging ji2022quantitative . Moreover, metasurface-based convolutional kernels zheng2022meta ; zheng2024multichannel can be directly integrated with camera photodetectors to enable fast and energy-efficient convolution operations luo2024meta . Despite this significant progress, most existing efforts have focused on two-dimensional (2D) image processing, and the extension of pre-sensor optical computing concepts to 3D vision sensing shi2021multiple ; yan2024nanowatt remains relatively underexplored. To address this gap, we propose and experimentally demonstrate a metasurface-encoder-integrated optoelectronic neural network ( MONN ) architecture capable of simultaneously performing target classification and depth (distance from the metasurface) estimation with low latency. The metasurface encoder compresses 3D volumetric information into 2D images by encoding depth with a designed double-helix point spread function (DH-PSF) jin2019dielectric ; shen2023monocular . The encoded images are captured by a conventional camera and subsequently processed by a pre-trained shadow ResNet neural network wen2020transfer . We evaluate the proposed system using the MNIST handwritten digit dataset and a Vehicle-Image dataset (see Experimental Section ), achieving ¿97.6% accuracy in target classification and depth estimation errors around 1% within the designed measurement range, comparable to sophisticated AI models. Real-time tracking of the target trajectories is also experimentally demonstrated. The robustness of the approach is further validated by changing image size and partially occluding the targets, while maintaining high detection accuracy. Although the demonstrated metasurface encoder represents a specific encoding strategy, the proposed processing architecture is readily extendable to a broad range of alternative optical encoding designs. Overall, this work establishes a scalable optoelectronic paradigm for low-latency, energy-efficient 3D perception by jointly leveraging metasurface-based optical encoding and alleviating the complexity of deep neural networks, offering a promising pathway toward next-generation intelligent vision systems. II Results and Discussions II.1 Metasurface encoder design and numerical validation of the MONN architecture The MONN processing architecture is depicted in Figure S1 a . The target image and its depth information are optically encoded by the specially designed metasurface before captured by the CMOS camera. A narrow band-pass filter is placed before the camera for selecting the designed working wavelength of 785 nm. After optical to electronic conversion, the encoded image data is fed into the ResNet neural network for further processing, which simultaneously functions as the target classifier and the corresponding depth decoder. Firstly, we present the design of the depth-encoding metasurface and its experimental characterization. Many different encoding principles have been proposed for depth detection. For in-house experimental validation, we design a metalens-integrated DH-PSF phase mask jin2019metasurface as the depth encoder (see Section S1.1 of the Supporting Information (SI) for design details). Under plane-wave illumination, this metasurface generates two focal spots whose relative orientation rotates monotonically along the propagation axis, enabling quasi-linear depth encoding. With the distance between the phase mask and the CMOS camera fixed at 115 mm, the designed depth measurement range spans from 430 mm to 930 mm. The DH-PSF metasurface is realized using silicon nanoposts as the fundamental meta-units to faithfully implement the calculated phase profile. Figures S1 b and S1 c show the corresponding unit-cell parameters and the photo together with a scanning electron micrograph (SEM) of the fabricated metasurface, which has an overall

Read the complete text