Single-Eye View: Monocular Real-time Perception Package for Autonomous Driving
Haixi Zhang, Aiyinsi Zuo, Zirui Li, Chunshu Wu, Tong Geng, Zhiyao Duan · 2026
Read Single-Eye View: Monocular Real-time Perception Package for Autonomous Driving by Haixi Zhang, Aiyinsi Zuo, Zirui Li, Chunshu Wu, Tong Geng, Zhiyao Duan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-21061.txt.
Introduction
Zhang and co-authors present LRHPerception, a real-time perception package for self-driving that uses one camera. Camera based driving work often favors accuracy over compute cost. Their system turns single view video into a five channel tensor with color, road segmentation, and pixel level depth. It adds object detection and trajectory prediction in the same frame. It runs at twenty nine frames per second on one GPU, a fivefold plus speedup over the fastest mapping based rival.
Catalog record
| CREATOR | Haixi Zhang, Aiyinsi Zuo, Zirui Li, Chunshu Wu, Tong Geng, Zhiyao Duan |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40298 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.21061
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Single-Eye View: Monocular Real-time Perception Package for Autonomous Driving Haixi Zhang, Aiyinsi Zuo, Zirui Li, Chunshu Wu, Tong Geng, Zhiyao Duan Abstract Amidst the ascendency of camera based autonomous driving technology, sometimes effectiveness is overally focused with limited attention to computational demand. To address these issues, this paper introduces LRHPerception, a real-time monocular perception package for autonomous driving that uses single view monocular camera videos to produce interpretation of the surroundings. It innovatively fuses the computational efficiency of end-to-end learning with the comprehensive details intrinsic to local mapping methodologies: With significant enhancements across object tracking and prediction, road segmentation, and depth estimation encapsulated in a cohesive system, LRHPerception proficiently processes monocular image data, yielding a five-channel tensor comprising the original RGB, road segmentation, and pixel-level depth estimation channels, embellished with object detection and trajectory prediction. Empirical evaluations substantiate its superior performance, showcasing a single-GPU real-time processing rate of 29 FPS, a 555% acceleration over the fastest mapping technique, attributed equally to modular enhancements and our unique amalgamation technique. The code is available at LRHPerception . Index Terms— Computer Vision for Transportation, Deep Learning for Visual Perception, Visual Recognition Fig. 1 : Innovation and architecture blueprint a) Paradigm of end-to-end solution b) Paradigm of camera-fusion for local map solution c) Paradigm of our LRHPerception package, extracts essences from monocular camera for cost-info trade-off. I Introduction In recent years, the domain of autonomous driving has witnessed significant progress, particularly in cost-effective camera-based technologies. Such research is also driven by the aspiration to mirror human driving, ultimately leading to safer and more understandable vehicles [ 40 ] . Predominantly, two methodologies have emerged: 1) end-to-end neural networks that interpret raw images to directly produce steering commands [ 4 , 14 ] , and 2) multi-camera fusion techniques that generate a bird’s eye view or 3D occupancy map for path planning [ 12 , 39 , 6 ] . While groundbreaking, they have their respective constraints. End-to-end learning methods transform raw imagery into driving decisions, lauded for their computational prowess [ 14 ] . Yet, they often face scrutiny due to limited interpretability and unpredictability from simplified input processing design [ 29 ] . These systems, despite performing admirably under training-similar conditions, might falter in unfamiliar, dynamic traffic scenarios [ 36 ] , potentially leading to unsafe decisions. Conversely, systems utilizing imagery from multiple cameras provide an encompassing view of the environment, aiding technicians in comprehension and system enhancement. Their computational demands, however, can sometimes impede real-time processing on standard hardware with single GPU, limiting practicality in live scenarios. Our paper presents a novel monocular real-time perception system for autonomous driving, addressing key challenges in this domain. Traditional approaches have extensively researched individual areas like road detection, pixel-depth estimation, object detection, and trajectory predictions. However, these domains have often been studied in isolation. We bridge this gap by not only enhancing each module but also introducing novel integration techniques that seamlessly fuse these components. By processing single-camera video feeds, our system, LRHPerception ( L ow-cost, R eal-time, H igh Information richness), offers a robust blend of resource efficiency with an information-rich perception of the driving scenario. LRHPerception processes RGB images, providing road segmentation, pixel-depth estimation, object detection, and trajectory predictions. Unique modules for each task integrate computationally efficient blocks and structures, ensuring comparable or superior accuracy. Integration involves shared backbones and skip feature map connections, reducing repetitive input data processing. This pioneering effort represents the first instance of amalgamating these modules into a comprehensive package. Our key contributions include: 1) We introduce “LRHPerception”, a unified and pragmatic approach to autonomous perception that efficiently implements object tracking, trajectory prediction, road segmentation, and depth estimation, all derived from a monocular camera’s video input. To our knowledge, this is the first work to integrate these modules into a cohesive package for real-time processing. 2) We implement substantial innovation across each module of monocular image perception, consistently surpassing contemporary state-of-the-art benchmarks. These advancements not only contribute to faster processing speeds but also achieve either comparable or superior perception accuracy. 3) We present an integration technique that consolidates all modules within the package, facilitating information sharing to reduce redundant processing. This integration, combined with module-specific innovations, achieves a 555% acceleration compared to the fastest local-mapping method. 4) The LRHPerception package constitutes the first block in the robotic pipeline of “Perception-Cognition-Action” under our vision of creating a practical and efficient monocular-camera-based autonomous driving system. II Related Works II-A End-to-End Training Contrary to traditional methods, which segregate processes like localization and mapping, planning, and control [ 30 ] , end-to-end algorithms seek to unify these processes into a single learned model, directly translating raw sensory data into output control commands. This idea was first exemplified by the ALVINN system [ 31 ] , which employed a multilayer perceptron to learn the vehicle’s steering direction. With the rise of convolutional neural networks (CNNs), the ability to learn deterministic [ 4 ] or probabilistic [ 2 ] driving commands from raw imagery has significantly evoloved, enabling sustained driving [ 36 ] and intricate lane-change maneuvers [ 16 ] . Existing models often struggle to address the inherent ambiguity, commonly described as the “black-box nature,” [ 29 ] in yielding steering possibilities, which can create obscure limitations and pose a potential safety risk. Moreover, they frequently neglect the importance of considering interactions with other traffic participants [ 31 , 2 ] , which can hinder performance, especially in complex and dynamic traffic scenarios. II-B Local Mapping via Multi-Camera Fusion The fusion of multi-camera data to enable simultaneous localization and mapping (SLAM) and thus construct and update maps of uncharted environments, has attracted substantial research attention [ 12 , 6 , 15 ] . Its capacity to furnish a panoramic view of the surroundings enriches trajectory prediction and distance computation: Numerous methodologies have been proffered to accomplish multi-camera fusion to link with downstream tasks of object tracking and prediction [ 47 , 11 ] . Despite their remarkable results, the inherent complexity of the tasks precludes deployment on a singular, unified system [ 15 ] . Specifically, current state-of-the-art models only reach a processing rate of less than 10 frames per second [ 15 ] . Fig. 2 : Granular Model Structure.1 Design of convolution decoder, object tracking, trajectory prediction, and depth estimation; magnify for details. BTAE mechanism in Algorithm 1. Remaining components are shown in Fig. 3. III Method: LRHPerception We present a state-of-the-art autonomous perception package designed for monocular image inputs, striking an optimal balance among interpretability, information richness, and computational efficiency, ensuring real-time processing capabilities. Our package’s architecture (see Fig. 1 ) encompasses key functionalities of object tracking, trajectory prediction, road segmentation, and depth estimation. Unlike serial connections, we’ve integrated these modules within our model to facilitate information sharing, reducing redundant input processing. For common feature recognition, we have chosen transformer backbones, particularly Swin Transformer [ 24 ] . Acknowledged for its versatility, Swin Transformer outperforms convolutional backbones, notably in depth estimation [ 28 ] . This backbone ingests an RGB image to generate feature-extracted maps \Phi_{2^{k}} with stride size 2^{k} , where k\in\{2,3,4,5\} . The feature maps, namely \{\Phi_{4},\Phi_{8},\Phi_{16},\Phi_{32}\} , are routed to the aforementioned modules for subsequent tasks. Specifically, \{\Phi_{8},\Phi_{16},\Phi_{32}\}