LiDAR-Guided Monocular 3D Object Detection for Long-Range Railway Monitoring
Raul David Dominguez Sanchez, Xavier Diaz Ortiz, Xingcheng Zhou, Max Peter Ronecker, Michael Karner, Daniel Watzenig, Alois Knoll · 2025
Read LiDAR-Guided Monocular 3D Object Detection for Long-Range Railway Monitoring by Raul David Dominguez Sanchez, Xavier Diaz Ortiz, Xingcheng Zhou, Max Peter Ronecker, Michael Karner, Daniel Watzenig, Alois Knoll (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-18203.txt.
Introduction
Raul Dominguez Sanchez and colleagues built a vision system in 2025 to help driverless trains spot obstacles far ahead. Trains need perception at extreme range, beyond a kilometer, because their braking distances dwarf a car's. Their approach detects objects in three dimensions from a single camera, a monocular method that borrows depth knowledge from LiDAR only during training, so the deployed system needs no LiDAR at all.
The pipeline chains four modules: a modified detector that finds objects with partial depth information, a depth estimation network, and separate detection heads tuned for short and long ranges. Evaluated on a public railway dataset, the system detected objects out to a few hundred meters. The authors present it as a step toward the perception autonomy German railways need, while noting the remaining gap to true kilometer-range performance.
Catalog record
| CREATOR | Raul David Dominguez Sanchez, Xavier Diaz Ortiz, Xingcheng Zhou, Max Peter Ronecker, Michael Karner, Daniel Watzenig, Alois Knoll |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 27 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-18203
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LiDAR-Guided Monocular 3D Object Detection for Long-Range Railway Monitoring Raul David Dominguez Sanchez, Xavier Diaz Ortiz, Xingcheng Zhou, Max Peter Ronecker, Michael Karner, Daniel Watzenig, Alois Knoll Abstract Railway systems, particularly in Germany, require high levels of automation to address legacy infrastructure challenges and increase train traffic safely. A key component of automation is robust long-range perception, essential for early hazard detection, such as obstacles at level crossings or pedestrians on tracks. Unlike automotive systems with braking distances of 70 meters, trains require perception ranges exceeding 1 km. This paper presents an deep-learning-based approach for long-range 3D object detection tailored for autonomous trains. The method relies solely on monocular images, inspired by the Faraway-Frustum approach, and incorporates LiDAR data during training to improve depth estimation. The proposed pipeline consists of four key modules: (1) a modified YOLOv9 for 2.5D object detection, (2) a depth estimation network, and (3-4) dedicated short- and long-range 3D detection heads. Evaluations on the OSDaR23 dataset demonstrate the effectiveness of the approach in detecting objects up to 250 meters. Results highlight its potential for railway automation and outline areas for future improvement. I INTRODUCTION Railways are a key mode of transport in Europe, facilitating the movement of goods and passengers efficiently. However, the system faces capacity limits and legacy issues [ 1 ] . Initiatives such as Digital Schiene Deutschland [ 2 ] , Shift2Rail [ 3 ] , and safe.trAIn [ 4 ] aim to modernize railway operations and increase automation. A crucial requirement for autonomous rail systems is a robust perception module capable of detecting objects and locating the train with high precision. This is particularly important for improving safety, as most railway accidents involve track intrusions (61%) or level crossings (24%) [ 5 ] . Furthermore, the long braking distances of trains, reaching lengths as high as 1000 meters [ 6 , 7 ] , necessitate early detection of obstacles, significantly beyond the typical 70-meter braking distance for cars [ 8 ] . LiDAR provides accurate depth information but is limited in range (250-500 meters), while monocular cameras capture high-resolution images but lack direct depth measurement [ 9 ] . The fusion of both sensors has shown significant improvements in 3D object detection [ 10 , 11 , 12 , 13 ] . This work proposes a monocular 3D object detection method tailored for railway environments with the following contributions: Contributions 1. An AI-based 3D object detection approach for autonomous railway vehicles using only monocular images, addressing long-range perception challenges. 2. A novel pipeline integrating a modified YOLOv9 for 2.5D detection with depth estimation module for accurate frustum-based 3D localization. 3. A training strategy leveraging LiDAR point clouds and 3D labels to enhance depth estimation and improve monocular-based 3D detection, reducing the gap with LiDAR-based methods. II RELATED WORK The following provides a brief introduction to 3D object detection, including relevant detector families such as frustum-based methods and the YOLO family, as well as an overview of depth estimation. II-A 3D Object Detection 3D object detection involves identifying objects in a scene and estimating their enclosing 3D bounding boxes. Typically, 8 to 11 parameters are predicted, including the object class, center coordinates, dimensions, and orientation. In autonomous driving, orientation prediction is often simplified to yaw rotation, as objects are assumed to rotate only about the axis perpendicular to the road [ 14 ] . Methods for 3D object detection are categorized by: • Input data type : Input data typically includes images or point clouds, which may be further processed into formats like depth maps, voxel grids, or BEV maps. Camera-based methods are divided into monocular [ 8 , 15 ] , stereo [ 16 ] , and multi-camera [ 17 ] approaches. Multimodal methods fuse data from multiple sensors, such as combining LiDAR and monocular images, to improve performance [ 10 , 13 ] . • Detection environment : Detection methods are tailored to either indoor or outdoor scenes. Indoor point clouds are dense and provide rich context, while outdoor point clouds are sparse and influenced by external factors. Recent work explores environment-agnostic methods [ 18 ] . • Detection approach : Deep learning dominates the field, though classical methods remain relevant for efficiency and simpler parameterization. Hybrid methods combine classical techniques (e.g., preprocessing) with deep learning for improved performance [ 19 ] . This classification is not standardized, but it highlights the factors most relevant to this work. II-B Frustum-Based Methods for 3D-Detection In the context of this paper, a frustum refers to a portion of a 3D point cloud that is pyramid-shaped and is used to narrow down a region of interest in a point cloud. Frustum-based methods leverage 2D object detections to narrow the 3D search space by projecting 2D bounding boxes into 3D frustums [ 20 ] . Faraway-Frustum [ 21 ] combines monocular images and LiDAR point clouds to address the challenges of long-range 3D detection. It uses a dual-head architecture: one head employs standard methods like PV-RCNN [ 22 ] or F-PointNet [ 20 ] for short-range objects, while the second head (FF-Net) processes long-range frustums in BEV format using MobileNet [ 23 ] . FF-Net handles sparsity in long-range LiDAR data and predicts 3D bounding boxes, including yaw orientation, object center, and dimensions. Frustums are classified as short- or long-range based on their centroid’s distance, with thresholds (e.g., 60 m for pedestrians, 75 m for cars) derived from the KITTI dataset [ 24 ] . By reducing search space and leveraging multimodal data, frustum-based methods improve detection accuracy and efficiency, making them suitable for autonomous driving applications. II-C The YOLO Object Detector Family and Extensions The You Only Look Once (YOLO) family of object detectors simplifies 2D detection by combining classification and localization into a single network [ 25 ] . YOLOv1 predicts bounding boxes and class probabilities on a 7 \times 7 grid, with non-maximal suppression filtering low-confidence predictions. Subsequent versions introduced significant improvements in accuracy, efficiency, and feature extraction. YOLOv9 [ 26 ] replaced fixed bounding box dimensions with probabilistic distributions, eliminating non-maximum suppression through one-to-one label assignment. Architectural upgrades include Spatial Pyramid Pooling (SPP), Path Aggregation Networks (PANet), Generalized Efficient Layer Aggregation Networks (GELANs), and Programmable Gradient Information (PGI), enhancing multi-scale feature extraction and gradient flow while maintaining real-time performance [ 27 ] . YOLO has also been adapted for 2.5D tasks, integrating depth estimation. YOLO-D and Dist-YOLO extend YOLOv3 to directly predict depth in the detection head [ 28 , 29 ] . Another approach overlays YOLOv5 bounding boxes on depth maps to estimate distances via polynomial regression [ 30 ] . While effective for short ranges, these methods require recalibration for new domains. II-D Depth Estimation Monocular depth estimation converts 2D images into depth maps, enabling point cloud generation. Classical methods relied on geometric cues such as structure-from-motion and multi-view stereo, but deep learning now dominates due to its ability to infer depth from single images. Supervised approaches use ground truth depth, while self-supervised methods exploit geometric consistency. The following methods are relevant to this work. MiDAS [ 31 ] trains on multiple datasets for cross-domain generalization using zero-shot transfer. Fine-tuning is required for absolute depth estimation. DenseDepth [ 32