Railway Artificial Intelligence Learning Benchmark (RAIL-BENCH): A Benchmark Suite for Perception in the Railway Domain*

Annika Bätz, Pavel Klasek, Seo-Young Ham, Philipp Neumaier, Martin Köppel, Martin Lauer · 2026

Read Railway Artificial Intelligence Learning Benchmark (RAIL-BENCH): A Benchmark Suite for Perception in the Railway Domain* by Annika Bätz, Pavel Klasek, Seo-Young Ham, Philipp Neumaier, Martin Köppel, Martin Lauer (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-22507.txt.

Introduction

Bätz and five colleagues built RAIL-BENCH, a test suite that grades camera sight for self-driving trains on existing tracks. Railway work had no public yardstick, so the suite sets tasks, data, and scoreboards for comparison. It covers five jobs: spotting rails, finding objects, marking plants, tracking things, and judging motion from one camera. The team adds LineAP, a score for how closely a rail line matches the true shape apart from grouping.

Catalog record

CREATORAnnika Bätz, Pavel Klasek, Seo-Young Ham, Philipp Neumaier, Martin Köppel, Martin Lauer
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37766
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.22507

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Railway Artificial Intelligence Learning Benchmark (RAIL-BENCH): A Benchmark Suite for Perception in the Railway Domain* Annika Bätz, Pavel Klasek, Seo-Young Ham, Philipp Neumaier, Martin Köppel, Martin Lauer Abstract Automated train operation on existing railway infrastructure requires robust camera-based perception, yet the railway domain lacks public benchmark suites with standardized evaluation protocols that would enable reproducible comparison of approaches. We present RAIL-BENCH, the first perception benchmark suite for the railway domain. It comprises five challenges — rail track detection, object detection, vegetation segmentation, multi-object tracking, and monocular visual odometry — each tailored to the specific characteristics of railway environments. RAIL-BENCH provides curated training and test datasets drawn from diverse real-world scenarios, evaluation metrics, and public scoreboards ( https://www.mrt.kit.edu/railbench ). For the rail track detection challenge we introduce LineAP, a novel segment-based average precision metric that evaluates the geometric accuracy of polyline predictions independently of instance-level grouping, addressing key limitations of existing line detection metrics. I Introduction The automation of passenger and freight transportation is gaining increasing relevance in the context of future mobility concepts. In the railway sector, automated and fully driverless operation has been progressively implemented since the 1970s, predominantly on systems that are completely segregated from their surrounding environment, e.g., metro systems and people movers. In these infrastructures, track access is inherently restricted by design and further controlled through platform doors, thereby enabling the system to reliably assume that the tracks are free. Existing railway infrastructure, however, often cannot meet these requirements. Level crossings, open platform edges, or shared-traffic environments of urban railways create substantially higher complexity. Automated operation on such infrastructure demands sensor systems capable of detecting obstacles on or near the tracks, interpreting the behavior of persons on platforms, recognizing vegetation intruding into the clearance gauge, identifying blocked ego tracks, and detecting railway signals and their aspects. Addressing these perception challenges is a fundamental prerequisite for safe, fully automated train operation [ 1 ] . State-of-the-art environmental perception systems tackle these tasks primarily by leveraging camera-based imaging, owing to its capability to acquire rich, high-resolution visual data, in combination with machine learning models to interpret the visual data. In the automotive sector, which faces related perception challenges, model development has been substantially advanced by public benchmark suites such as KITTI [ 2 ] , Waymo [ 3 ] , and nuScenes [ 4 ] . These benchmark suites provide annotated datasets and standardized evaluation protocols that enable the development, assessment, and systematic comparison of perception models, making empirical research reproducible and comparable while lowering the barrier to entry for new contributors. High-quality benchmark suites can thus considerably accelerate research in a given field. Comparable benchmarks are still missing for the railway domain. Publicly available datasets are limited in size and representativeness, and domain-specific evaluation protocols are not rigorously defined, hindering unbiased comparison and reproducible evaluation. To fill this gap, we present RAIL-BENCH, the first environmental perception benchmark suite for the railway domain. It provides datasets and evaluation protocols for five challenges: rail track detection, object detection, vegetation segmentation, object tracking, and monocular visual odometry — all addressing highly relevant perception tasks in automated railway operation. These tasks differ systematically from their automotive counterparts due to the distinct structural layout (tracks and turnouts vs. roads and lanes), object categories (railway vehicles, signals, catenary poles), typical motion patterns, and geometric constraints such as narrow lateral clearances between tracks and infrastructure objects. Our contributions are outlined as follows: • We present the first benchmark suite for environmental perception in the railway domain, comprising five challenges — rail track detection, object detection, vegetation segmentation, multi-object tracking, and ego-motion estimation — each tailored to the specific structural and operational characteristics of railway environments. • We present the novel LineAP metric for evaluating polyline predictions, e.g., for rail tracks. Unlike existing metrics, LineAP handles partial detections, penalizes over-detection consistently, and incorporates joint distance and orientation thresholds, making it applicable beyond the railway domain. TABLE I: Public railway object and rail detection datasets with real-world RGB sensor data recorded from the driver’s point of view. Annotations Scenarios Data Classes with Instance-Based Dense Annotations Train [-0.3ex]Stations Urban Suburban/ [-0.3ex]High-speed Rural Staged Image [-0.3ex]Resolution [-0.3ex]in MPx [-0.3ex](min - max) # Images Sparse [-0.3ex]Image [-0.3ex]Sampling Railway Infrastructure Potential Obstacles Rail Signal Signal [-0.3ex]Pole Catenary [-0.3ex]Pole Train Person Road [-0.3ex]Vehicle Bicycle/ [-0.3ex]Cyclist RailSem19 [ 5 ] \checkmark \times \times \times \times \times \times \times \checkmark \checkmark \checkmark \checkmark \times 2.07 8{,}500 \checkmark RailSet ∗1 [ 6 ] \checkmark \times \times \times \times \times \times \times \checkmark \checkmark \checkmark \checkmark \times 0.41-3.69 729 \checkmark Rail-DB [ 7 ] \checkmark \times \times \times \times \times \times \times \times \times \checkmark \times \checkmark 0.92-8.29 7{,}432 \times L4R_NLB [ 8 ] \checkmark \times \times \times \times \times \times \times \checkmark \times \checkmark \checkmark \times 2.07 10{,}253 \checkmark FRSign [ 9 ] \times \checkmark \times \times \times \times \times \times \checkmark \checkmark \checkmark \checkmark \times 2.3-3.15 105{,}352 \times GERALD [ 10 ] \times \checkmark \times \times \times \times \times \times \checkmark \checkmark \checkmark \checkmark \times 0.92-2.07 5{,}000 \checkmark RailGoerl24 [ 11 ] \times \times \times \times \times \checkmark \times \checkmark \times \times \times \times \checkmark 2.07 12{,}205 \times ESRORAD ∗2 [ 12 ] \times \times \times \times \times \checkmark \checkmark \checkmark \times \checkmark \times \times \times 2.07 \sim 100{,}000 \times MRSI ∗3 [ 13 ] \times \times \times \times \times \checkmark \checkmark \checkmark \times \checkmark \times \times \checkmark 0.52 2{,}983 \times OSDaR23 ∗4 [ 14 ] \checkmark \checkmark \checkmark \checkmark \checkmark \checkmark \checkmark \checkmark \checkmark \checkmark \checkmark \times \checkmark 3.94-10.3 1{,}534 m-frames ∗5 \times OSDaR26 ∗4 [ 15 ] \checkmark \checkmark \checkmark \checkmark \checkmark \checkmark \checkmark \checkmark \checkmark \checkmark \checkmark \checkmark \checkmark 3.94-19.96 15{,}646 m-frames ∗5 \times RAIL-BENCH Object+Rail \checkmark \checkmark \checkmark \checkmark \checkmark \checkmark \checkmark \checkmark \checkmark \checkmark \checkmark \checkmark \checkmark 2.07-19.96 2{,}500 \checkmark ∗1 Published real-world subset of RailSet dataset. ∗2 Only a small subset of these data shows tram scenes. ∗3 Subset of the MRSI dataset with instance-based annotations. ∗4 Our RAIL-BENCH Object+Rail dataset includes specifically sampled images from OSDaR23 and OSDaR26. ∗5 M-frame stands for multi-sensor frame, which is the collection of several sensor frames. II Related work In recent years a couple of datasets have been published which contain camera-based perception data from the railway domain. In this survey, we focus on data sets that are publicly available without restrictions and that come with annotations, e.g. for tracks or objects. Table I provides a synopsis. RailSem19 [ 5 ] provides images extracted from 530 video sequences from the train driver’s perspective. The dataset exhibits substantial inter-image variability and encompasses a wide spectrum of operational and environmental scenarios. Annotations are available as geometric annotations for 21 object classes with bounding boxes, polylines and polygons and dense semantic annotations for 20 classes. Except for the rail class, geometric annotations are highly sparse, with only a few objects labeled in each image. The datasets RailSet [ 6 ] , Rail-DB [ 7 ] , and L4R_NLB [ 8 ] provide annotations for railway tracks. L4R_NLB constitutes the largest and most diverse collection in terms of scope and variability, but it does not address complex urban scenarios. FRSign [ 9 ] and GERALD [ 10 ] , contain annotations for railway signals

Read the complete text