ARCANE-PedSynth: Synthetic Multi-Pedestrian Datasets with Behavioural Crossing Annotations Journal: SoftwareX

Muhammad Naveed Riaz, Maciej Wielgosz, Antonio M. López Peña · 2026

Read ARCANE-PedSynth: Synthetic Multi-Pedestrian Datasets with Behavioural Crossing Annotations Journal: SoftwareX by Muhammad Naveed Riaz, Maciej Wielgosz, Antonio M. López Peña (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-24950.txt.

Introduction

Muhammad Naveed Riaz leads ARCANE-PedSynth, an open-source CARLA-based toolkit for building synthetic multi-pedestrian data. It trains crossing prediction for self-driving cars. CARLA on its own yields crossings in only a small share of cases. The toolkit mixes AI and manual pedestrian control to lift the crossing rate high. A twelve-state behavior machine with five character types drives varied crossing styles. Outputs sync color, LiDAR, and event-camera streams with per-frame crossing labels, behavior states, and pose points. Reproducible by command line..

Catalog record

CREATORMuhammad Naveed Riaz, Maciej Wielgosz, Antonio M. López Peña
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37117
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.24950

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ARCANE-PedSynth: Synthetic Multi-Pedestrian Datasets with Behavioural Crossing Annotations Journal: SoftwareX Muhammad Naveed Riaz, Maciej Wielgosz, Antonio M. López Peña Abstract We present ARCANE-PedSynth, an open-source CARLA-based software framework for generating synthetic multi-pedestrian datasets with dense behavioural annotations for pedestrian crossing prediction in autonomous driving. The framework overcomes CARLA’s native 9% crossing rate through a hybrid AI-manual pedestrian control architecture, enabling configurable target rates up to 75%. A 12-state behavioural finite state machine with five character archetypes produces diverse crossing behaviours. The framework generates synchronised RGB, LiDAR, and DVS data with per-frame crossing labels, behavioural states, and estimated 2D pose keypoints. We demonstrate ARCANE-PedSynth through PedSynth++, an example dataset generated with the framework, comprising 533 multi-pedestrian clips across 12 weather conditions with RGB, LiDAR, and DVS streams. ARCANE-PedSynth is fully reproducible via CLI parameterisation and Docker containerisation. Metadata Table 1 summarises the code metadata for the ARCANE-PedSynth software release. Nr. Code metadata description Metadata C1 Current code version v2.0.0 (Git tag; includes submodules pedestrians-scenarios and pedestrians-video-2-carla ; clone with --recurse-submodules ) C2 Permanent link to code/repository used for this code version https://github.com/wielgosz-info/carla-pedestrians ; C3 Permanent link to Reproducible Capsule – C4 Legal Code License MIT Licence (core framework). Third-party components (OpenPose, SMPL/VPoser) have separate licences; see repository README for details. C5 Code versioning system used git C6 Software code languages, tools, and services used Python 3.8, CARLA 0.9.13, PyTorch, Docker, Docker Compose C7 Compilation requirements, operating environments & dependencies Linux (Ubuntu 20.04), NVIDIA GPU with CUDA 11.x, Docker \geq 20.10, CARLA 0.9.13 server C8 If available, link to developer documentation/manual https://github.com/wielgosz-info/carla-pedestrians/blob/main/README.md C9 Support email for questions [email protected] Table 1: Code metadata 1 Motivation and significance Predicting whether a pedestrian will cross in front of an autonomous vehicle (AV) is a safety-critical perception task. The vulnerability is particularly acute in urban environments where complex vehicle-pedestrian interactions occur frequently. Accurate crossing/non-crossing (C/NC) prediction enables proactive braking and comfortable maneuver planning; errors directly translate to collision risk. Unlike static obstacles, pedestrian behaviour exhibits high variability influenced by demographics, environmental conditions, traffic patterns, and individual risk perception [ 15 ] . Real-world pedestrian datasets such as JAAD [ 17 , 16 ] and PIE [ 14 ] have been instrumental in advancing the field, providing behavioural annotations from dashboard cameras [ 9 ] . However, crossing events constitute fewer than 10% of observed frames, and safety-critical behaviours such as jaywalking, sudden dashes, and mid-road pauses are severely underrepresented. Manual annotation is expensive and prone to inconsistencies in annotation criteria, particularly for subjective attributes such as crossing intention and attention state. Moreover, real-world recording cannot ethically stage hazardous edge cases. The nuScenes dataset [ 3 ] provides multi-modal sensor data but lacks C/NC annotations; the TITAN dataset [ 11 ] offers detailed pedestrian actions but is not designed for binary crossing prediction. Several pose-based approaches [ 2 , 10 ] have demonstrated that skeleton keypoints provide valuable cues for crossing intention, motivating the integration of 2D pose estimates into synthetic data pipelines. Synthetic data generation via simulators such as CARLA [ 5 ] offers a principled path forward, producing unlimited data with pixel-perfect ground truth, controllable class balance, and systematic coverage of rare events. Earlier synthetic datasets—Virtual KITTI [ 7 ] , SYNTHIA [ 22 ] , and GTA-V-based collections [ 21 ] —demonstrated the feasibility of training perception models on simulator output. More recently, HABIT introduced a CARLA benchmark that retargets real human motion from motion capture and video into physically consistent pedestrian behaviours, highlighting the importance of realistic interactive pedestrian motion for evaluating autonomous-driving agents [ 13 ] . In the pedestrian domain, Virtual-Pedcross-4667 [ 1 ] introduced CARLA-generated clips with C/NC labels and a 61% crossing ratio, but pedestrians follow scripted paths without naturalistic pre-crossing behaviours (hesitation, traffic checking, retreat) and the pipeline lacks multi-modal sensing or parametric control. DVS-PedX [ 23 ] provides paired synthetic (CARLA) and real (JAAD-to-DVS conversion) event-camera sequences with binary crossing labels, advancing event-based pedestrian perception; however, it focuses on single or limited pedestrian scenarios without behavioural FSM modelling, LiDAR sensing, or controllable crossing-rate generation. Critically, CARLA’s built-in AI pedestrian system routes walkers along sidewalks, yielding a baseline crossing rate of only 9%—insufficient for training balanced C/NC models. We previously developed the ARCANE framework and used it to generate PedSynth [ 18 ] : 947 single-pedestrian, RGB-only clips (approximately 20 s each at 30 FPS, totalling 5.5 hours) with binary C/NC labels. PedSynth demonstrated that combining synthetic and real data improves C/NC prediction, particularly for rare crossing events, and can be used to develop consistent C/NC self-annotation procedures [ 19 ] . However, its single-agent, stationary-vehicle setup does not capture the complexity of real driving, where multiple pedestrians interact with dynamic traffic simultaneously. To our knowledge, no existing open-source tool provides all of the following: (a) controllable crossing rates, (b) multi-pedestrian dynamic traffic, (c) multi-modal sensing, (d) dense behavioural annotations beyond binary labels, and (e) a reproducible open-source generation pipeline. Table 2 compares representative datasets and frameworks across modality, annotation, and scenario dimensions. Table 2: Comparison of pedestrian crossing datasets and frameworks. ✓ = fully supported, – = not supported. “Action labels” indicates action classes are provided but not binary crossing/non-crossing labels. “Partial” behavioural states denote coarse attributes such as crossing intention or attention rather than explicit per-frame finite-state-machine states. “Estimated” pose indicates 2D keypoints obtained from RGB frames using external pose estimators rather than simulator-native or motion-capture skeleton ground truth; ARCANE-PedSynth currently uses this post-processing route for consistency with prior C/NC pipelines. Dataset Synthetic C/NC Labels Multi-Ped. Moving Ego RGB LiDAR DVS Pose Behav. States JAAD [ 17 ] – ✓ ✓ ✓ ✓ – – Estimated Partial PIE [ 14 ] – ✓ ✓ ✓ ✓ – – Estimated Partial nuScenes [ 3 ] – – ✓ ✓ ✓ ✓ – – – TITAN [ 11 ] – Action labels ✓ ✓ ✓ – – – ✓ V-Pedcross-4667 [ 1 ] ✓ ✓ Limited – ✓ – – – – HABIT [ 13 ] ✓ Action labels ✓ ✓ ✓ – – MoCap ✓ DVS-PedX [ 23 ] ✓ ✓ Limited ✓ ✓ – ✓ – – PedSynth [ 18 ] ✓ ✓ – – ✓ – – Estimated – PedSynth++ ✓ ✓ ✓ ✓ ✓ ✓ ✓ Estimated ✓ ARCANE-PedSynth addresses all of these gaps. The software has already been used to generate the PedSynth++ dataset (533 clips across 12 weather conditions with three sensor modalities), on which the PedGT model [ 20 ] achieves F1=89 on the held-out Town05 test split after training only on synthetic PedSynth++ clips from Town01 and Town02. The framework is designed for reuse: researchers can generate arbitrarily many custom dataset variants by changing CLI parameters without modifying the source code. Figure 1: Software architecture of ARCANE-PedSynth. Three Git repositories are orchestrated via Docker Compose ( carla-pedestrians ): the unmodified CARLA server, the pedestrians-scenarios generation container, and the pedestrians-video-2-carla pose extraction container. Pose keypoints are extracted from rendered RGB frames, rather than taken directly from CARLA’s internal skeleton, to keep the annotation pipeline consistent with real-video processing and reduce a potential synthetic-to-real gap; see Sec. 2.2.4 . Containers communicate through Docker networking and share generated data via mounted volumes. 2

Read the complete text