Li-ViP3D++: Query-Gated Deformable Camera–LiDAR Fusion for End-to-End Perception and Trajectory Prediction
Matej Halinkovic 1 , Nina Masarykova 1 , Alexey Vinel 2 3 , Marek Galinski 1 · 2026
Read Li-ViP3D++: Query-Gated Deformable Camera–LiDAR Fusion for End-to-End Perception and Trajectory Prediction by Matej Halinkovic 1 , Nina Masarykova 1 , Alexey Vinel 2 3 , Marek Galinski 1 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-20720.txt.
Introduction
Matej Halinkovic and three co-authors work on how self-driving cars see the road and predict what comes next. Their 2026 paper joins camera images with LiDAR scans. LiDAR measures distance with laser light. The system must do perception and trajectory prediction in one pass from raw sensor data. That end-to-end design is the paper's core aim.
Older designs split the job into modules that pass results along. Errors then grow as they move down the chain. Newer query-based models fix some of this by letting parts share information. Yet how cameras and LiDAR best complement each other in that shared space is still open. The team's method uses gated, deformable fusion to join the two. It speaks to the push for cars that read a scene whole, not in pieces.
Catalog record
| CREATOR | Matej Halinkovic 1 , Nina Masarykova 1 , Alexey Vinel 2 3 , Marek Galinski 1 |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-20720
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Li-ViP3D++: Query-Gated Deformable Camera–LiDAR Fusion for End-to-End Perception and Trajectory Prediction Thanks: This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible. This work was funded by the Excellence Strategy of the German Federal and State Governments and the Helmholtz Program “Engineering Digital Futures”. Additionally, it was funded the EU NextGenerationEU through the Recovery and Resilience Plan for Slovakia under the project No. 09I05-03-V02-00014 and by APVV-23-0519 project “Legal and Technical Challenges of Smart Mobility to Increase Road Traffic Safety”. Finally, the support of the ELLIIT strategic research network and the Helmholtz Association Initiative and Networking Fund on the HAICORE@KIT partition is gratefully acknowledged. Matej Halinkovic 1 , Nina Masarykova 1 , Alexey Vinel 2 3 , Marek Galinski 1 Abstract End-to-end perception and trajectory prediction from raw sensor data is one of the key capabilities for autonomous driving. Modular pipelines restrict information flow and can amplify upstream errors. Recent query-based, fully differentiable perception-and-prediction (PnP) models mitigate these issues, yet the complementarity of cameras and LiDAR in the query-space has not been sufficiently explored. Models often rely on fusion schemes that introduce heuristic alignment and discrete selection steps which prevent full utilization of available information and can introduce unwanted bias. We propose Li-ViP3D++, a query-based multimodal PnP framework that introduces Query-Gated Deformable Fusion (QGDF) to integrate multi-view RGB and LiDAR in query space. QGDF (i) aggregates image evidence via masked attention across cameras and feature levels, (ii) extracts LiDAR context through fully differentiable BEV sampling with learned per-query offsets, and (iii) applies query-conditioned gating to adaptively weight visual and geometric cues per agent. The resulting architecture jointly optimizes detection, tracking, and multi-hypothesis trajectory forecasting in a single end-to-end model. On nuScenes, Li-ViP3D++ improves end-to-end behavior and detection quality, achieving higher EPA (0.335) and mAP (0.502) while substantially reducing false positives (FP ratio 0.147), and it is faster than the prior Li-ViP3D variant (139.82 ms vs. 145.91 ms). These results indicate that query-space, fully differentiable camera–LiDAR fusion can increase robustness of end-to-end PnP without sacrificing deployability. Email: {matej.halinkovic, nina.masarykova, marek.galinski}@stuba.sk, [email protected] I Introduction Forecasting the positions and future behavior of agents in a traffic scene is one of the key capabilities needed for autonomous driving, for road-infrastructure management, and ensuring the safety of vulnerable road users. For downstream planning and control, anticipating how surrounding participants will evolve is as important as estimating their current state. As trajectory prediction methods improve in autonomous driving environments, they contribute directly to safer interactions among road users and to smoother traffic flow. Among Perception and Prediction (PnP) approaches in autonomous driving, the most advanced direction is to predict trajectories directly from raw sensor inputs. Accordingly, end-to-end models that unify object detection, tracking, and prediction have become an important component of embodied perception [ 23 ] . However, leading solutions still omit several practical considerations. In particular, there remains a lack of methods that can fully exploit both RGB and LiDAR streams in a single, fully differentiable framework. Although unimodal methods have historically dominated computer vision for dynamic scenes, they exhibit fundamental shortcomings [ 1 , 11 ] . LiDAR-centric approaches often emphasize computational efficiency, whereas camera-based systems can be deployed using comparatively inexpensive hardware. Despite these advantages, unimodal pipelines inherit the failure modes and constraints of the single sensor they depend on. These limitations reduce the reliability of PnP systems in challenging conditions that involve diverse object types and changing environments. Recent research has shown that object queries proposed by DETR3D [ 4 , 25 ] , a learned sparse set of embedding vectors (object priors) which can be decoded into a 3D reference point, are an effective interface for using RGB data for 3D detections. Query-based approaches have expanded to showcase the suitabilty of the interface for object tracking and trajectory prediction [ 12 ] . However, while they underline the value of RGB for 3D detection and forecasting, they do not resolve the broader uncertainty issues faced by PnP systems that rely exclusively on vision. This is especially relevant when the goal is to protect vulnerable road users. Recent research on traffic monitoring with social robots [ 8 , 7 ] indicates that RGB sensing can support basic detection and intention inference, yet it does not provide sufficiently accurate estimates of agent position and timing. This manuscript extends our earlier proceedings publication [ 14 ] . In that work, we introduced Li-ViP3D, a novel end-to-end and fully differentiable approach to trajectory prediction from combined RGB and LiDAR data. Li-ViP3D expanded the agent-query paradigm into a multimodal framework capable of predicting trajectories over a 6-second horizon for a broad set of agents, including cars, bicycles, and pedestrians. We also redesigned the 3D agent-query interface to incorporate LiDAR information and showed that this interface enables effective fusion of RGB and LiDAR cues. In the present extension, our primary contributions focus on a new query-gated deformable fusion (QGDF) strategy that further increases the practicality of agent queries for end-to-end perception and prediction. We modify our Li-ViP3D architecture to use the QGDF mechanism. The resulting Li-ViP3D++ network improves computational efficiency and detection quality, while also enhancing the explainability of the base Li-ViP3D. II Related Work II-A Traditional Perception and Prediction Pipelines Research on the detection-tracking-forecasting pipeline is extensive and spans many facets of these challenging problems [ 16 ] . Although numerous methods address parts of the pipeline, the shift toward fully end-to-end formulations is still in its early stages. As a result, there is not yet broad agreement on how to design end-to-end systems that simultaneously account for the full range of requirements. Conventional PnP systems typically decompose the task into separate components [ 29 , 26 , 27 , 28 ] : object detection, object tracking, and trajectory prediction. Advocates of modular designs emphasize their adaptability, ease of upgrading individual modules, and clearer attribution of errors, arguing that these benefits can outweigh those of end-to-end models. Many such points are well supported: for example, vehicle recognition can be handled efficiently with lightweight region filtering [ 22 ] , and geometric cues from camera frames can improve scene understanding [ 9 ] . Nevertheless, existing modular pipelines have not yet demonstrated a practical way to counteract the propagation and accumulation of errors across stages when modules are integrated, while still meeting real-time computational constraints. Moreover, modular interfaces typically enforce compact information exchange, often passing only selected descriptors such as position, velocity, acceleration, distance, and related features. II-B End-to-End solutions End-to-end formulations, by contrast, can carry informative cues about agent intent from perception to prediction, including signals such as turn indicators and brake lights, as well as body pose and other rich context. They are also appealing from an engineering standpoint due to single-loop training and simpler deployment. However, until recently, methods such as FaF [ 21 ] , IntentNet [ 5 ] , and PnPNet [ 18 ] did not fully realize the promise of end-to-end PnP, because they depended on intermediate feature-map representations and non-differentiable operations (e.g., non-maximum suppression). This design choice restricts error propagation between PnP components and often leads to only partial coupling through a composite loss.