Out-of-Sight Trajectories: Tracking, Fusion, and Prediction PubID: pubid: 0000–0000/00$00.00 © 2025 IEEE
Haichao Zhang, Yi Xu, Yun Fu · 2025
Read Out-of-Sight Trajectories: Tracking, Fusion, and Prediction PubID: pubid: 0000–0000/00$00.00 © 2025 IEEE by Haichao Zhang, Yi Xu, Yun Fu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-15219.txt.
Introduction
Haichao Zhang, Yi Xu, and Yun Fu extend a tracking task for agents that cameras cannot currently see in a 2025 computer vision paper. Trajectory prediction feeds driving, robotics, and surveillance, yet most methods assume clean, complete observations. Real sensors lose people and vehicles behind occlusions and coverage gaps, and their raw tracks are noisy.
The Out-of-Sight Trajectory task asks for clean visual trajectories of hidden objects inferred from noisy sensing. This extended study enlarges the earlier benchmark from pedestrians alone to pedestrians and vehicles, matching the needs of autonomous driving more closely. Multimodal tracking, sensor fusion, and forecasting combine to guess where an unseen agent travels and when it reappears. Robust prediction for the invisible is framed as a safety feature, not a curiosity.
Catalog record
| CREATOR | Haichao Zhang, Yi Xu, Yun Fu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 70 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-15219
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Out-of-Sight Trajectories: Tracking, Fusion, and Prediction PubID: pubid: 0000–0000/00$00.00 © 2025 IEEE Haichao Zhang, Yi Xu, Yun Fu Abstract Trajectory prediction is a critical task in computer vision and autonomous systems, playing a key role in applications such as autonomous driving, robotics, surveillance, and virtual reality. Existing methods often rely on complete and noise-free observational data, overlooking the challenges associated with out-of-sight objects and the inherent noise in sensor data caused by limited camera coverage, obstructions, and the absence of ground truth for denoised trajectories. These limitations pose significant safety risks and hinder reliable trajectory prediction in real-world scenarios. In this extended work, we present significant advancements in Out-of-Sight Trajectory (OST), a novel task that predicts the noise-free visual trajectories of out-of-sight objects using noisy sensor data. Building on our previous research, we broaden the scope of Out-of-Sight Trajectory Prediction (OOSTraj) to include both pedestrians and vehicles, extending its applicability to autonomous driving, robotics, surveillance, and virtual reality. Our enhanced Vision-Positioning Denoising Module leverages camera calibration to establish a vision-positioning mapping, addressing the lack of visual references, while effectively denoising noisy sensor data in an unsupervised manner. Through extensive evaluations on the Vi-Fi and JRDB datasets, our approach achieves state-of-the-art performance in both trajectory denoising and prediction, significantly surpassing previous baselines. Additionally, we introduce comparisons with traditional denoising methods, such as Kalman filtering, and adapt recent trajectory prediction models to our task, providing a comprehensive benchmark. This work represents the first initiative to integrate vision-positioning projection for denoising noisy sensor trajectories of out-of-sight agents, paving the way for future advancements in this domain. The code and preprocessed datasets are available at https://github.com/Hai-chao-Zhang/OST . I Introduction (a) (b) (c) (d) Fig. 1 : Illustrations of typical scenarios where out-of-sight trajectory prediction is critical. (The lights in the figure indicate the field of view of each embedded camera.) (a) Autonomous vehicles face challenges with blind spots caused by limited sensors and cameras, often missing out-of-sight pedestrians or vehicles. Out-of-sight trajectory prediction aids in detecting and forecasting their future locations. (b) Surveillance cameras with fixed angles frequently fail to capture vehicles entering or leaving their field of view. Out-of-sight trajectory prediction enables tracking of vehicles outside these restricted views. (c) Traditional virtual reality systems depend on specific devices for location tracking, while recent VR devices use ego-cameras for hand position recognition. These systems lose track of gestures when hands move out of view, but out-of-sight trajectory prediction helps maintain tracking even in such cases. (d) Robotics systems encounter occlusions where trajectories of IoT devices or other robots move out of sight. Out-of-sight trajectory prediction combines data from various viewpoints to track the positions and future paths of robots and IoT devices. Figures generated with [ 22 ] . Trajectory prediction plays a crucial role across a wide range of research areas, including computer vision, surveillance, virtual reality, robotics, vehicle safety systems, and autonomous driving, as illustrated in Fig. 1 . This capability is essential not only for understanding human behavior through deep learning models but also for empowering autonomous systems to predict and react to the movements of pedestrians and vehicles. By forecasting future trajectories from multimodal signals—often noisy or incomplete—trajectory prediction significantly enhances safety and helps prevent potential collisions. However, real-world environments introduce a myriad of complexities, such as occlusions, blind spots, and noisy sensor inputs, which make reliable prediction exceedingly difficult. While prior work has made considerable progress in this domain, existing methods predominantly rely on computer vision-based algorithms under the assumption of ideal conditions, such as noise-free and consistently visible observations. This reliance neglects two critical challenges often encountered in practical scenarios: the presence of objects that fall outside the camera’s field of view and the pervasive noise in multimodal sensor data. To address these challenges, our research proposes an innovative vision-positioning denoising framework, specifically designed to enhance trajectory prediction by leveraging both visual and sensor data in challenging real-world conditions. Let us consider a representative case where neglecting out-of-sight objects leads to critical issues in the domain of autonomous driving and robotics. The inherent limitations of camera-based systems pose significant challenges in such scenarios. As illustrated in Fig. 2 , pedestrians or vehicles are often obscured by other vehicles, physical barriers, or fall into blind spots, making them temporarily or entirely invisible to cameras. These blind spots can result in agents suddenly reappearing within the camera’s field of view, leaving insufficient time for trajectory prediction algorithms to process the visual data and respond effectively. The problem is further exacerbated by the high speeds at which autonomous vehicles operate; rapid movement allows pedestrians or other vehicles to emerge from occluded regions faster than the system can register and act. Traditional camera systems face additional difficulties in situations where the visual history of an object is entirely absent, significantly impairing their ability to predict future trajectories. This limitation compromises the reliability of these systems, as their functionality relies heavily on constant visibility. Consequently, the inability to account for out-of-sight objects introduces severe safety risks, such as delayed reactions and potential collisions. Beyond immediate safety concerns, these challenges also hinder the scalability and practical implementation of autonomous driving technologies in complex and dynamic environments where occlusions are inevitable. To bridge these gaps, our research introduces a vision-positioning denoising framework specifically designed to address the out-of-sight problem in trajectory prediction. By integrating multimodal data, including both noisy sensor inputs and visual information, our approach enhances the system’s ability to predict the trajectories of occluded agents with greater accuracy. This multimodal fusion not only mitigates safety risks but also offers a scalable and robust solution for autonomous systems, enabling them to operate reliably in unpredictable real-world scenarios. Recent advancements in trajectory prediction research have increasingly recognized the challenges posed by incomplete observations, which remain a critical bottleneck for achieving accurate predictions. The majority of existing studies, however, continue to operate under the assumption of complete and noise-free observational data. Notable exceptions include the works of Xu et al. [ 29 ] and Fujii et al. [ 6 ] , which have made strides in addressing this limitation by focusing on imputing missing views or predicting trajectories from partially observed data. Similarly, Zhang et al. [ 30 ] introduced a method for predicting layout trajectories based on a single visual timestamp, leveraging supplementary information from mobile sensor modalities. Despite their contributions, these approaches share a fundamental constraint: they rely on the availability of at least some visual data to support trajectory prediction. To the best of our knowledge, no existing research has fully explored the prediction of trajectories solely based on out-of-sight observations, where visual data is entirely absent. This gap represents not only a significant limitation in the academic domain but also an urgent safety challenge in real-world applications such as autonomous driving. Our research directly addresses this void by proposing a novel framework capable of predicting trajectories exclusively from out-of-sight observations, paving the way for safer and more robust systems in complex environments. Addressing the out-of-sight problem in trajectory prediction introduces two primary challenges, the first of which is managing noisy sensor measurements. Without access to visual timestamps, our approach relies on widely used localization sensors, such as GPS and odometers, to provide