CmIVTP: Cross-modal Interaction-based Vessel Trajectory Prediction for Maritime Intelligence
Yuxu Lu, Dong Yang, Xiaoyu Li, Mengwei Bao, Congcong Zhao · 2026
Read CmIVTP: Cross-modal Interaction-based Vessel Trajectory Prediction for Maritime Intelligence by Yuxu Lu, Dong Yang, Xiaoyu Li, Mengwei Bao, Congcong Zhao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-26524.txt.
Introduction
Yuxu Lu and four colleagues predict vessel paths with CmIVTP. AIS tracks run sparse for small vessels, while CCTV alone misses full dynamics. Their framework adds a target scene encoder and a cross-modal transformer fusing AIS motion, CCTV environmental, and scene features. Cross-modal attention captures within-mode and cross-mode links at once, keeping predictions consistent and feasible. A trajectory bank from clustered past AIS tracks supplies candidate paths, and Maritime-MmD+ syncs AIS and CCTV data. Tests beat multimodal benchmarks.
Catalog record
| CREATOR | Yuxu Lu, Dong Yang, Xiaoyu Li, Mengwei Bao, Congcong Zhao |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 98199 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.26524
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CmIVTP: Cross-modal Interaction-based Vessel Trajectory Prediction for Maritime Intelligence Yuxu Lu, Dong Yang, Xiaoyu Li, Mengwei Bao, Congcong Zhao Abstract Maritime intelligent transportation systems (MITS) are essential for ensuring navigation safety and efficiency in busy waterways. However, accurate vessel trajectory prediction remains challenging due to the limitations of single-source data. Automatic identification system (AIS) data is often sparse or unavailable for small vessels, while closed-circuit television (CCTV) data alone cannot fully capture dynamic vessel behavior. To mitigate these challenges, we propose a cross-modal interaction-based vessel trajectory prediction (named CmIVTP) framework to model the intricate interactions between vessel dynamics and environmental constraints. Specifically, we introduce a target-aware scene encoder to extract scene semantic features, effectively capturing vessel-environment interactions and enhancing trajectory prediction accuracy. In addition, we propose a cross-modal interaction transformer, which integrates AIS-derived motion features, CCTV-based environmental features, and scene representations. It leverages cross-modal attention mechanisms to simultaneously capture intra-modal semantics and inter-modal interactions, ensuring dynamically consistent and environmentally feasible predictions. Furthermore, we construct a vessel group trajectory bank by clustering historical AIS trajectories into representative motion patterns, providing an efficient and scalable approach for candidate trajectory generation. Additionally, we introduce the maritime multimodal dataset plus (named Maritime-MmD + ), a large-scale dataset that synchronizes AIS data and CCTV video data, providing robust support for multimodal trajectory prediction research. Extensive experiments demonstrate that CmIVTP achieves better performance on multimodal-driven vessel trajectory prediction benchmarks. The code resources for this work can be available at https://github.com/LouisYxLu/CmIVTP . I Introduction Achieving omission-free and real-time situational awareness is a fundamental prerequisite for maritime intelligent transportation systems (MITS), serving as the basis for intelligent alarms and collision prevention to ensure navigational safety [ 45 , 64 ] . As an essential component of MITS, maritime supervision involves monitoring and regulating vessel movements in areas such as ports or busy waterways [ 32 ] , leveraging technologies like the automatic identification system (AIS) [ 34 ] and closed-circuit television (CCTV) [ 35 ] to enforce safe navigation practices. Beyond ensuring compliance, MITS plays a critical role in advancing maritime autonomous surface ship (MASS) [ 6 ] and ship-shore collaboration [ 7 ] . As illustrated in Fig. 1 , by synergizing multi-source data like AIS and CCTV to overcome individual sensor limitations, MITS not only guarantees operational safety and efficiency but also catalyzes the evolution of next-generation vessel traffic systems [ 59 ] . Fig. 1: The MITS integrates advanced infrastructure and artificial intelligence-driven analytics to enable cross-modal interaction-based vessel trajectory prediction (e.g., using AIS and CCTV data), ultimately enhancing maritime intelligence and MASS operations for safe, sustainable navigation. Fig. 2: The flowchart of the proposed cross-modal interaction-based vessel trajectory prediction (named CmIVTP) framework. It consists of four main modules: the visual scene target-aware encoder (VSTaE) to extract environmental interaction features, the cross-modal interaction-based encoder (CmIE) to fuse AIS and CCTV data for modeling complex interactions, and the uncertainty-aware variational decoder (UaVD) to generate future trajectories. Additionally, a vessel group trajectory bank (VGTB) is constructed to improve the efficiency and accuracy of trajectory generation. In modern maritime surveillance, relying on a single data source is increasingly insufficient for comprehensive situational awareness [ 36 ] . Traditionally, AIS serves as the primary cooperative tracking tool, providing essential macroscopic vessel kinematics. However, as a radio-dependent system, its reliability is inherently vulnerable to signal loss, transmission delays, data sparsity, and the presence of uncooperative vessels [ 55 , 47 ] . CCTV acts as a non-cooperative visual sensor, capturing rich local semantics and environmental context independently of vessel broadcasts. In addition, high-performance graphics processing unit and rapid iterative optimization of learning methods make CCTV widely used in many fields of maritime affairs [ 19 ] . Nevertheless, relying solely on real-time static and dynamic information from video footage is inadequate for ensuring safety supervision of the vessel when visual targets are occluded or suffer from imaging interference [ 35 ] . To advance from passive monitoring to proactive collision avoidance, effective safety supervision fundamentally relies on multimodal data fusion to anticipate future vessel movements. I-A Motivation Multimodal-driven vessel trajectory prediction emerges as a critical prerequisite for MITS, which is essential for improving both the efficiency and safety of navigation in complex waterways [ 63 ] . AIS-based trajectory prediction methods [ 31 ] have been used in different navigable waters. However, AIS limitations make it challenging to effectively monitor vessels navigating complex restricted waters, which include • AIS may provide incomplete or unavailable data which limits the ability to track vessels accurately. • AIS may experience delays in data transmission which hinders timely trajectory predictions. • AIS may be intentionally turned off or not installed on a small number of vessels to evade detection. Although previous efforts have explored AIS latency [ 37 , 2 ] , tackling the complete unavailability of AIS signals remains a critical hurdle. To mitigate such inherent limitations, CCTV video serves as an indispensable modality by providing high-resolution visual semantics and environmental context. Currently, significant progress has been made in multimodal data-driven trajectory prediction for person [ 39 , 60 ] and vehicle [ 11 , 46 ] . However, transferring land-based methods to the maritime domain is sub-optimal. Unlike lane-constrained vehicles, vessels navigate in unstructured, open-water environments where their movements are governed by complex hydrodynamic behaviors and implicit interactions with the surroundings. For instance, while AIS provides sparse, macroscopic kinematic priors, it is challenging to capture the dense, microscopic visual cues necessary to perceive uncooperative vessels or complex spatial-temporal interactions in congested waterways [ 3 ] . Consequently, trajectory prediction can be significantly enhanced by fusing macroscopic AIS kinematics with microscopic CCTV semantics to robustly capture complex vessel dynamics and compensate for the missing data of a single modality. In addition, the absence of synchronized AIS-CCTV datasets severely hinders the training of such models, making a foundational benchmark urgently needed. I-B Contribution Fig. 3: Vessel trajectory prediction has evolved from traditional model-based methods to unimodal, and subsequently to advanced multimodal learning methods. Advancing this trend, our CmIVTP proposes a novel multimodal framework fusing AIS and CCTV data, ensuring robust prediction through cross-modal compensation, even when single-modal data (e.g., AIS) is sparse or completely missing. To mitigate these challenges, as shown in Fig. 2 , we propose a cross-modal interaction-based vessel trajectory prediction (named CmIVTP) framework that effectively fuses heterogeneous data from AIS and CCTV to model the complex interactions between vessel dynamics and environmental constraints. Specifically, we propose a visual scene target-aware encoder (VSTaE) to extract object-level scene features, effectively modeling vessel-environment interactions. Furthermore, we introduce the cross-modal interaction-based encoder (CmIE) that integrates AIS-derived motion features, CCTV-based environmental features, and scene representation features. We propose an uncertainty-aware variational decoder (UaVD) that shifts the prediction paradigm from a deterministic mapping to probabilistic distribution modeling. By learning a latent space to represent unobserved maneuvering intentions, it effectively captures multi-modal uncertainties to predict diverse future trajectories. To further enhance trajectory prediction, we construct a vessel group trajectory bank (VGTB) by clustering historical AIS trajectories into representative motion patterns, providing an efficient and scalable approach for candidate trajectory generation. Additionally, we introduce the maritime multimodal dataset plus (Maritime-MmD + ), a large-scale dataset that synchronizes AIS and