Multimodal Radio and Vision Fusion for Robust Localization in Urban V2I Communications
Can Zheng 1 , Jiguang He 2 , Chung G. Kang 1 , Guofa Cai 3 , Henk Wymeersch 4 · 2025
Read Multimodal Radio and Vision Fusion for Robust Localization in Urban V2I Communications by Can Zheng 1 , Jiguang He 2 , Chung G. Kang 1 , Guofa Cai 3 , Henk Wymeersch 4 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-17640.txt.
Introduction
Can Zheng and co-authors fuse radio and camera data to pin down vehicle spots in city streets, in a 2025 paper. Cars that talk to road gear need to know just where they are. GPS falters among tall towers, where signals bounce and fade. Wireless channel data and street images each hold part of the answer.
Their method learns from both streams at once with a contrastive setup that ends in a direct spot estimate. Radio traits show how signals warp on their paths. Images show the scene the car sits in. Joined, the two cover each other's blind spots. In simulations of tangled city blocks, the fused model beat both classic tools and single-stream models on accuracy and precision. The authors aim the work at self-driving cars and smart city gear.
Catalog record
| CREATOR | Can Zheng 1 , Jiguang He 2 , Chung G. Kang 1 , Guofa Cai 3 , Henk Wymeersch 4 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 30 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-17640
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Multimodal Radio and Vision Fusion for Robust Localization in Urban V2I Communications Can Zheng 1 , Jiguang He 2 , Chung G. Kang 1 , Guofa Cai 3 , Henk Wymeersch 4 Abstract Accurate localization is critical for vehicle-to-infrastructure (V2I) communication systems, especially in urban areas where GPS signals are often obstructed by tall buildings, leading to significant positioning errors, necessitating alternative or complementary techniques for reliable and precise positioning in applications like autonomous driving and smart city infrastructure. This paper proposes a multimodal contrastive learning regression based localization framework for V2I scenarios that combines channel state information (CSI) with visual information to achieve improved accuracy and reliability. The approach leverages the complementary strengths of wireless and visual data to overcome the limitations of traditional localization methods, offering a robust solution for V2I applications. Simulation results demonstrate that the proposed CSI and vision fusion model significantly outperforms traditional methods and single modal models, achieving superior localization accuracy and precision in complex urban environments. I Introduction Accurate localization is a cornerstone of vehicle-to-infrastructure (V 2 I) communication systems, particularly in urban environments where the global positioning system (GPS) often fails due to signal obstructions from tall buildings, resulting in significant positioning errors [ 1 ] . In such scenarios, traditional GPS-based localization methods are inadequate, requiring alternative or complementary techniques for reliable and precise positioning in applications like autonomous driving and smart city infrastructure. The evolution of wireless localization from 4 G to 5 G networks highlights the increasing demand for high-precision positioning. In 4 G networks, localization techniques such as observed time difference of arrival (OTDOA) and uplink time difference of arrival (UL-TDOA) offered accuracy in the tens of meters, mainly for emergency services [ 2 ] . The advent of 5 G networks has marked a significant leap forward, introducing dedicated positioning reference signals (RSs) and advanced techniques such as round trip time (RTT) and angle-based methods [ 3 ] . These advancements enable positioning accuracies down to centimeters, driven by wider bandwidths, massive antenna arrays, and enhanced signal processing capabilities. (a) Miss detection of the target. (b) False positive of the target. Fig. 1 : Two common failure modes observed in a YOLO-based object detection system, highlighting the inherent limitations of relying solely on visual input for accurate localization. Despite the potential of V 2 I communication for localization, several challenges persist. V 2 I systems rely on interactions between vehicles and roadside units (RSUs), but multipath propagation, non-line-of-sight (NLOS) conditions, and the nonlinear, non-Gaussian nature of wireless measurements can severely degrade performance [ 4 ] . Traditional angle-of-arrival (AoA) estimation methods, such as the multiple signal classification (MUSIC) algorithm, struggle with multipath interference due to the rank deficiency of the covariance matrix [ 5 ] . Techniques like spatial smoothing can mitigate some of these issues, but their effectiveness diminishes when the line-of-sight (LoS) path is weak [ 6 ] . Furthermore, range estimation accuracy is closely tied to system bandwidth. In narrowband systems, achieving high precision requires substantial RS overhead, which is resource-intensive and inefficient. These issues highlight the need for innovative approaches to improve localization accuracy in complex urban environments. To address the limitations of current localization systems, integrating channel state information (CSI) with visual data from RSU-mounted cameras offers a promising solution. Visual data provides positional cues from environmental landmarks, unaffected by wireless signal distortions, while CSI ensures stability when visual data is unreliable. (see Fig. 1 ). Fusing these modalities creates a robust, accurate localization system for complex environments. Current research has explored various avenues to enhance localization in challenging environments. For instance, one approach leverages beam received power and visual data for GPS denoising [ 7 ] , aiming to refine positioning in scenarios where GPS signals are unreliable. Another notable effort involves a meta-learning framework for joint localization using multiple cameras and RSUs [ 8 ] , showcasing the potential of advanced machine learning techniques in complex multi-sensor systems. In this work, we propose a multimodal contrastive learning regression based localization framework for V 2 I scenarios that combines CSI with visual information to achieve improved accuracy and reliability. Our main contributions are: (i) proposing a novel localization framework that utilizes only a single RSU with a monocular camera to achieve high-precision positioning, eliminating the need for multiple RSUs; (ii) developing an innovative approach for feature alignment and fusion between CSI and visual modalities, enabling complementary multimodal integration for robust localization. This distinct approach aims to demonstrate robust localization performance even in highly constrained sensing environments. Fig. 2 : Demonstration of considered urban V 2 I scenario. The collected CSI and the image data in RSU are fed into a neural network to estimate the 3 D position of the UE. Notations : Bold lowercase letters denote vectors (e.g., \mathbf{x} ), and bold uppercase letters denote matrices or tensors (e.g., \mathbf{X} ). \text{diag}(\mathbf{x}) denotes a square diagonal matrix with entries of \mathbf{X} on its diagonal. [\mathbf{X}]_{a,:} and [\mathbf{X}]_{:,b} denote the a -th row, b -th column vectors of \mathbf{X} . The superscripts (\cdot)^{\mathsf{T}} and (\cdot)^{\mathsf{H}} represent the transpose and Hermitian (conjugate transpose) operations, respectively. The operator [\cdot;\cdot] denotes the concatenation operation, while \|\cdot\|_{2} denotes the Euclidean norm of a vector and |\cdot| denotes element-wise magnitude of a vector or a matrix. \otimes denotes the Kronecker product. Unless otherwise specified, \mathbb{R} and \mathbb{C} denote the sets of real and complex numbers, respectively. \mathrm{Cov}(\cdot,\cdot) denotes the covariance. II System Model In this section, we delve into the system model considered, as shown in Fig. 2 . The RSU integrated with monocular cameras and other sensors use uplink (UL) CSI from the UE and sensing data such as visual data to enable high-precision localization. Meanwhile, the RSU employs orthogonal frequency division multiplexing (OFDM) with K subcarriers and is equipped with a uniform planar array (UPA) comprising M antenna elements. We aim to estimate the UE position \mathbf{p}_{\text{UE}}=[p_{\text{UE}}^{x},p_{\text{UE}}^{y},p_{\text{UE}}^{z}]^{\mathsf{T}} in a multipath environment. In practical deployments, the RSU can instruct the UE to transmit an RS, enabling UL channel estimation. Let \hat{h}_{m,k} represent the channel response associated with the m -th antenna and k -th subcarrier. It can be modeled as the product of an amplitude response |\hat{h}_{m,k}| and a phase response \angle\hat{h}_{m,k} , expressed as a summation over multiple propagation paths. Specifically, the channel gain can be written as \displaystyle\hat{h}_{m,k}=|\hat{h}_{m,k}|e^{\angle\hat{h}_{m,k}}=\sum_{l=1}^{L}\alpha_{l}e^{-j2\pi(f_{c}+k\Delta f)\tau_{l}}{a}_{m}(\theta_{l},\phi_{l}), (1) where f_{c} denotes the carrier frequency and L is the total number of propagation paths; for the l -th path, \alpha_{l} , \tau_{l} , and {a}_{m}(\theta_{l},\phi_{l}) denote the complex gain, the time of arrival (ToA), and the m -th element of the corresponding steering vector, respectively. Here, \theta_{l} represents the elevation angle and \phi_{l} represents the azimuth angle for the l -th path’s direction of arrival. The term \Delta f represents the subcarrier spacing. Let \hat{\mathbf{H}}_{\text{F}}\in\mathbb{C}^{M\times K} represent the sampled frequency-domain CSI matrix. We can then separate the matrix into an outer product of array steering vector \mathbf{a}(\phi_{l},\theta_{l})\in\mathbb{C}^{M\times 1} and delay steering vector \mathbf{d}(\tau_{l})\in\mathbb{C}^{K\times 1} for path l : \displaystyle\mathbf{a}(\theta_{l},\phi_{l})=\begin{bmatrix}a_{1}(\theta_{l},\phi_{l})\\ a_{2}(\theta_{l},\phi_{l})\\ \vdots\\ a_{M}(\theta_{l},\phi_{l})\end{bmatrix},\ \mathbf{d}(\tau_{l})=\begin{bmatrix}e^{-j2\pi f_{c}\tau_{l}}\\ e^{-j2\pi(f_{c}+1\Delta f)\tau_{l}}\\ \vdots\\ e^{-j2\pi(f_{c}+(K-1)\Delta f)\tau_{l}}\end{bmatrix}. (2) Assuming no mutual coupling between antenna elements, we consider a UPA with M_{x} elements along the x -axis and M_{y} elements along