M 2 BeamLLM: Multimodal Sensing-empowered mmWave Beam Prediction with Large Language Models

Can Zheng, Jiguang He, Chung G. Kang, Guofa Cai, Zitong Yu, Mérouane Debbah · 2025

Read M 2 BeamLLM: Multimodal Sensing-empowered mmWave Beam Prediction with Large Language Models by Can Zheng, Jiguang He, Chung G. Kang, Guofa Cai, Zitong Yu, Mérouane Debbah (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-14532.txt.

Introduction

Can Zheng and co-authors aim millimeter wave beams with help from a language model in a 2025 paper. Fast vehicle links need beams pointed right before data flows. Cameras, radar, depth sensors, and position data all hint where to aim. A language model fuses those hints into one beam choice.

Sensors are encoded, aligned, and merged before the model decides. More sensor types steadily improved the aim in their tests. The system beat plain deep learning in normal and few example cases. The work targets car to roadside links in busy streets.

Catalog record

CREATORCan Zheng, Jiguang He, Chung G. Kang, Guofa Cai, Zitong Yu, Mérouane Debbah
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-14532

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

M 2 BeamLLM: Multimodal Sensing-empowered mmWave Beam Prediction with Large Language Models Can Zheng, Jiguang He, Chung G. Kang, Guofa Cai, Zitong Yu, Mérouane Debbah Abstract This paper introduces a novel neural network framework called M 2 BeamLLM for beam prediction in millimeter-wave (mmWave) massive multi-input multi-output (mMIMO) communication systems. M 2 BeamLLM integrates multi-modal sensor data, including images, radar, LiDAR, and GPS, leveraging the powerful reasoning capabilities of large language models (LLMs) such as GPT- 2 2 for beam prediction. By combining sensing data encoding, multimodal alignment and fusion, and supervised fine-tuning (SFT), M 2 BeamLLM achieves significantly higher beam prediction accuracy and robustness, demonstrably outperforming traditional deep learning (DL) models in both standard and few-shot scenarios. Furthermore, its prediction performance consistently improves with increased diversity in sensing modalities. Our study provides an efficient and intelligent beam prediction solution for vehicle-to-infrastructure (V 2 2 I) mmWave communication systems. I Introduction Millimeter-wave (mmWave) frequency bands offer broader channel bandwidths compared to traditional sub- 6 GHz bands, enabling higher capacity for communication systems. However, their high propagation loss introduces new challenges for system design. While high-gain narrow beams formed by large antenna arrays can effectively compensate for the high path loss, the resulting strong directional characteristics make the beams highly sensitive to user mobility and environmental obstructions [ 1 ] . This sensitivity can lead to a significant decline in the stability of mmWave connectivity. In this context, beam alignment (BA) becomes a crucial technology for determining the optimal/sub-optimal beamforming direction, playing a key role in establishing reliable communication links. In vehicle-to-everything (V 2 X) scenarios, the high mobility of user equipment (UE) imposes even more stringent requirements on BA mechanisms. Compared to general mobile scenarios, the rapid topology changes unique to mmWave V 2 X significantly increase the complexity of real-time alignment for beams. Frequent beam misalignment can lead not only to communication link interruptions but also to severe degradation of communication quality. Existing BA solutions rely on repeated beam training processes to determine the optimal beam direction, but the resource-intensive nature of this process can encroach upon time slot resources allocated for data transmission, resulting in reduced system throughput. Therefore, developing intelligent BA mechanisms that offer efficiency and robustness has become a key research direction for ensuring the reliability of mmWave V 2 X communications. The recent advancements in integrated sensing and communication (ISAC) have brought significant attention to sensing-empowered beam prediction. The optimal beam pair between the base station (BS) and UE is determined by their instantaneous spatial positions and the surrounding environment. Proactively utilizing UE positioning data alongside sensing data measured by the BS for beam prediction is a viable strategy to eliminate beam training overhead. Beam prediction based on historical sensing data essentially involves analyzing the complex interplay between the temporal dimension (historical state evolution) and the spatial dimension (environmental physical characteristics) of sensing data. This analysis is aimed at constructing mathematical models that reveal hidden dynamic patterns and establish a dynamic mapping between sensing data and future beam states. This task is particularly well-suited for deep learning (DL) methods, which can effectively capture and model such intricate relationships [ 2 ] . Prior studies have explored the use of various communication and sensing modalities—such as sub- 6 GHz channel state information (CSI) [ 3 ] , RGB images [ 4 ] , radar [ 5 ] , LiDAR [ 6 ] , and GPS [ 7 ] —in conjunction with DL architectures for beam prediction. The rapid development of large language models (LLMs), such as ChatGPT [ 8 ] and DeepSeek [ 9 ] , introduces new possibilities for physical-layer applications that go beyond the limitations of conventional methods. LLMs, with their billions of parameters, exhibit exceptional capabilities in natural language understanding and logical reasoning. Owing to extensive pretraining on diverse and large-scale datasets, LLMs can achieve strong performance with minimal task-specific fine-tuning. Compared to training ransformer models from scratch, LLMs require significantly less labeled data, making them particularly advantageous in real-world applications where collecting and annotating large-scale supervised datasets is often impractical. Recent efforts have explored leveraging LLMs for physical-layer tasks, such as mmWave beam prediction. A representative example is BeamLLM [ 10 ] , which utilizes LLMs to predict beam directions in vehicle-to-infrastructure (V 2 I) scenarios. While BeamLLM demonstrates strong prediction performance, its reliance on a single sensing modality-namely RGB images-limits its adaptability in complex and dynamic environments where visual information alone may be insufficient due to occlusions, lighting changes, or adverse weather conditions. To address this limitation, a more recent study [ 11 ] has attempted to integrate GPS positioning data and RGB images for multimodal beam prediction. This work highlights the potential of cross-modal fusion in enhancing robustness. However, it suffers from two limitations. First, it relies on large-scale GPT- 4 models whose inference latency renders the approach unsuitable for real-time applications. For practical deployment, a more feasible solution would leverage historical data to predict future beam, aligning better with real-world requirements. Second, still only a limited number of modalities and lacking adaptability to environmental variations. Beyond this, some prior works have investigated more generalized multimodal fusion frameworks to improve prediction robustness. For example, Shi et al. [ 12 ] explored combining multimodal sensing data, yet the fusion was static and based on naive concatenation. In contrast, Zhang et al. [ 13 ] introduced a more dynamic mixture-of-experts (MoE) framework, in which a gating network assigned weights to each modality. In addition, Cui et al. [ 14 ] established the relationship between different modalities through cross-attention. In this paper, we introduce M 2 BeamLLM, a multimodal sensing empowered beam prediction framework built upon LLMs. Our major contributions are summarized as follows: 1. Our M 2 BeamLLM framework integrates multiple data sources—images, radar, LiDAR, and GPS. This multimodal approach enables a more comprehensive understanding of the environment, significantly enhancing the accuracy and robustness of beam prediction, particularly in dynamic scenarios. 2. We pioneer the application of LLMs, specifically GPT- 2 , to the beam prediction task, which has traditionally been addressed using DL architectures, e.g., recurrent neural networks (RNNs) [ 4 ] . By leveraging the superior generalization and reasoning capabilities of LLMs, we effectively model complex relationships within multimodal data, an approach that remains largely unexplored in existing studies. 3. Through supervised fine-tuning (SFT) of a pretrained LLM [ 15 ] , we enable the model to adapt to the beam prediction task with minimal training data and computational cost. This contrasts sharply with the resource-intensive approaches of training models from scratch, offering a more efficient solution for practical deployment. 4. Experimental results demonstrate that M 2 BeamLLM outperforms state-of-the-art DL models in both standard and few-shot prediction scenarios. Our framework achieves a Top- 1 accuracy of 68.9\% , surpassing the next-best method by 13.9\% , and maintains superior performance even under challenging few-shot conditions. 5. Through ablation studies, we provide empirical evidence on the impact of different sensing modality combinations on prediction performance. This analysis reveals that modality diversity generally enhances prediction accuracy, offering new insights into the value of multimodal fusion in beam prediction and advancing the understanding of multi-source data contributions in this domain. The rest of

Read the complete text