BeamLLM: Vision-Empowered mmWave Beam Prediction with Large Language Models

Can Zheng, Jiguang He, Guofa Cai, Zitong Yu, Chung G. Kang · 2025

Read BeamLLM: Vision-Empowered mmWave Beam Prediction with Large Language Models by Can Zheng, Jiguang He, Guofa Cai, Zitong Yu, Chung G. Kang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-10432.txt.

Introduction

Can Zheng, Jiguang He, Guofa Cai, Zitong Yu, and Chung G Kang use camera images and large language models to aim millimeter-wave beams, in a 2025 paper. Such beams carry fast wireless data but are narrow, so finding the right direction costs time and training signals. Their framework watches the scene instead of probing the air.

A vision step pulls the position of the user's device from ordinary photos. A reprogramming step then translates those visual and time features into a form the language model can reason over. On a realistic vehicle-to-base-station test, the method picked the exact best beam about sixty-one percent of the time and landed in the top three nearly always, beating standard deep learning models. With very few examples, its scores slipped only modestly over time.

Catalog record

CREATORCan Zheng, Jiguang He, Guofa Cai, Zitong Yu, Chung G. Kang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE24 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-10432

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

BeamLLM: Vision-Empowered mmWave Beam Prediction with Large Language Models Can Zheng, Jiguang He, Guofa Cai, Zitong Yu, Chung G. Kang Abstract In this paper, we propose BeamLLM, a vision-aided millimeter-wave (mmWave) beam prediction framework leveraging large language models (LLMs) to address the challenges of high training overhead and latency in mmWave communication systems. By combining computer vision (CV) with LLMs’ cross-modal reasoning capabilities, the framework extracts user equipment (UE) positional features from RGB images and aligns visual-temporal features with LLMs’ semantic space through reprogramming techniques. Evaluated on a realistic vehicle-to-infrastructure (V2I) scenario, the proposed method achieves 61.01 % 61.01\% top- 1 1 accuracy and 97.39 % 97.39\% top- 3 3 accuracy in standard prediction tasks, significantly outperforming traditional deep learning models. In few-shot prediction scenarios, the performance degradation is limited to 12.56 % 12.56\% (top- 1 1 ) and 5.55 % 5.55\% (top- 3 3 ) from time sample 1 1 to 10 10 , demonstrating superior prediction capability. I Introduction Millimeter-wave (mmWave) communication has garnered significant attention due to its abundant spectrum resources above 26 GHz, enabling high-speed data transmission. However, the high operating frequency results in substantial path loss. To address this challenge, massive multiple-input multiple-output (mMIMO) antenna arrays are extensively employed, which utilize highly directional beamforming techniques to mitigate propagation losses. Furthermore, the short wavelength of mmWave signals facilitates compact antenna spacing, which enables the integration of large-scale antenna arrays within constrained physical dimensions. The effectiveness of directional beamforming depends on precise alignment between transmit and receive beams. Beam training addresses this challenge by scanning predefined codebooks at both the transmitter and receiver to identify the optimal beam pair, thereby maximizing received signal power without requiring the exhaustive acquisition of full channel state information (CSI). Compared to legacy sub-6 GHz MIMO systems, beam training in mmWave systems faces heightened challenges: 1) Large antenna arrays lead to high-dimensional channel matrices, increasing training overhead; 2) Frequent beam tracking, especially in high-mobility scenarios (e.g., vehicle-to-everything (V2X) and unmanned aerial vehicles (UAVs)), introduces prohibitive latency. Recent studies [ 1 , 2 , 3 , 4 ] have explored sensing-aided beam prediction, leveraging multimodal sensor data such as RGB images, radar, LiDAR, and GPS to improve efficiency and reduce training overhead. As a key enabler of integrated sensing and communication (ISAC) in 6G, this approach holds significant potential to enhance the performance of mmWave MIMO systems. To maintain beam prediction performance, deep learning (DL) is commonly used to extract user equipment (UE) movement features from received sensing data, enabling more accurate future beam selection. Due to its powerful non-linear feature extraction capability, DL has been widely explored in wireless communication tasks, including channel estimation [ 5 ] and beam prediction. Recent breakthroughs in large language models (LLMs), such as GPT-4 [ 6 ] and DeepSeek [ 7 ] , have demonstrated remarkable contextual reasoning and few-shot generalization abilities. While LLMs are originally designed for natural language processing (NLP), LLMs have shown strong cross-modal learning capabilities, thus extending their applications to time series for forecasting and computer vision (CV) tasks. Inspired by these advantages of the LLMs, several methods applying LLMs have been proposed for channel prediction [ 8 ] , beam prediction [ 9 ] , and port prediction for fluid antennas [ 10 ] . Built on these developments, in this paper, we propose a vision-aided beam prediction framework, named BeamLLM, which utilizes LLMs to process RGB images, thereby enabling more efficient and adaptive beam selection. Unlike [ 9 ] , our method does not rely on historical beam indices or angle of departure (AoD) information. Instead, BeamLLM relies solely on visual features for beam prediction. Furthermore, to ensure robust performance and practical applicability, we validate our framework using real-world measurement datasets, to demonstrate its potential for deployment in real-world scenarios. The rest of this paper is organized as follows: Section II provides a system model and problem formulation of the beam prediction task. The proposed BeamLLM framework is presented in Section III. Section IV presents extensive simulation results, including performance comparisons with benchmark methods, along with detailed discussions. Finally, we conclude our work in Section V. Fig. 1: Illustration of the system model considered. II System Model and Problem Formulation II-A System Model Fig. 1 illustrates the system model considered for vehicle-to-infrastructure (V2I) mmWave communication. In this model, the base station (BS) deploys a mmWave phased-array receiver with M elements of half-wavelength spacing and an RGB camera. The antenna array enables the BS to perform beamforming, while the camera captures images within its field of view at a certain frame rate for sensing and downstream applications. Fig. 2: The model framework of BeamLLM. We assume that the BS has a predefined beamforming codebook \mathcal{F}=\{\mathbf{f}_{1},\cdots,\mathbf{f}_{|\mathcal{F}|}\} , containing |\mathcal{F}| beams, where \mathbf{f}_{m}\in\mathbb{C}^{N\times 1},m=1,\cdots,M represents the m -th beamforming vector. The UE is assumed to have a single antenna. At time step t , the user transmits a single symbol s[t]\in\mathbb{C} that satisfies the power constraint \mathbb{E}[|s[t]|^{2}]=P , where P represents the transmit power. At the BS, the received signal y[t] can be expressed as: \displaystyle y[t]=\mathbf{h}^{H}[t]\mathbf{f}_{m}[t]s[t]+n[t], (1) where \mathbf{h}[t] is the channel vector, \mathbf{f}_{m}[t] is the m -th beamforming vector from the codebook in time step t , and n[t]\sim\mathcal{CN}(0,\sigma^{2}) is the additive white Gaussian noise (AWGN) with variance \sigma^{2} . II-B Problem Formulation This paper mainly focuses on the beam prediction problem at the BS. Given the available sensing information up to time t-1 , the BS attempts to determine the optimal beams for H\in\mathbb{Z}^{+} future time steps, specifically for t,\cdots,(t+H-1) . We define the optimal beam at time step t as the beam that provides the highest beamforming gain, given by: \displaystyle\mathbf{f}_{m}^{*}[t]=\arg\max_{\mathbf{f}_{m}[t]\in\mathcal{F}}|\mathbf{h}^{H}[t]\mathbf{f}_{m}[t]|^{2}. (2) When perfect CSI knowledge is unavailable, beam training serves as an alternative method for determining the optimal beam. However, with a narrow beam codebook, the training overhead can be significant, and the likelihood of identifying the optimal beam is often low when the pre-beamforming SNR is poor. Because the optimal beam selection at the transmitter and receiver depends on the surrounding environment of the transceiver, our work aims to leverage visual information at the BS to assist beam selection and develop a beam prediction framework. III Large Laguage Model-Based Beam Prediction In this section, we introduce BeamLLM to tackle the vision-assisted beam prediction task outlined in Section II. Fig. 2 illustrates the proposed BeamLLM. The architecture mainly comprises three components, i.e., the visual data feature extraction module and the backbone module. III-A Visual Data Feature Extraction Module To process raw RGB data for the vision-aided beam prediction task, we employ the YOLOv4 object detector [ 11 ] . This detector identifies potential UEs within RGB images and extracts bounding box vectors \mathbf{b} . For a single image \mathbf{X}_{I}\in\mathbb{R}^{W\times H\times C} , bounding box vector is given as: \displaystyle\mathbf{b}=\text{YOLO}(\mathbf{X}_{I})=[x_{c},y_{c},w,h]^{T}, (3) which consists of the detected object’s center coordinates ( x -axis, y -axis), width, and height within the RGB image. Since the optimal beam selection is highly dependent on the direction and position of the transmission target, we use a sequence of bounding box vectors as the extracted visual feature. The objective is to predict the optimal beam index for the next H steps based on the historical

Read the complete text