Multi-Agent Reinforcement Learning Counteracts Delayed CSI in Multi-Satellite Systems

Marios Aristodemou 4 , Yasaman Omid 1 , Sangarapillai Lambotharan 2 , Mahsa Derakhshani 1 , Lajos Hanzo 3 · 2026

Read Multi-Agent Reinforcement Learning Counteracts Delayed CSI in Multi-Satellite Systems by Marios Aristodemou 4 , Yasaman Omid 1 , Sangarapillai Lambotharan 2 , Mahsa Derakhshani 1 , Lajos Hanzo 3 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-16470.txt.

Introduction

Aristodemou and co-authors fight stale channel data in multi-satellite downlinks. Long travel times mean satellites act on outdated channel state. Their multi-agent learning method aims to lift total user data rates. A two-stage optimizer, DS-PPO, handles the large action space. Stage one tunes each satellite alone for sum-rate. Stage two tunes the fleet acting as one distributed antenna. Tests show robustness to channel errors plus higher rates. Convergence and cost analyses come with the method.

Catalog record

CREATORMarios Aristodemou 4 , Yasaman Omid 1 , Sangarapillai Lambotharan 2 , Mahsa Derakhshani 1 , Lajos Hanzo 3
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57658
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.16470

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Multi-Agent Reinforcement Learning Counteracts Delayed CSI in Multi-Satellite Systems Marios Aristodemou 4 , Yasaman Omid 1 , Sangarapillai Lambotharan 2 , Mahsa Derakhshani 1 , Lajos Hanzo 3 Abstract The integration of satellite communication networks with next-generation (NG) technologies is a promising approach towards global connectivity. However, the quality of services is highly dependant on the availability of accurate channel state information (CSI). Channel estimation in satellite communications is challenging due to the high propagation delay between terrestrial users and satellites, which results in outdated CSI observations on the satellite side. In this paper, we study the downlink transmission of multiple satellites acting as distributed base stations (BS) to mobile terrestrial users. We propose a multi-agent reinforcement learning (MARL) algorithm which aims for maximising the sum-rate of the users, while coping with the outdated CSI. We design a novel bi-level optimisation, procedure themes as dual stage proximal policy optimisation (DS-PPO), for tackling the problem of large continuous action spaces as well as of independent and non-identically distributed (non-IID) environments in MARL. Specifically, the first stage of DS-PPO maximises the sum-rate for an individual satellite and the second stage maximises the sum-rate when all the satellites cooperate to form a distributed multi-antenna BS. Our numerical results demonstrate the robustness of DS-PPO to CSI imperfections as well as the sum-rate improvement attached by the use of DS-PPO. In addition, we provide the convergence analysis for the DS-PPO along with the computational complexity. I Introduction Non-terrestrial networks (NTN) and particularly satellite communications (SatComs) are emerging technologies that are expected to find their way in the next-generation (NG) communication systems. Low-earth orbit (LEO) satellites provide extended coverage for numerous devices, while maintaining a relatively low latency compared to geostationary satellites [ 26 ] . Examples like the Starlink mega-constellation are already providing a dense network of satellites, enabling users roaming in off-the-grid areas to stay connected [ 14 , 19 ] . The high-altitude platforms, such as unmanned aerial vehicles, are unable to provide continuous, long-term connectivity. Thus, the integration of terrestrial networks with satellites is a vital step for NG systems. Integrating SatComs with terrestrial networks poses significant challenges in achieving high-throughput connectivity. Some of these challenges include high Doppler shifts, frequent satellite handovers, and long propagation delays [ 17 ] . Specifically, propagation delays pose challenges for the channel estimation process in the uplink, as the delay exceeds the channel’s coherence interval. To estimate the CSI, users transmit unique pilot sequences, allowing the satellites to estimate the channels between the two parties. However, the propagation delay results in a mismatch between the estimated channel state information (CSI) and the actual CSI. This issue is known as the “outdated/delayed CSI problem,” which we aim to address. In the literature, several authors have considered the effect of delayed CSI in downlink satellite-user transmission. The authors of [ 31 ] proposed a deep learning (DL)-based satellite channel predictor that used long short-term memory units for eliminating the detrimental impact of the outdated CSI. In [ 30 ] , a pair of deep neural networks (DNN) were proposed, one for satellite channel prediction (SatCP) and another for satellite hybrid beamforming (SatHB). In SatCP, the correlation between the uplink and the downlink CSI was exploited for predicting the downlink CSI by a DNN. By feeding the output of the SatCP network into the SatHB network, the SatHB maps the predicted CSI to a desired hybrid beamformer. Furthermore, in [ 18 ] , an RL algorithm using the deep deterministic policy gradient (DDPG) technique is introduced for coping with the effects of delayed CSI. Unlike previous research, in [ 18 ] , the delayed CSI is directly mapped to the optimised transmit precoding matrix (TPM), skipping over the channel prediction entirely. In LEO satellite networks, the dense constellation of satellites ensures that multiple satellites are visible to terrestrial users at any given time [ 5 ] . This allows the satellites to operate as a distributed multiple-input multiple-output (MIMO) base station (BS), providing significant diversity gain to improve the throughput, enhance reliability, and mitigate fading effects. These satellites are connected by inter-satellite links, enabling them to share their information effectively [ 1 ] . To maximise throughput, the satellites must collaboratively form a distributed TPM, while accounting for the challenges of outdated CSI. To the best of our knowledge, the consideration of outdated CSI in a cooperative multi-satellite system has only been discussed in [ 20 ] , where the authors harnessed the statistical information of the channel estimation errors to cope with the deleterious effects of outdated CSI. The simplicity of the technique makes it compatible with systems having low operating frequencies of up to 1 GHz. While it is theoretically possible to model such delays probabilistically and apply robust or stochastic optimisation [ 24 ] , in practice, high user/satellite mobility and the increasing carrier frequency (e.g., above 1 GHz) introduce rapidly time-varying channels. As shown in [ 20 ] , the error induced by delayed CSI under these conditions becomes excessive to model accurately via a simple statistical distribution. This renders conventional convex optimisation approaches ineffective due to high statistical uncertainty. For higher operating frequencies, the current literature of multi-satellite systems either considers the unrealistic scenario of perfect CSI or they only rely on statistical CSI. To fill the knowledge gaps, we aim for directly mapping the delayed CSI to an optimised TPM in a multi-satellite system. However, the technique conceived in [ 18 ] cannot be applied here due to the independent and non-identically distributed (non-iid) nature of the environment, where each satellite’s path uniquely determines its CSI. Hence, the RL agent(s) must track and adapt to the dynamic changes across multiple channels, significantly increasing the complexity of the task. In this paper, we explore the potential of multi-agent reinforcement learning (MARL) in counteracting the outdated CSI challenge in cooperative multi-satellite systems, where satellites act as distributed agents. Our contributions are as follows: 1. We address the challenge of channel aging in multi-satellite downlink communication to mobile devices. In constrast to [ 31 , 30 ] , our work bypasses channel prediction altogether by directly mapping the delayed CSI to the TPMs. Furthermore, in contrast to [ 20 ] , our approach is specifically designed for high-frequency scenarios and incorporates distributed TPM optimisation. This eliminates the reliance on a high-performance network controller and effectively distributes the processing load across satellites. 2. We propose a novel algorithm termed as Dual-Stage Proximal Policy optimisation (DS-PPO). Unlike the method in [ 18 ] , the DS-PPO algorithm is specifically designed for cooperative multi-satellite communications with non-iid environments. We introduce a bi-level optimisation framework for handling the complexity of the environment: In a given satellite, the first proximal policy optimisation (PPO) optimises the satellite’s TPM to maximise its individual sum-rate. The second PPO leverages the singular values of the first stage TPM across other satellites to optimise its TPM as part of a distributed satellite system. Sharing these singular values limits the information exchanged among satellites, enabling them to handle the non-iid environment through distributed learning. 3. We provide the computational complexity and the convergence analysis of the DS-PPO, proving that DS-PPO provides a performance improvement to the

Read the complete text