Learning to Focus: CSI-Free Hierarchical MARL for Reconfigurable Reflectors

Hieu Le, Jian Tao, Mostafa Ibrahim, Sabit Ekin, Oguz Bedir · 2026

Read Learning to Focus: CSI-Free Hierarchical MARL for Reconfigurable Reflectors by Hieu Le, Jian Tao, Mostafa Ibrahim, Sabit Ekin, Oguz Bedir (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-05165.txt.

Introduction

Le and four colleagues give a layered design that controls mirror panels without per-element channel guessing. The control problem splits into two tiers. A top unit assigns users to mirrors through discrete slots, while low-level units tune smooth focal points with MAPPO under central training, split runs. Indoor tests show signal gains up to 7.79 dB over central rivals. The system keeps aiming under sub-meter errors for up to four users and two mirror arrays.

Catalog record

CREATORHieu Le, Jian Tao, Mostafa Ibrahim, Sabit Ekin, Oguz Bedir
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE25872
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.05165

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Learning to Focus: CSI-Free Hierarchical MARL for Reconfigurable Reflectors Thanks: This material is based upon work supported by the U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing Research, Early Career Research Program under Award Number DE-SC-0023957. Hieu Le, Jian Tao, Mostafa Ibrahim, Sabit Ekin, Oguz Bedir Abstract Reconfigurable Intelligent Surfaces (RIS) has a potential to engineer smart radio environments for next-generation millimeter-wave (mmWave) networks. However, the prohibitive computational overhead of Channel State Information (CSI) estimation and the dimensionality explosion inherent in centralized optimization severely hinder practical large-scale deployments. To overcome these bottlenecks, we introduce a “CSI-free” paradigm powered by a Hierarchical Multi-Agent Reinforcement Learning (HMARL) architecture to control mechanically reconfigurable reflective surfaces. By substituting pilot-based channel estimation with accessible user localization data, our framework leverages spatial intelligence for macro-scale wave propagation management. The control problem is decomposed into a two-tier neural architecture: a high-level controller executes temporally extended, discrete user-to-reflector allocations, while low-level controllers autonomously optimize continuous focal points utilizing Multi-Agent Proximal Policy Optimization (MAPPO) under a Centralized Training with Decentralized Execution (CTDE) scheme. Comprehensive deterministic ray-tracing evaluations demonstrate that this hierarchical framework achieves massive RSSI improvements of up to 7.79 dB over centralized baselines. Furthermore, the system exhibits robust multi-user scalability and maintains highly resilient beam-focusing performance under practical sub-meter localization tracking errors. By eliminating CSI overhead while maintaining high-fidelity signal redirection, this work establishes a scalable and cost-effective blueprint for intelligent wireless environments. I Introduction The unprecedented surge in wireless traffic demand has driven conventional communication architectures to their theoretical limits [ 1 ] . In response, reconfigurable intelligent surfaces (RIS) have emerged as a transformative technology, turning the previously passive radio propagation medium into a dynamic, controllable environment. Conventional RIS architectures leverage electronically controlled phase shifters to induce constructive interference at receivers [ 1 , 2 ] . However, these systems depend critically on accurate channel state information (CSI) for each reflecting unit [ 3 ] . As deployments scale to hundreds of elements, the pilot overhead required for cascaded channel estimation becomes a prohibitive computational bottleneck, causing high spectral efficiency loss [ 4 ] . While deep reinforcement learning (DRL) and multi-agent reinforcement learning (MARL) have been increasingly adopted to address the optimization complexity of RIS coordination, most existing frameworks still mandate explicit channel estimation [ 5 , 6 , 7 ] . Efforts to relax this CSI dependence often require massive offline training datasets or the integration of dedicated sensing hardware into the RIS, which substantially increases system cost, power consumption, and hardware complexity [ 8 , 9 ] . To circumvent these fundamental limitations, we shift focus from electronic metasurfaces to mechanically reconfigurable metallic reflectors. Unlike electronic architectures that require complex RF circuitry, metallic reflectors offer inherent wideband operation and simplified mechanical actuation [ 10 , 11 , 12 ] . In our prior work, we established the physical foundation and control viability of this mechanical approach. We demonstrated that arrays of metallic flat reflectors, acting as linear Fresnel reflectors, can provide substantial coverage and gain enhancements in non-line-of-sight (NLOS) environments, offering a low-cost, frequency-versatile alternative to electronic phase-shifters [ 10 ] . Building upon this hardware design, we introduced a MARL framework to guide these reflector arrays, demonstrating that distributed control outperforms centralized DRL baselines in multi-user scenarios [ 12 ] . However, while standard MARL mitigates the dimensionality explosion of centralized approaches, coordinating simultaneous discrete user allocation and continuous beam-focusing across massive multi-reflector environments remains a formidable computational challenge. To overcome this remaining complexity bottleneck while leveraging the mechanical hardware, we propose a fundamentally different CSI-free methodology that eliminates pilot-based electromagnetic channel estimation. Instead of coordinating fine-grained electromagnetic interference, our framework exploits spatial awareness and readily available user localization data to manage macro-scale signal propagation in these NLOS environments. To manage the massive combinatorial complexity of joint user assignment and continuous control, we formulate the problem as a hierarchical multi-agent reinforcement learning (HMARL) framework [ 13 ] . The architecture decomposes the task into two temporal abstraction levels: a centralized high-level controller that performs discrete user-to-reflector allocations, and low-level controllers that autonomously optimize continuous focal points for their assigned users. Trained using multi-agent proximal policy optimization (MAPPO) [ 14 ] under a centralized training with decentralized execution (CTDE) paradigm [ 15 ] , this decomposition ensures rapid learning and practical deployment scalability. Furthermore, the practical deployment viability is highly supported by the widespread availability of specialized deep learning accelerators in modern communication systems [ 16 , 17 ] . In this paper, we demonstrate the efficacy of this CSI-free paradigm. The primary contributions are: • CSI-free optimization: A fully functional HMARL framework that eliminates CSI estimation overhead, utilizing spatial localization to achieve significant received signal strength indicator (RSSI) improvements over centralized baselines. • Hardware and algorithmic robustness: Comprehensive validation proving the framework’s resilience across practical sub-meter localization errors. II System Model and Problem Formulation Figure 1: Reflector Design. II-A Mechanically Reconfigurable Reflector & Hierarchical Coordination The system comprises an access point (AP) located at s\in\mathbb{R}^{3} , K user equipment (UE) devices, and L independent reflector segments operating in a NLOS millimeter-wave (mmWave) environment. Unlike standard electronic phase-shifters, the reflector is a mechanical device consisting of many small metallic tiles arranged in an N_{r}\times N_{c} grid (Fig. 1 ). To bypass the severe computational bottleneck of per-tile optimization, we optimize for beam-focusing with a focal point so that the reflector’s tiles focus energy toward that point. For a focal point f_{l}(t)\in\mathbb{R}^{3} associated with reflector segment l , the mechanical orientation of tile (i,j) at position r_{i,j} is deterministically governed by its normal vector: \vec{n}_{i,j}(f_{l})=\frac{1}{2}\left(\frac{f_{l}-r_{i,j}}{\|f_{l}-r_{i,j}\|_{2}}+\frac{s-r_{i,j}}{\|s-r_{i,j}\|_{2}}\right). (1) This geometric formulation allows us to derive the necessary elevation \theta_{i,j} and azimuth \phi_{i,j} angles without requiring instantaneous electromagnetic CSI. To manage the massive combinatorial complexity of a multi-user, multi-reflector environment, we decompose the control problem into a two-tier HMARL architecture. There are two levels: a high-level user–reflector assignment controller and low-level reflector focal point control agents. Operating at an extended timescale T , the high-level controller observes the global spatial state to determine the discrete user-to-reflector allocation b=\{b_{1},\dots,b_{L}\} . Given this assignment, the decentralized low-level controllers autonomously execute continuous focal-point displacements a_{l,t}=[\Delta f_{l,x},\Delta f_{l,y},\Delta f_{l,z}]^{T} (2) at every environmental timestep to maximize the signal strength for their assigned users. II-B Signal Propagation & Optimization Objective In the considered NLOS mmWave environment, the direct path between the AP and the users is assumed to be obstructed. Consequently, the controllable RSSI at user k relies entirely on the reflected paths facilitated by its assigned reflector segment. For a user k assigned to segment l under the high-level allocation b , the received power is formulated as: P_{r,k}(u_{k},f)=P_{t}\sum_{(i,j)\in\mathcal{S}_{b_{l}}}\left|h_{r,k}^{(i,j)}(u_{k},f)+h_{other,k}(u_{k})\right|^{2}, (3) where P_{t} is the transmit power, h_{r,k}^{(i,j)}(u_{k},f) represents the reflected channel coefficient from tile (i,j) as a function of the user, u_{k} , and focal point, f , locations, and h_{other,k}(u_{k}) accounts for other environmental propagation paths. Crucially, these coefficients are derived from deterministic ray tracing of a fixed propagation environment based on user and focal-point localization. The system objective is to maximize the aggregate received power across all K users. We define the system-wide performance metric at time step t as a state-dependent reward function: R_{sys}(s(t),b(t))=\sum_{k=1}^{K}P_{r,k}(u_{k}(t),f(t)).

Read the complete text