Dynamics of Resource Allocation in O-RANs: An In-depth Exploration of On-Policy and Off-Policy Deep Reinforcement Learning for Real-Time Applications

Manal Mehdaoui 1, Amine Abouaomar 1 · 2024

Read Dynamics of Resource Allocation in O-RANs: An In-depth Exploration of On-Policy and Off-Policy Deep Reinforcement Learning for Real-Time Applications by Manal Mehdaoui 1, Amine Abouaomar 1 (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-01839.txt.

Introduction

Open radio access networks let operators mix gear from many vendors, but someone must hand out spectrum and compute second by second. Manal Mehdaoui and Amine Abouaomar test two deep reinforcement learning styles for that job in their 2024 paper. The work is a deliberate replication of an earlier study, run to check whether its claims hold.

One learner, PPO, improves by trying fresh actions; the other, ACER, replays past experience to learn faster. Both beat plain greedy rules. The rerun confirmed the first study's pattern: PPO struck the better balance between energy use and user delay, while ACER settled into its stride sooner. The authors say the match strengthens trust in both methods for real-time network control.

Catalog record

CREATORManal Mehdaoui 1, Amine Abouaomar 1
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE22 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-01839

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Dynamics of Resource Allocation in O-RANs: An In-depth Exploration of On-Policy and Off-Policy Deep Reinforcement Learning for Real-Time Applications Manal Mehdaoui 1, Amine Abouaomar 1 Abstract Deep Reinforcement Learning (DRL) is a powerful tool used for addressing complex challenges in mobile networks. This paper investigates the application of two DRL models, on-policy and off-policy, in the field of resource allocation for Open Radio Access Networks (O-RAN). The on-policy model is the Proximal Policy Optimization (PPO), and the off-policy model is the Sample Efficient Actor-Critic with Experience Replay (ACER), which focuses on resolving the challenges of resource allocation associated with a Quality of Service (QoS) application that has strict requirements. Motivated by the original work of Nessrine Hammami and Kim Khoa Nguyen, this study is a replication to validate and prove the findings. Both PPO and ACER are used within the same experimental setup to assess their performance in a scenario of latency-sensitive and latency-tolerant users and compare them. The aim is to verify the efficacy of on-policy and off-policy DRL models in the context of O-RAN resource allocation. Results from this replication contribute to the ongoing scientific research and offer insights into the reproducibility and generalizability of the original research. This analysis reaffirms that both on-policy and off-policy DRL models have better performance than greedy algorithms in O-RAN settings. In addition, it confirms the original observations that the on-policy model (PPO) gives a favorable balance between energy consumption and user latency, while the off-policy model (ACER) shows a faster convergence. These findings give good insights to optimize resource allocation strategies in O-RANs. I Introduction Reinforcement Learning (RL) is an approach that examines the dynamic interaction between agents and environments, which aims to achieve the highest possible cumulative reward. The mix of RL with Deep Neural Networks (DNN) has led to the technique of Deep Reinforcement Learning (DRL) in addressing high-complexity problems. DRL includes two model types: on-policy and off-policy and they have different characteristics and advantages [1]. On-policy models like Proximal Policy Optimization (PPO), are known for stability and ease of implementation but suffer from data inefficiency. In contrast, off-policy models, for instance, Sample Efficient Actor-Critic with Experience Replay (ACER), show data efficiency but often face stability challenges [2]. The usage of both on-policy and off-policy DRL models in solving resource allocation problems within mobile networks has given a critical examination of their performance. While past works have applied singular DRL techniques for resource allocation in scenarios like cloud RANs and multi-tenant networks, a comparative analysis of on-policy and off-policy models in the context of real-time surveillance video resource allocation remains important [3]. Motivated by this research, our study is a replication study of ”On-Policy vs. Off-Policy Deep Reinforcement Learning for Resource Allocation in Open Radio Access Networks.”. Our goal is to investigate how effective the on-policy (PPO) and off-policy (ACER) DRL models are in the scenario of dynamic slicing within Open Radio Access Networks (O-RAN). It focuses on real-time surveillance video with the metric of latency as important for Quality of Service (QoS). The use of wireless video surveillance cameras (VSCs) in domains of public safety and intelligent transportation systems requires innovative solutions to meet the QoS requirements[4]. Many researchers addressed resource allocation in video surveillance, however, they mainly focused on a single type of DRL model without comparing the effectiveness of on-policy and off-policy approaches [5]. In reference [6], the authors examined video surveillance within construction sites, where computational and networking resources are frequently constrained. The proposal was an edge-based approach using a graph-assisted hierarchical Deep Q-Network (DQN) algorithm to ensure both dependable accuracy and minimal delay. The goal of this paper is to answer: Which type of DRL models, on-policy or off-policy, is more efficient in solving the resource allocation problem for real-time surveillance video in O-RANs? To answer this, we use the O-RAN architecture. This architecture is known for its openness, intelligence, and interoperability. The O-RAN architecture provides a suitable platform for hosting AI workflows, especially DRL models. Our contributions include the evaluation of PPO and ACER in finding an efficient network resource allocation strategy for real-time video surveillance in a dynamic slicing O-RAN system, considering both latency-sensitive and latency-tolerant users. We formulate the problem as a Mixed-Integer Programming (MIP) model, providing optimal bounds to assess the performance of on-policy and off-policy DRL models. The remainder of this paper is organized as follows: Section II provides a review of related work to gain more understanding and explore different approaches to our problem. Section III, which is the System Model talks about the O-RAN architecture and problem formulation. Section IV introduces the DRL-based resource allocation solution or the proposed solution, followed by the Evaluation and results in Section V, where we evaluate the models’ performance and results. Finally, we conclude our findings in Section VI. II Related Work In the topic of Dynamics of Resource allocation in O-RANs, many studies have explored and covered many techniques and approaches to find a solution to different problems and challenges found in this domain. In this paper [7] authors propose a UK-means-based clustering and deep reinforcement learning (DRL) approach to address localization uncertainties and dynamic traffic demands in 5G mmWave networks. Moreover, this study [8] examines how mixing on-policy Monte Carlo updates with off-policy, bootstrap Q-Learning updates affects the performance of deep reinforcement learning methods. The results point out partial improvements in some scenarios and give insight into reinforcement learning optimization strategies. Additionally, a study [9] investigates reinforcement learning optimization strategies, which contributes to the ongoing exploration of effective resource allocation techniques in O-RANs. Furthermore, an additional study [10] explores mobility-aware load balancing for optimizing O-RAN networks using machine learning algorithms. The proposed multi-agent multi-armed bandit for load balancing and resource allocation (mmLBRA) scheme aims to distribute loads across the network, to prevent congestion and improve the effective sum-rate performance of the O-RAN network. Simulation results show significant enhancements in network sum rate and load balancing compared to existing heuristic methods. These studies give insight into different methods of optimizing resource allocation in O-RANs, tackling challenges around QoS, network performance, and congestion prevention. III System Model A. System model In Fig1, we use a sliced O-RAN system for our replication study. This system is composed of N O-RUs linked to a pool of M virtualized O-DUs (vO-DUs) These vO-DUs, implemented as virtual network functions (VNFs), operate in a cloud environment. End-users with varying Quality of Service (QoS) needs can connect to a nearby O-RU, gaining access to network services through wireless communication links. Fig.1 System model We are considering two service types, which are distinguished by their latency characteristics. The first is a latency-sensitive service with a specified maximum latency constraint, and the second is a latency-tolerant service. Accordingly, two slices are established: one catering to VSCs for real-time video transmission via O-RAN to the 5G Core Network (CN) and the other to a Control Center (CC) like a police station for real-time monitoring; the other serving latency-tolerant users. These slices are overseen by an O-RAN cross-slice resource orchestrator hosted by the SMO. The set of VSCs is represented as W, and the set of latency-tolerant users is represented as V. The demands of VSC w and latency-tolerant user v

Read the complete text