Multi-agent Robust and Optimal Policy Learning for Data Harvesting
Shili Wu, Yancheng Zhu, Aniruddha Datta, Sean B. Andersson · 2025
Read Multi-agent Robust and Optimal Policy Learning for Data Harvesting by Shili Wu, Yancheng Zhu, Aniruddha Datta, Sean B. Andersson (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-16490.txt.
Introduction
Shili Wu, Yancheng Zhu, Aniruddha Datta, and Sean Andersson tackle data harvesting by robot teams in 2025. Sensor nodes scattered over an area hold data that mobile agents must gather while flying to set end points. The agents steer smoothly, so the control problem is continuous.
The team trained the agents with a reinforcement learning method that adds penalty terms for breaking rules. A smoothing step at each state made the learned policy tougher against noise. Simulations checked speed and robustness across setups. The learned plans gathered data well while reaching the end points, and held up when conditions shifted.
Catalog record
| CREATOR | Shili Wu, Yancheng Zhu, Aniruddha Datta, Sean B. Andersson |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 26 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-16490
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Multi-agent Robust and Optimal Policy Learning for Data Harvesting Shili Wu, Yancheng Zhu, Aniruddha Datta, Sean B. Andersson Abstract We consider the problem of using multiple agents to harvest data from a collection of sensor nodes (targets) scattered across a two-dimensional environment. These targets transmit their data to the agents that move in the space above them, and our goal is for the agents to collect data from the targets as efficiently as possible while moving to their final destinations. The agents are assumed to have a continuous control action, and we leverage reinforcement learning, specifically Proximal Policy Optimization (PPO) with Lagrangian Penalty (LP), to identify highly effective solutions. Additionally, we enhance the controller’s robustness by incorporating regularization at each state to smooth the learned policy. We conduct a series of simulations to demonstrate our approach and validate its performance and robustness. I INTRODUCTION Wireless Sensor Networks (WSNs) play a significant role in many applications, including environmental monitoring [ 1 ] , smart cities [ 2 ] and surveillance [ 3 ] . WSNs are composed of a (relatively) large number of fixed sensor nodes deployed over a large geographical area. When it is impractical or infeasible to broadcast the sensor data out to users, a (relatively) small number of mobile agents, such as unmanned ground or aerial vehicles, can be used to move among the nodes and harvest their data, delivering it to a data sink. This approach can overcome energy, power, and computational limitations at the nodes [ 4 ] . At the same time, these agents have their own constraints, and as a result, it is important to determine efficient trajectories for the agents to perform their mission successfully. When the problem takes place in one dimension, it is possible to derive a provably optimal controller [ 5 ] . However, while there are applications where one dimension is sufficient [ 6 ] , in most practical situations, the agents and sensor nodes operate in a higher dimensional space. While there has been significant effort in addressing optimal data harvesting in higher-dimensional spaces, it is a far more challenging problem. This is primarily due to the lack of a globally optimal parameterization of a controller that would allow for straightforward optimization [ 7 ] . An alternative to the global parameterization approach is to formulate the problem as a Combinatorial Optimization (CO) problem. In this approach, the agent performs rollouts using a model and applies a search technique to identify the optimal action [ 8 ] . However, the high dimensionality of the data harvesting problem makes it computationally infeasible for the agent to explore all possible paths, as the time complexity of such an algorithm is O(b^{H}) , where b represents the size of the action space and H is the planning horizon. To overcome this challenge and reduce the complexity, a more feasible strategy is to use random rollouts combined with a reinforcement learning approach [ 9 , 10 ] . In our previous work, we developed an RL-based controller that directly optimized the binary reward function [ 10 ] . This approach allowed us to avoid the need for reward engineering and reliance on intuitive approaches to formulate the reward function to produce a “desired result” that is not encoded in the rewards. It was limited, however, to a single agent with a finite number of available control actions and ignored issues of measurement noise, modeling error, and disturbances. In practice, the use of multiple agents can be beneficial, especially when the WSN is distributed over a large area and the control actions are better modeled as bounded and continuous. In this paper, we build upon our previous effort, allowing multiple agents to do data harvesting simultaneously and move away from a discrete representation of the available controls into a continuous action space. When considering data harvesting in practical scenarios, it is important to recognize that the system is subject to noise and disturbances, both in the measurements and in the control, arising from environmental factors and modeling errors. Particularly when applying learning-based methods, these disturbances can lead to undesirable behavior, such as policy vibrations, that lead to poor performance. One can address this in the learning process by, for example, modifying the critic in an actor-critic formulation to account for the noise and ensuring disturbances are modeled in the sampling process during learning [ 11 ] . However, these approaches typically assume a Gaussian distribution on the noise, an assumption that may not hold in practice. We take a different approach based on network smoothing. Smoothing has been shown to be an effective method for finding a robust estimator in supervised learning [ 12 ] , as well as for finding a robust controller in trajectory tracking tasks [ 13 ] . Our goal in this paper is to find a control policy over a continuous action space for a collection of agents so as to harvest all the data from a set of target sensor nodes in a two-dimensional environment in minimal time and in a way that is robust against disturbances. The primary contributions of our work are: • A deep reinforcement solution to the time-optimal, two-dimensional, multi-agent data harvesting problem. • An exploration of the trade-off between different terminal penalty assignments that shows the advantage of using a Lagrangian Penalty in terms of finding a higher-performing solution. • A learned policy that exhibits robustness to disturbances through the use of regularization for policy smoothing. Through simulations, we demonstrate improvement over both our prior work and a classical shortest path finding algorithm as a baseline. We also demonstrate the robustness of the scheme with respect to disturbances in the measured state. Note that while we focus on the data harvesting problem, our approach can also be applied to similar multi-agent control problems like pickup and delivery or persistent monitoring. II Problem Formulation II-A System Design As illustrated in Fig. 1 , we consider a collection of M_{s} sensor nodes (also referred to as targets ) located at positions s_{i}\in\mathbb{R}^{2} , i=1,2,\dots,M_{s} . The initial data volume to be harvested from each target i is given by D_{i}\in\mathbb{R}_{\geq 0} where \mathbb{R}_{\geq 0} denotes the set of non-negative real numbers. The joint initial data volume collection is denoted as \mathbf{D}\in\mathbb{R}^{M_{s}}_{\geq 0} . Harvesting is performed by M_{a} agents flying at a constant height h over the workspace. Agent j ( j=1,\dots,M_{a} ) starts at location p_{start}^{j}\in\mathbb{R}^{2} , and must end at p_{final}^{j}\in\mathbb{R}^{2} . For example, the final location may be at a data sink so that the agent can offload the acquired data. The joint position of the agents is given by \mathbf{p}\in\mathbb{R}^{2M_{a}} . Fig. 1: Illustration of the data harvesting problem. Multiple agents, flying at constant heights, harvest data from sensor nodes to bring the information back to data sinks for use. The circles represent the effective communication range. We model the data transmission rate \psi_{ij} between target i and agent j by combining the Shannon-Hartley Theorem [ 14 ] with the Friis Transmission Formula [ 15 ] , yielding \psi_{ij}(p_{j},s_{i})={B}\log_{2}\left(1+\frac{K}{\parallel p_{j}-s_{i}\parallel_{2}^{2}}\right), (1) where B is the bandwidth of the channel, and K is a constant proportional to the output power of