Adaptive Scheduling: A Reinforcement Learning Whittle Index Approach for Wireless Sensor Networks

Sokipriala Jonah 1 , Seong Ki Yoo 2 , Saurav Sthapit 3 · 2026

Read Adaptive Scheduling: A Reinforcement Learning Whittle Index Approach for Wireless Sensor Networks by Sokipriala Jonah 1 , Seong Ki Yoo 2 , Saurav Sthapit 3 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-01179.txt.

Introduction

Sokipriala Jonah, Seong Ki Yoo, and Saurav Sthapit schedule wireless sensors that must decide, moment by moment, which ones to check. Each sensor's state keeps changing whether it is checked or not, a setup known as a restless bandit problem. Classic solutions lean on the Whittle index, a score for how worthwhile checking each sensor is, but learning it well has needed careful hand tuning.

Their method learns the index by reinforcement learning and explores options using an optimism rule, with no problem-specific tuning. Tests cover standard benchmarks and a sensor task that tracks how stale each reading is. The method performs near the best known, while using far less memory and deciding in under a millisecond. For one test size it needed only about six hundred bytes of memory, where deep learning rivals needed hundreds of kilobytes or more. That economy suits small edge devices.

Catalog record

CREATORSokipriala Jonah 1 , Seong Ki Yoo 2 , Saurav Sthapit 3
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE67 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-01179

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Adaptive Scheduling: A Reinforcement Learning Whittle Index Approach for Wireless Sensor Networks Sokipriala Jonah 1 , Seong Ki Yoo 2 , Saurav Sthapit 3 Abstract We propose a reinforcement learning–based scheduling framework for Restless Multi-Armed Bandit (RMAB) problems, centred on a Whittle Index Q-Learning policy with Upper Confidence Bound ( Whittle index Q-Learning (WIQL) - upper confidence bound (UCB) ) exploration. Unlike existing approaches that rely on fixed or adaptive ϵ \epsilon -greedy strategies and require careful hyperparameter tuning, the proposed method eliminates problem-specific tuning and is therefore more generalisable across diverse RMAB settings. We evaluate WIQL - UCB on standard RMAB benchmarks and on a practical sensor scheduling application based on the Age of Incorrect Information (AoII), using an edge-based estimation scheme that requires no prior knowledge of system dynamics. Experimental results demonstrate that WIQL - UCB achieves near-optimal performance while significantly improving efficiency. At a representative problem size ( N = 15 , M = 3 N=15,M=3 ), the proposed method requires only about 600 bytes of memory, compared to several kilobytes for tabular Q-learning and hundreds of kilobytes to megabytes for deep reinforcement learning baselines. In addition, WIQL - UCB attains sub-millisecond per-decision runtimes and is several times faster than deep RL approaches, while maintaining competitive performance. These results demonstrate that WIQL - UCB consistently outperforms both non–Whittle-based and Whittle-index learning baselines across diverse RMAB settings. Version note: This manuscript is a substantially extended version of our earlier arXiv preprint [ 22 ] . The present version introduces a UCB-based Whittle index learning framework and includes a significantly expanded experimental evaluation across multiple RMAB problems, with systematic comparisons against both Whittle-index-based and non-Whittle-index policies. I Introduction In many remote monitoring applications, various sensors are deployed to track environmental conditions in applications such as smart homes, agricultural, and industrial monitoring. As sensor nodes are resource-constrained and typically battery-powered, continuous data collection and packet transmission consume significant amounts of energy. Additionally, frequent and indiscriminate transmissions increase storage requirements at the remote monitoring system without necessarily improving the accuracy of the state estimation. These challenges have led to the adoption of goal oriented scheduling techniques, which prioritise transmissions based on the value of the information they provide or the goal of the remote monitoring system [ 51 , 20 ] . In a standard application, the network objective is to transmit packets with high reliability from the transmitter to the receiver. For instance, consider a simple monitoring scenario where the goal is to keep the room temperature warm. In a traditional approach, all temperature readings would typically be transmitted. However, in a goal-oriented system, data is transmitted only when there is a change in the current state of the room. In this case, processing can be performed at the edge, on an Internet of things (IoT) or wireless sensor networks (WSNs) node where the data is generated. This enables the transmission of only information that is relevant to the underlying control or monitoring objective. Transmitting all raw sensing data in such scenarios is often highly inefficient and unnecessary, particularly in bandwidth- and energy-constrained industrial deployments. In [ 13 ] , an edge-mining approach termed Class-Act is proposed, in which accelerometer data are transmitted only when a change in activity state is detected, such as transitions between standing, sitting, or walking. By suppressing redundant transmissions during periods of unchanged behaviour, the approach significantly reduces communication overhead while preserving task-relevant information. Related goal-driven communication principles have also been applied to real-time tracking and control of autonomous systems, where transmissions are triggered only when necessary to maintain accurate state estimation or control performance [ 39 ] . Similarly, scheduling based on the value of information in wireless sensing systems has been investigated in [ 5 , 21 ] . These approaches prioritise task relevance over raw data fidelity and demonstrate that selective, goal-oriented information transmission can significantly improve efficiency under stringent communication constraints. A key challenge in this problem is that the sink (or gateway), which can only poll a subset of the sensors at each time step due to channel constraints, does not have perfect knowledge of the monitored process at the nodes. To address this, we propose a reinforcement learning-based polling strategy that selects the sensors expected to provide the most valuable updates, thereby optimising state estimation at the sink. We build upon the edge mining approach used in [ 13 ] , where raw sensor data is processed at the node side before transmission, ensuring that only meaningful information which improves the state estimation of the node at the sink side is sent to the sink. Building on this, we employ semantics-aware age of incorrect information (AoII) , which quantifies how outdated the state estimate at the sink is and serves as a metric for prioritising which sensors to poll. To solve this scheduling problem, we develop a reinforcement learning solution to learn the Whittle index, a technique that has been shown to yield near-optimal performance in scheduling problems where the transition probabilities of the underlying system are known. However, even under known transition probabilities, computing an optimal scheduling policy for selecting M out of N processes in systems with large state spaces remains computationally expensive. In contrast to previous works discussed in Section II , which assume known transition probabilities or complete knowledge of the system state at the sink, our approach makes no such assumptions. Although this increases the complexity of obtaining an optimal solution, it better reflects real-world sensor monitoring scenarios where state transitions are typically unknown. Consequently, we adopt the edge mining approach from [ 13 ] to directly estimate AoII . We use an online reinforcement learnin based approach to learn an optimal scheduling policy. Contrary to previous work, we conduct extensive experiments using synthetic data that represents typical sensor monitoring and control applications to evaluate the performance of our approach. We further validate the generalisability of the technique across diverse domains, including maintenance scheduling and preventive healthcare applications. Specifically, we propose a UCB -based Q-learning method to learn the WIQL policy, which does not require hyperparameter tuning in contrast to works in [ 11 , 3 ] . We compare the performance of our WIQL method with other Q-learning based Whittle index policies and state-of-the-art adaptive scheduling algorithms across various scenarios. The primary contributions of this work are as follows: • We present a practical technique for accurately estimating the AoII at the sink without relying on assumptions about the underlying node dynamics. This is accomplished using an edge mining approach that enables local processing at the data source. • We introduce WIQL - UCB , an asymptotically optimal learning algorithm that eliminates the need for manual hyperparameter tuning and achieves lower average AoII compared to existing techniques. • We demonstrate that WIQL - UCB generalises effectively across diverse application domains, outperforming state-of-the-art methods for online learning of the Whittle index. II Background and Related Works In a standard sensor monitoring process, particularly in less dense networks, packets are transmitted either at regular time intervals or in an event-based manner, depending on the occurrence of an event, usually in a distributed fashion. However, in large-scale sensor networks or Internet of Things (IoT) systems, as

Read the complete text