DRTA: Dynamic Reward Scaling for Reinforcement Learning in Time Series Anomaly Detection PubID: pubid: 2025 Conference on AI, Science, Engineering, and Technology (AIxSET)
1 st Bahareh Golchin, 2 nd Banafsheh Rekabdar, 3 rd Kunpeng Liu · 2025
Read DRTA: Dynamic Reward Scaling for Reinforcement Learning in Time Series Anomaly Detection PubID: pubid: 2025 Conference on AI, Science, Engineering, and Technology (AIxSET) by 1 st Bahareh Golchin, 2 nd Banafsheh Rekabdar, 3 rd Kunpeng Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-18474.txt.
Introduction
Bahareh Golchin, Banafsheh Rekabdar, and Kunpeng Liu teach a learning agent to spot odd points in time series. Their 2025 paper notes that classic methods drown in false alarms and scarce labels. Their system, DRTA, joins reinforcement learning with a variational autoencoder and active learning. A reward rule shifts with the task, balancing search against certainty.
The reward blends rebuild error from the encoder with class feedback. That lets the agent learn from few labeled cases while keeping precision and recall high. On two Yahoo benchmark sets, the method beat top unsupervised and semi-supervised rivals. The authors present it as a scalable fix for watch tasks in finance, health, sensors, and industry.
Catalog record
| CREATOR | 1 st Bahareh Golchin, 2 nd Banafsheh Rekabdar, 3 rd Kunpeng Liu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-18474
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DRTA: Dynamic Reward Scaling for Reinforcement Learning in Time Series Anomaly Detection PubID: pubid: 2025 Conference on AI, Science, Engineering, and Technology (AIxSET) 1 st Bahareh Golchin, 2 nd Banafsheh Rekabdar, 3 rd Kunpeng Liu Abstract Anomaly detection in time series data is important for applications in finance, healthcare, sensor networks, and industrial monitoring. Traditional methods usually struggle with limited labeled data, high false-positive rates, and difficulty generalizing to novel anomaly types. To overcome these challenges, we propose a reinforcement learning-based framework that integrates dynamic reward shaping, Variational Autoencoder (VAE), and active learning, called DRTA. Our method uses an adaptive reward mechanism that balances exploration and exploitation by dynamically scaling the effect of VAE-based reconstruction error and classification rewards. This approach enables the agent to detect anomalies effectively in low-label systems while maintaining high precision and recall. Our experimental results on the Yahoo A1 and Yahoo A2 benchmark datasets demonstrate that the proposed method consistently outperforms state-of-the-art unsupervised and semi-supervised approaches. These findings show that our framework is a scalable and efficient solution for real-world anomaly detection tasks. 1 1 1 Code is available at GitHub Repository . I Introduction Anomaly detection in time series is crucial across various domains, including data centers, sensor networks, cyber-physical systems, healthcare, demand forecast, and finance [ 27 ] , [ 25 ] , [ 8 ] , and [ 31 ] . Most existing algorithms in the literature require manually tuning the problem parameters and selecting features. However, this is often tailored to specific data characteristics. Additionally, identifying anomalies manually is both time-consuming and labor-intensive, and it is prone to human error. In today’s world, given the vast amounts of data involved, an automated system is essential for efficiently detecting anomalies in large-scale time series datasets [ 5 ] . In practice, anomaly detection faces two key challenges: first, anomalies are rare. Therefore, it is difficult to train models effectively. Second, real-world data is often time-dependent. To address the former challenge, many anomaly detection algorithms have been developed, which use unsupervised approaches that do not rely on labeled data. However, these methods are not generic. They often operate based on specific assumptions about anomaly patterns in the data, and these assumptions may not always hold in real-world scenarios, which would result in high false-positive rates. This issue stems from diverse user perspectives and varying definitions of anomalies [ 18 ] , and [ 35 ] . To overcome this issue, supervised methods have also been developed to detect anomalies, and they perform well when ample labeled data is available. However, they struggle in situations with limited or no labels. Even with labeled data, these algorithms assume a stable underlying distribution. When the distribution is shifted for any reasons, they require retraining to maintain accuracy [ 34 ] . Moreover, semi-supervised learning algorithms have also been developed to detect anomalies in time series datasets. However, these methods are limited in scenarios where diverse anomaly types are present. The reason is that semi-supervised learning algorithms rely on a small set of known anomalies, and they may fail to identify novel anomalies in unlabeled data. Consequently, they are ineffective at detecting new types of anomalies [ 14 ] . To address the aforementioned challenges of weakly-supervised anomaly detection in time series data, in this paper, we propose using Deep Reinforcement Learning (DRL). A key aspect of Reinforcement Learning (RL) is the balance between exploration and exploitation [ 28 ] . Exploitation refers to selecting actions based on existing knowledge to maximize rewards. Exploration involves trying new actions to discover potentially better strategies. The agent must continuously navigate this trade-off to balance the immediate optimal decisions with the need to acquire more knowledge. The DRL model in our study is designed to effectively use a limited set of labeled anomalous data ( D_{la} ) while extensively exploring a large pool of unlabeled data ( D_{u} ). This approach enables us to detect new anomaly types not present in the labeled dataset. The exploration process is improved by incorporating a Variational Autoencoder (VAE), which strengthens the DRL framework. Additionally, given the high cost and limited availability of fully labeled data in real-world scenarios, our system uses active learning. This allows the RL agent to: 1) efficiently explore the environment and gather experience, and 2) make informed decisions based on the knowledge gained during exploration. Finally, the core of our DRL agent is a Long Short-Term Memory (LSTM) network, which 1) models sequential time series data and 2) captures long-term dependencies between events [ 10 ] . To improve the learning efficiency of the RL agent, we use the novel idea of reward shaping, which provides additional guidance to the reinforcement learning process. By designing a well-structured reward function, we help the RL agent learn meaningful patterns more quickly. This reduces the time needed to converge to an optimal policy. Reward shaping enables the agent to receive intermediate feedback. This is particularly helpful in complex environments where rewards are sparse or delayed. This approach prevents the agent from relying only on trial-and-error exploration. It instead encourages the agent to follow a more informed learning trajectory, which ultimately improves anomaly detection performance in time series data [ 22 ] . To summarize, our key contributions include the following. • Dynamic Reward Mechanism for Adaptive Exploration and Exploitation . This study introduces a novel dynamic reward shaping mechanism that adaptively balances exploration and exploitation, enabling the RL agent to achieve robust anomaly detection performance. • Efficient Active Learning Integration with Adaptive Querying . Our proposed method improves active learning by dynamically querying the most uncertain samples during training. This, combined with adaptive reward shaping, reduces the need for labeled data. • Unified Framework for Semi-Supervised Anomaly Detection . By integrating RL with the dynamic reward function, active learning, and VAE, this study proposes a unified framework that effectively handles low-label systems. This paper is structured as follows: Section II surveys relevant literature, Section III establishes the theoretical foundation for time series anomaly detection, Section IV describes our proposed methodology, Section V analyzes the implementation process, and Section VI presents our conclusions. II Related Work In this section, we review machine learning-based approaches in the literature for time series anomaly detection. II-A Machine Learning-Based Anomaly Detection Machine learning approaches address the limitations of statistical methods by using labeled training data to distinguish between normal and anomalous instances through classification or clustering techniques. Supervised and Unsupervised Learning . Commonly used algorithms include Bayesian networks [ 3 ] , support vector machines (SVM) [ 19 ] , rule-based systems [ 30 ] , and neural networks [ 23 ] . Clustering techniques, such as k-means, have also been used for anomaly detection [ 26 ] . Reinforcement Learning in Anomaly Detection . More recently, RL has been used in anomaly detection due to its ability to learn from interactions with the environment. Bourdonnaye et al. proposed using convolutional autoencoders within an RL framework to detect anomalies [ 2 ] . Huang et al. introduced a value-based DRL approach using the Deep Q-Network (DQN) algorithm [ 13 ] . Furthermore, reward shaping has emerged as an important technique in RL, particularly for enhancing anomaly detection