NS-Gym: Open-Source Simulation Environments and Benchmarks for Non-Stationary Markov Decision Processes

Nathaniel S. Keplinger, Baiting Luo, Iliyas Bektas, Yunuo Zhang, Kyle Hollins Wray, Aron Laszka, Abhishek Dubey, Ayan Mukhopadhyay · 2025

Read NS-Gym: Open-Source Simulation Environments and Benchmarks for Non-Stationary Markov Decision Processes by Nathaniel S. Keplinger, Baiting Luo, Iliyas Bektas, Yunuo Zhang, Kyle Hollins Wray, Aron Laszka, Abhishek Dubey, Ayan Mukhopadhyay (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-09646.txt.

Introduction

Keplinger and co-authors release an open toolkit for testing decision algorithms in worlds that keep changing. Their 2025 paper deals with non stationary decision processes, where the rules drift over time. Real agents face such drift from outside forces. Classic models assume fixed dynamics and can fail badly. A lack of shared test beds has slowed progress in this young field, they argue.

The toolkit, NS-Gym, plugs into a popular simulation framework. It separates the drifting world parameters from the agent's decision module. That split lets researchers swap drift patterns and agents freely. The paper reviews past work and packages the key problem types. Six known algorithms are benchmarked inside the toolkit. The authors hope standard interfaces will make results comparable across labs. Reproducible tests of adaptability are the stated goal.

Catalog record

CREATORNathaniel S. Keplinger, Baiting Luo, Iliyas Bektas, Yunuo Zhang, Kyle Hollins Wray, Aron Laszka, Abhishek Dubey, Ayan Mukhopadhyay
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-09646

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

NS-Gym: Open-Source Simulation Environments and Benchmarks for Non-Stationary Markov Decision Processes Nathaniel S. Keplinger, Baiting Luo, Iliyas Bektas, Yunuo Zhang, Kyle Hollins Wray, Aron Laszka, Abhishek Dubey, Ayan Mukhopadhyay Abstract In many real-world applications, agents must make sequential decisions in environments where conditions are subject to change due to various exogenous factors. These non-stationary environments pose significant challenges to traditional decision-making models, which typically assume stationary dynamics. Non-stationary Markov decision processes (NS-MDPs) offer a framework to model and solve decision problems under such changing conditions. However, the lack of standardized benchmarks and simulation tools has hindered systematic evaluation and advance in this field. We present NS-Gym, the first simulation toolkit designed explicitly for NS-MDPs, integrated within the popular Gymnasium framework. In NS-Gym, we segregate the evolution of the environmental parameters that characterize non-stationarity from the agent’s decision-making module, allowing for modular and flexible adaptations to dynamic environments. We review prior work in this domain and present a toolkit encapsulating key problem characteristics and types in NS-MDPs. This toolkit is the first effort to develop a set of standardized interfaces and benchmark problems to enable consistent and reproducible evaluation of algorithms under non-stationary conditions. We also benchmark six algorithmic approaches from prior work on NS-MDPs using NS-Gym. Our vision is that NS-Gym will enable researchers to assess the adaptability and robustness of their decision-making algorithms to non-stationary conditions. NS-Gym can be downloaded from: https://github.com/scope-lab-vu/ns_gym 1 Introduction Many real-world problems involve agents making sequential decisions over time under exogenous sources of uncertainty. Such problems exist in autonomous driving ( Kiran et al., 2021 ) , medical diagnosis and treatment ( Yu et al., 2021 ) , emergency response ( Mukhopadhyay et al., 2022 ) , vehicle routing ( Li et al., 2021 ) , and financial portfolio optimization ( Pendharkar and Cusatis, 2018 ) . We define an agent as an entity capable of computation that acts based on observations from the environment ( Kochenderfer et al., 2022 ) . Decision-making for such agents is widely modeled by Markov decision processes (MDPs), a general mathematical model for stochastic control processes. A canonical challenge in such problems, motivated by practical scenarios, is non-stationarity, where the distribution of environmental conditions can change over time. While non-stationarity has been well-explored from both control and decision-theoretic perspectives, several conceptual paradigms of non-stationarity exist, which lead to different mathematical formalisms for how the environmental parameters change and how the agent (and the control process) interacts with the changes. Ackerson and Fu (1970) provide one of the earliest conceptualizations of a system operating in “switching” environments, where the mean and covariance of the underlying process can change over time. Campo et al. (1991) formalize the switching process, where some environment parameters can change after a random sojourn time, as a sojourn-time-dependent Markov chain, which is semi-Markovian. Recent investigations of non-stationary stochastic control processes involve two major threads: the first problem deals with an agent trying to adapt to a single change in the environment (which can either be observed Pettet et al. (2024) or unobserved ( Luo et al., 2024 ) ); and the second problem models situations where environmental parameters can change continuously over time ( Lecarpentier and Rachelson, 2019 ) . In an orthogonal line of work, Chandak et al. (2020b) present a problem formulation where the agent’s goal is to maximize a forecast of future performance (of the control policy) instead of directly modeling the non-stationarity. Notably, these problem classes provide fundamentally different formalisms (or treatments) for non-stationarity. Indeed, not only are the formalisms different, but we point out another interesting observation from prior work on non-stationary stochastic control processes: while recent prior work on stationary Markov decision processes (MDP) use standard benchmark problems, e.g., by using the popular Gymnasium toolkit ( Towers et al., 2023 ) , there are no standard problems or benchmarks for non-stationary MDPs. For example, Lecarpentier and Rachelson (2019) evaluate non-stationarity using a custom non-stationary bridge environment (an abstract problem where an agent must navigate on a grid-based slippery maze where the properties of the surface change over time), Chandak et al. (2020b) use problems motivated by real-world applications such as recommendation systems and diabetes treatment, and Pettet et al. (2024) use well-known benchmark problems used for stationary MDPs (e.g., the cartpole problem from Gymnasium Towers et al. (2023) ) and introduce non-stationarity manually. In this paper, we identify key characteristics of non-stationary MDPs that affect decision-making, review prior work in this area, and present the first simulation toolkit specifically tailored for non-stationary MDPs. We argue that four key considerations affect decision-making in non-stationary MDPs, where environmental factors can change over time: what changes? how does it change? can the agent detect the change? can the agent know the updated parameter that has changed? These questions summarize the nature of the change and the key properties of modeling approaches from prior work. Based on these questions, we present NS-Gym (Non-Stationary Gym), the first collection of simulation environments for non-stationary MDPs. Inspired by the seminal work of Campo et al. (1991) , we segregate the evolution of the environmental parameters that characterize non-stationarity and the agent’s decision-making module. This modularization enables us to configure various components (and types) of non-stationary MDPs seamlessly. The NS-Gym toolkit is based on Python and is completely compatible with the widely popular Gymnasium framework. Instead of developing a new simulation environment from scratch, we build upon the existing Gymnasium toolkit due to its popularity and ensure that the large user base already familiar with Gymnasium can easily use NS-Gym (we keep all standard Gymnasium functionalities and interfaces intact). Specifically, we make the following contributions. We make the following contributions: 1. We present the first simulation toolkit for NS-MDPs that provides a tailored, standardized, and principled set of interfaces for non-stationary environments. 2. We identify canonical problem instances for decision-making in non-stationary environments, e.g., decision-making where the agent knows about the change but is not aware of exactly what the change is, or decision-making where the agent is aware of the change. 3. We present an overview of prior work on non-stationary decision-theoretic models and a programming interface that unifies prior work. 4. Our simulation framework extends the widely popular Gymnasium toolkit, thereby requiring minimal added efforts from researchers in online planning, reinforcement learning, and decision-making in using our toolkit. 5. We present the first set of benchmark results (and open-source implementations using NS-Gym) that compares six algorithmic approaches for solving NS-MDPs. 6. Our benchmark results are presented across a series of problem types in non-stationary environments. The rest of the paper is organized as follows. We begin by describing characteristics of NS-MDPs and prior work. Then, we identify canonical problem instances, describe our framework, and present a tutorial of how to use it. Finally, we present benchmark results using NS-Gym. 2 Characteristics of NS-MDPs and Prior Work Figure 1: An overall framework for non-stationary Markov decision processes. At time t , the agent observes the state s_{t}\in S and takes an action a\in A . The environment emits a reward signal r(s_{t},a) and transitions to the next state s_{t+1} . The transition and the reward are governed by parameters \theta , which do not necessarily have

Read the complete text