Domain-driven Metrics for Reinforcement Learning: A Case Study on Epidemic Control using Agent-based Simulation
Rishabh Gaur Gaurav Deshkar Jayanta Kshirsagar Harshal Hayatnagarkar Janani Venugopalan Engineering for Research (e4r) Thoughtworks Technologies, Pune, INDIA · 2025
Read Domain-driven Metrics for Reinforcement Learning: A Case Study on Epidemic Control using Agent-based Simulation by Rishabh Gaur Gaurav Deshkar Jayanta Kshirsagar Harshal Hayatnagarkar Janani Venugopalan Engineering for Research (e4r) Thoughtworks Technologies, Pune, INDIA (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-05154.txt.
Introduction
Rishabh Gaur and four co-authors craft domain-shaped scorecards for learning algorithms that steer epidemic simulations, in a paper from twenty twenty-five. Agent-based models let each virtual person act by rules, and learning algorithms can tune the policies inside them. Judging those learners is hard because the systems are complex and random, and standard scores miss what matters in the field.
Their case study models masking, vaccination, and lockdown choices during a pandemic. The new metrics blend classic learning scores with measures drawn from epidemic practice. Tests across scenarios, such as uneven mask supply, show how domain rewards change the verdict on each policy. The authors pitch the scorecards for fairer comparison of control strategies.
Catalog record
| CREATOR | Rishabh Gaur Gaurav Deshkar Jayanta Kshirsagar Harshal Hayatnagarkar Janani Venugopalan Engineering for Research (e4r) Thoughtworks Technologies, Pune, INDIA |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-05154
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Domain-driven Metrics for Reinforcement Learning: A Case Study on Epidemic Control using Agent-based Simulation Rishabh Gaur Gaurav Deshkar Jayanta Kshirsagar Harshal Hayatnagarkar Janani Venugopalan Engineering for Research (e4r) Thoughtworks Technologies, Pune, INDIA Abstract For the development and optimization of agent-based models (ABMs) and rational agent-based models (RABMs), optimization algorithms such as reinforcement learning are extensively used. However, assessing the performance of RL-based ABMs and RABMS models is challenging due to the complexity and stochasticity of the modeled systems, and the lack of well-standardized metrics for comparing RL algorithms. In this study, we are developing domain-driven metrics for RL, while building on state-of-the-art metrics. We demonstrate our “Domain-driven-RL-metrics” using policy optimization on a rational ABM disease modeling case study to model masking behavior, vaccination, and lockdown in a pandemic. Our results show the use of domain-driven rewards in conjunction with traditional and state-of-the-art metrics for a few different simulation scenarios such as the differential availability of masks. \fancyhead \WSCpagesetup Gaur, Deshkar, Kshirsagar, Hayatnagarkar, and Venugopalan 1 INTRODUCTION Agent Based-Models (ABMs) and Rational Agent-Based Models (RABMs) are being used extensively as modeling tools in various fields, including economics ( \shortciteNP charpentier2021reinforcement, \citeNP arthur2021foundations, \shortciteNP cristelli2011critical), social sciences ( \shortciteNP bedson2021review, \citeNP peysakhovich2019reinforcement, \citeNP railsback2019agent, \shortciteNP hansen2019agent), and behavior studies ( \citeNP mabaso2021computationally, \shortciteNP kountouriotis2014agent, \shortciteNP taberna2020tracing), due to their ability to simulate complex systems. The development and optimization of ABMs and RABMs involve various modeling techniques and approaches, including rule-based models ( \shortciteNP schluter2019potential, \citeNP macal2009agent), reinforcement learning (RL) models ( \shortciteNP charpentier2021reinforcement, \shortciteNP lee2017agent, \shortciteNP jalalimanesh2017simulation), and network-based models \shortcite carley2006biowar and other AI models \shortcite cervantes2020artificial RL is increasingly becoming the preferred method for developing RABMs and optimization of ABMs policy, owing to its ability to handle complex decision-making problems in dynamic environments. However, assessing the performance of RL-based ABMs and RABMS models is challenging due to the complexity and stochasticity of the modeled systems, and the lack of well-standardized metrics for comparing RL algorithms ( \shortciteNP le2021metrics, \shortciteNP henderson2018deep, \shortciteNP chan2019measuring). Unlike standard RL applications, RL-based ABMs and RABMs involve multiple agents interacting with each other and the environment, making it difficult to assess the performance of individual agents or the overall system \shortcite SALTELLI201929. Moreover, RL-based ABMs can generate multiple optimal policies or trajectories, further complicating the comparison of different algorithms. Thus, there is a need for standardized metrics and evaluation procedures that can provide a fair and consistent assessment of RL-based ABMs and RABMs. In this study, we propose metrics to compare different RL algorithms used for the development and optimization of ABMs and RABMs. We illustrate our performance metrics using a case study on the public policy optimization of a rational agent-based epidemiological model for simulating sociological behavior during a COVID-19 epidemic. The remainder of the paper is structured as follows: Section 2 provides a review of the related literature, Section 3 describes the metrics for evaluating RL-based RABMs, followed by an epidemiology case study in Section 4 . We present the our case study on epidemic control using agent-based simulation and experimental results and finally, in Section 5 , we conclude the paper with a summary and discuss future work. 2 RELATED WORKS The optimization of the outputs of ABMs and RABMs has gained traction in recent years, as the use of these models for decision-making has grown. As there are no standardized metrics in ABMs and RABMs while in Reinforcement Learning (RL) space, there are some common practices to evaluate and compare RL algorithms by using plots or tables of average cumulative reward (average returns), standard deviation of rewards, maximum mean-reward and recently, maximum reward over a fixed number of runs (episodes) \shortcite henderson2018deep, oliveira2019difference, fernando2017pathnet, rusu2016progressive, parisotto2015actor, macua2017diff. In RL space, the main problems stem from a knowledge gap of hyper-parameters, misleading and unstandardized evaluation metrics for the RL algorithms \shortcite henderson2018deep, vamplew2011empirical. Therefore, researchers in RL space have explored different facets to unfold the black box nature of reinforcement learning and to compare and evaluate algorithms, e.g.: i) Explainability : Within this work, the authors analyze the RL algorithm’s interaction with the environment and understand the RL algorithm’s underlying characteristics and aptitude in a task which make reinforcement learning more transparent, understandable, trustable and debuggable [ 30 ] ii) Reliability : In this work, they proposed a set of metrics that measure reliability and aspects of the variability of RL algorithms. \shortcite chan2019measuring, iii) Algorithm completeness : Here, they propose an algorithm evaluation metric of completeness, which states that an algorithm is complete on an environment, if the only required inputs to the algorithm is the meta-information about the environment( the number of state features and actions) and without hyperparameter tuning algorithm should reliably solve multiple tasks at different environments. \shortcite jordan2020evaluating etc. Some researchers also tried comparing RL algorithms on basis of data efficiency, goal-changing handling capability, trajectories, and policies [ 3 ] . Other than the evaluation aspect, researchers are assessing existing reinforcement learning practices by numerous other factors i.e.: i) Reproducibility : For reproducibility, the authors focus on the difficulties faced by RL practitioners in reproducing a state-of-the-art deep RL method and similar results as most publications do not report all hyperparameters, proper in-depth implementation details, experimental setup, and evaluation methods for both baseline comparison methods and novel state-of-the-art work \shortcite henderson2018deep. ii) Experimentation techniques : Here, they are highlighting the issue of no standardized experimentation procedures in the RL community \shortcite henderson2018deep. iii) Result reporting : Here, researchers are pointing out one of the major issues in reinforcement learning literature which is the diversity of metrics, lack of significance testing, and statistical uncertainty leading to deceptive results reporting. ( \shortciteNP agarwal2021deep, \shortciteNP henderson2018deep, \shortciteNP chan2019measuring) iv) Usability of the algorithm across multiple environments including the time and effort spent in hyperparameter-tuning of an algorithm, and v) Computational tractability meaning that an RL practitioner should be able to run the procedure and repeat experiments found in the literature \shortcite jordan2020evaluating. In practice, RL algorithms are often evaluated and compared using one or the other metric based on reward, which is inadequate for a good comparison because of the volatile nature of the RL algorithm, environment stochasticity, and impact of noise on mean-reward ( \shortciteNP henderson2018deep, \shortciteNP chan2019measuring, \shortciteNP islam2017reproducibility). Reward alone may not provide a proper insight into an algorithm’s performance, it should be incorporated with domain knowledge to build more concrete evaluation metrics. Some researchers have tried comparing algorithms only using domain knowledge only \shortciteNP nagendra2017comparison but that is also inadequate for proper algorithm comparison. To the best of our knowledge, this is the first work towards evaluation metrics based on both domain knowledge and rewards. By considering both reward and domain knowledge, it adds more confidence and trust in the results. Our framework builds on this prior work by providing a set of metrics for analyzing the different RL algorithms used in RABMs and for optimizing ABMs outputs. 3 METRICS TO EVALUATE RL-BASED RABMs AND ABMs OPTIMIZATION Figure 1 shows the workflow of our algorithm ranking, it starts with the “simulation module” where we perform agent-based simulations for a small community of 1000 rational individuals. The “policy discovery module” is responsible for generating optimal policy to control