Reinforcement Learning for Machine Learning Model Deployment: Evaluating Multi-Armed Bandits in ML Ops Environments
S. Aaron McClendon, Vishaal Venkatesh, Juan Morinelli · 2025
Read Reinforcement Learning for Machine Learning Model Deployment: Evaluating Multi-Armed Bandits in ML Ops Environments by S. Aaron McClendon, Vishaal Venkatesh, Juan Morinelli (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-22595.txt.
Introduction
McClendon and co-authors test whether bandit algorithms can run machine learning model deployment better than fixed rules, in a 2025 paper. Shops today promote or roll back models by hand checked scores and split tests. Drift and sudden drops demand faster hands.
Six selection strategies faced off on two real world data sets. Bandit driven managers watched live performance and swapped out weak models in real time. They matched or beat the traditional methods. The authors argue such automation cuts manual toil and lowers the risk of a bad model lingering in production. Deployment joins training as a place where learning helps.
Catalog record
| CREATOR | S. Aaron McClendon, Vishaal Venkatesh, Juan Morinelli |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 65 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-22595
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Reinforcement Learning for Machine Learning Model Deployment: Evaluating Multi-Armed Bandits in ML Ops Environments Thanks: Preprint Citation : McClendon, S.A., Venkatesh, V., Morinelli, J. (2025). Reinforcement Learning for Machine Learning Model Deployment: Evaluating Multi-Armed Bandits in ML Ops Environments. Aimpoint Digital Labs Technical Report. S. Aaron McClendon, Vishaal Venkatesh, Juan Morinelli Abstract In modern ML Ops environments, model deployment is a critical process that traditionally relies on static heuristics such as validation error comparisons and A/B testing. However, these methods require human intervention to adapt to real-world deployment challenges, such as model drift or unexpected performance degradation. We investigate whether reinforcement learning, specifically multi-armed bandit (MAB) algorithms, can dynamically manage model deployment decisions more effectively. Our approach enables adaptive production environments by continuously evaluating deployed models and rolling back underperforming ones in real-time. We test six model selection strategies across two real-world datasets and find that RL based approaches match or exceed traditional methods in performance. Our findings suggest that reinforcement learning (RL)-based model management can improve automation, reduce reliance on manual interventions, and mitigate risks associated with post-deployment model failures. K eywords ML Ops \cdot Reinforcement Learning \cdot Online Learning 1 Introduction Machine learning (ML) models are only as valuable as their ability to perform well in production environments. However, selecting and deploying models in real-world settings presents significant challenges, particularly in ML Operations (ML Ops) . A core issue in ML Ops is performance degradation , often caused by shifts in data distribution over time, commonly referred to as drift [ 1 ] . Drift can manifest in various forms, including concept drift and data drift . Concept drift occurs when the statistical properties of the target variable change over time, affecting the relevance of previously learned patterns. A canonical example is spam detection: as adversarial actors adapt their strategies to evade spam filters, models trained on historical spam patterns may gradually lose effectiveness [ 2 ] . Data drift , on the other hand, refers to shifts in the underlying distribution of input features, leading to discrepancies between training and production data. This phenomenon is well-documented in ML literature [ 3 ] , and it poses a major challenge for models deployed in dynamic environments. These factors contribute to a critical issue in ML Ops: a model that performs well in an offline evaluation setting may fail to generalize effectively once deployed . Despite substantial research on mitigating model drift in domains such as predictive maintenance [ 2 ] and healthcare [ 4 ] , many ML Ops workflows still rely on static deployment heuristics that fail to dynamically adjust to real-world conditions. Currently, model deployment decisions in ML Ops are typically made using fixed statistical methods , such as A/B testing [ 5 , 6 ] or simple validation set error comparisons [ 3 ] . These approaches introduce limitations: • They assume that the distribution of test data remains consistent with the training environment, which is often not the case [ 4 ] . • Once a model is deployed, its performance can be sparsely reassessed in real time , often requiring manual intervention and extended delays before performance issues are identified and corrected [ 7 ] . These challenges raise fundamental questions: When should we deploy a newly trained model into a production environment? How long should we wait until we determine we have a reliable indicator of real world performance? Despite these challenges, reinforcement learning-based approaches to model deployment remain largely unexplored in ML Ops research. In this work, we investigate whether reinforcement learning (RL) [ 8 ] methods, specifically multi-armed bandits (MABs) [ 9 ] , can replace traditional static deployment strategies in ML Ops. Unlike static approaches, MABs dynamically balance exploration (deploying new models) and exploitation (favoring models with strong real-world performance) [ 9 ] , allowing for continuous optimization of model deployment [ 10 ] . Through empirical evaluation on two real-world datasets— census wage prediction and fraud detection —we compare RL-based model deployment to industry-standard methods such as validation error comparisons and A/B testing. Our findings suggest that RL-based model selection can offer significant improvements in adaptability, reducing reliance on manual monitoring and mitigating the risks associated with model drift . 2 Background Effective model deployment in ML Ops requires a structured approach to evaluating and selecting machine learning models for production. Traditional deployment strategies rely on statistical heuristics such as validation set error comparisons and A/B testing, which assume that a model’s offline performance is indicative of its real-world effectiveness. However, these approaches fail to dynamically adjust to changing data distributions and model drift, often leading to performance degradation over time. Reinforcement learning (RL) methods, particularly Multi-Armed Bandits (MABs) [ 9 ] , provide an alternative framework for adaptive model selection by continuously balancing exploration of new models with exploitation of known high-performing models [ 9 ] . In this study, we compare RL-based [ 8 ] model deployment strategies against several commonly used baselines, including naive model replacement, validation error heuristics, and A/B testing. To provide context for the experimental results, this section outlines each of these deployment strategies, explaining their theoretical foundations, industry relevance, and associated advantages and limitations. This discussion serves as a foundation for understanding how reinforcement learning-based approaches differ from traditional ML Ops methodologies and why they may offer improvements in adaptive model selection. 2.1 Naïve Model Deployment One of the simplest strategies for model deployment in ML Ops is the naïve method , in which every newly trained model is automatically deployed to production, replacing the previous version. This approach is frequently used in industry due to its simplicity and automation-friendly workflow, eliminating the need for manual approval or extensive evaluation. Organizations following continuous integration/continuous deployment (CI/CD) practices sometimes adopt this method to ensure models are always using the most recent data and training methodologies. Despite its ease of implementation, the naïve method carries significant risks. Because it assumes that a newer model will always perform better than its predecessor, it fails to account for model drift, overfitting to recent data, or unexpected distribution shifts in production. Without safeguards, this approach can lead to sudden performance degradation if a newly deployed model generalizes poorly to real-world data. Moreover, organizations using the naïve approach often lack robust rollback mechanisms, making it difficult to recover from unintended performance regressions. While the naïve method is observed in industry, there is limited academic literature explicitly documenting its usage. However, its prevalence can be inferred from discussions in ML Ops best practices, where automated deployment pipelines are a common feature [ 3 , 7 ] . This method serves as a baseline for comparison against more adaptive deployment strategies, such as A/B testing and reinforcement learning-based approaches. 2.2 Validation Testing In the validation testing deployment strategy, model selection is based on comparative performance using historical validation scores. Each deployed model is associated with a recorded validation metric, which serves as a benchmark for future model iterations. During each retraining cycle, historical data is partitioned into training and validation sets. Let \mathcal{D}_{t}=\{(x_{i},y_{i})\}_{i=1}^{N_{t}} represent the dataset available at time t , where x_{i} are input features and y_{i} are target labels. This dataset is split into: \mathcal{D}_{t}^{\text{train}},\mathcal{D}_{t}^{\text{val}}\sim\mathcal{D}_{t},\quad\mathcal{D}_{t}^{\text{train}}\cap\mathcal{D}_{t}^{\text{val}}=\emptyset (1) where