Adaptive queue management in healthcare using supervised Q-learning with time-varying reward and cost structures Authors: Kartick Bag, Raj Panda, Sweety Suman Jha, Arista Lahiri, Debjani Chakraborty Published: 2026 ABSTRACT Efficient patient management in hospitals requires adaptive decision-making under time-varying demand and dynamic service environments. This study proposes a heterogeneous medical patient queueing model that integrates reinforcement learning with stochastic queue dynamics to minimize overall patient waiting time. The model distinguishes between two categories of service providers (SPs): those attending first-time patients and those serving returning patients. Each category may differ in service rate but not in medical specialty. Patient arrivals follow a non-homogeneous Poisson process (NHPP) to capture realistic time-dependent flow variations. A Q-learning framework with a supervised ε-greedy policy is developed to determine optimal operational actions, such as adding or reallocating service providers, based on system state and event type. Separate Q-tables are maintained for arrival and departure events to account for differing cost and reward dynamics. Simulation results demonstrate that the proposed model significantly reduces total waiting time and system cost compared with conventional homogeneous queue models. This approach provides a data-driven mechanism for dynamic hospital queue management and can be extended to broader healthcare resource optimization scenarios. Introduction Effective patient management is one of the most critical challenges in modern healthcare systems, where fluctuating patient demand, limited resources, and service heterogeneity often lead to congestion and prolonged waiting times. There in outpatient departments (OPDs), the arrival rate of patients varies significantly throughout the day due to various external factors. In traditional queuing models, particularly those assuming homogeneous types service structures and constant Poisson arrivals, often fail to capture this dynamic and heterogeneous nature of the real-world hospital operations. Consequently, these types of models may not offer sufficient decision support for adaptive workforce management or resource allocation in the face of evolving circumstances. To address these types of challenges, queuing-based learning models gained increasing attention in healthcare operations research. Reinforcement learning (RL), specifically Q-learning, offers a data-driven mechanism to optimize sequential decisions under uncertainty without requiring complete knowledge of the system dynamics. Several studies have been applied in RL to healthcare service management, including appointment scheduling, dynamic staffing, and emergency response optimization. However, several existing approaches rely on simplified service and arrival assumptions, which may limit their adaptability in dynamic medical environments. Moreover, few models explicitly account for time-varying arrival rates or distinct service patterns associated with different patient types, both of which are fundamental to real-life OPD operations scenarios. The problem of controlling queueing systems with heterogeneous servers has been extensively studied in operations research literature. Early foundational work by Larsen1 conjectured that the optimal service policies in heterogeneous server systems could be characterized by threshold-type structures. This conjecture was subsequently verified for systems with two heterogeneous servers using policy iteration techniques2, value iteration methods3, and sample-path arguments4. Attempts to extend these results to systems having multiple heterogeneous servers were made by Rykov5, but the proposed proof was later shown to be incomplete by De Vericourt and Zhou6. Similar concerns were expressed with alternative formulations7. Recent studies emphasize that the characterization of optimal control policies for multi-server heterogeneous queueing systems remains an open and challenging problem8. Due to the analytical intractability of such systems, particularly under stochastic arrival processes and service times that follow a general distribution, learning-based and approximate dynamic programming approaches have emerged as practical alternatives. Learning-based approaches have been successfully applied to a range of queueing and service system problems, including network scheduling9, inventory control10, and emergency service deployment11. Comprehensive surveys further illustrate the increasing significance of RL in queueing and service systems12,13. Such methods are particularly attractive in settings where system dynamics are complex or partially unknown, as they enable adaptive policy learning without relying on explicit analytical characterizations14. In healthcare operations, queueing-based models have been widely employed to analyze patient flow, waiting times, and resource utilization in outpatient departments and emergency units. Several studies have emphasized the importance of capturing time-varying patient arrival patterns using non-homogeneous Poisson processes (NHPPs)15,16 to reflect peak-hour congestion and off-peak variability17,18. However, most existing RL-based healthcare queueing models continue to rely on homogeneous service assumptions or stationary arrival processes. Relatively limited attention has been given to integrating time-dependent arrival dynamics within a unified reinforcement learning framework, particularly in OPD environments where patient types and service requirements differ substantially. Here, in this work, we propose a heterogeneous medical patient queuing model that integrates Q-learning with a non-homogeneous Poisson process (NHPP)-based arrival framework guided by a heuristic supervised ε-greedy policy to optimize long-term operational costs and system efficiency. This proposed model captures the dynamic interplay between patient inflow and service provider (SP) allotment through a reinforcement learning (RL)-based adaptive policy. Service providers are categorized in two different functional groups: those attending the first-time patients (FP-SPs) and those serving the returning or follow-up patients (RP-SPs). Each server from each server category operates with distinct service rates, reflecting the practical differences in consultation or service duration, thereby introducing heterogeneity into the system. The arrival process is modeled as a time-dependent NHPP, described by a rate function λ (t), which varies with time to replicate realistic patient inflow patterns in OPDs. This allows the model to represent the peak-hour congestion and off-peak idle periods within a single stochastic framework. By coupling NHPP arrivals with a Q-learning–based decision mechanism, the proposed model dynamically adjusts operational actions to minimize the long-term total system cost, which encompasses both waiting and operational components. The reinforcement learning framework is designed to autonomously learn optimal policies through trial-and-error interactions with the environment. Specifically, the agent selects actions such as adding new service providers, swapping between FP-SP and RP-SP queues, or maintaining the current configuration, depending on the event type and current system state. Unlike conventional approaches, the proposed learning strategy employs a supervised ε-greedy policy, where exploration decisions are guided by event-specific cost evaluations rather than purely random selection. This structure enhances learning stability and accelerates convergence toward optimal queue management policies. Comprehensive simulation experiments were conducted to evaluate the proposed framework. Results demonstrate that the Q-learning–based heterogeneous queue significantly reduces total operational cost compared with a homogeneous baseline model under equivalent conditions. The adaptive nature of the learned policy enables the system to respond effectively to fluctuating patient arrival rates, balancing service efficiency and resource utilization. The findings highlight the potential of reinforcement learning–based adaptive queue management for improving healthcare service quality and operational resilience in OPD environments. The primary contributions of this study can be summarized as follows: A novel heterogeneous queueing model that distinguishes service providers based on patient type (first-time or returning) while incorporating time-varying arrival dynamics using an NHPP framework.An adaptive Q-learning mechanism with a supervised ε-greedy policy to optimize decisions across heterogeneous event types.Simulation-based validation showing measurable improvement in total system cost and waiting performance compared with traditional homogeneous queue models.The remainder of this paper is structured as follows. The Results section presents the simulation outcomes and analytical observations. The Discussion section then provides an in-depth discussion of the findings and their implications. The detailed model formulation, including system structure, arrival–service dynamics, assumptions, and the Supervised Q-learning (SQL) framework, is described in the Methods section. Finally, key references supporting this work are listed at the end of the paper. A novel heterogeneous queueing model that distinguishes service providers based on patient type (first-time or returning) while incorporating time-varying arrival dynamics using an NHPP framework. A novel heterogeneous queueing model that distinguishes service providers based on patient type (first-time or returning) while incorporating time-varying arrival dynamics using an NHPP framework. An adaptive Q-learning mechanism with a supervised ε-greedy policy to optimize decisions across heterogeneous event types. An adaptive Q-learning mechanism with a supervised ε-greedy policy to optimize decisions across heterogeneous event types. Simulation-based validation showing measurable improvement in total system cost and waiting performance compared with traditional homogeneous queue models. Simulation-based validation showing measurable improvement in total system cost and waiting performance compared with traditional homogeneous queue models. Results In this section, key outcomes are obtained from the heterogeneous Q-learning-based hospital queue management framework. The analysis focuses on three components: (i) the learned action-value landscape, (ii) the emergent optimal service provider (SP) allocation policy across patient queue states, and (iii) comparative cost performance under time-varying arrival intensity. The results are obtained through simulations for scenarios comprising a maximum of 10 and a maximum of 30 (F and/or R) patients, and the total number of maximum SPs is 5 (FP-SP and/or RP-SP). The results for the simulation with a maximum of 30 patients are depicted in Table 1, which is obtained following the simulation with a maximum of 10 patients (see Supplementary ).Table 1Representative sections of the learned Q-tables for different arrival mix scenarios (maximum 30 patients).012345678⋯10⋯60(a) Arrival mix 30:700-5.5647ϕϕϕϕϕϕϕϕ⋯ϕ⋯ϕ1ϕ-3.7015ϕϕϕϕϕϕϕ⋯ϕ⋯ϕ2ϕϕ-5.0751ϕϕϕϕϕϕ⋯ϕ⋯ϕ3ϕ-3.1378ϕ-3.7791ϕϕϕϕϕ⋯ϕ⋯ϕ4ϕ-3.6325-3.5487ϕ-4.1625ϕϕϕϕ⋯ϕ⋯ϕ5ϕϕ-5.4357ϕϕ-5.4709ϕϕϕ⋯ϕ⋯ϕ6ϕ-1.9578ϕ-2.1933ϕϕ-3.5655ϕϕ⋯ϕ⋯ϕ7ϕ-2.6607-2.6726-3.5352-3.8273ϕϕ-4.4820ϕ⋯ϕ⋯ϕ8ϕ-3.5597-3.1304ϕ-4.1660-3.9238ϕϕ-4.9789⋯ϕ⋯ϕ⋮⋮⋮⋮⋮⋮⋮⋮⋮⋮⋱⋮⋱⋮10ϕ-0.5130ϕ-2.3045ϕϕ-2.9883ϕϕ⋯-4.0435⋯ϕ⋮⋮⋮⋮⋮⋮⋮⋮⋮⋮⋱⋮⋱⋮1860ϕϕϕϕϕϕϕϕϕ⋯ϕ⋯-38.7557(b) Arrival mix 50:500-5.4151ϕϕϕϕϕϕϕϕ⋯ϕ⋯ϕ1ϕ-4.3775ϕϕϕϕϕϕϕ⋯ϕ⋯ϕ2ϕϕ-5.4668ϕϕϕϕϕϕ⋯ϕ⋯ϕ3ϕ-2.6037ϕ-4.0130ϕϕϕϕϕ⋯ϕ⋯ϕ4ϕ-2.7649-4.1416ϕ-4.1444ϕϕϕϕ⋯ϕ⋯ϕ5ϕϕ-6.5433ϕϕ-6.3231ϕϕϕ⋯ϕ⋯ϕ6ϕ-1.7586ϕ-3.1643ϕϕ-4.4264ϕϕ⋯ϕ⋯ϕ7ϕ-2.0728-2.0196-3.3502-3.4856ϕϕ-4.5043ϕ⋯ϕ⋯ϕ8ϕ-3.7477-3.0712ϕ-4.0756-4.0292ϕϕ-5.8461⋯ϕ⋯ϕ⋮⋮⋮⋮⋮⋮⋮⋮⋮⋮⋱⋮⋱⋮10ϕ-0.7993ϕ-1.5530ϕϕ-3.5437ϕϕ⋯-4.9214⋯ϕ⋮⋮⋮⋮⋮⋮⋮⋮⋮⋮⋱⋮⋱⋮1860ϕϕϕϕϕϕϕϕϕ⋯ϕ⋯-60.3832(c) Arrival mix 70:300-6.1615ϕϕϕϕϕϕϕϕ⋯ϕ⋯ϕ1ϕ-3.1327ϕϕϕϕϕϕϕ⋯ϕ⋯ϕ2ϕϕ-5.2855ϕϕϕϕϕϕ⋯ϕ⋯ϕ3ϕ-3.0061ϕ-3.5966ϕϕϕϕϕ⋯ϕ⋯ϕ4ϕ-2.7236-3.3408ϕ-3.7272ϕϕϕϕ⋯ϕ⋯ϕ5ϕϕ-6.1511ϕϕ-6.4949ϕϕϕ⋯ϕ⋯ϕ6ϕ-1.4127ϕ-2.8845ϕϕ-4.3883ϕϕ⋯ϕ⋯ϕ7ϕ-1.5705-1.3678-2.7043-2.9114ϕϕ-4.2073ϕ⋯ϕ⋯ϕ8ϕ-2.4491-2.2687ϕ-3.9149-4.1053ϕϕ-4.5377⋯ϕ⋯ϕ⋮⋮⋮⋮⋮⋮⋮⋮⋮⋮⋱⋮⋱⋮10ϕ-0.6575ϕ-2.1138ϕϕ-2.7664ϕϕ⋯-4.2392⋯ϕ⋮⋮⋮⋮⋮⋮⋮⋮⋮⋮⋱⋮⋱⋮1860ϕϕϕϕϕϕϕϕϕ⋯ϕ⋯-44.6542 Table 1Representative sections of the learned Q-tables for different arrival mix scenarios (maximum 30 patients).012345678⋯10⋯60(a) Arrival mix 30:700-5.5647ϕϕϕϕϕϕϕϕ⋯ϕ⋯ϕ1ϕ-3.7015ϕϕϕϕϕϕϕ⋯ϕ⋯ϕ2ϕϕ-5.0751ϕϕϕϕϕϕ⋯ϕ⋯ϕ3ϕ-3.1378ϕ-3.7791ϕϕϕϕϕ⋯ϕ⋯ϕ4ϕ-3.6325-3.5487ϕ-4.1625ϕϕϕϕ⋯ϕ⋯ϕ5ϕϕ-5.4357ϕϕ-5.4709ϕϕϕ⋯ϕ⋯ϕ6ϕ-1.9578ϕ-2.1933ϕϕ-3.5655ϕϕ⋯ϕ⋯ϕ7ϕ-2.6607-2.6726-3.5352-3.8273ϕϕ-4.4820ϕ⋯ϕ⋯ϕ8ϕ-3.5597-3.1304ϕ-4.1660-3.9238ϕϕ-4.9789⋯ϕ⋯ϕ⋮⋮⋮⋮⋮⋮⋮⋮⋮⋮⋱⋮⋱⋮10ϕ-0.5130ϕ-2.3045ϕϕ-2.9883ϕϕ⋯-4.0435⋯ϕ⋮⋮⋮⋮⋮⋮⋮⋮⋮⋮⋱⋮⋱⋮1860ϕϕϕϕϕϕϕϕϕ⋯ϕ⋯-38.7557(b) Arrival mix 50:500-5.4151ϕϕϕϕϕϕϕϕ⋯ϕ⋯ϕ1ϕ-4.3775ϕϕϕϕϕϕϕ⋯ϕ⋯ϕ2ϕϕ-5.4668ϕϕϕϕϕϕ⋯ϕ⋯ϕ3ϕ-2.6037ϕ-4.0130ϕϕϕϕϕ⋯ϕ⋯ϕ4ϕ-2.7649-4.1416ϕ-4.1444ϕϕϕϕ⋯ϕ⋯ϕ5ϕϕ-6.5433ϕϕ-6.3231ϕϕϕ⋯ϕ⋯ϕ6ϕ-1.7586ϕ-3.1643ϕϕ-4.4264ϕϕ⋯ϕ⋯ϕ7ϕ-2.0728-2.0196-3.3502-3.4856ϕϕ-4.5043ϕ⋯ϕ⋯ϕ8ϕ-3.7477-3.0712ϕ-4.0756-4.0292ϕϕ-5.8461⋯ϕ⋯ϕ⋮⋮⋮⋮⋮⋮⋮⋮⋮⋮⋱⋮⋱⋮10ϕ-0.7993ϕ-1.5530ϕϕ-3.5437ϕϕ⋯-4.9214⋯ϕ⋮⋮⋮⋮⋮⋮⋮⋮⋮⋮⋱⋮⋱⋮1860ϕϕϕϕϕϕϕϕϕ⋯ϕ⋯-60.3832(c) Arrival mix 70:300-6.1615ϕϕϕϕϕϕϕϕ⋯ϕ⋯ϕ1ϕ-3.1327ϕϕϕϕϕϕϕ⋯ϕ⋯ϕ2ϕϕ-5.2855ϕϕϕϕϕϕ⋯ϕ⋯ϕ3ϕ-3.0061ϕ-3.5966ϕϕϕϕϕ⋯ϕ⋯ϕ4ϕ-2.7236-3.3408ϕ-3.7272ϕϕϕϕ⋯ϕ⋯ϕ5ϕϕ-6.1511ϕϕ-6.4949ϕϕϕ⋯ϕ⋯ϕ6ϕ-1.4127ϕ-2.8845ϕϕ-4.3883ϕϕ⋯ϕ⋯ϕ7ϕ-1.5705-1.3678-2.7043-2.9114ϕϕ-4.2073ϕ⋯ϕ⋯ϕ8ϕ-2.4491-2.2687ϕ-3.9149-4.1053ϕϕ-4.5377⋯ϕ⋯ϕ⋮⋮⋮⋮⋮⋮⋮⋮⋮⋮⋱⋮⋱⋮10ϕ-0.6575ϕ-2.1138ϕϕ-2.7664ϕϕ⋯-4.2392⋯ϕ⋮⋮⋮⋮⋮⋮⋮⋮⋮⋮⋱⋮⋱⋮1860ϕϕϕϕϕϕϕϕϕ⋯ϕ⋯-44.6542 Representative sections of the learned Q-tables for different arrival mix scenarios (maximum 30 patients). In Table 1 Rows represent patient number cantor pairs19 and columns represent Service Provider(SP) numbers cantor pairs; e.g., patient cantor pair 10 represents 2 new(F) patients and 2 follow up patients (R-patients), SP cantor pair 10 represent 2 FP-SP and 2 RP-SP Entries marked ϕ denote undefined or unvisited state–action pairs. Only representative rows and columns are shown; the complete learned Q-table has size 1860 × 60, corresponding to a maximum patient capacity of 30. The Q-learning agent produces an action-value table. Due to the high dimensionality of the state space, we need at least a five-dimensional space to properly visualize these Q-values in relation to the optimal state components \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_1(F), n_1(R), n_2(FP), n_2(RP)$$\end{document}, and the Q-values from the Q table; direct numerical inspection from the above Q table is impractical; therefore, the converged values are visualized by using heatmaps and three-dimensional surfaces. State-action entries that remain unvisited or infeasible (i.e., \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(n_2(FP) > n_1(F))$$\end{document} or \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(n_2(RP) > n_1(R))$$\end{document}) are masked explicitly, ensuring that artificially high values from initialization do not distort the landscape. Owing to the stochastic nature of episodic exploration, several state–action pairs remain unvisited during training, resulting in remaining the same as initial entries in the learned Q table. This sparsity becomes more pronounced for \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\text {max patients}=30$$\end{document}, where the state space expands quadratically, and the probability of random visitation decreases. In this model, the learned Q-values in the Q table represent the expected cumulative cost of transitioning from any given state to the terminal state under the optimal policy (Fig. 1).Fig. 1Three-dimensional surfaces of the optimal state-value (maximum Q-value) for different patient arrival compositions, considering a maximum of 30 patients and 5 service providers in each category. Three-dimensional surfaces of the optimal state-value (maximum Q-value) for different patient arrival compositions, considering a maximum of 30 patients and 5 service providers in each category. Optimal server allocation under varying arrival compositions To further examine the behavior of the learned policy, we analyze the optimal service-provider (SP) allocations under three representative patient-mix scenarios. These scenarios correspond to different proportions of first-time (F) and returning (R) patients in the arrival stream, reflecting typical variations observed in outpatient clinic operations. Figure 2 presents the optimal SP allocation surfaces for (i) a returning-patient–dominated scenario (30% F, 70% R), (ii) a balanced arrival scenario (50% F, 50% R) and (iii) a first-time–patient–dominated scenario (70% F, 30% R). The learned Q-learning policy dynamically adapts the allocation of service providers across all three scenarios in response to the prevailing patient composition. Since the system state is characterized by four variables, namely \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_1(F)$$\end{document}, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_1(R)$$\end{document}, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_2(FP)$$\end{document}, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_2(RP)$$\end{document}, direct visualization would require a four-dimensional representation. To facilitate interpretation, the allocation dimensions are aggregated by considering the total number of optimal service providers, defined as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_2(FP) + n_2(RP)$$\end{document}. Accordingly, the resulting three-dimensional surfaces depict the total optimal SP allocation as a function of the number of first-time and returning patients in the system. As an illustrative example, in the first scenario shown in Fig. 2,Fig. 2Optimal service provider allocation policies with a maximum of 30 patients and 5 service providers in each category under different patient arrival scenarios. The right-column figures present magnified views of high-load regions to enhance the visualization of local policy behavior. Here, we plot the total number of optimal SP (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_2(FP) + n_2(RP)$$\end{document}) against the total number of patients in each category(\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_1(F)$$\end{document},\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_1(R)$$\end{document}).In the left column figures, we show the results for maximum patients = 30 and max SP = 5 in each category. Since the data points are very congested, we add a zoomed portion of the left figure in the right column for patients numbers 25 to 30 in each category. For example, in the balanced scenario shown in Fig. 2, when the system has 25 first-time patients and 25 returning patients, the learned policy recommends a total of three service providers, corresponding to the allocation \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(n_2(FP), n_2(RP)) = (2,1)$$\end{document}. Under the F-Patients dominant arrival scenario, the same system state (25,25) yields a higher total allocation of 4 service providers, given by \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(n_2(FP), n_2(RP)) = (4,0)$$\end{document} and in the R-Patients dominant scenario, the total allocation is 2 with \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(n_2(FP), n_2(RP)) = (0,2)$$\end{document}. These surfaces collectively summarize how the system learns to vary optimal total service capacity in different arrival scenarios. Optimal service provider allocation policies with a maximum of 30 patients and 5 service providers in each category under different patient arrival scenarios. The right-column figures present magnified views of high-load regions to enhance the visualization of local policy behavior. Here, we plot the total number of optimal SP (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_2(FP) + n_2(RP)$$\end{document}) against the total number of patients in each category(\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_1(F)$$\end{document},\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_1(R)$$\end{document}).In the left column figures, we show the results for maximum patients = 30 and max SP = 5 in each category. Since the data points are very congested, we add a zoomed portion of the left figure in the right column for patients numbers 25 to 30 in each category. For example, in the balanced scenario shown in Fig. 2, when the system has 25 first-time patients and 25 returning patients, the learned policy recommends a total of three service providers, corresponding to the allocation \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(n_2(FP), n_2(RP)) = (2,1)$$\end{document}. Under the F-Patients dominant arrival scenario, the same system state (25,25) yields a higher total allocation of 4 service providers, given by \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(n_2(FP), n_2(RP)) = (4,0)$$\end{document} and in the R-Patients dominant scenario, the total allocation is 2 with \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(n_2(FP), n_2(RP)) = (0,2)$$\end{document}. These surfaces collectively summarize how the system learns to vary optimal total service capacity in different arrival scenarios. Scalability analysis and policy stability To evaluate the scalability of the proposed Q-learning framework and the robustness of the supervised ε-greedy policy, a dynamic scaling experiment was conducted. The maximum patient capacity (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_{max}$$\end{document}) for each patient category was progressively increased from 10 up to 140, while keeping the number of service providers within practical bounds. For each scale, the model was trained over multiple learning episodes. Two key performance indicators were analyzed: computational execution time and policy stability, measured by the change in optimal Q-values (Δ Q) across consecutive scales. As shown in Fig. 3, the execution time increases non-linearly with system size, with a sharp rise observed for larger values of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_{max}$$\end{document}. This reflects the curse of dimensionality in tabular Q-learning. In terms of policy behavior, the supervised ε-greedy policy remains stable for moderate system sizes.Fig. 3Scalability analysis: (top) execution time vs. maximum patient capacity, (bottom) policy stability measured by Δ Q. Scalability analysis: (top) execution time vs. maximum patient capacity, (bottom) policy stability measured by Δ Q. System-level performance and baseline comparison While the convergence of the Q-learning algorithm minimizes expected system costs, evaluating practical hospital performance metrics is essential. To this end, a 100-day Monte Carlo simulation was conducted to compare the trained dynamic Reinforcement Learning (RL) policy against a conventional static allocation baseline. The simulation environment was subjected to time-varying demand using a non-homogeneous Poisson process (NHPP). To rigorously test adaptability, the traffic was asymmetrically weighted, simulating a surge of first-time patients in the early hours and a shift toward returning patients later in the shift. The Static Baseline was restricted to a fixed provider allocation (reserving a strict number of providers for each patient class), whereas the RL model dynamically adjusted provider allocations from a shared pool based on real-time queue lengths. The action-switching penalty, utilized during training to prevent hyperactive reallocation, was excluded from the simulation to isolate core operational costs. System-level performance was evaluated across four key operational metrics: Total Cost, Average Waiting Time, System Throughput, and Jain’s Fairness Index. The comparative results are summarized in Table 2 and visualized in Fig. 4.Table 2Performance comparison between the static baseline and the dynamic RL policy.MetricStatic baselineDynamic RLTotal system cost9903.258587.90Average wait time (total)18.48 min11.76 minSystem throughput (pts/min)0.18300.1956Jain’s fairness index0.99560.9971Significant values are in bold. Table 2Performance comparison between the static baseline and the dynamic RL policy.MetricStatic baselineDynamic RLTotal system cost9903.258587.90Average wait time (total)18.48 min11.76 minSystem throughput (pts/min)0.18300.1956Jain’s fairness index0.99560.9971Significant values are in bold. Performance comparison between the static baseline and the dynamic RL policy. Significant values are in bold. Fig. 4Performance comparison between the static baseline and dynamic RL policy across total cost, average wait time, throughput, and Jain’s fairness index. Performance comparison between the static baseline and dynamic RL policy across total cost, average wait time, throughput, and Jain’s fairness index. The empirical data demonstrates that the dynamic RL policy significantly outperforms the static baseline across all measured metrics. Most notably, the RL model reduced the total average waiting time from 18.48 minutes to 11.76 minutes, achieving an approximate 36% improvement. Additionally, the system’s empirical throughput increased from 0.1830 to 0.1956 patients per minute, while the total operational cost was reduced by approximately 13.3%. Both models maintained a high Jain’s Fairness Index, with the RL model scoring slightly higher at 0.9971. Collectively, these results demonstrate that the proposed heterogeneous Q-learning model learns stable, feasible, and resource-efficient scheduling policies that adapt to heterogeneous patient queues and time-varying demand. Discussion The experimental results provide several important insights into the behavior and robustness of the proposed heterogeneous Q-learning–based scheduling framework. Unlike classical homogeneous queueing formulations, which typically assume constant arrival rates and uniform waiting-cost structures20, the present model incorporates both time-varying arrival intensity and class-specific service characteristics. The NHPP-driven arrival process enables the model to capture empirically observed multi-peak outpatient inflow patterns, while the heterogeneous cost structure allows first-time and returning patients to influence the optimal scheduling policy in distinct ways. A meaningful comparison emerges when examining the behavior of the learned policy across states containing a total of three patients. In the homogeneous Q-learning model reported by Ni et al.21, the optimal policy assigns two service providers for all such states and yields an approximate Q-value of -5.61. In contrast, the heterogeneous model produces differentiated optimal costs and allocations for the four feasible patient compositions (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_1(F)$$\end{document},\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_1(R)$$\end{document}): (0, 3), (1, 2), (2, 1), and (3, 0), and corresponding expected total cumulative costs of -6.33, -3.07, -2.01 and -1.75 respectively. We observed here that while increasing in F-patient number, the optimal total cumulative cost is also decreasing. Under a comparable simulation environment, the heterogeneous model achieves improved cost performance for most states, as we use time-dependent cost functions and SP setup cost instead of relying on a constant cost rate as in Ni et al.21. The scenario-based optimal server allocation results further highlight the adaptability of the proposed Q-learning framework under varying patient arrival compositions. As illustrated by the allocation surfaces in Fig. 2, the learned policy dynamically adjusts service-provider (SP) allocation in response to changes in the relative proportions of first-time (F) and returning (R) patients, rather than relying on a fixed staffing rule. To illustrate this behavior quantitatively, Table 3 reports the optimal cumulative cost and the corresponding optimal SP allocation at the representative system state (25, 25) under three different arrival mix scenarios. The results indicate that the learned policy responds not only to the total system load but also to the underlying composition of arrivals.Table 3Optimal cumulative cost and optimal service-provider allocation at state (25, 25) under different arrival mix scenarios.Arrival scenarioOptimal costOptimal SP allocation\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_1(F)$$\end{document}\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_1(R)$$\end{document}50:50-3.32170:30-4.34030:70-7.902 Table 3Optimal cumulative cost and optimal service-provider allocation at state (25, 25) under different arrival mix scenarios.Arrival scenarioOptimal costOptimal SP allocation\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_1(F)$$\end{document}\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_1(R)$$\end{document}50:50-3.32170:30-4.34030:70-7.902 Optimal cumulative cost and optimal service-provider allocation at state (25, 25) under different arrival mix scenarios. When F-patients dominate the arrival process (70:30 scenario), the learning model allocates four F-patient service providers, resulting in an optimal cumulative cost of -4.3. Although this configuration incurs a higher SP allocation cost due to increased staffing, it effectively reduces waiting costs for the dominant patient class, leading to a moderate overall cost. In contrast, under the R-patient–dominated scenario (30:70), the policy allocates only two R-patient service providers. While this reduces the SP allocation cost, the increased waiting cost dominates, resulting in a higher cumulative cost of -7.9. For the balanced arrival scenario (50:50), the optimal policy allocates two F-patient service providers and one R-patient service provider, achieving a cumulative cost of -3.3. This configuration reflects a balanced trade-off between service capacity and waiting costs. These observations confirm that the proposed Q-learning policy is sensitive not only to absolute queue lengths but also to the underlying arrival composition, enabling flexible and context-aware service-provider allocation. It therefore follows that the adaptive supervised Q-learning (SQL) policy is capable of responding effectively to different patient arrival compositions and can dynamically adjust the optimal service-provider allocation in accordance with the prevailing patient-mix conditions. Overall, these results indicate that the Q-learning–based policy generalises effectively across heterogeneous arrival environments. By continuously adapting service-provider allocation in line with both congestion levels and patient-mix dynamics, the framework achieves a flexible and operationally realistic scheduling strategy suitable for outpatient systems subject to fluctuating demand patterns. The scalability analysis (Fig. 3) indicates that although computational complexity increases significantly with system size, the proposed supervised ε-greedy policy maintains stable performance within moderate system scales. Two aspects were evaluated: computational execution time and policy stability, measured using the variation in optimal Q-values (Δ Q). The results clearly demonstrate the curse of dimensionality in tabular Q-learning. In terms of policy behaviour, the supervised ε-greedy policy remains stable for moderate system sizes, with Δ Q showing a decreasing trend up to \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_{\max } \approx 90$$\end{document}. For larger scales (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_{\max } > 100$$\end{document}), slight fluctuations are observed, indicating reduced exploration efficiency in very large state spaces. It is important to note that, although the computational limits can be explored by increasing \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_{\max }$$\end{document}, in practical outpatient department (OPD) settings the number of patients is naturally bounded due to operational and resource constraints. Therefore, the proposed model operates effectively and efficiently within realistic system scales. For large-scale systems, future work may focus on incorporating advanced approaches such as deep reinforcement learning (DRL) to address these limitations. The empirical results in Fig. 4 highlight the operational advantage of the proposed RL framework over static allocation. The model reduces average waiting time by approximately 36% (from 18.48 to 11.76 minutes) while improving system throughput from 0.1830 to 0.1956 patients per minute. These improvements are driven by the dynamic reallocation of service providers under time-varying NHPP demand, which avoids bottlenecks and reduces idle capacity. The Jain’s fairness index remains close to unity (0.9971), indicating balanced service across patient categories. Overall, the results demonstrate that the proposed RL approach provides an efficient and cost-effective solution for dynamic healthcare environments. The proposed approach integrates reinforcement learning with heuristic-guided supervision to enhance decision-making efficiency. Although classical Q-learning ensures convergence under specific conditions, the inclusion of heuristic policies may affect strict theoretical guarantees. In this study, convergence is evaluated empirically through the stability of the learned policy and consistent performance observed across simulation experiments. This behavior is further supported by the scalability analysis, where the policy remains stable across increasing system sizes. Future research may focus on developing theoretical convergence analysis and incorporating advanced learning approaches to further improve accuracy and scalability. In this study, first-visit and revisit patients are modeled as separate categories, without explicitly tracking the same patient across multiple visits. This abstraction enables a tractable representation of the system and allows focus on cost-driven dynamic service provider allocation under time-varying demand. In practice, however, patients may undergo multiple visits, forming a multi-stage service process involving an initial consultation followed by one or more revisits. Extending the proposed framework to incorporate such patient-level transitions would enhance the realism of the model and require additional state representation and scheduling complexity. Recent studies, such as Shen et al. (2023)22 and Shen et al. (2026)23, have explored scheduling and capacity allocation under revisit uncertainty and multi-stage service systems. Integrating such mechanisms into the present model represents a promising direction for future work. Overall, the proposed heterogeneous Q-learning–based scheduling framework demonstrates strong adaptability and robustness under time-varying and heterogeneous healthcare environments. By integrating NHPP-based arrivals and dynamic service-provider allocation, the learned policy effectively balances waiting costs and service allocation costs while maintaining stable performance across realistic system scales. The framework also demonstrates practical effectiveness through improved system-level performance metrics, including reduced waiting time, increased throughput, and high fairness. While tabular Q-learning faces scalability limitations for very large systems, the model remains efficient within naturally bounded OPD settings. Future extensions may incorporate advanced learning approaches such as deep reinforcement learning (DRL), multi-stage patient modeling, and time-varying service capacity to further enhance applicability in complex healthcare systems. Methods Proposed queueing model in the light of traditional approaches Traditional queuing theory provides a well-established framework for analyzing service systems, in which performance measures such as average waiting time, queue length, and system utilization are derived under specific assumptions, typically involving stationary arrival processes and fixed service mechanisms. Classical models such as M/M/1 or M/G/K allow analytical derivation of these metrics under simplified conditions. However, real-world healthcare systems often exhibit time-varying arrivals. In the present study, patient arrivals are modeled using a non-homogeneous Poisson process (NHPP)15,16, and service times follow a general distribution. Furthermore, the system incorporates dynamic decision-making through adaptive service provider allocation. These features result in a non-stationary and state-dependent queueing system, making closed-form analytical expressions for performance measures, such as average waiting time, intractable. However, an approximate expression for the queue length can be obtained using standard multi-server queueing approximations, such as the Allen-Cunneen formula24. The corresponding waiting time is then estimated using Little’s law:\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L_q(t) \approx \frac{\rho (t)^{\sqrt{2(K(t)+1)}}}{1 - \rho (t)} \cdot \frac{C_s^2 + 1}{2}, \quad W_q(t) \approx \frac{L_q(t)}{\lambda (t)},$$\end{document}where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\rho (t) = \frac{\lambda (t)}{K(t)\mu }$$\end{document} denotes the time-dependent traffic intensity, λ (t) is the arrival rate, μ is the service rate, and K(t) is the number of active servers. To address these challenges, a reinforcement learning (RL)-based framework is adopted. The RL agent operates on the queueing system by observing the system state, which includes queue lengths and service provider configurations, and dynamically adjusts resource allocation to minimize total system cost. Problem description This study considers a heterogeneous healthcare queueing system representing an outpatient department (OPD), where patients arrive over time and require consultation services. The system distinguishes between two categories of patients: first-visit patients (F-patients) and returning or follow-up patients (R-patients), each forming separate queues upon arrival. Patient arrivals are modeled as a non-homogeneous Poisson process (NHPP), allowing the capture of time-varying arrival intensities commonly observed in real healthcare settings. Upon arrival, patients are directed to their respective queues based on their classification. Each queue is served by a designated group of service providers (SPs), and patients wait if no provider is immediately available. To efficiently manage system congestion and balance service allocation, a supervised Q-learning framework is integrated into the decision-making process. The learning agent observes the system state and selects actions that dynamically adjust the number and allocation of service providers, including adding new service providers, swapping providers between queues, or maintaining the existing configuration. The complete operational workflow, including patient arrival, queuing, service, and decision-making processes, is illustrated in Fig. 5. Model formulation This is a heterogeneous queuing model for outpatient department (OPD) services, designed to minimize the long-term total system cost while concurrently reducing patient waiting times. The model integrates a time-dependent arrival pattern regulated by a non-homogeneous Poisson process (NHPP) and classifies both patients and service providers (SPs) into two types. Reinforcement learning (supervised Q-learning) is employed to determine adaptive allocation and swapping policies in response to evolving queue states. System structure The OPD system accommodates two patient classes: first-time patients (F-patients) requiring initial consultation, and returning patients (R-patients) seeking follow-up care. Correspondingly, two categories of SPs are considered: F-patient SPs (FP-SPs) primarily attending first-time patients, and R-patient SPs (RP-SPs) focusing on returning patients. This structural division allows the model to realistically emulate OPD service allocation where continuity and responsiveness are both operational priorities. The figure illustrates the overall patient flow and service interaction.Fig. 5Heterogeneous system model. Heterogeneous system model. Assumptions: A limited number of service providers (SP) and medical resources are available in the system, and a limited maximum number of patients come on a particular day.The model functions during a certain daily time frame (e.g., 10:00–14:00 h), subsequent to which the system resets for the next working day. A limited number of service providers (SP) and medical resources are available in the system, and a limited maximum number of patients come on a particular day. A limited number of service providers (SP) and medical resources are available in the system, and a limited maximum number of patients come on a particular day. The model functions during a certain daily time frame (e.g., 10:00–14:00 h), subsequent to which the system resets for the next working day. The model functions during a certain daily time frame (e.g., 10:00–14:00 h), subsequent to which the system resets for the next working day. Arrival and service rate dynamics Patient arrivals to the OPD exhibit clear temporal variability, with multiple daily peaks corresponding to patient convenience. Conventional queueing studies often assume a homogeneous Poisson process with constant arrival intensity, which oversimplifies real hospital inflow behavior. To address this, we employ a non-homogeneous Poisson process (NHPP) that allows the arrival rate to vary continuously over time, capturing realistic fluctuations in demand. The intensity function is defined as:1\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \lambda (t) = \lambda _0 + \sum _{j=1}^{n} A_j \exp \!\left( -\frac{(t - t_j)^2}{2\sigma _j^2} \right) , \end{aligned}$$\end{document}where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lambda _0$$\end{document} represents the baseline arrival rate, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$A_j$$\end{document} denotes the amplitude of the j-th arrival peak, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$t_j$$\end{document} is the time of occurrence, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sigma _j$$\end{document} defines its temporal spread. Here, n represents the number of observable peaks during the OPD session, corresponding to distinct demand surges such as morning and midday registration periods. This representation effectively models both short-term bursts and gradual transitions in patient inflow intensity in a real scenario. Each SP \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$D_i$$\end{document} possesses an effective average service rate defined by:2\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} AS(D_i) = \frac{C_i^p}{t_i}, \end{aligned}$$\end{document}where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C_i^p$$\end{document} is the number of patients served during the previous working day and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$t_i$$\end{document} is the cumulative service duration. Service durations are modeled as lognormal random variables to capture the right-skewed, heterogeneous consultation times typically observed in healthcare systems. Assumptions: Patients randomly arrive at the OPD registration counter and then enter one at a time; the registration time at the OPD registration counter is nearly the same and negligible for all the arriving patients. Each SP can serve only one patient at a time, following a first-come, first-served (FCFS) discipline.Once a physician begins service within a working session, they remain active until their capacity or the end of the queue.The system assumes a time-varying patient arrival rate λ (t), modeled as a non-homogeneous Poisson process (NHPP), reflecting the realistic fluctuations in patient flow throughout the day. The service rate of all service providers (physicians) is considered nearly identical, denoted by a constant κ. Additionally, there is no predefined quota or maximum service capacity imposed on physicians; they continue to serve patients until the entire queue is cleared.Model assumptions and improvements:Distribution and assumptions for variables/processes.Variables/processesDistribution/assumptionsArrival rateLiterature: Poisson distribution Proposed: non-homogeneous Poisson process (NHPP)Service rateLognormal distribution (both)Waiting cost and SP setup cost parametersLiterature: constant cost assumptions Proposed: time-dependent sigmoid and piecewise linear functionsService cost parametersLiterature: constant cost assumptions Proposed: constant cost with activation thresholdQueue disciplineFCFS (both)Service provider allocationLiterature: SPs can be added/removed at any moment Proposed: an SP remains until maximum capacity or end of queue Patients randomly arrive at the OPD registration counter and then enter one at a time; the registration time at the OPD registration counter is nearly the same and negligible for all the arriving patients. Each SP can serve only one patient at a time, following a first-come, first-served (FCFS) discipline. Patients randomly arrive at the OPD registration counter and then enter one at a time; the registration time at the OPD registration counter is nearly the same and negligible for all the arriving patients. Each SP can serve only one patient at a time, following a first-come, first-served (FCFS) discipline. Once a physician begins service within a working session, they remain active until their capacity or the end of the queue. Once a physician begins service within a working session, they remain active until their capacity or the end of the queue. The system assumes a time-varying patient arrival rate λ (t), modeled as a non-homogeneous Poisson process (NHPP), reflecting the realistic fluctuations in patient flow throughout the day. The service rate of all service providers (physicians) is considered nearly identical, denoted by a constant κ. Additionally, there is no predefined quota or maximum service capacity imposed on physicians; they continue to serve patients until the entire queue is cleared. The system assumes a time-varying patient arrival rate λ (t), modeled as a non-homogeneous Poisson process (NHPP), reflecting the realistic fluctuations in patient flow throughout the day. The service rate of all service providers (physicians) is considered nearly identical, denoted by a constant κ. Additionally, there is no predefined quota or maximum service capacity imposed on physicians; they continue to serve patients until the entire queue is cleared. Distribution and assumptions for variables/processes.Variables/processesDistribution/assumptionsArrival rateLiterature: Poisson distribution Proposed: non-homogeneous Poisson process (NHPP)Service rateLognormal distribution (both)Waiting cost and SP setup cost parametersLiterature: constant cost assumptions Proposed: time-dependent sigmoid and piecewise linear functionsService cost parametersLiterature: constant cost assumptions Proposed: constant cost with activation thresholdQueue disciplineFCFS (both)Service provider allocationLiterature: SPs can be added/removed at any moment Proposed: an SP remains until maximum capacity or end of queue Distribution and assumptions for variables/processes. The underlying system dynamics follow a queueing-theoretic structure governed by the arrival and service processes. In particular, patient arrivals are modeled using a non-homogeneous Poisson process, and service times follow a general distribution. The reinforcement learning (RL) framework is built on top of this queueing system, where the system state captures the queue lengths and service provider allocation. The RL agent interacts with this evolving queueing system to dynamically adjust service allocation decisions in order to optimize system performance. In classical queueing models such as M/G/K systems, analytical expressions for performance measures like average waiting time can be derived under stationary assumptions. However, in the present study, the arrival process follows a non-homogeneous Poisson process, and the service times follow a general distribution. Additionally, the system incorporates dynamic control through reinforcement learning, resulting in a non-stationary and state-dependent queueing system. Due to these complexities, deriving closed-form analytical expressions for performance metrics such as average waiting time is intractable. Therefore, a simulation-based reinforcement learning approach is adopted, and performance measures are evaluated empirically. Supervised Q-learning framework (SQL) To enable adaptive and data-driven decision-making, the patient queueing environment is modeled as a Markov Decision Process (MDP). In this formulation, the system evolves over discrete decision epochs where actions correspond to scheduling or allocation decisions, and rewards reflect the balance between diminished waiting costs and operating expenditures. The Q-learning algorithm is employed as the reinforcement learning mechanism to identify optimum allocation strategies via environmental interaction. The value of executing an action a in a state s is represented by the Q-function Q(s, a), which is progressively revised according to the following rule:3\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} Q(s,a) \leftarrow Q(s,a) + \alpha \Big [ r(s,a) + \gamma \max _{a'} Q(s',a') - Q(s,a) \Big ], \end{aligned}$$\end{document}where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha \in (0,1]$$\end{document} is the learning rate, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\gamma \in [0,1]$$\end{document} is the discount factor, r is the immediate reward obtained from executing the action a in state s, s' denotes the next state reached after the transition, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\max _{a'} Q(s',a')$$\end{document} represents the best achievable value in the subsequent state. To balance between exploration and exploitation, a Supervised ε-greedy strategy is adopted within this heterogeneous hospital patient queue model, Q-learning dynamically learns policies for when to add service providers, when to swap service providers between groups, and when to remain idle. After several training events (simulated OPD sessions), the system converges to adaptive scheduling rules that reduce cumulative waiting costs while constraining operational effort. The major parts of SQL are discussed after the pseudo-code below. The subsequent subsection details the system state representation that forms the foundation for this learning process. Algorithm 1 Q-learning with supervised ε-greedy policy for HQ model. Q-learning with supervised ε-greedy policy for HQ model. System state representation In the heterogeneous queueing environment is modeled as a Markov Decision Process (MDP), where the state at any decision epoch is described as\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$s = (N_1, N_2, e) = \big ((n_1(F),n_1(R)),(n_2(FP),n_2(RP)), e\big ),$$\end{document}with the following components:\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N_1$$\end{document} current patient population in the system,\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_1(F)$$\end{document} number of first-time (new) patients,\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_1(R)$$\end{document} is Number of returning (follow-up) patients. \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N_2$$\end{document}is the number of active service providers, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_2(FP)$$\end{document} is the number of physicians assigned to first-time patients, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_2(RP)$$\end{document} is the number of physicians assigned to returning patients. e is event type,A is arrival of a patient,D is departure of a patient after service. For efficient implementation of Q-learning, the two-dimensional patient and provider states must be mapped into unique scalar indices to index the Q-tables. This achieved using the Cantor pairing function, a bijective mapping \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\pi : \mathbb {N}\times \mathbb {N} \rightarrow \mathbb {N}$$\end{document} widely used for encoding multidimensional discrete state spaces 19,25,26, defined as\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\pi (x,y) = \frac{(x+y)(x+y+1)}{2} + y.$$\end{document}Accordingly, the row and column indices are calculated as,\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N_1 = \pi \big (n_1(F), n_1(R)\big ), \quad N_2 = \pi \big (n_2(FP), n_2(RP)\big ),$$\end{document}and the Q-table is represented as\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Q[(N_1,N_2,e),a] = Q\big [(\pi (n_1(F),n_1(R)), \pi (n_2(FP),n_2(RP), e), a\big ],$$\end{document}where a is the action taken from state s. This Cantor mapping ensures compact indexing, reduced memory usage, and efficient lookup for Q-value updates during training. Action space representation Inside the heterogeneous patients queuing architecture, the decision-making agent functions inside a defined action space that regulates the allocation or reallocation of service providers (SPs) in response to system events. The aim of the action set is to dynamically equilibrate the allocation of first-time patient physicians (FP-SP) and returning patient physicians (RP-SP) based on the patient composition of first-time patients (F-patients) and returning patients (R-patients). The action space is defined as:\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$A_s = \{ +1_{FP}, +1_{RP}, 0, -1_{FP}, -1_{RP}, +1_{SFP}, +1_{SRP} \},$$\end{document}where each symbol denotes a specific type of adjustment: \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$+1_{FP}$$\end{document}: add one First time patient physician (FP-SP), \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$+1_{RP}$$\end{document}: add one returning patient physician (RP-SP), \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-1_{FP}$$\end{document}: remove one first time patient physician (FP-SP), \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-1_{RP}$$\end{document}: remove one returning patient physician (RP-SP), \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$+1_{SFP}$$\end{document}: swap one RP-SP to FP-SP server category, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$+1_{SRP}$$\end{document}: swap one FP-SP to RP-SP server category, 0: no change in the allocation of SPs. The allowable actions are contingent upon the nature of the system event e, which may signify either an arrival (e=A) or a departure (e=D) of a patient. Arrival event (e=A): for this event, the system may require additional physicians to manage the increased demand of patients. Here, this event can be categorized into two parts: one is the coming patient is a first-time patient (i.e., AF), or the patient is a returning patient (i.e., AR). Hence, the action set for the arrival event is\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$A(A) = \{ +1_{FP}, +1_{RP}, 0, +1_{SFP}, +1_{SRP} \}.$$\end{document}and for two categories the action set can be subdivided as follows: \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$A(AF) = \{ +1_{FP}, 0, +1_{SFP} \}.$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$A(AR) = \{ +1_{RP}, 0, +1_{SRP} \}.$$\end{document} Here, physician removal is not permitted, as reducing capacity in the presence of new arrivals could increase patient dissatisfaction. Departure Event (e=D): with the release of patients from system resources, physicians may be reallocated or reduced. This departure event can be categorized in two parts: one is the departing patient is a first-time patient (i.e., AF), or the patient is a returning patient (i.e., AR). The corresponding action set for the departure event is\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$A(D) = \{ -1_{FP}, -1_{RP}, 0, +1_{SFP}, +1_{SRP} \}.$$\end{document}and for two categories the action set can be subdivided as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$A(DF) = \{ -1_{FP}, 0, +1_{SRP} \}.$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$A(DR) = \{ -1_{RP}, 0, +1_{SFP} \}.$$\end{document} In this case, removal of SPs is permitted to avoid unnecessary overstaffing in a particular SP category, while swapping operations remain available for balancing physician assignments. This process guarantees that actions are contingent upon events and operationally viable, allowing the learning agent to adaptively optimize the number of physician allocations in response to uncertain patient flows. Assumptions: The system has mainly three types of costs: the SP setup cost required for assigning an SP at a particular time, the waiting cost for those patients waiting in the queue, and the service cost needed by the SP for operating the system, and our main focus is to minimize these total costs. The system has mainly three types of costs: the SP setup cost required for assigning an SP at a particular time, the waiting cost for those patients waiting in the queue, and the service cost needed by the SP for operating the system, and our main focus is to minimize these total costs. The system has mainly three types of costs: the SP setup cost required for assigning an SP at a particular time, the waiting cost for those patients waiting in the queue, and the service cost needed by the SP for operating the system, and our main focus is to minimize these total costs. Reward function The reward function offers a quantifiable indicator to direct the reinforcement learning agent in balancing patient waiting costs with the operational efforts of service providers (SPs). The net benefit is derived from an activity a in a specific state s defined as follows:r(s,a) = w(s,a) - c(s,a),where the immediate reward/penalty depends on the chosen action:\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$w(s,a) = {\left\{ \begin{array}{ll} 0, & a \in \{0, -1_{FP}, -1_{RP}, +1_{SFP}, +1_{SRP}\}, \\ - V(t), & a \in \{+1_{FP}, +1_{RP}\}, \end{array}\right. }$$\end{document}Thus, the introduction of a new SP (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$+1_{FP}$$\end{document}, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$+1_{RP}$$\end{document}) carries a penalty V(t), representing an SP setup cost, while other actions incur no direct penalty. The setup cost function V(t) is modeled as a time-dependent decaying function and defined as:\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$V(t) = {\left\{ \begin{array}{ll} C_0, & 0< t \le t_0, \\ C_0' + \dfrac{C_0 - C_0'}{1 + e^{\beta (t - t_m)}}, & t_0 < t \le T, \end{array}\right. }$$\end{document}where: \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C_0$$\end{document}: maximum SP setup cost, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C_0'$$\end{document}: minimum long-run SP setup cost, t: current time since OPD start, T: total OPD session duration, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$t_0$$\end{document}: threshold time after which setup costs begin to decay, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$t_m$$\end{document}: mid-transition time of the decay, β: steepness parameter for controlling the rate of decay. In the context of our Markov Decision Process (MDP), the objective is to derive a policy that minimizes the expected total operational cost. To this end, we define an immediate cost function, c(s, a), which represents the total cost incurred when taking action a in a given state s during the time interval \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$t_q$$\end{document} until the next system event. This cost is the cumulative cost of patients waiting in the queue and the operational cost of the service infrastructure. The total immediate cost is formulated as:4\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} c(s, a) = t_q \left( L_q \cdot U(t) + S(t) \right) \end{aligned}$$\end{document}Below, we define each component of this cost function in detail. Here, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$t_q$$\end{document} is the transition time from the current state to next state, related to the arrival time in the system. The queue length, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L_q$$\end{document}, is represents the total number of patients awaiting for getting service.5\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} L_q = (n_1(F) + n_1(R)) - (n_2(FP) + n_2(RP)) \end{aligned}$$\end{document}The server operational cost per unit time, S(t), includes both a fixed cost for maintaining server capacity and a variable cost related to server activity. It is defined as6\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} S(t) = K \cdot C_2 + n_2 \cdot C_2^{\text {a}} \end{aligned}$$\end{document}where: K is the total number of servers available in the system, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C_2$$\end{document} is the base service cost per unit time for each available server, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_2$$\end{document} is total number of active servers, given as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_2 = n_2(FP) + n_2(RP)$$\end{document}, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C_2^{\text {a}}$$\end{document} is the activation cost per unit time for each server currently active in service. To address the increasing urgency or disutility associated with longer waiting times, the waiting cost per patient per unit time, U(t), is defined as a non-static function of time. This function establishes a baseline cost for an initial period and then transitions smoothly to a higher maximum cost. Here, the function is defined as:7\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} U(t) = {\left\{ \begin{array}{ll} C_1, & \text {if } 0< t \le t_1 \\ C_1 + \dfrac{C_1' - C_1}{1 + e^{-\beta (t - t_m)}}, & \text {if } t_1 < t \le T \end{array}\right. } \end{aligned}$$\end{document}Where the components of this function are: \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C_1$$\end{document}: the initial, constant waiting cost per patient per unit time, applicable during the early phase of the operational period \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(0, t_1]$$\end{document}. \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C_1'$$\end{document}: the maximum waiting cost per patient per unit time, which the system approaches as time progresses. t: current time since waiting time started, T: maximum waiting time for a patient, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$t_1$$\end{document}: threshold time after which the waiting costs begin to escalate, β: a slope parameter that controls the steepness of the cost transition. \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$t_m$$\end{document}: The midpoint of the sigmoid transition, representing the time at which the waiting cost is exactly halfway between \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C_1$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C_1'$$\end{document}. This logistic (sigmoid) function models a smooth and non-linear increase in waiting cost after a threshold time \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$t_1$$\end{document}. This reflects more precisely the principle that in real life, patient impatience or the priority of their service does not increase linearly but often grows more rapidly after a certain tolerance threshold has been passed. The reward function incorporates both service-related costs and waiting costs. The waiting cost is modeled as a function of the queue length and time-dependent penalties, ensuring that longer queues and higher delays result in increased cost. This enables the learning agent to implicitly minimize patient waiting time through appropriate service allocation decisions. Supervised ε-greedy policy In the proposed SQL model, action selection is governed by a supervised ε-greedy strategy.The fundamental concept is that the exploration phase is supervised by domain-specific cost rules, rather than being conducted entirely at random. This approach enhances the practical stability and learning efficiency of the heterogeneous patient queue management process. In the reinforcement learning literature, similar heuristic-guided exploration approaches have been investigated (e.g., heuristic-guided RL and preference-based ε-greedy methods 27,28), but in the present formulation adapts These ideas are specifically for the heterogeneous patient queue model by introducing event-wise cost evaluation functions for supervised exploration. With probability ε, the algorithm explores new actions; however, unlike the conventional ε-greedy policy where exploration is purely random, here, it is guided by heuristic cost evaluations corresponding to the current event type. For instance, during an arrival at a first-time patient queue (AF), if we have to add one SP in FP-SP, then the candidate exploratory actions \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(+1_{FP}, +1_{SFP})$$\end{document} are evaluated through the incremental cost functions \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Delta C(+1_{FP})$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Delta C(+1_{SFP})$$\end{document}, and the action that gives a lower incremental cost is selected. Similar cost-based comparisons are applied for other arrival or departure event categories (AR, DF, DR). With probability 1-ε, the algorithm performs exploitation by selecting the action with the maximum Q-value for the current state. This heuristic supervision ensures that exploratory decisions remain feasible and cost-effective, reducing impractical random choices while preserving the essential balance between exploration and exploitation.\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$a = {\left\{ \begin{array}{ll} \text {guided action from heuristic supervision}, & \text {with probability } \epsilon , \\ \arg \max \limits _{a'} Q(s,a'), & \text {with probability } 1-\epsilon \end{array}\right. }$$\end{document} Algorithm 2Supervised ε-greedy policy. Supervised ε-greedy policy. Notations list: state(s)= (n1(F),n1(R),n2(FP),n2(RP),e), event(e)= {A,D} ≈ {AF, AR, DF, DR}, Here ‘AF’ event type means the arriving patients is F-Patients type, Q is Q-table type based on event, ε is predefined random number ∈ [0,1], a′ is selected action from action setAs = {+1FP ,+1RP,0,−1FP,−1RP,+1SFP,+1SRP} , \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$^{\rm argmax}_{a^\prime}$$\end{document}Q(s,a′) is the maximum Q-value in state s among all possible actions in Q-table. Heuristic guided supervision rules The system allows for the swapping of service providers (SPs) between the FP-SP and RP-SP groups in order to balance workloads and minimize patient waiting costs. This mechanism dynamically reallocates physicians across patient categories depending on the current demand conditions. We consider that patient arrivals and departures occur sequentially, i.e., one patient at a time. The arrival process can be represented through the following scenarios: At any given time:Scenario 1: one patient arrives in the F-patient category.Scenario 2: one patient arrives in the R-patient category.Now we consider Scenario 1: one patient arrives in the F-patient category. At that moment, let there be \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_2(FP)$$\end{document} active SPs in the FP-SP group and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_2(RP)$$\end{document} active SPs in the RP-SP group. The numbers of patients just before this arrival event are \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_1(F)$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_1(R)$$\end{document} in the F-patient and R-patient categories, respectively. Scenario 1: one patient arrives in the F-patient category. Scenario 1: one patient arrives in the F-patient category. Scenario 2: one patient arrives in the R-patient category. Scenario 2: one patient arrives in the R-patient category. Two possible actions can be taken: No change in the number of SPs in the FP-SP category (+0).Change in the number of SPs, either by adding a new server \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(+1_{FP})$$\end{document} or by swapping an SP from RP-SP to FP-SP \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(+1_{SFP})$$\end{document}.During random exploration, if the chosen random action is +0, then no change occurs in the number of SPs. However, if the chosen action is either \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$+1_{FP}$$\end{document} or \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$+1_{SFP}$$\end{document}, the more beneficial option is selected by comparing the incremental cost function Δ C. No change in the number of SPs in the FP-SP category (+0). No change in the number of SPs in the FP-SP category (+0). Change in the number of SPs, either by adding a new server \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(+1_{FP})$$\end{document} or by swapping an SP from RP-SP to FP-SP \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(+1_{SFP})$$\end{document}. Change in the number of SPs, either by adding a new server \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(+1_{FP})$$\end{document} or by swapping an SP from RP-SP to FP-SP \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(+1_{SFP})$$\end{document}. Now, we calculate this incremental cost for the action \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$+1_{SFP}$$\end{document}. After applying the \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$+1_{SFP}$$\end{document} action, the changes in the number of patients and SPs in both categories can be expressed as follows:\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_1'(F) = n_1(F) + 1, \quad n_1'(R) = n_1(R),$$\end{document}\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_2'(FP) = n_2(FP) + 1, \quad n_2'(RP) = n_2(RP) - 1.$$\end{document}Average service speed: The average service speed for a group of physicians is defined as the mean service capacity per unit time. It is given for the F-patient and R-patient queues, respectively, as follows:8\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} AS^{SFP}(FP) = \frac{\sum _{i=1}^{n_2'(FP)} \frac{C_i}{t_i}}{n_2'(FP)}, \end{aligned}$$\end{document}9\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} AS^{SFP}(RP) = \frac{\sum _{i=1}^{n_2'(RP)} \frac{C_i}{t_i}}{n_2'(RP)}. \end{aligned}$$\end{document}Average waiting time: The expected waiting time in each queue is computed as the ratio of excess patients over the number of available physicians to the corresponding average service speed. For the F-patient queue:10\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} wt_{FP}^{SFP} = \frac{(n_1'(F) - n_2'(FP))}{AS^{SFP}(FP)}. \end{aligned}$$\end{document}For the R-patient queue:11\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} wt_{RP}^{SFP} = \frac{(n_1'(R) - n_2'(RP))}{AS^{SFP}(RP)}. \end{aligned}$$\end{document}Waiting cost calculation: The waiting cost is determined by multiplying the number of waiting patients by a utility-weighted measure of the average waiting time. For the F-patient queue:12\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} W_C^{SFP}(FP) = (n_1'(F) - n_2'(FP)) \cdot U\big (wt_{FP}^{SFP}\big ), \end{aligned}$$\end{document}For the R-patient queue:13\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} W_C^{SFP}(RP) = (n_1'(R) - n_2'(RP)) \cdot U\big (wt_{RP}^{SFP}\big ). \end{aligned}$$\end{document}Total cost evaluation: At this decision epoch, the total cost for waiting associated with the \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$+1_{SFP}$$\end{document} action is given by\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T_{WC}^{SFP} = W_C^{SFP}(FP) + W_C^{SFP}(RP).$$\end{document}Similarly, the total costs corresponding to other actions are:\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T_{WC}^{AFP} = W_C^{AFP}(FP) + W_C^{AFP}(RP) + v(t),$$\end{document}\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T_{WC}^{NC} = W_C^{NC}(FP) + W_C^{NC}(RP).$$\end{document}Here, “SFP” denotes the swapping of one SP from RP-SP to FP-SP category, “AFP” denotes the addition of a new SP in the FP-SP category, and “NC” represents no change in SP allocation across both categories and v(t) is the cost for allocating a new fresh SP in the system at time t. Decision rule: The decision policy is formulated based on a cost–benefit comparison between the alternative actions. The marginal reductions in total waiting cost are expressed as:\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Delta C_{SFP} = T_{WC}^{NC} - T_{WC}^{SFP}, \quad \Delta C_{AFP} = T_{WC}^{NC} - T_{WC}^{AFP}.$$\end{document}The following heuristic rule is applied:If both \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Delta C_{SFP}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Delta C_{AFP}$$\end{document} are positive:Perform swapping if \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Delta C_{AFP} < \Delta C_{SFP}$$\end{document}.Otherwise, add a new physician if \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Delta C_{SFP} < \Delta C_{AFP}$$\end{document}.If only one action yields a positive reduction, that action is selected.If \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Delta C_{SFP} = \Delta C_{AFP}$$\end{document}, either action is chosen randomly.This policy ensures that operational decisions are made by balancing patient waiting cost reduction against additional operational effort. By dynamically comparing alternative interventions, the proposed approach promotes efficient and resource-conscious queue management. A similar analytical procedure is followed for Scenario 2, where one patient arrives in the R-patient category. The decision-making process, including the computation of the average service speed, expected waiting time, and corresponding waiting cost and total cost, remains structurally identical to that described in Scenario 1. The only distinction lies in the interchange of the F-patient and R-patient categories within the respective expressions. Furthermore, during departure events, two analogous scenarios arise corresponding to patient departures from the F-patient or R-patient categories. In both cases, the incremental changes in the number of patients and service providers are evaluated, and the total waiting cost functions are updated accordingly. The same cost–benefit comparison framework and decision policy, as established for the arrival scenarios, are applied to determine whether to retain the same, swap, or reduce service providers. If both \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Delta C_{SFP}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Delta C_{AFP}$$\end{document} are positive:Perform swapping if \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Delta C_{AFP} < \Delta C_{SFP}$$\end{document}.Otherwise, add a new physician if \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Delta C_{SFP} < \Delta C_{AFP}$$\end{document}. If both \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Delta C_{SFP}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Delta C_{AFP}$$\end{document} are positive:Perform swapping if \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Delta C_{AFP} < \Delta C_{SFP}$$\end{document}.Otherwise, add a new physician if \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Delta C_{SFP} < \Delta C_{AFP}$$\end{document}. Perform swapping if \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Delta C_{AFP} < \Delta C_{SFP}$$\end{document}. Perform swapping if \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Delta C_{AFP} < \Delta C_{SFP}$$\end{document}. Otherwise, add a new physician if \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Delta C_{SFP} < \Delta C_{AFP}$$\end{document}. Otherwise, add a new physician if \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Delta C_{SFP} < \Delta C_{AFP}$$\end{document}. If only one action yields a positive reduction, that action is selected. If only one action yields a positive reduction, that action is selected. If \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Delta C_{SFP} = \Delta C_{AFP}$$\end{document}, either action is chosen randomly. If \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Delta C_{SFP} = \Delta C_{AFP}$$\end{document}, either action is chosen randomly. Thus, for both arrival and departure events across patient categories, the decision logic remains consistent, governed by the heuristic supervision rule aimed at minimizing the total waiting costs while maintaining balanced system utilization.