Comparative Analysis of Discrete and Continuous Action Spaces in Reservoir Management and Inventory Control Problems
Sravani Boddepalli, Prathamesh Kothavale · 2025
Read Comparative Analysis of Discrete and Continuous Action Spaces in Reservoir Management and Inventory Control Problems by Sravani Boddepalli, Prathamesh Kothavale (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-02093.txt.
Introduction
Sravani Boddepalli and Prathamesh Kothavale compare two styles of decision-making in sequential problems: choosing from a discrete menu of actions, or from a continuous range. Their test problems are reservoir management, where water release decisions matter over long horizons, and inventory control under both fixed and random demand. The comparison focuses on how time and memory costs grow.
Finer discrete menus approached the quality of continuous choices but at rising computational cost, and in inventory problems the cost grew exponentially with more actions and items. The authors also introduce a symbolic method for continuous problems in hybrid decision processes, using a new data structure to represent value functions piece by piece. Their results map the trade-offs facing anyone scaling up exact decision methods.
Catalog record
| CREATOR | Sravani Boddepalli, Prathamesh Kothavale |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 65 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-02093
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Comparative Analysis of Discrete and Continuous Action Spaces in Reservoir Management and Inventory Control Problems Sravani Boddepalli, Prathamesh Kothavale Abstract This paper presents a comparative analysis of discrete and continuous action spaces within the contexts of reservoir management and inventory control problems. We explore the computational trade-offs between discrete action discretizations and continuous action settings, focusing on their effects on time complexity and space requirements across different horizons. Our analysis includes a detailed evaluation of discretization levels in reservoir management, highlighting that finer discretizations approach the performance of continuous actions but at increased computational costs. For inventory control, we investigate deterministic and stochastic demand scenarios, demonstrating the exponential growth in time and space with increasing discrete actions and inventory items. We also introduce a novel symbolic approach for solving continuous problems in hybrid MDPs (H-MDPs), utilizing a new XADD data structure to manage piecewise symbolic value functions. Our results underscore the challenges of scaling solutions and provide insights into efficient handling of discrete and continuous action spaces in complex decision problems. Future research directions include exploring heuristic search methods and improved approximations for enhancing the practicality of exact solutions. I Introduction Some genuine stochastic arranging issues, for example, those in Mars Wanderer route, stock control, and water supply the executives, include ceaseless factors in their state and activity portrayals. For instance, in Mars Meanderer errands, a wanderer explores inside a consistent spatial climate while performing logical undertakings; in stock control issues for nonstop assets like oil, a business chooses the amount of every thing to arrange in view of unsure interest, limit imperatives, and reordering costs; and in water supply issues, utilities oversee water levels continuously to keep away from undercurrent while improving power age revenue [ bresina02 , Scarf_Karlin58 , reservoir ] . Definite answers for multivariate consistent state and activity Half and half Markov Choice Cycles (HMDPs) have been restricted. While direct quadratic Gaussian (LQG) control gives precise answers for straight frameworks with Gaussian noise [ lqgc ] , most issues including piecewise elements have seen little improvement, particularly in situations where the progress elements include various persistent factors. For discrete activity HMDPs, ideal piecewise straight worth capabilities have been derived [ feng04 ] , yet broad classes of HMDPs with direct elements for ceaseless factors stay inexplicable. This work proposes advances to the cutting edge with novel emblematic powerful programming (SDP) strategies, which broaden the class of HMDPs for which careful arrangements can be inferred. These procedures consider taking care of issues with piecewise straight elements and compensations in consistent activity settings and erratic elements in discrete activity settings. As a key application, the methodology empowers the specific arrangement of complicated stock control issues, denoting the initial time in north of 50 years that such an answer has been determined for consistent state situations in this domain [ Scarf_Karlin58 ] . Discrete Action Inventory Control ( DAIC ): A multi-item ( K -item) inventory consists of continuous amounts of specific items x_{i} where i\in[0,K] is the number of items and x_{i}\in[0,200] . The customer demand is a stochastic boolean variable d for low or high demand levels. The order action a_{j} takes two values of (0,200) where the first indicates no ordering and the second assumes maximum amount of ordering which is 200. There are linear reorder costs and also a penalty for holding items. The transition and reward functions have to be defined for each continuous item x_{i} and action j . Continuous Action Inventory Control ( CAIC ): In a more general continuous action HMDP setting to this problem, the inventory can order any of the i items a_{i}\in[0,200] considering the stochastic customer demand. The transition functions for the continuous state x_{i} and actions a_{i} is defined as: \displaystyle x^{\prime}_{i} \displaystyle=\begin{cases}d&:x_{i}+a_{i}-150\\ \neg d&:x_{i}+a_{i}-50\end{cases} \displaystyle P(d^{\prime}=\mathit{true}|d,\vec{x},\vec{x^{\prime}}) \displaystyle=\begin{cases}d&:0.7\\ \neg d&:0.3\end{cases} (1) The reward is the sum of K functions R=\sum_{i=0}^{K}R_{i} as below: \displaystyle R \displaystyle=\begin{cases}\sum_{j}x_{j}\geq C&:-\infty\\ \sum_{j}x^{\prime}_{j}\geq C&:-\infty\\ \sum_{j}x_{j}\leq C&:0\\ \sum_{j}x^{\prime}_{j}\leq C&:0\\ \end{cases}+\sum_{i=0}^{K}\Bigg(\begin{cases}d\wedge x_{i}\geq 150&:150-0.1\cdot a_{i}-0.05\cdot x_{i}\\ d\wedge x_{i}\leq 150&:x_{i}-0.1\cdot a_{i}-0.05\cdot x_{i}\\ \neg d\wedge x_{i}\geq 50&:50-0.1\cdot a_{i}-0.05\cdot x_{i}\\ \neg d\wedge x_{i}\leq 50&:x_{i}-0.1\cdot a_{i}-0.05\cdot x_{i}\\ \end{cases} \displaystyle+\sum_{i=0}^{K}\Bigg(\begin{cases}x_{i}\leq 0&:-\infty\\ x^{\prime}_{i}\leq 0&:-\infty\\ x_{i}\geq 0&:0\\ x^{\prime}_{i}\geq 0&:0\end{cases}\Bigg) where C is the total capacity for K items in the inventory. The first and last cases check the safe ranges of the capacity such that the inventory capacity of each item above zero and the sum of total capacity below C is desired. Fig. 1: The ideal worth capability V^{2}(x) for the CAIC issue is addressed by a XADD. To assess V^{2}(x) , follow the choice tree to a leaf, where the non-coincidental enunciation gives the worth, and the coincidental verbalization gives the ideal strategy a=\pi^{*,2}(x) . The right diagram shows the superior arrangement \pi^{2} , lined up with Scarf’s answer. Note that illegal state values are defined using -\infty , in this case having the capacity lower than zero at any time and having capacity higher than that of the total C . If our objective is to maximize the long-term value V (i.e. the sum of rewards received over an infinite horizon of actions), we show that the optimal value function can be derived in closed-form. For a single-item CAIC problem the optimal value function for the second horizon is defined as below: \displaystyle V=\begin{cases}(x<0\vee x>500)&:-\infty\\ d\land(0\leq x\leq 500)\land(x\geq 300)&:277.5-0.1*x\\ d\land(150\leq x\leq 300)&:232.5+0.05*x\\ d\land(0\leq x\leq 150)&:82.5+1.05*x\\ \neg d\land(200\leq x\leq 500)&:132.5-0.1*x\\ \neg d\land(50\leq x\leq 200)&:102.5+0.05*x\\ \neg d\land(0\leq x\leq 50)&:52.5+1.05*x\\ \end{cases}\vskip-19.91692pt (2) The policy obtained from this piecewise and linear value function and V^{2} itself are shown in Figure 1 using an extended algebraic decision diagram (XADD) representation which allows efficient implementation of the case calculus for arbitrary functions. According to Scarf’s policy for the Inventory Control problem, if the holding and storage costs are linear the optimal policy in each horizon is always of (S,s) [ Scarf_Karlin58 ] . In general this means if ( x>s ) the policy should be not to order any items and if ( x<s ) then ordering S-s-x items is optimal. According to this we can rewrite Scarf’s policy where each slice of the state space matches with this general rule: \displaystyle\pi^{*,2}(x)=\begin{cases}(x<0\vee x>500)&:-\infty\\ d\land(300\leq x\leq 500)&:0\\ d\land(0\leq x\leq 300)&:300-x\\ \neg d\land(200\leq x\leq 500)&:0\\ \neg d\land(0\leq x\leq 200)&:200-x\\ \end{cases} While this simple example illustrates the power of using continuous variables, for a multi-variate problem it is the very first solution to exactly solving problems such as the DAIC and CAIC. We propose novel ideas to work around some of the expressiveness limitations of previous approaches, significantly generalizing the range of HMDPs that can be solved exactly. To achieve this more general solution, this paper contributes a number of important advances: • The use of case calculus allows us to perform Symbolic dynamic programming (SDP) [ fomdp ] used to solve MDPs with piecewise transitions and reward functions defined in first-order logic. We define all required operations for SDP such as \oplus,\ominus,max,min as well as new operations such as the continuous maximization of an action parameter y defined as max_{y} and integration of discrete noisy transition. • We perform value iteration for two different settings. In the first setting of DA-HMDP we consider continuous state variables with a discrete action set while in the second setting CA-HMDP we