Constrained Reinforcement Learning for the Dynamic Inventory Routing Problem under Stochastic Supply and Demand
Umur Hasturk, Kees Jan Roodbergen, Evrim Ursavas, Albert H. Schrotenboer · 2025
Read Constrained Reinforcement Learning for the Dynamic Inventory Routing Problem under Stochastic Supply and Demand by Umur Hasturk, Kees Jan Roodbergen, Evrim Ursavas, Albert H. Schrotenboer (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-05276.txt.
Introduction
Green fuel gas made from sun and wind comes in gusts, while its call runs its own clock, so its store and road plan must bend by the day. Hasturk and co-authors school a plan tool for that bend in a 2025 study. Their tool learns with the bars built in, not pasted on: stock caps and road rules sit in the school sums.
A state worth read past each choice sharpens the tool's aim. A wide look form then reads more days at once, and beats the plain form by a tenth on the mean, at more cost. Small sets sit near the best known. Stock school rules and plain rolls fall short of both forms.
Catalog record
| CREATOR | Umur Hasturk, Kees Jan Roodbergen, Evrim Ursavas, Albert H. Schrotenboer |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 114 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-05276
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Constrained Reinforcement Learning for the Dynamic Inventory Routing Problem under Stochastic Supply and Demand Umur Hasturk, Kees Jan Roodbergen, Evrim Ursavas, Albert H. Schrotenboer Abstract Green hydrogen has multiple use cases and is produced from renewable energy, such as solar or wind energy. It can be stored in large quantities, decoupling renewable energy generation from its use, and is therefore considered essential for achieving a climate-neutral economy. The intermittency of renewable energy generation and the stochastic nature of demand are, however, challenging factors for the dynamic planning of hydrogen storage and transportation. This holds particularly in the early-adoption phase when hydrogen distribution occurs through vehicle-based networks. We therefore address the Dynamic Inventory Routing Problem (DIRP) under stochastic supply and demand with direct deliveries for the vehicle-based distribution of hydrogen. To solve this problem, we propose a Constrained Reinforcement Learning (CRL) framework that integrates constraints into the learning process and incorporates parameterized post-decision state value predictions. Additionally, we introduce Lookahead-based CRL (LCRL), which improves decision-making over a multi-period horizon to enhance short-term planning while maintaining the value predictions. Our computational experiments demonstrate the efficacy of CRL and LCRL across diverse instances. Our learning methods provide near-optimal solutions on small scale instances that are solved via value iteration. Furthermore, both methods outperform typical deep learning approaches such as Proximal Policy Optimization, as well as classical inventory heuristics, such as ( s , S ) (s,S) -policy-based and Power-of-Two-based heuristics. Furthermore, LCRL achieves a 10 % 10\% improvement over CRL on average, albeit with higher computational requirements. Analyses of optimal replenishment policies reveal that accounting for stochastic supply and demand influences these policies, showing the importance of our addition to the DIRP. keywords Reinforcement Learning, Approximate Dynamic Programming, Inventory Management, Green Hydrogen, Direct Deliveries, Lookahead Policies † † runningauthor: Hasturk † † runningtitle: CRL for DIRP † † authors: Department of Operations, Faculty of Economics and Business, University of Groningen, Groningen, the Netherlands [email protected], [email protected], [email protected] Operations, Planning, Accounting and Control Group, School of Industrial Engineering, Eindhoven University of Technology, Eindhoven, the Netherlands [email protected] † † affiliation: † † affiliation: † † affiliation: † † affiliation: 1 Introduction Achieving carbon neutrality by 2050, as targeted by the European Union and other global initiatives ( IPCC 2023 ) , requires a transition to renewable energy sources such as wind and solar ( Holechek et al. 2022 ) . While these sources play an essential role in decarbonizing energy systems, their output is inherently intermittent and uncertain ( Drücke et al. 2021 ) , creating significant challenges in maintaining the balance between supply and demand. Conventional gas-fired power plants are often used to address this imbalance, as they can be quickly activated to produce electricity when supply is low ( Safari et al. 2019 ) . However, reliance on such fossil fuel-based solutions contradicts the carbon-neutral goals and pinpoints the need for alternative mechanisms. Hydrogen, particularly green hydrogen produced through renewables, provides a promising solution to these challenges ( Abe et al. 2019 ) . It enables the storage of excess energy as gas during periods of high renewable generation via electrolysis, and its later conversion back to electricity when needed ( Oliveira et al. 2021 ) . Beyond energy storage, hydrogen can be transported from one location to another as an energy carrier and used directly in applications such as transportation, residential heating, and industrial processes ( Ball and Wietschel 2009 ) . Its versatility and ability to integrate with multiple sectors make hydrogen a critical component in the development of sustainable energy systems. A hydrogen distribution network typically consists of a supplier and multiple demand locations. The supplier generates hydrogen from renewable energy sources, resulting in a stochastic green hydrogen supply. Demand locations, such as hydrogen refueling stations, residential areas, and industrial plants, have stochastic demand, requiring periodic replenishments to maintain operations. To meet these demands, hydrogen is transported by vehicles from the supplier to these geographically dispersed locations. In the near term, distribution is vehicle-based due to small volumes; pipelines may be added later, while last-mile deliveries remain vehicle-based as in conventional fuel logistics. For instance, the Hydrogen Energy Applications in Valley Environments for Northern Netherlands (HEAVENN) project exemplifies this by incorporating both vehicle-based distribution and planning the future pipeline development ( New Energy Coalition 2020 , HEAVENN 2022 ) . The supplier and customers can create inventories to buffer between supply and demand. This results in a complex stochastic dynamic planning problem that requires balancing between transportation, storage, and operational costs. In this study, we conceptualize vehicle-based hydrogen distribution by introducing a dynamic inventory routing problem (DIRP) that considers stochastic supply and demand. The supplier uses a homogeneous vehicle fleet to replenish the customers. Both the supplier and the customers have limited inventory capacity for hydrogen storage. The replenishments are, thus, subject to constraints, such as vehicle fleet size and customer inventory capacities. As is common in the inventory routing literature, we assume a direct delivery setting in our problem ( Kleywegt et al. 2002 , Bertazzi 2008 , Archibald et al. 2009 , Coelho et al. 2012 , Bertazzi et al. 2013 , Bertazzi et al. 2015 ) . Moreover, this is highly practical considering the relatively small quantities that can be transported in a full truckload. If needed, multiple vehicles may be dispatched to a customer at the same time, to accommodate larger replenishments. When demand cannot be met, it is considered as a lost sale. In contrast, the supplier can sell excess hydrogen to other local suppliers. This formulation is intended as a focused operational abstraction of early-stage vehicle-based hydrogen distribution, rather than a complete representation of all hydrogen logistics. It builds on a setting in which distribution is organized around a producer and multiple demand locations ( Staffell et al. 2019 , Hasturk et al. 2024 ) . In such early-adoption settings, lower volumes and limited dedicated infrastructure make customer-specific vehicle deliveries a reasonable first-order planning setting; the direct-delivery and homogeneous-fleet assumptions then keep the analysis focused on the core inventory-routing trade-off under stochastic supply and demand. This problem is modeled as a Markov decision process (MDP) over an infinite time horizon, with each period representing a day. The goal of the problem is to determine an optimal, state-dependent, replenishment policy that uses the current state information, i.e. inventory levels on locations, to minimize future expected discounted costs, consisting of transportation, holding, lost sales, and profits from sales. Finding optimal state-dependent replenishment policies is difficult due to the complexity of the action space. First, the action space grows exponentially with the number of customers and the maximum size of the replenishment quantities, making explicit enumeration or probabilistic sampling intractable. Second, the feasible actions are not explicitly listed but are instead defined implicitly by constraints within a Mixed-Integer Programming (MIP) formulation. Third, since the feasibility of the MIP depends on the current inventory levels, these MIP formulations change at each state, requiring redefinition at every decision. While achieving optimality in such complex settings is unrealistic, various learning techniques show good performance in the literature ( Vanvuchelen et al. 2020 , Ulmer et al. 2022 ) . Traditional reinforcement learning techniques, such as TD( \lambda ) and Proximal Policy