Integrating Human Knowledge Through Action Masking in Reinforcement Learning for Operations Research
Mirko Stappert, Bernhard Lutz, Niklas Goby, Dirk Neumann · 2025
Read Integrating Human Knowledge Through Action Masking in Reinforcement Learning for Operations Research by Mirko Stappert, Bernhard Lutz, Niklas Goby, Dirk Neumann (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-02662.txt.
Introduction
Learning agents that plan factory or stock tasks often clash with the managers who must trust them. Mirko Stappert, Bernhard Lutz, Niklas Goby, and Dirk Neumann let managers fence the agent in a 2025 study. Fences come as action masks drawn from expert rules of thumb.
Three testbeds cover paint shops, power peaks, and stock rooms. Masked agents beat unmasked ones by wide margins. In tight action spaces, masks proved key to learning any good policy at all. Over-tight masks did trap agents below their best. The balance point is expert input without a cage. Trust, the team argues, is a design feature.
Catalog record
| CREATOR | Mirko Stappert, Bernhard Lutz, Niklas Goby, Dirk Neumann |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 55 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-02662
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Integrating Human Knowledge Through Action Masking in Reinforcement Learning for Operations Research Mirko Stappert, Bernhard Lutz, Niklas Goby, Dirk Neumann Abstract Reinforcement learning (RL) provides a powerful method to address problems in operations research. However, its real-world application often fails due to a lack of user acceptance and trust. A possible remedy is to provide managers with the possibility of altering the RL policy by incorporating human expert knowledge. In this study, we analyze the benefits and caveats of including human knowledge via action masking. While action masking has so far been used to exclude invalid actions, its ability to integrate human expertise remains underexplored. Human knowledge is often encapsulated in heuristics, which suggest reasonable, near-optimal actions in certain situations. Enforcing such actions should hence increase trust among the human workforce to rely on the model’s decisions. Yet, a strict enforcement of heuristic actions may also restrict the policy from exploring superior actions, thereby leading to overall lower performance. We analyze the effects of action masking based on three problems with different characteristics, namely, paint shop scheduling, peak load management, and inventory management. Our findings demonstrate that incorporating human knowledge through action masking can achieve substantial improvements over policies trained without action masking. In addition, we find that action masking is crucial for learning effective policies in constrained action spaces, where certain actions can only be performed a limited number of times. Finally, we highlight the potential for suboptimal outcomes when action masks are overly restrictive. ∎ 1 Introduction Reinforcement Learning (RL), a branch of machine learning, has been successfully applied to solve complex problems such as the games of Chess and Go by self-play only ( Silver et al., 2017 ; Silver et al., 2018 ) . The resulting performance even surpasses the skills of human experts by significant amounts. In RL, an agent interacts with an environment to maximize the expected reward ( Sutton et al., 2018 ) . The environment presents a numerical representation of the current state to the agent, who then decides upon the action to perform next. After receiving an action, the environment transforms to the next state and emits a numerical reward to the agent. Advances in deep reinforcement learning (DRL) allowed researchers to solve the problem of the exploding state space as the mapping of states to Q-values or action probabilities is now approximated by neural networks ( Mnih et al., 2015 ) . Over the past few years, RL has also been increasingly used as a powerful tool for addressing complex problems in operations research (OR). The considered problem areas include production planning and control ( Esteso et al., 2023 ; Panzer and Bender, 2022 ; Zhang and Zhu, 2024 ) , supply chain management and inventory management ( Rolf et al., 2023 ) , machine scheduling ( Kayhan and Yildiz, 2023 ) , machine maintenance planning ( Ogunfowora and Najjaran, 2023 ) , and quality control ( Paraschos and Koulouriotis, 2024 ) , among many others. A major advantage of RL over traditional solution methods (e.g., hand-crafted rules, heuristics, mathematical solvers) is that a trained RL policy presents a fast and adaptive solution method. Businesses often encounter similar decision problems where the parameters follow known distributions like package delivery problems in the same district ( Bengio et al., 2021 ; Schiewe and Stinzendörfer, 2024 , e.g.,) , or sequencing problems with similar demand plans ( Brammer et al., 2022a ; Brammer et al., 2022b , e.g.,) . A RL policy can be trained in a simulated environment and subsequently transfer knowledge from prior learning experiences to unseen problem instances ( Taylor and Stone, 2009 ) . By contrast, the aforementioned traditional solution methods are designed to solve each problem instance individually without relying on prior experience. In addition, RL can be trained to account for the uncertainty in real-world operations regarding delivery times or customer demand. The paradigm of RL hence aligns particularly well with the goals of Industry 4.0, where adaptive and fast decision-making under uncertainty are crucial ( Choi et al., 2022 ; Weckenborg et al., 2024 ) . A major challenge of implementing RL in real-world operations is the task of converting the problem presumably formulated as a mixed-integer linear programming (MILP) problem to a Markov decision process, including states, actions, and reward function, that is suitable for RL policy learning ( Sutton et al., 2018 ) . Here, businesses need to ensure that i) the policy generates valid solutions, and that ii) a (near-)optimal policy of the MDP also generates (near-)optimal solutions of the actual MILP problem. A common approach to guiding the RL policy towards generating only valid solutions is given by returning the same state and penalizing invalid actions with large negative rewards ( Brammer et al., 2022b ; Valet et al., 2022 , e.g.,) . While this method generally achieves the desired outcome, the policy will still perform invalid actions during the training process, thereby increasing the time required for learning effective policies. An alternative is given by action masking ( Huang and Ontañón, 2022 ; Luo et al., 2022 ; Wang et al., 2024 , e.g.,) , a technique in RL that constrains the action space by limiting the set of available actions in specific states. In addition to these technical challenges, the implementation of RL, like other AI-based solution methods, often fails in real-world operations due to a lack of human trust and acceptance of AI decisions ( Dietvorst et al., 2015 ; Lehmann et al., 2022 ) . Although problems from OR are often considered as repetitive low-stake problems ( Notz and Pibernik, 2024 ) , human decision-makers still want to be able to make appropriate adjustments, particularly when models are wrong ( Dietvorst et al., 2018 ) . Therefore, a possible remedy to increase trust and user acceptance is given by allowing managers to modify the RL policy by including human domain knowledge. Human knowledge can prescribe reasonable heuristic actions in certain states, or even provably optimal actions. By ensuring that a policy incorporates human knowledge, the workforce should be less likely to suffer from algorithm aversion and instead rely on the model’s decisions. In this study, we analyze the inclusion of human domain knowledge via action masking. While traditionally used to prevent invalid actions, its potential for enforcing heuristic-suggested or provably optimal actions remains unexplored. Although a RL policy could, in theory, learn the actions suggested by the action mask implicitly after sufficient training time, there is no such guarantee in DRL due to the “deadly triad” of function approximation, bootstrapping, and off-policy learning ( Van Hasselt et al., 2018 ; Sutton et al., 2018 ) . Unlike tabular-based Q-learning, where convergence to a global maximum is theoretically guaranteed ( Sutton et al., 2018 ) , DRL approximates the values of states using a neural network, introducing function approximation errors. In fact, an update of the network’s weights influences several states instead of only a single state-action entry of the Q-value table. In addition, policy updates depend on previous estimates (bootstrapping) and are often applied to past experience (off-policy learning), thereby possibly amplifying approximation errors. Including action masks in the training process could hence lead to considerably superior