Controlling the low-temperature Ising model using spatiotemporal Markov decision theory
M.C. de Jongh, Richard J. Boucherie, M.N.M. van Lieshout · 2025
Read Controlling the low-temperature Ising model using spatiotemporal Markov decision theory by M.C. de Jongh, Richard J. Boucherie, M.N.M. van Lieshout (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-03668.txt.
Introduction
De Jongh, Richard Boucherie, and Van Lieshout let a decision maker steer a cold magnet spin by spin. Their spatiotemporal Markov decision process joins choices across time and across a lattice, then they build one from the two-dimensional Ising model under Metropolis dynamics. The controller may flip chosen spins at chosen moments to push the grid toward the all-plus state.
Solving the full space head-on is hopeless, so they shrink it to the energy landscape's local minima. That auxiliary problem yields its Bellman equations recursively, exposing the optimal policy's shape. Simulations then grade the policy back on the original lattice model.
Catalog record
| CREATOR | M.C. de Jongh, Richard J. Boucherie, M.N.M. van Lieshout |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 221 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-03668
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Controlling the low-temperature Ising model using spatiotemporal Markov decision theory M.C. de Jongh, Richard J. Boucherie, M.N.M. van Lieshout Abstract We introduce the spatiotemporal Markov decision process (STMDP), a special type of Markov decision process that models sequential decision-making problems which are not only characterized by temporal, but also by spatial interaction structures. To illustrate the framework, we construct an STMDP inspired by the low-temperature two-dimensional Ising model on a finite, square lattice, evolving according to the Metropolis dynamics. We consider the situation in which an external decision maker aims to drive the system towards the all-plus configuration by flipping spins at specified moments in time. In order to analyze this problem, we construct an auxiliary MDP by means of a reduction of the configuration space to the local minima of the Hamiltonian. Leveraging the convenient form of this auxiliary MDP, we uncover the structure of the optimal policy by solving the Bellman equations in a recursive manner. Finally, we conduct a numerical study on the performance of the optimal policy obtained from the auxiliary MDP in the original Ising STMDP. A Preprint Keywords Bellman equations \cdot Ising model \cdot Metastability \cdot Sequential decision making \cdot Spatiotemporal Markov decision process 1 Introduction The Markov decision process (MDP) is a well-established framework for modeling and solving problems that involve sequential decision-making under uncertainty ( Puterman, 2014 ) . Various domains, such as epidemic management ( Diaz-Infante et al., 2023 ; Palopoli et al., 2023 ) , wild fire prevention ( Altamimi et al., 2022 ; Roozbeh et al., 2021 ) and agricultural economics ( Chi et al., 2022 ; Swinton, 2002 ) , face decision-making problems that feature not only temporal but also spatial structures. If the state space, and possibly the action space, can be factored, such problems can be covered by the Factored MDP (FMDP) ( Boutilier et al., 2000 ) or graph-based MDP (GMDP) ( Sabbadin et al., 2012 ) frameworks. However, a framework for problems that do not obey such a factorisation assumption is lacking in the literature. In this article, we introduce the spatiotemporal Markov decision process (STMDP) as a special type of MDP in which the state variables adhere to local interaction structures that cannot necessarily be written in factored form. Unlike the FMDP and GMDP models, the STMDP framework includes processes with asynchronous dynamics, providing a higher level of flexibility. To illustrate the framework, we formulate an STMDP based on the two-dimensional Ising model on a finite, square lattice that evolves according to asynchronous Metropolis dynamics ( Newman and Barkema, 1999 , Section 3) . We assume that an external decision maker has the power to flip a spin at specified points in time, with the aim to drive the process towards a predetermined target configuration. In our analysis, we focus on the low-temperature regime, where the behaviour of the Ising model is well studied ( Cerf and Manzo, 2013 ; Neves and Schonmann, 1991 ; Cirillo et al., 2022 ; Arous and Cerf, 1996 ; Kotecký and Olivieri, 1993 ; Nardi and Olivieri, 1996 ; Nardi and Zocca, 2019 ; Cirillo and Lebowitz, 1998 ) . Specifically, if the system is initialized from the all-minus configuration and subjected to a small positive magnetic field, it will take an exponentially long time to reach the stable all-plus configuration. This phenomenon is known as metastability . The transition to the stable state typically occurs through the formation of a critical droplet of + -spins, which eventually nucleates the lattice. In this article, we use the Ising STMDP to optimize the trajectory for reaching the stable state in the low-temperature regime. To analyze the STMDP, we construct an auxiliary MDP by reducing the configuration space to the local minima of the Hamiltonian and show that this MDP is an accurate approximation of the original process if the time between two actions of the decision maker is sufficiently long. Exploiting the convenient form of this auxiliary MDP, we recursively solve the Bellman equations to unravel the structure of the optimal policy to optimally speed up the nucleation process. The structure of this paper is as follows. In Section 2 , we introduce the STMDP framework and discuss its relation to the FMDP and GMDP models. In addition, we provide some insights in the connection between the value function and first hitting times for MDPs with a reachability objective. In Section 3 , we formulate the two-dimensional low-temperature Ising STMDP and outline the construction of the auxiliary MDP. In addition, we state our main result, which concerns the structure of the optimal policy in the auxiliary MDP. Section 4 provides several numerical experiments that give insight in the performance of the optimal policy derived for the auxiliary MDP in the original Ising STMDP and draws a comparison with two alternative policies. In Section 5 , we formalize our analysis of the low-temperature Ising STMDP and give an outline of the proof of the result stated in Section 3.1 . Details of the proof are deferred to the Supplementary Material. 2 The spatiotemporal Markov decision process 2.1 Introduction to Markov decision processes A Markov decision process is defined as a tuple (S,A,P,r) , in which the elements are specified as follows ( Puterman, 2014 ) . • The state space S contains the possible states that the system can occupy. We assume S to be finite. • Let A_{s} denote the set of possible actions that can be taken from a state s\in S . The action space is defined as A=\cup_{s\in S}A_{s} . We assume the action space to be finite. • The transition probability kernel P:S\times A\times S specifies the dynamics of the MDP. Here, P(s^{\prime}|s,a) denotes the probability that the system will make a transition to state s^{\prime}\in S given that it is currently in state s\in S and the decision maker selected action a\in A_{s} . • The reward function r:S\times A\rightarrow\mathbb{R} specifies the immediate reward r(s,a) which the decision maker receives if he selects action a\in A_{s} from state s\in S . We assume that the reward function is bounded, i.e., that there is a constant M>0 such that |r(s,a)|\leq M for each s\in S , a\in A . Let T denote the set of decision epochs , i.e., the moments in time at which a decision can be made. We assume that T=\{0,1,2,\ldots\} . The behaviour of the decision maker is governed by a policy . We restrict our attention to policies that are stationary and deterministic . Such a policy applies at each decision epoch a deterministic decision rule d:S\rightarrow A that specifies the action d(s) that ought to be selected in state s\in S . The resulting policy is written as \pi=d^{\infty} . Let \Pi denote the set of stationary deterministic policies. The expected total discounted reward, or the value function v_{\lambda}^{\pi}(s) of a policy \pi=d^{\infty} in a state s\in S is defined as v^{\pi}_{\lambda}(s)=\mathbb{E}^{\pi}_{s}\left[\sum\limits_{t=0}^{\infty}\lambda^{t}r_{t}(X_{t},Y_{t})\right], where X_{t} denotes the state of the system at decision epoch t , Y_{t} the selected action at decision epoch t and \lambda\in(0,1) is a discount factor. By conditioning on the state reached at the first decision epoch, we obtain the following set of equations for