Sampling two-dimensional spin systems with transformers Journal: Computer Physics Communications

Piotr Białas, Piotr Korcyl, Tomasz Stebel, Adam Stefański, Dawid Zapolski · 2026

Read Sampling two-dimensional spin systems with transformers Journal: Computer Physics Communications by Piotr Białas, Piotr Korcyl, Tomasz Stebel, Adam Stefański, Dawid Zapolski (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-27738.txt.

Introduction

Białas and four colleagues propose a transformer sampler for two-dimensional spin systems that generates groups of spins per step instead of one. They also build an approximated-probability model, further improving efficiency. Despite heavier computation than dense or convolutional networks, their sampler handles systems up to 180 by 180 spins for the Ising model. Its effective sample size is about 20 times the previous best neural sampler's on the 128 by 128 Ising model at critical temperature. They also test the method on the two-dimensional Edwards-Anderson model with 64 by 64 spin systems.

Catalog record

CREATORPiotr Białas, Piotr Korcyl, Tomasz Stebel, Adam Stefański, Dawid Zapolski
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45035
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.27738

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Sampling two-dimensional spin systems with transformers Journal: Computer Physics Communications Piotr Białas, Piotr Korcyl, Tomasz Stebel, Adam Stefański, Dawid Zapolski Abstract Autoregressive Neural Networks based on dense or convolutional layers have recently been shown to be a viable strategy for generating classical spin systems. Unlike these methods, sampling with transformers is commonly considered to be computationally inefficient. In this work, we propose a novel approach to transformer-based neural samplers in which we generate not a single spin per step but groups of spins. As an additional improvement, we construct a model of approximated probabilities, further improving the efficiency of the algorithm. Despite our approach being computationally heavier than dense networks or CNN-based approaches, we were able to sample larger systems of up to 180 × 180 180\times 180 spins in case of the Ising model. The Effective Sample Size of our sampler is ∼ 20 \sim 20 times larger than that of the previous state-of-the-art neural sampler when trained for the 128 × 128 128\times 128 Ising model at critical temperature. Finally, we also test our algorithm on the 2D Edwards-Anderson model, where we train 64 × 64 64\times 64 spin systems. 1 Introduction Simulating physical systems is one of the central challenges that led to the development of Monte Carlo (MC) algorithms. Beginning with the pioneering work of Nicholas Metropolis and collaborators [ 1 ] , the past 70 years have witnessed remarkable progress in Markov Chain Monte Carlo (MCMC) methods, resulting in a wide range of applications across science, engineering, finance, and beyond. Despite this tremendous success, several fundamental limitations of MCMC methods remain. Chief among these are autocorrelations between successive samples and potential issues with ergodicity, which can hinder efficient exploration of complex probability landscapes. With the advent of modern neural network–based approaches, new opportunities have emerged for the design of more efficient and scalable sampling algorithms, raising hopes of overcoming some of the longstanding challenges associated with traditional MCMC techniques. Sampling classical spin systems using neural networks was proposed in [ 2 ] , where the autoregressive architecture models the conditional probabilities of consecutive spins. The loss function used to train those models is the variational free energy; hence the name of the algorithm: Variational Autoregressive Networks (VAN). In [ 2 ] authors used two architectures of autoregressive networks: dense layers with masked connections, known as Masked Autoencoder for Distribution Estimation (MADE) [ 3 ] and convolutional layers with masked kernels. The main limitations of the VAN are the unfavorable scaling of numerical cost with the system size and poor training efficiency at large system sizes. For the simplest spin model, the 2-dimensional Ising model, the original VAN cannot be effectively trained for system sizes larger than 32\times 32 spins. On the other hand, the algorithm is very flexible concerning the type of interactions – as far as the energy of the spin configuration can be calculated, there are no limitations. Motivated by the problems of VAN in simulating large system sizes, we proposed in Ref. [ 4 ] a modification of this algorithm that utilizes the properties of the nearest neighbors’ interactions and the translational symmetry of the Ising model. Hierarchical Autoregressive Networks (HAN) replaced one large dense network from VAN with a hierarchy of much smaller networks, which can be used multiple times to fix the spins in different parts of the configuration. With that change, the algorithm can effectively simulate a 2D Ising system with 64\times 64 spins. The HAN algorithm was later applied to the calculations of mutual information [ 5 ] , entanglement entropy [ 6 ] , density matrix [ 7 ] , as well as to the simulation of the Potts model [ 8 ] and the 3D Ising model [ 9 ] . On the other hand, the more complicated models, like spin glasses, cannot be sampled. An alternative approach to simulations of spin systems with autoregressive networks by Biazzo et al. [ 10 , 11 ] takes into consideration also the form of the interactions between spins. By incorporating parts of these interactions into the model, the authors obtained much better performance of the model compared to VAN (MADE) when simulating spin glass systems. Recently, yet another neural network algorithm for sampling the Ising model was proposed [ 12 ] . It is inspired by the renormalization group technique and performs sampling from low-resolution to high-resolution lattice configurations using conditional probability. Authors report that the Renormalization-informed Generative Critical Sampler (RiGCS) can effectively sample 2D Ising systems of size 128\times 128 spins. The above-mentioned algorithms utilize the physical properties of spin systems, but the neural network architectures that are used are usually just dense or convolutional networks. In this contribution, we shall present a version of the VAN algorithm that is based on the transformer architecture [ 13 ] . We call our algorithm "transformer VAN" (tVAN), to distinguish it from the original proposal. We show that even with a relatively small transformer model, a sampler can efficiently simulate the 2D Ising model with 180\times 180 spins and the spin glass system with 64\times 64 spins. With such system sizes trained, tVAN is a state of the art neural sampler of spin systems. It enjoys the flexibility of the original VAN concerning the variety of interactions it can simulate. We test our tVAN architecture on the 2D classical spin system with a square lattice of size L\times L spins and periodic boundary conditions. The spins have values s^{i}=\pm 1 . The energy of the configuration is given by: E(\mathbf{s})=-\sum_{\langle i,j\rangle}J_{i,j}s^{i}\,s^{j}\,, (1) where the sum is performed over the nearest neighbor interactions. Our default choice in this manuscript is the Ising model, where J_{i,j}=1 for all i,j . We shall also consider the Edwards-Anderson (EA) spin glass model with J_{i,j}=\pm 1 , where the value of each J_{i,j} is sampled with probability 1/2 . The probability of the spin configuration is given by the Boltzmann distribution: \displaystyle p(\mathbf{s})=\frac{1}{Z}\exp(-\beta E(\mathbf{s}))\,, (2) where \beta is an inverse temperature, \beta=1/k_{B}T ; Z is the partition function, Z=\sum_{\mathbf{s}}\exp(-\beta E(\mathbf{s})) , and the sum is performed over all 2^{L^{2}} configurations. The free energy is defined as: F=-\frac{1}{\beta}\ln Z. (3) In the Ising model, the exact value of F can be calculated for arbitrary \beta and L [ 14 ] . We take it as our benchmark, which we compare with variational values obtained from the models, F_{q} . The Ising model in 2D exhibits a second order phase transition when L\to\infty . The critical inverse temperature is given by \beta_{c}=\frac{1}{2}\ln(1+\sqrt{2}) . In what follows, when considering the Ising model, we use \beta=\beta_{c} . In case of the spin glass models like the EA model considered here, one has to average over the link variables J_{ij} and equation ( 3 ) becomes F=-\frac{1}{\beta}\left\langle{\ln Z}\right\rangle_{J}. (4) As the aim of this reference is not the study of spin glasses but a proof of concept of the method, we only present results for several sets of coupling J without any averaging. In general, the spin glass models do not exhibit spontaneous symmetry breaking but rather a glassy transition associated with replica symmetry breaking [ 15 , 16 ] . However, the two-dimensional system considered here only exhibits

Read the complete text