Convolutional transformer wave functions

Ao Chen, Vighnesh Dattatraya Naik, Markus Heyl · 2025

Read Convolutional transformer wave functions by Ao Chen, Vighnesh Dattatraya Naik, Markus Heyl (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-10462.txt.

Introduction

Ao Chen, Vighnesh Naik, and Markus Heyl blend two network styles into a new wave function for solving quantum many-body puzzles. Their 2025 paper uses neural quantum states, where a network stands in for the wave that describes a quantum system. Transformer networks looked strong for the job, yet past tries had not drawn out their full power.

Their design feeds the transformer's input through a convolutional front, which reads local patterns first. In tests, the blend beat past transformer wave functions both at finding the lowest energy state of a system and at tracking how it moves out of balance. The authors pitch the gain as a step toward harder quantum problems.

Catalog record

CREATORAo Chen, Vighnesh Dattatraya Naik, Markus Heyl
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE24 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-10462

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Convolutional transformer wave functions Ao Chen, Vighnesh Dattatraya Naik, Markus Heyl Abstract Deep neural quantum states have recently achieved remarkable performance in solving challenging quantum many-body problems. While transformer networks appear particularly promising due to their success in computer science, we show that previously reported transformer wave functions haven’t so far been capable to utilize their full power. Here, we introduce the convolutional transformer wave function (CTWF). We show that our CTWFs exhibit superior performance in ground-state search and non-equilibrium dynamics compared to previous results, demonstrating promising capacity in complex quantum problems. * * footnotetext: These authors contributed equally to this work Introduction. — The accurate numerical solution of strongly correlated quantum matter remains as an outstanding challenge in modern quantum physics. This concerns in particular the regime of large two-dimensional quantum many-body systems despite of impressive theoretical developments. In recent years the neural quantum state (NQS) has emerged as a promising numerical method to solve the quantum many-body problem. The NQS is based on utilizing artificial neural networks (ANNs) to encode the quantum many-body wave functions Carleo and Troyer 2017 . As ANNs are universal function approximators the NQS becomes a numerically exact approach converging, in principle, to the exact solution upon increasing the size of the ANN. Until now, the NQS has shown great potential in various quantum many-body problems, including quantum spin liquids Nomura and Imada 2021 ; Astrakhantsev et al. 2021 ; Chen and Heyl 2024 , Fermi-Hubbard models Nomura et al. 2017 ; Luo and Clark 2019 ; Moreno et al. 2022 , electronic structures Hermann et al. 2023 ; Choo et al. 2020 ; Pfau et al. 2020 ; Hermann et al. 2020 ; Pfau et al. 2024 , open quantum systems Nagy and Savona 2019 ; Hartmann and Carleo 2019 ; Vicentini et al. 2019 , and quantum dynamics Schmitt and Heyl 2020 ; Schmitt et al. 2022 ; Sinibaldi et al. 2023 ; Mendes-Santos et al. 2023 ; Nys et al. 2024 ; Mendes-Santos et al. 2024 . The key enabling potential for the NQS technique is the expressive power of the underlying ANNs, in particular when it comes to modern deep architectures. While the initial starting points were based on still relatively shallow networks such as restricted Boltzmann machines (RBMs) Carleo and Troyer 2017 ; Nomura 2021 , in the recent years many deeper networks have already been investigated, including convolutional neural networks (CNNs) Choo et al. 2019 , variational autoregressive networks Sharir et al. 2020 , recurrent neural networks (RNN) Hibat-Allah et al. 2020 , group CNNs Roth et al. 2023 , and deep CNNs Liang et al. 2023 ; Chen and Heyl 2024 . As a consequence of recent advances in training algorithms, it has now also become possible to optimize deep NQSs with up to 10^{6} parameters thereby pushing the NQS approach more towards exploiting the full power of ANNs. In particular, with such deep NQSs unprecedented numerical accuracies have been reached for multiple frustrated quantum magnets Chen and Heyl 2024 . In the field of machine learning, transformer neural networks have developed into the most powerful architecture for many tasks Vaswani et al. 2017 ; Dosovitskiy et al. 2021 . Consequently, it is a natural immediate question to which extent such transformer architectures could have a similar powerful potential for NQS. So far, in the context of NQS transformers have already been applied in electronic structure problems von Glehn et al. 2023 ; Pescia et al. 2024 ; Shang et al. 2024 ; Pfau et al. 2024 and quantum lattice models Viteritti et al. 2023 ; Zhang and Di Ventra 2023 . However, as we will discuss in this work, these previously introduced design choices in particular for lattice models are not in such a form yet so as to fully exploit the power of transformer architectures: they either are actually equivalent to CNNs, and therefore don’t fully exploit the key attention mechanism in transformers, or break translation symmetries. Thus, it has so far remained open how to fully utilize the full potential of transformer architectures in the context of quantum lattice models. Figure 1: (a) Network architecture of the CNN (GELU) improved from Ref. Chen and Heyl 2024 and the convolutional transformer wave function (CTWF). (b) Building blocks of CNN (GELU) and CTWF. The main block of CTWF consists of three consecutive parts, namely the convolutional unit (Conv Unit), the multi-head self-attention (MHSA), and the inverted residual feed-forward network (IRFFN). A normalization step (Norm) is applied at the beginning of all these parts by dividing the inputs by the expected initial standard deviation. The MHSA contains multiple self-attention (SA) blocks, each processed by Eq. ( 1 ). The IRFFN consists of 3 consecutive convolutional layers, and the middle one with an expanded channel dimension is depthwise (DW) to reduce the computing cost. Finally, the pair complex activation function is applied to convert real values to complex outputs Roth et al. 2023 . The illustration shown here has channel dimension c=4 , self-attention token dimension d=2 , and the number of heads h=2 . In this work, we introduce the convolutional transformer wave function (CTWF) inspired by recently developed variants of transformers in computer vision tasks Wu et al. 2021a ; Guo et al. 2022 . In particular, we develop design choices motivated by physics principles and apply them to quantum spin systems. The performance of CTWF for ground-state search is compared with the best previous results in the literature as well as a CNN (GELU) architecture, which we introduce here as an improvement from our previous work Chen and Heyl 2024 . As a first challenging benchmark we consider the ground state search of the prototypical 10\times 10 J_{1} - J_{2} Heisenberg model realizing a frustrated quantum magnet. We find that both CTWF and CNN (GELU) outperform the best previous result given a similar amount of parameters in the underlying ANN. Furthermore, we also benchmark the performance of CNN (GELU) and CTWF for quantum quench dynamics in the two-dimensional quantum Ising model. We observe that the CTWF and CNN (GELU) provide reliable evolution trajectories for a much longer time as compared to the best existing result in the literature. Overall, for the considered physics problems and design choices we conclude that both the CTWF and CNN (GELU) achieve superior performance as compared to existing literature. Transformer architectures. — In the following, we revisit previously utilized transformer wave functions and introduce our CTWF whose structure is illustrated in Fig. 1 . The core ingredient of transformer neural networks is the multi-head self-attention (MHSA), which can be viewed as a parallelization of h attention heads Vaswani et al. 2017 . Each attention head takes the same input x\in\mathbb{R}^{n\times c} , where n=l_{x}\cdot l_{y} represents the flattened spatial dimension and c represents the embedded token dimension (or channel dimension in the context of CNN). The attention output is A\in\mathbb{R}^{n\times d} given by A_{i}=\sum_{j}\alpha(Q_{i},K_{j},P_{ij})V_{j} , where \alpha\in\mathbb{R}^{n\times n} is the attention coefficient, and Q\in\mathbb{R}^{n\times d} , K\in\mathbb{R}^{n\times d} , V\in\mathbb{R}^{n\times d} , and P\in\mathbb{R}^{n\times n} are query, key, value, and positional encoding, respectively. Here d is the so-called token dimension chosen as d=c/h in this work. Below, we list multiple

Read the complete text