Enhancing Reinforcement learning in 3-Dimensional Hydrophobic-Polar Protein Folding Model with Attention-based layers
Peizheng Liu, Hitoshi Iba · 2025
Read Enhancing Reinforcement learning in 3-Dimensional Hydrophobic-Polar Protein Folding Model with Attention-based layers by Peizheng Liu, Hitoshi Iba (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-15634.txt.
Introduction
Folding a protein chain on a lattice is a classic hard puzzle, simplified by sorting amino acids into two kinds: water-fearing and water-loving. Peizheng Liu and Hitoshi Iba teach an agent to fold such chains in three dimensions by trial and error. Their learner is a deep Q-network fitted with attention layers borrowed from transformer models, so it can weigh which parts of the growing chain matter most.
The walk the agent takes cannot cross itself, and rewards grow when water-fearing units end up touching. Tricks such as double and dueling Q-learning, prioritized replay of key moments, and symmetry-breaking rules sharpen the training. On standard test sequences the method hit several known best folds for short chains and came close on longer ones. The authors offer the system as a prototype for attention-based learning on lattice models.
Catalog record
| CREATOR | Peizheng Liu, Hitoshi Iba |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-15634
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Enhancing Reinforcement learning in 3-Dimensional Hydrophobic-Polar Protein Folding Model with Attention-based layers Peizheng Liu, Hitoshi Iba Abstract Transformer-based architectures have recently propelled advances in sequence modeling across domains, but their application to the hydrophobic-hydrophilic (H-P) model for protein folding remains relatively unexplored. In this work, we adapt a Deep Q-Network (DQN) integrated with attention mechanisms (Transformers) to address the 3D H-P protein folding problem. Our system formulates folding decisions as a self-avoiding walk in a reinforced environment, and employs a specialized reward function based on favorable hydrophobic interactions. To improve performance, the method incorporates validity check including symmetry-breaking constraints, dueling and double Q-learning, and prioritized replay to focus learning on critical transitions. Experimental evaluations on standard benchmark sequences demonstrate that our approach achieves several known best solutions for shorter sequences, and obtains near-optimal results for longer chains. This study underscores the promise of attention-based reinforcement learning for protein folding, and created a prototype of Transformer-based Q-network structure for 3-dimensional lattice models. 1 Introduction H-P model has been considered as a simplified model for protein structure prediction. However, optimizing the structure of H-P model still requires efficient algorithms due to the large solution space. Determining the optimal structure of proteins under the hydrophobic–hydrophilic (HP) model has been rigorously shown to be NP-complete [ 1 ] , highlighting the necessity for powerful heuristic or approximation methods in lieu of brute-force searches. Among heuristic approaches, Monte Carlo simulations are particularly popular and exhibit a wide range of implementations [ 2 ] [ 3 ] . In particular, the pruned-enriched Rosenbluth method (PERM)—a sequential Monte Carlo algorithm with resampling—has emerged as a strong performer for HP folding, producing several of the best-known solutions reported to date [ 4 ] . Likewise, replica exchange Monte Carlo (REMC) has also demonstrated top-level performance [ 5 ] . Beyond Monte Carlo–based methods, various evolutionary algorithms have also shown efficacy in this domain. For instance, ant colony optimization (ACO) has been successfully applied to both two-dimensional and three-dimensional formulations of the HP model, yielding competitive results [ 6 ] . Other well-known evolutionary and swarm-based methods—including genetic algorithms [ 7 ] [ 8 ] and particle swarm optimization [ 8 ] —have been tested on HP folding problems, also achieving effective approximations. Recently, machine learning approaches have also been employed to tackle HP folding. Among these approaches, deep reinforcement learning (DRL) has emerged as a promising method due to its ability to handle sequential decision-making tasks in high-dimensional and complex environments. One pioneering study applying RL to the HP folding task was ”FoldingZero” by Li et al. in 2018 [ 9 ] . That work utilized a deep convolutional neural network (CNN) integrated with a Residual Upper Confidence Tree (R-UCT) approach, all trained using an RL algorithm. Further advancements in the application of RL to HP folding were explored by Yang et al. [ 10 ] , who evaluated several foundational approaches from recent RL studies. These included the use of Long Short-Term Memory (LSTM) networks [ 11 ] , replay memory, and the Double Deep Q-Network (Double DQN) [ 12 ] . Transformer models , first introduced by Vaswani et al. in the landmark paper ”Attention Is All You Need” (2017) [ 13 ] , have emerged as a pivotal innovation in the field of machine learning. At their core, Transformers leverage the attention mechanism, a method that allows the model to assign varying levels of importance to different parts of the input data. This approach addresses key limitations of earlier sequence-based architectures such as recurrent neural networks (RNNs) and long short-term memory networks (LSTMs), which are constrained by their sequential processing nature and difficulties in capturing long-range dependencies [ 13 ] . In this study, we extend the application of the transformer models to address the H-P model. While deep reinforcement learning techniques, such as Deep Q-Networks (DQNs) combined with RNNs or LSTMs, have been explored in H-P model research, transformer architectures remain underexplored. This gap presents an opportunity to investigate the potential advantages of transformers in modeling the unique characteristics of the H-P problem. The H-P model encapsulates complex, non-sequential dependencies between folding steps driven by the spatial constraints. Traditional sequential models, such as RNN-LSTM structure, often struggle to capture these intricate dependencies due to their reliance on step-by-step processing. In contrast, the transformer model, with its self-attention mechanism, is inherently able to model such non-local interactions by attending to all relevant steps in parallel. This capability aligns well with the demands of H-P folding, which we aim to leverage in this study. 2 Methodology To address the H-P protein folding problem, we use a self-avoiding walk (SAW) process. This environment emulates the folding process by enabling folding actions to be applied, and it generates observations encapsulating the resulting structural, serving as a dynamic platform for simulating and analyzing folding pathways. Within this framework, a folding agent is employed to interact with the environment. The agent, powered by a Deep Q-Network (DQN) with attention-based layers, makes sequential decisions that guide the folding process. Through iterative interaction, the agent learns to optimize folding strategies by leveraging feedback from the environment, ultimately improving its decision-making capabilities. Figure 1 : Overall Training Data Flow. The process begins with an empty folding environment, where the initial state s_{0} is obtained by the agent. The agent iteratively follows a state-action-reward cycle until an episode terminates. After termination, the reward is backpropagated through each step and the environment is reset empty for the next episode. Training of the policy network is performed on a batch of samples from the replay memory after each step. The target network is periodically updated by copying the parameters of the policy network. A brief overview of the entire training data flow is shown in Figure 1 . The detailed methodology, including the environment’s design, the structure of the neural network of the folding agent, and the training protocol, will be described in the following sections. 2.1 H-P Model Folding Environment The protein folding environment is constructed on a three-dimensional cubic lattice implemented with Python lists to represent the spatial arrangement of protein structures. Within this lattice, amino acids are positioned at specific, discrete coordinates, ensuring that each residue occupies a unique and fixed location in space. The input sequence of amino acids is encoded as a string of ’H’ (hydrophobic) and ’P’ (polar) characters. Placements and observations interact with lattice data through class functions defined in the standard format of the gymnasium.Env class from the OpenAI Gymnasium framework. The OpenAI Gymnasium framework provides a standardized interface for defining and interacting with reinforcement learning environments. Each environment is implemented as a Python class that defines an action space and an observation space . The action space specifies the set of all possible actions an agent can take, while the observation space describes the format and bounds of the data received by the agent. This framework allows agents to interact with the environment in discrete time steps by selecting actions, observing resulting states, and receiving rewards, facilitating the design and evaluation of reinforcement learning tasks. The following subsections elucidate the core components and operational mechanics of the folding environment. 2.1.1 Amino Acid Placement Action The initial two