A Comprehensive Review of Transformer-based language models for Protein Sequence Analysis and Design

Nimisha Ghosh, Daniele Santoni, Debaleena Nawn, Eleonora Ottaviani and Giovanni Felici · 2025

Read A Comprehensive Review of Transformer-based language models for Protein Sequence Analysis and Design by Nimisha Ghosh, Daniele Santoni, Debaleena Nawn, Eleonora Ottaviani and Giovanni Felici (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-13646.txt.

Introduction

Transformer language models, built for sentences, turn out to read protein sequences too, treating amino acids as words. Nimisha Ghosh and four co-authors review the young field in 2025, gathering work on what these models can do with proteins. The uses span naming a protein's job and shape, designing new proteins, and predicting how proteins bind.

The review weighs each line of work for its strengths and gaps. Shortcomings in the current research are named plainly, with routes for fixing them. The authors aim to give newcomers a map of the state of the art. They close with directions they judge most promising for protein design.

Catalog record

CREATORNimisha Ghosh, Daniele Santoni, Debaleena Nawn, Eleonora Ottaviani and Giovanni Felici
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE98 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-13646

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Comprehensive Review of Transformer-based language models for Protein Sequence Analysis and Design Nimisha Ghosh, Daniele Santoni, Debaleena Nawn, Eleonora Ottaviani and Giovanni Felici Abstract The impact of Transformer-based language models has been unprecedented in Natural Language Processing (NLP). The success of such models has also led to their adoption in other fields including bioinformatics. Taking this into account, this paper discusses recent advances in Transformer-based models for protein sequence analysis and design. In this review, we have discussed and analysed a significant number of works pertaining to such applications. These applications encompass gene ontology, functional and structural protein identification, generation of de novo proteins and binding of proteins. We attempt to shed light on the strength and weaknesses of the discussed works to provide a comprehensive insight to readers. Finally, we highlight shortcomings in existing research and explore potential avenues for future developments. We believe that this review will help researchers working in this field to have an overall idea of the state of the art in this field, and to orient their future studies. † † impactstatement: Understanding the full potential of Transformer-based models to use them in a fruitful way in bioinformatics is the need of the hour. This review provides a thorough and critical examination of Transformer-based language models in the context of protein sequence analysis and design, unifying recent advancements in this domain. By synthesizing current methodologies and identifying key limitations, the paper offers a roadmap for leveraging Transformer architectures and accelerate the use of such protein language models. This survey further outlines pressing research gaps and proposes future directions, positioning the work as a foundational resource for interdisciplinary innovation in protein science. I Introduction Starting from the 1940s, scientists began to study the amino acid composition of proteins in order to characterize tissues and species by their amino acid frequencies [ 1 ] . Insulin was the first protein to be fully sequenced in 1951 and 1952 (chains of bovine insulin B and A, respectively) by Sanger [ 2 ] that led to him winning the Nobel prize in chemistry (1958). In the following years, as new technologies were developed, many other sequences became available thereby opening new frontiers in Biology and Chemistry. In fact, in the 70s a substantial number of sequences was made available, and several new issues related to protein sequences arose. Besides the simple analysis of amino acid frequencies [ 3 ] , scientists tried to infer functional and structural features of proteins from amino acid sequence patterns, their contexts and their combinatorics properties. Many works showed that classification methods were capable of separating proteins into different families [ 4 , 5 , 6 , 7 , 8 ] ) based exclusively on their sequences. In recent times, the influence of Natural Language Processing (NLP) in the field of bioinformatics is unprecedented. Neural networks have been applied to NLP tasks for over two decades, with word embeddings like Word2Vec [ 9 ] playing a key role in text representation. Word2Vec, using either Continuous Skip-Gram (CSG) or Continuous Bag-of-Words (CBOW), learns word vectors by predicting context from a target word or vice versa. The development of deep learning further advanced language modelling through architectures such as RNNs, Bi-RNNs [ 10 ] , LSTMs [ 11 ] , and GRUs [ 12 ] , which encode sequences into fixed-length vectors. However, these models often struggled with the vanishing gradient problem, which hindered effective training on long sequences. To address this, Transformer models [ 13 ] were introduced, replacing recurrent structures with a multi-head attention mechanism (see Section 2). Unlike RNNs, Transformers process sequences in parallel and capture word dependencies without relying on past hidden states, effectively mitigating gradient issues and improving performance on long-range dependencies. The applications of machine learning and deep learning, especially Transformers, in bioinformatics have brought about a huge change. A protein sequence can be viewed as a collection of overlapping/non-overlapping subsequences of a given length, that can be treated as words in a sentence, drawing a parallel between natural language processing (NLP) and protein sequences. Just as in human language, where the words are formed from letters, proteins can be represented as sequences of words constructed from the 20 amino acids. This kind of representation allows to apply NLP and Transformer-based architectures to protein sequences. In this regard, many Transformer-based models have been applied to nucleotide sequences [ 14 ] . In the nucleotide context, genome classification, starting from New Generation Sequencing (NGS) reads to species typing are often the major focus, while in the context of proteins the focus is on three-dimensional structure prediction or protein-protein binding as well as generation of synthetic proteins. Some significant milestones achieved in the context of protein research for Transformer-based models are outlined in Table I . Year Model Focus 2019 TAPE [ 15 ] Protein sequence classification and prediction 2020-2021 AlphaFold [ 16 ] , AlphaFold2 [ 17 ] Protein structure prediction 2022 ProtBERT [ 18 ] Protein sequence function prediction 2022 ProteinBERT [ 19 ] Protein function and structure prediction 2021-2022 ESM-models [ 20 , 21 , 22 ] Protein structure and function prediction 2020-2023 ProGen [ 23 ] and ProGen2 [ 24 ] Novel and viable protein sequence generation 2022 ProtGPT2 [ 25 ] , ESM-3 [ 26 ] De novo protein sequence generation TABLE I: Timeline of Transformer-based Models for Protein Sequences It is to be noted that almost all modern Transformer-based protein models use 1-mer (single amino acid) tokenization to preserve biological granularity. In the recent past, few reviews have been published based on protein structure prediction and protein design considering Transformers. In this regard, [ 27 ] has put forth algorithmic modelling techniques for protein structure prediction especially based on deep neural networks, where the authors have summarised works that cover MSA Transformer [ 28 ] , RoseTTAFold [ 29 ] , ProteinBERT [ 19 ] , RGN2 [ 30 ] etc.; all based on some form of Transformers. Dhanuka et al. [ 31 ] have also put forth a survey based on the applications of deep learning for the prediction of protein functions. In this regard, they have provided an insight into the interpretability of the models used in such prediction. Le [ 32 ] has also discussed the recent progress of Transformer-based NLP models in proteome bioinformatics to showcase their advantages, limitations and possible applications for improving their accuracy and efficiency. The review also provides an idea of the various challenges and the future directions of using the discussed models. Use of large scale Transformer models for predicting protein characteristics has been explored in [ 33 ] . The review in [ 34 ] examines the applications of large language models, particularly the ones based on Transformers, across protein design. The protein language models (pLMs) discussed in this work include ESM-based models, ProtGPT2, ProGen and ProGen2, PepMLM etc. Wang et al. [ 35 ] have also provided a comprehensive review of the pLMs encompassing the model architectures, positional encoding, scaling laws as well as the datasets. However, Wang et al. provides a macro perspective of the works being done in pLMs. The work presented here particularly addresses the use of Transformer-based models for

Read the complete text