PETRA: Pretrained Evolutionary Transformer for SARS-CoV-2 Mutation Prediction
Xu Zou · 2025
Read PETRA: Pretrained Evolutionary Transformer for SARS-CoV-2 Mutation Prediction by Xu Zou (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-03976.txt.
Introduction
Xu Zou, writing alone, introduces PETRA, a pretrained evolutionary transformer for predicting future mutations of SARS-CoV-2, the virus whose rapid, uneven evolution keeps challenging public health and vaccine design. Raw genomic sequences are noisy, so PETRA trains instead on evolutionary trajectories drawn from phylogenetic trees.
Tree-based training captures the hierarchy of viral change and reduces sequencing noise, while weighted training counters geographic and temporal imbalance in global data. Recall scores for nucleotide and spike amino-acid mutations far exceeded the best baseline reported. PETRA also helped track major recent clades in real time. Code was released openly for others to test and extend.
Catalog record
| CREATOR | Xu Zou |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-03976
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
PETRA: Pretrained Evolutionary Transformer for SARS-CoV-2 Mutation Prediction Xu Zou Abstract Since its emergence, SARS-CoV-2 has demonstrated a rapid and unpredictable evolutionary trajectory, characterized by the continual emergence of immune-evasive variants. This poses persistent challenges to public health and vaccine development. While large-scale generative pre-trained transformers (GPTs) have revolutionized the modeling of sequential data, their direct applications to noisy viral genomic sequences are limited. In this paper, we introduce PETRA ( P retrained E volutionary TRA nsformer), a novel transformer approach based on evolutionary trajectories derived from phylogenetic trees rather than raw RNA sequences. This method effectively mitigates sequencing noise and captures the hierarchical structure of viral evolution. With a weighted training framework to address substantial geographical and temporal imbalances in global sequence data, PETRA excels in predicting future SARS-CoV-2 mutations, achieving a weighted recall@ 1 1 of 9.45 % 9.45\% for nucleotide mutations and 17.10 % 17.10\% for spike amino-acid mutations, compared to 0.49 % 0.49\% and 6.64 % 6.64\% respectively for the best baseline. PETRA also demonstrates its ability to aid in the real-time mutation prediction of major clades like 24F(XEC) and 25A(LP.8.1). The code is open sourced on 1 1 1 https://github.com/xz-keg/PETra 1 Introduction Since its introduction to the human population, SARS-CoV-2 evolves at an extraordinary rate. ( Amicone et al. 2022 ) Although the World Health Organization (WHO) stops designating new SARS-CoV-2 variants with new greek letters since 2022, novel Nextstrain major clades ( Aksamentov et al. 2021 ) , variants that possess significant additional mutations and achieve substantial regional or global prevalence at rapid speed, continue to emerge rapidly and persistently. Such mutations often confer strong immune evasion against antibodies elicited by previous infections or vaccinations ( Cao et al. 2022 ; Rubio-Casillas et al. 2022 ; Uraki et al. 2023 ) , contributing to increasing rates of repeated infections and long-term sequelae. ( Bowe et al. 2023 ) Consequently, the ongoing viral evolution and its health sequelae represent escalating societal challenges. ( Al-Aly et al. 2024 ) Simultaneously, the technology of Generative Pretrained Transformers (GPTs) ( Radford et al. 2018 ) have undergone rapid development. Initially developed for natural language generation, GPTs have demonstrated remarkable effectiveness in handling various domains of sequential data including image, video, audio ( Hurst et al. 2024 ) and genetic sequences ( Avsec et al. 2025 ) . However, these models still struggle to model SARS-CoV-2 mutations. Genetic models are usually trained on DNA/RNA sequences. In genetic sequencing, the rate of sequence errors ranges from 0.4\% to 13\% depending on different sequencing methods ( Dohm et al. 2020 ) . This translates to 100-4,000 errors per sequence, far exceeding the number of mutations of SARS-CoV-2 variants. In this research, we demonstrate the feasibility of training a generative pretrained transformer based on evolutionary trajectories extracted from phylogenetic trees instead of raw RNA sequences, reducing noise caused by artifacts during sequencing of individual sequences. We term the model P retrained E volutionary TRA nsformer( PETRA ), and apply the model on SARS-CoV-2 mutation prediction. Experimental results show that PETRA achieves significantly better performance in predicting future mutations compared to established baseline methods. Furthermore, PETRA demonstrates practical utility by aiding the real-time prediction of the evolutionary trajectories of major clades like 24F(XEC) and 25A(LP.8.1). Our investigation into the global distribution of SARS-CoV-2 sequence data reveals substantial geographical and temporal imbalances, for which we develop a weighted training framework to address. While this approach can mitigate the effects of dataset imbalance, significantly enhancing the predictive performance of PETRA , we contend that the underlying data imbalance remains a significant challenge for comprehensive SARS-CoV-2 evolution tracking. 2 Backgrounds and Related Works 2.1 SARS-CoV-2 Mutations and Phylogenetic Tree Since SARS-CoV-2 starts to infect humans, people begin to collect its sequences. ( Wang et al. 2020 ) SARS-CoV-2 is an RNA virus with a reference genome of 29,903 nucleotides. It contains multiple open reading frames (ORFs) that encode the proteins to be synthesized. Mutations are changes to the nucleotide sequence, typically involving substitutions, insertions, and deletions. Some mutations within ORFs cause changes in the synthesized proteins, some even make large differences like frameshifts, early stops, or removal of whole ORFs, while others have no obvious functional impact. There exist some researches on the effects of some mutations ( Plante et al. 2021 ; Motozono et al. 2021 ; Saito et al. 2022 ) , but the exact functions of most mutations remain unknown. More recent studies show that the advantages of some of the most important mutations like S:Q493E on recent lineages rely on the existence of some other mutations. ( Taylor & Starr 2024 ) Therefore, different lineages may adopt different mutation patterns, trending towards mutation spectrum. ( Bloom et al. 2023 ) The Bloom estimator ( Bloom & Neher 2023 ) is an open-source deep-mutational-scanning(DMS) based project that scans through all mutations on every major variant and computes the likelihood to appear and potential fitness for mutation analysis and prediction. The project 2 2 2 https://github.com/jbloomlab/SARS2-mut-fitness is updated on a regular basis, but slowed recently. The latest version of the project is released in Nov 2024. Separating different variants is vital given the mutation spectrum phenomenon. There are two variant classification systems, Pango ( O’Toole et al. 2022 ) and Nextstrain ( Aksamentov et al. 2021 ) for SARS-CoV-2. Nextstrain focuses on major clades that flourish on continental or global level, while Pango focuses on any novel lineages that could be related to possible epidemic events. Both systems are mostly maintained by a group of volunteer researchers who analyse sequences and discover and suggest new variants from time to time. The variant discovery process is now partially automated. ( Zou 2024 ) Details of the systems are introduced in the appendix. Once a variant is designated, its relative growth can be estimated according to the sampling time and location of its sequences. New SARS-CoV-2 waves are usually driven by fast-growing new variants. Cov-spectrum ( Chen et al. 2022 ) analyzes the growth of these new variants. Most of the global SARS-CoV-2 sequences are primarily shared on the following platforms: GISAID ( Shu & McCauley 2017 ) , GenBank ( Benson et al. 2012 ) , Cog-UK ( Marjanovic et al. 2022 ) , and CNCB ( Members et al. 2024 ) . Ultrafast Sample placement on Existing tRee (UShER) ( Turakhia et al. 2021 ) aggregates sequence data on multiple databases to build a unified phylogenetic mutation tree. It maintains an existing mutation tree and attempts to add new sequences at the most likely positions, resulting in a continuously updated phylogenetic tree that represents the estimated mutation trajectories of all sequences. Genetic sequences of SARS-CoV-2 are known to be imperfect since the virus is introduced to the human population. ( De Maio et al. 2020 ) To make things worse, the virus mutates and different variants start to co-infect people, expanding the source of artifacts. ( Pipek et al. 2024 ) In genetic sequencing, the rate of sequence errors ranges from 0.4\% to 13\% depending on different sequencing methods ( Dohm et al. 2020 ) , which is 100-4,000 considering the reference length of 29,903 for SARS-CoV-2. On the other hand, SARS-CoV-2 variants have