Atom-level Protein Representation Learning Improves Protein Structure Prediction
Taewon Kim Hyosoon Jang Hyunjin Seo Seonghwan Seo Hyeongwoo Kim, [0.45em] Wonho Zhung Mingyeong Shin Wooyoun Kim Sungsoo Ahn [0.65em] KAIST {maxkim139, hyosoon.jang, sungsoo.ahn}@kaist.ac.kr · 2026
Read Atom-level Protein Representation Learning Improves Protein Structure Prediction by Taewon Kim Hyosoon Jang Hyunjin Seo Seonghwan Seo Hyeongwoo Kim, [0.45em] Wonho Zhung Mingyeong Shin Wooyoun Kim Sungsoo Ahn [0.65em] KAIST {maxkim139, hyosoon.jang, sungsoo.ahn}@kaist.ac.kr (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-22133.txt.
Introduction
Kim and co-authors propose TriProRep, an atom-level protein representation method that improves protein structure prediction. Pretrained representations can improve generation as features or targets, and the authors study them beyond conventional function annotation. It jointly models three aligned views: amino-acid identity, backbone geometry, and local full-atom shape. Their RepSP tests cover homodimer co-folding from apo-chain representations, values of homodimer-derived properties, and monomer results. Across these tasks it beats sequence-only and prior models.
Catalog record
| CREATOR | Taewon Kim Hyosoon Jang Hyunjin Seo Seonghwan Seo Hyeongwoo Kim, [0.45em] Wonho Zhung Mingyeong Shin Wooyoun Kim Sungsoo Ahn [0.65em] KAIST {maxkim139, hyosoon.jang, sungsoo.ahn}@kaist.ac.kr |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 66041 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.22133
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Atom-level Protein Representation Learning Improves Protein Structure Prediction Taewon Kim Hyosoon Jang Hyunjin Seo Seonghwan Seo Hyeongwoo Kim, [0.45em] Wonho Zhung Mingyeong Shin Wooyoun Kim Sungsoo Ahn [0.65em] KAIST {maxkim139, hyosoon.jang, sungsoo.ahn}@kaist.ac.kr Abstract Recent advances in generative modeling show that pretrained representations can improve generation as conditioning features or alignment targets. Motivated by this, we study protein representations for predicting structures beyond conventional function annotation. We propose TriProRep , a structure-aware pretraining method that jointly models three aligned residue-level views: amino-acid identity, backbone geometry, and local full-atom geometry, discretely encoded via VQ-VAE tokenizers. By pretraining to recover original tokens from generator-corrupted views, TriProRep learns to distinguish plausible but incorrect cross-view augmentations from the original protein. We further introduce RepSP , a benchmark for evaluating protein representations in structure-predictive settings. RepSP tests three uses of representations: homodimer co-folding from apo-chain representations, residue-level prediction of homodimer-derived interaction properties, and representation-aligned monomer structure prediction. Across these tasks, TriProRep improves over sequence-only and prior structure-aware representation models, while maintaining competitive performance on conventional benchmarks. † † footnotetext: ∗ Equal contribution. Author order was determined randomly. † Corresponding author. 1 Graduate School of AI, KAIST; 2 Department of Chemistry, KAIST; 3 HITS Inc.; 4 Department of AX, KAIST; 5 InnoCORE AI-CRED Institute, KAIST. Code is available at https://holymollyhao.github.io/TriProRep/ . 1 Introduction Structure-aware protein representation learning aims to produce residue-level features that can be reused by downstream models for three-dimensional reasoning such as complex prediction, interface modeling, and structure prediction. Yet this goal is only indirectly tested by many common protein representation benchmarks. Enzyme commission (EC) and gene ontology (GO) prediction ( 9 ) measure broad biological utility, but they do not ask whether a representation exposes the geometric information needed to predict interfaces, assemble complexes, or supervise structure-prediction models. As a result, sequence-only representation models can remain competitive with explicitly structure-aware models on these benchmarks, which we also observe in Table 6 , without resolving whether structural supervision has improved the representation for its intended use. This mismatch is increasingly important as protein representation learning moves beyond sequence-only pretraining. SaProt ( 28 ) and ProstT5 ( 12 ) augment amino-acid sequences with Foldseek 3Di tokens ( 30 ) . ESM3 represents proteins with coupled sequence, structure, and function tracks ( 11 ) ; and related methods condition sequence modeling on backbone structure or align sequence and structure representations ( 37 ; 33 ) . These approaches provide increasingly rich structural supervision, but their evaluation often remains centered on tasks that only weakly isolate residue-level geometry. A useful structure-aware representation should not merely improve functional annotation: it should provide features that downstream structure models can directly exploit. This perspective mirrors a broader trend in representation learning. In computer vision, pretrained representations are increasingly used not only as task features, but also to improve structured-output models through conditioning ( 19 ; 38 ; 25 ) and representation alignment-based distillation ( 39 ; 17 ) . We ask the analogous question for proteins: Question. Can pretrained protein representations serve as useful geometric signals for structure-predictive modeling? Figure 1 : TriProRep . (a) Three-view tokenization. A protein is independently tokenized into amino-acid, backbone, and full-atom token sequences at the residue level. (b) ELECTRA-style discriminative pretraining. A small generator corrupts each of the three sequences, and a large discriminator predicts the original token at every position. We study this question from both a methodological and evaluation perspective. Methodologically, we propose TriProRep , a structure-aware protein representation model that jointly encodes three aligned residue-level views: amino-acid identity, backbone geometry, and full-atom residue geometry. The two geometric views are discretized by VQ-VAE tokenizers ( 29 ) into backbone-geometry and full-atom-geometry tokens. Unlike prior tokenizations that mainly capture backbone-level geometry ( 30 ; 11 ; 40 ) , our full-atom tokens preserve local atomic information, including side-chain arrangements that are important for residue environments and interfaces. To train these coupled views, we use corrective pretraining with generator-corrupted token sequences ( 4 ) : a lightweight generator replaces masked tokens in each view with plausible alternatives, and the representation model recovers the original tokens. For evaluation, we introduce RepSP , the first benchmark specifically designed for structure-predictive protein representation learning. Built on homodimer complexes from the recent AFDB multimer release ( 10 ) , RepSP provides a uniformly processed million-scale setting for controlled evaluation of binding-relevant geometry from apo-chain representations, while acknowledging that homodimers are simpler than general heteromeric complexes. RepSP contains three complementary tasks: homodimer co-folding from frozen apo-chain representations, homodimer-derived residue-level prediction of binding sites, solvent-accessibility changes, interface regions, and interaction types, and representation-aligned structure prediction , where pretrained representations serve as dense alignment targets for structure prediction learning ( 39 ) . We pretrain and evaluate TriProRep at four scales, 35M, 150M, 650M, and 2.8B. On RepSP , TriProRep improves over sequence-only and prior structure-aware representation models across homodimer co-folding, homodimer-derived residue-level prediction, and representation-aligned structure prediction. On the EC/GO benchmarks, TriProRep remains competitive with the strongest overall results, preserving broad biological representation quality. Our contributions are summarized as follows: • We propose TriProRep , a three-view structure-aware protein representation model trained with generator-corrupted token recovery. • We introduce RepSP , a benchmark that evaluates whether protein representations are useful for structure-predictive tasks. • We show that TriProRep improves homodimer co-folding, homodimer-derived residue-level prediction, and representation-aligned structure prediction, while remaining competitive on EC/GO. 2 Related work Protein structure tokenization. A line of work uses VQ-VAE-based tokenizers ( 29 ) to convert protein structures into discrete tokens for scalable structure-aware protein representation modeling. Foldseek defines 3Di tokens as a structural alphabet for residue-residue interaction geometry ( 30 ) . ESM3 also uses tokens that discretize local backbone geometry into reconstructable tokens ( 11 ) , and AminoAseed improves tokenizer quality through codebook reparameterization and Pareto-optimal codebook configuration ( 40 ) . However, these tokenizations mainly focus on backbone-level geometry and omit full-atom geometry within residues, such as side-chain packing, and rotameric states. Structure-aware protein representation learning. Recent protein representation models incorporate 3D structural information into protein language modeling. SaProt ( 28 ) pairs amino-acid tokens with Foldseek’s 3Di tokens ( 30 ) , and ProstT5 ( 12 ) uses a T5 architecture to model bidirectional translation between sequence and 3Di tokens. ESM3 ( 11 ) introduces structure tokens alongside sequence and function tracks. In addition, MIF-ST ( 37 ) performs masked inverse folding with backbone-structure conditioning, while S-PLM ( 33 ) aligns sequence and structure representations through contrastive learning. These methods show that structural information can improve protein representations. We study whether such representations transfer to structure-predictive tasks. Protein structure and complex prediction. Protein structure prediction has progressed from single-chain folding to protein-protein complex prediction. AlphaFold-Multimer ( 7 ) , AlphaFold3 ( 1 ) , and Boltz ( 35 ; 22 ) predict protein complexes using MSA-derived representations, while SimpleFold ( 34 ) revisits single-chain folding with flow matching and a general-purpose Transformer architecture using protein sequence representation. Our work uses structure prediction as an evaluation setting, rather than proposing a new structure prediction model. We evaluate whether structure-aware representations extracted from a single apo protein support flexible-docking prediction of the protein complex and serve as distillation targets for training single-protein folding