Yeti : A compact protein structure tokenizer for reconstruction and multi-modal generation
Nabin Giri Steven Farrell Kristofer E. Bouchard [email protected] [email protected] [email protected] Lawrence Berkeley National Laboratory 1 Cyclotron Road, Berkeley, CA 94720 · 2026
Read Yeti : A compact protein structure tokenizer for reconstruction and multi-modal generation by Nabin Giri Steven Farrell Kristofer E. Bouchard [email protected] [email protected] [email protected] Lawrence Berkeley National Laboratory 1 Cyclotron Road, Berkeley, CA 94720 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-09981.txt.
Introduction
Giri and two colleagues built Yeti, a compact protein structure tokenizer for reconstruction and multi-modal generation. Multi-modal models that jointly reason over protein sequences, structures, and function notes need a tokenizer turning smooth atomic coordinates into discrete tokens for large training. Models are only as good as their tokens, yet existing tokenizers favor reconstruction over generative ability.
Yeti uses look-up free quantization trained end to end with a flow matching goal. It led on codebook use and token diversity, and placed second on rebuild accuracy with 10 times fewer parameters than ESM3. A small multi-modal model trained from scratch over Yeti tokens plus amino-acid sequence made plausible structures under unconditional co-generation, matching models 10 times larger. Yeti is thus a small, expressive tokenizer for co-making plausible sequences and structures.
Catalog record
| CREATOR | Nabin Giri Steven Farrell Kristofer E. Bouchard [email protected] [email protected] [email protected] Lawrence Berkeley National Laboratory 1 Cyclotron Road, Berkeley, CA 94720 |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 65 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.09981
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Yeti : A compact protein structure tokenizer for reconstruction and multi-modal generation Nabin Giri Steven Farrell Kristofer E. Bouchard [email protected] [email protected] [email protected] Lawrence Berkeley National Laboratory 1 Cyclotron Road, Berkeley, CA 94720 Abstract Multi-modal models that jointly reason over protein sequences, structures, and function annotations within a unified representation hold immense potential for integrating multi-modal data and generating new proteins with designed functional properties. To utilize transformer architectures, such models require a tokenizer that converts protein structure from continuous atomic coordinates into discrete representations suitable for scalable multi-modal training. The quality of such models are fundamentally upper-bounded by the fidelity and expressiveness of the underlying tokenized structure. However, existing tokenizers prioritize reconstruction over generative abilities. To address these gaps, we introduce Yeti , a simple and compact protein structure tokenizer based on look-up free quantization and trained end-to-end with a flow matching objective for multi-modal learning. Compared to existing models, Yeti generally achieves the best codebook utilization and token diversity, and second best reconstruction accuracy (with 10x fewer parameters than ESM3) on diverse datasets. To validate Yeti ’s generative capability, we trained a compact multi-modal model jointly over its structure tokens and amino acid sequence entirely from scratch, with no pretrained initialization. The resulting multi-modal model generates plausible structures under unconditional co-generation of protein sequence and structures, achieving comparable results to 10x larger models. Together, these results demonstrate that Yeti is a compact and expressive protein structure tokenizer suitable for training multi-modal models that co-generates highly plausible sequences and structures. 1 Introduction Biological systems are among the most complex and multifaceted systems in nature, shaped by billions of years of evolution and governed by intertwined physical, chemical, and biological processes. Understanding and modeling such systems has long been a central challenge in computational biology, driving the development of innovative methods that balance accuracy, flexibility, and scalability. However, current methods struggle along three interconnected areas; first, biological data is inherently heterogeneous spanning biophysical constraints, biochemical annotations, and experimentally derived modalities such as function-annotated text, two-dimensional imaging, and three-dimensional volumetric data from cryo-electron microscopy and tomography with each modality carrying distinct statistical properties and noise characteristics cryo2structdata ; cryodenoise_data . Second, real biological datasets are incomplete by nature; modalities are frequently missing, measurements are noisy, and coverage across the structure and function space of proteins remains uneven. Third, biological processes are multiscale and coupled, ranging from atomic interactions to cellular behavior, yet most models operate at a single scale and modality without mechanisms to transfer information across levels. These challenges are compounded by the growing demand for models that are not only accurate but also steerable, interpretable, and sample-efficient, a requirement that become critical as we move toward unified multi-modal systems capable of modeling the full complexity of biological system. Proteins sit at the center of this challenge. As the primary functional units of the cell, their function is determined not just by one-dimensional (1D) sequence alone but by three-dimensional (3D) atomic structure, and understanding that structure-function relationship requires representations that are both expressive and compatible with large-scale learning. Recent generative models have addressed this by treating proteins as 3D objects, utilizing complex architectures operating in SE(3) and diffusion processes in continuous coordinate space. A complementary and increasingly compelling alternative is to encode 3D atomic structure as discrete tokens, as in ESM3 esm3 , DPLM2 dplm2 , and SaProt saprot , which frame protein modeling as masked or auto-regressive language modeling over structure vocabularies jointly with inherently discrete modalities such as sequence and functional annotations. An advantage of auto-regressive model is that it allows protein structure generation without predefined length which is desirable in a case where given an experimental cryo-EM density map we want to automatically generate a 3D atomic model without any information about number of residues present in the structure cryoem_review ; cryo2struct . A discrete tokenization of modalities allows the training of a multi-modal model which is scalable, modality-agnostic, and extensible to diverse data types within a unified framework. In this spirit, we present a protein structure tokenizer, Yeti ( Y ielding E ncoded T okens for I ntermodality), that maps 3D atomic coordinates to discrete tokens suited for large-scale multi-modal learning, laying the groundwork for a universal molecular representation capable of capturing the full complexity of living systems. 2 Related Work Protein structure tokenizers protein_tokenizer_benchmark_receipe ; bio2token ; foldtoken work by converting the continuous 3D coordinates of proteins into a sequence of discrete tokens from a finite vocabulary utilizing vector quantization methods. We represent these structures using the input space \mathbb{R}^{L\times 3} . In this formulation, L corresponds to the number of C_{\alpha} residues present in the 3D atomic protein structure, and the three-dimensional vector denotes the coordinates of each C_{\alpha} atom. The standard paradigm for training protein structure tokenizers typically follows a two-stage process. First, an encoder and codebook are optimized alongside a lightweight decoder. In the second stage, the encoder and codebook are frozen while a larger, more expressive decoder is trained to maximize reconstruction accuracy. Building on the foundations of AlphaFold2 jumper2021highly , most architectures often rely on SE(3) -invariant components such as Invariant Point Attention (IPA) and SE(3) -invariant loss like Frame-Aligned Point Error (FAPE). ESM3 esm3 follows this two-stage approach with a VQ-VAE vqvae quantizer and a 4,096-entry codebook. The first stage uses five supervised losses including backbone distance, backbone direction, and inverse folding, and the second stage trains a deeper decoder on all-atom geometric losses across PDB pdb , AFDB, and ESMAtlas structures. The full tokenizer model contains \approx 600M parameters. DPLM-2 dplm2 similarly follows the two-stage paradigm. Its tokenizer consists of a pretrained GVP-Transformer gvp_transformer encoder which is frozen during the tokenizer training, a Lookup-Free Quantizer (LFQ) lfq , and an IPA-based decoder from AlphaFold2. Reconstruction is supervised with FAPE, violation, and distogram losses inherited from AlphaFold2, along with a sequence prediction cross-entropy head. This tokenizer contains approximately 100M parameters. Kanzi kanzi departs from the standard two-stage paradigm by using a flow-matching decoder over raw global coordinates, without SE(3) -invariant representations. It utilizes an asymmetric architecture (2-layer encoder, 8–12 layer decoder) with Finite Scalar Quantization (FSQ) fsq and a relatively small codebook ( \approx 1,000 entries). The publicly available kanzi model is limited to proteins with \leq 256 residues, and its generative capability is demonstrated via an auto-regressive model trained on its tokenized structures. However, as a single-modality approach, its multi-modal generative potential remains underexplored. Additionally, its use of sliding window attention can induce token repetition during encoding, which degrades downstream generative task dilip2026adaptive ; scaling_codebook . A common assumption is that better reconstruction translates to better generation. These objectives are not inherently aligned lfq ; scaling_visual_tokens_learnings ; yao2025reconstruction and a tokenizer can achieve high reconstruction accuracy while producing tokens poorly suited for generative modeling task, and vice versa. Despite this, most evaluation protocols treat reconstruction quality as the primary proxy for tokenizer quality. We suggest this is insufficient, and advocate evaluating tokenizers on both reconstruction accuracy and support for high-quality generation under multi-modal model paradigm. To address these gaps, we present Yeti , a single-stage protein structure tokenizer built on the StripedHyena stripehyena architecture and trained end-to-end under a flow-matching objective. Yeti shows high codebook utilization across an 8,192-entry