SpecMER: Fast Protein Generation with K-mer Guided Speculative Decoding
Thomas A. Walton, Darin Tsui, Aryan Musharaf, Amirali Aghazadeh · 2025
Read SpecMER: Fast Protein Generation with K-mer Guided Speculative Decoding by Thomas A. Walton, Darin Tsui, Aryan Musharaf, Amirali Aghazadeh (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-21689.txt.
Introduction
Autoregressive protein making is powerful but slow, since each token waits for the last. Thomas Walton and 3 co-authors speed it up with biologically guided speculative decoding in a 2025 paper. A draft model proposes tokens and a target model verifies them. Protein draft models usually ignore form and functional constraints. Implausible outputs and shifted odds follow.
SpecMER guides drafting with k-mer motifs extracted from multiple sequence alignments. K-mers are short sequence fragments that carry biological signal. Candidate series are scored in parallel, and promising ones are selected. Form and functional priors enter the decoding loop directly. Fast protein making for high throughput screening is the goal, without abandoning biological plausibility.
Catalog record
| CREATOR | Thomas A. Walton, Darin Tsui, Aryan Musharaf, Amirali Aghazadeh |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-21689
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SpecMER: Fast Protein Generation with K-mer Guided Speculative Decoding Thomas A. Walton, Darin Tsui, Aryan Musharaf, Amirali Aghazadeh Abstract Autoregressive models have transformed protein engineering by enabling the generation of novel protein sequences beyond those found in nature. However, their sequential inference introduces significant latency, limiting their utility in high-throughput protein screening. Speculative decoding accelerates generation by employing a lightweight draft model to sample tokens, which a larger target model then verifies and refines. Yet, in protein sequence generation, draft models are typically agnostic to the structural and functional constraints of the target protein, leading to biologically implausible outputs and a shift in the likelihood distribution of generated sequences. We introduce SpecMER (Speculative Decoding via k-mer Guidance), a novel framework that incorporates biological, structural, and functional priors using k-mer motifs extracted from multiple sequence alignments. By scoring candidate sequences in parallel and selecting those most consistent with known biological patterns, SpecMER significantly improves sequence plausibility while retaining the efficiency of speculative decoding. SpecMER achieves 24–32% speedup over standard autoregressive decoding, along with higher acceptance rates and improved sequence likelihoods. 1 Introduction Designing proteins with enhanced or novel biological functions is an important problem with wide-ranging applications in therapeutics, sustainability, and drug discovery [ 1 ] . Protein design is challenging due to the astronomically large space of possible protein sequences, of which only a tiny fraction is likely to exhibit the desired functions [ 2 ] . Generative models trained on the vast landscape of evolved protein sequences have emerged as powerful tools for navigating this combinatorial sequence space. Among them, autoregressive models have been especially successful in the design of proteins with functional properties comparable to natural proteins [ 3 ] . Their autoregressive nature integrates well with textual information [ 4 ] , ensures training stability [ 5 ] , enables variable-length sequence generation [ 5 ] , and effectively captures long-range dependencies that encode complex amino acid interactions [ 4 ] . Despite these advantages, autoregressive models struggle with pathological repetition [ 6 , 7 , 8 ] , early termination [ 7 ] , out-of-distribution generalization [ 8 ] , and most consequential, high computational cost during inference, resulting in slow sequence generation. This computational cost is a debilitating limitation for high-throughput protein screening, where thousands of protein sequences need to be generated and screened to create large-scale libraries of functional and structurally stable proteins [ 9 ] . For instance, generating 20,000 protein sequences of length 200 amino acids using ProGen2-XL, a 6.4-billion-parameter transformer-based autoregressive model, takes approximately 65 hours using a single NVIDIA A6000 GPU. Such slow sequence generation can delay high-throughput design workflows by days, not accounting for the additional time required for iterative design processes. Figure 1: Overview of speculative decoding and SpecMER. a , Given a context sequence, speculative decoding uses a draft model to sample tokens. The target model verifies generated tokens, accepting or rejecting them. In this example, two tokens were accepted from the draft model, and the token “ E ” was corrected to “ F ”. b , SpecMER augments the drafting process, generating multiple sequences which are then filtered by k-mers at near-zero cost. K-mers are formed through a multiple sequence alignment, capturing evolutionary patterns through homologous protein sequences. In this example, “ VLL ” was selected from the proposed candidates, where two tokens were accepted and the token “ L ” was corrected to “ K ”. c , Speculative decoding undersamples high log-likelihood sequences, missing the tail distribution of the target model. K-mers act as guidance for SpecMER, resulting in a distribution of generated sequences that more adequately samples high-likelihood, high-plausibility protein sequences. Additionally, SpecMER retains nearly the same generation speed as speculative decoding, with only a 3\% overhead observed at c=3 . Recently, speculative decoding [ 10 ] has proven effective in accelerating autoregressive generation in natural language processing. Speculative decoding utilizes a lightweight draft model to propose tokens, which are then verified by a larger, more expressive target model. This speculate-then-verify framework of generation has demonstrated wall-time speedup factors exceeding 2 \times by selecting outputs consistent with the target model distribution [ 11 ] . In protein sequence generation, however, target-consistent outputs may not align with the most biologically plausible proteins, particularly those that best satisfy structural or functional constraints. This misalignment is amplified in protein design workflows, where satisfying biochemical constraints is essential. While draft models may capture broad structural patterns, small deviations in local structure can lead to misfolded or nonfunctional proteins. Among target-consistent outputs, some may result in proteins with more plausible biological properties; however, such distinctions are not considered during speculative decoding. To address this, incorporating biologically meaningful local priors offers a principled way to guide generation. Notably, k-mers, or short contiguous amino acid subsequences, encode local structural motifs that correlate with both folding and function, and can be extracted with minimal computational cost. We hypothesized that incorporating structural cues from k-mers could guide speculative decoding toward more biologically plausible proteins, while retaining or even enhancing generation speed. To test this hypothesis, we develop spec ulative decoding via k- mer guidance (SpecMER), a novel decoding framework that leverages structural motifs from homologous protein sequences to improve both generation speed and quality. By incorporating structural priors into the decoding process, SpecMER provably selects tokens that are not only consistent with the target model but also biased toward completions likely to yield structured, biologically plausible proteins. This motif-informed guidance increases the likelihood that generated sequences will fold into stable structures. Our contributions are as follows: • We develop the first speculative decoding framework for protein generation. Using ProGen2-S as the draft model and ProGen2-M as the target model, we demonstrate an average increase in generation speed by 32\% . • We develop SpecMER, a novel framework that incorporates k-mer frequencies from natural proteins to guide draft token selection. SpecMER produces sequences more likely to resemble natural proteins while retaining increased generation speed from speculative decoding, enabling fast generation of plausible proteins. Software for SpecMER is available at https://github.com/amirgroup-codes/SpecMER.git . • We establish theoretical bounds on wall-time speedup and characterize how guidance from k-mers improves draft token selection, offering practical guidance for estimating speedups under varying system configurations and sequence constraints. 2 Preliminary Notation. Let draft model P and target model Q be autoregressive models with probability density functions p and q . A protein sequence x\sim P is denoted by sequence x(i),x(i+1),\dots,x(j) , where x(i) is the i^{th} token sampled from p . Tokens are defined by the shared vocabulary \mathcal{V} of P and Q , consisting of amino acids and special tokens. 2.1 Speculative Decoding Speculative decoding has recently emerged as a highly effective technique for accelerating autoregressive generation. Speculative decoding employs a smaller draft model to propose candidate tokens, which are then refined by a larger, more expressive target model [ 10 , 11 ] . This approach generates tokens consistent with the target model distribution while significantly reducing decoding latency. Various extensions to speculative decoding have explored architectural modifications [ 12 ] , tree-based verification [ 13 ] , distillation of the draft model [ 14 ] , and optimized scheduling or reference-free inference [ 15 ]