Beyond Conditional Computation: Retrieval-Augmented Genomic Foundation Models with Gengram
Huinan Xu, Xuyang Feng, Junhong Chen, Junchen Liu, Kaiwen Deng, Kai Ding, Shengning Long, Jiaxue Shuai, Zhaorong Li, Shiping Liu, Guirong Xue, Zhan Xiao · 2026
Read Beyond Conditional Computation: Retrieval-Augmented Genomic Foundation Models with Gengram by Huinan Xu, Xuyang Feng, Junhong Chen, Junchen Liu, Kaiwen Deng, Kai Ding, Shengning Long, Jiaxue Shuai, Zhaorong Li, Shiping Liu, Guirong Xue, Zhan Xiao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-22203.txt.
Introduction
Huinan Xu and co-authors give genomic foundation models a memory to look things up. Current models of DNA spend heavy neural computation to approximate conserved biological motifs from single-letter inputs. They rebuild the same patterns from scratch for every sequence they read.
The team's 2026 preprint proposes Gengram, a conditional memory module built into the model. It adds an explicit, efficient lookup for multi-base motifs, using a hashing scheme designed for genomes. And thereby establishes a genomic syntax the network can consult. Explicit memory could make genome models faster and sharper at the patterns evolution conserved, with payoffs for variant prediction and gene regulation studies.
Catalog record
| CREATOR | Huinan Xu, Xuyang Feng, Junhong Chen, Junchen Liu, Kaiwen Deng, Kai Ding, Shengning Long, Jiaxue Shuai, Zhaorong Li, Shiping Liu, Guirong Xue, Zhan Xiao |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-22203
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Beyond Conditional Computation: Retrieval-Augmented Genomic Foundation Models with Gengram Huinan Xu, Xuyang Feng, Junhong Chen, Junchen Liu, Kaiwen Deng, Kai Ding, Shengning Long, Jiaxue Shuai, Zhaorong Li, Shiping Liu, Guirong Xue, Zhan Xiao Abstract Current genomic foundation models (GFMs) rely on extensive neural computation to implicitly approximate conserved biological motifs from single-nucleotide inputs. We propose Gengram, a conditional memory module that introduces an explicit and highly efficient lookup primitive for multi-base motifs via a genomic-specific hashing scheme, establishing genomic "syntax". Integrated into the backbone of state-of-the-art GFMs, Gengram achieves substantial gains (up to 14%) across several functional genomics tasks. The module demonstrates robust architectural generalization, while further inspection of Gengram’s latent space reveals the emergence of meaningful representations that align closely with fundamental biological knowledge. By establishing structured motif memory as a modeling primitive, Gengram simultaneously boosts empirical performance and mechanistic interpretability, providing a scalable and biology-aligned pathway for the next generation of GFMs. The code is available at https://github.com/zhejianglab/Genos , and the model checkpoint is available at https://huggingface.co/ZhejiangLab/Gengram . 1 Introduction DNA sequences constitute the fundamental biological language of life, encoding the genetic information underlying cellular function and organismal development. Understanding the “syntax” of this language is central to deciphering biological processes and their dysfunctions [ 15 , 35 ] . Recent advances in GFMs on architectures and tokenizers (e.g., k-mer, Byte Pair Encoding(BPE), single-base) have substantially accelerated progress in this direction. Although single-nucleotide tokenization has demonstrated strong performance in downstream tasks, particularly those that evaluate fine-grained single-base effects [ 18 ] , the majority of biologically functional genomic elements are defined by multi-nucleotide motifs with specific sequence patterns. Consequently, constrained by the Transformer architecture, existing Transformer-based GFMs lack an explicit mechanism to store and retrieve such functional motifs, instead relying on large-scale pretraining and dense computation to implicitly infer them from combinations of single-base representations [ 29 ] . This indirect inference process is inefficient and often limits performance on motif-dominated functional element detection tasks. Inspired by DeepSeek’s Engram—a conditional memory module that stores n-gram knowledge in a hash-based memory [ 6 ] — we propose Gengram, a conditional memory module specifically designed for genomic motif modeling. In Gengram, k-mers (with k = 1,2,3,4,5,6) and their corresponding embeddings are explicitly stored and leveraged through a gate-controlled conditional memory mechanism. Unlike Engram, which retrieves single preceding n-grams per position, Gengram introduces a local window aggregation mechanism , which aggregates multiple k-mer embeddings within a fixed genomic window. This design allows the model to capture local contextual dependencies relevant to functional motifs while avoiding reliance solely on single-nucleotide composition. Owing to the small vocabulary size of genomic sequences (ATCGN), Gengram introduces negligible additional computational overhead while enabling larger and more expressive motif memory tables. More importantly, Gengram can be seamlessly integrated into mainstream Transformer architectures. Our contributions can be summarized as follows: We propose Gengram , a lightweight conditional motif memory module for Transformer-based GFMs that encodes multi-nucleotide motifs and integrates them via a structure-aware windowed aggregation mechanism, improving efficiency and interpretability without hard-coded biological rules. Gengram demonstrates consistent and substantial performance improvements across downstream genomic tasks—especially motif-dominated functional element detection—achieving up to 14% improvement over state-of-the-art GFMs architectures under identical training protocols. Gengram is architecture-agnostic, introducing negligible additional parameters (~60M) and computation, while consistently reducing training loss across diverse Transformer architectures, suggesting its potential as a reusable, standardized component for future GFMs. Gengram is computationally efficient, achieving benchmark performance comparable to state-of-the-art models while utilizing fewer activated parameters and less training data. The Gengram module exhibits robust capability in modeling long sequences and enables stable load balancing in sparse Mixture of Experts (MoE) models. Gengram spontaneously captures biologically meaningful structure, exhibiting reverse-complement symmetry in memory embeddings and context-dependent gating aligned with functional genomic regions (e.g., promoters and 5 ′ UTRs), indicating structural reasoning rather than static pattern memorization. 2 Related Works 2.1 Current Architectures and Representation Paradigms in GFMs 2.1.1 Genomic Foundation Models Current genomic modeling is dominated by Transformer-derived architectures [ 30 ] and non-Transformer alternatives, primarily represented by State Space Models (SSMs) [ 11 ] . Based on their underlying mechanisms, these models can be categorized into three paradigms: Bidirectional Encoders (e.g., DNABERT) [ 14 ] : Built upon the Transformer Encoder, these models utilize omnidirectional attention for local motif extraction. Autoregressive Decoders (e.g., Nucleotide Transformer) [ 8 , 3 ] : Leveraging the Transformer Decoder, these models offer robust long-range modeling capabilities. State Space Models (e.g., Caduceus) [ 25 ] : Based on the Mamba architecture, SSMs achieve linear scaling via selective scanning. 3 Method Figure 1: Overview of the Gengram architecture and evaluation. (Left) Gengram is integrated into a Transformer block via a residual connection and applied before the attention module. It maintains a motif memory implemented as a hash table with static keys derived from all possible k-mers ( k=1,2,3,4,5,6) and learnable embedding values. For a given local window, k-mers are mapped to memory entries, aggregated within each k-mer level, and concatenated across levels. The resulting representation is then passed through a gate-controlled module to modulate the hidden states. (Right) Evaluation and analysis of Gengram. We conduct extensive benchmarks across diverse genomic tasks and Transformer architectures, demonstrating strong compatibility with different attention mechanisms (e.g., MHA, GQA, MLA). Biological inspection further shows that Gengram’s activations align with functional genomic regions. 3.1 Architecture Overview As shown in Figure 1 , the Gengram module integrates efficient lookup with a motif prior: when predicting the current token, it uses a sliding-window mechanism within the Local Window immediately preceding it to retrieve all occurring contiguous substrings, i.e., motif priors. After deduplication, it looks up the corresponding values in a hash table; vectors with the same N are summed and averaged, vectors across different N are concatenated, and then, after a linear transformation, dot-product computation, and gating, the resulting signal is added to the backbone network via a residual connection. 3.2 Notation Let the backbone produce an input representation and the previous-layer hidden states are X\in\mathbb{R}^{n\times d_{\text{model }}},X^{(\ell-1)}\in\mathbb{R}^{n\times d_{\text{model}}} (1) where n is the sequence length, d_{\text{model}} the hidden dimension, and X_{\text{t}} . The N -gram lengths follow chapter 4.2 N\in\mathcal{N}=\{1,2,3,4,5,6\} (2) To capture multi-scale motifs. For each N , we maintain a learnable lookup hash table \left\{\mathbf{M}^{(N)}\right\}_{N\in\mathcal{N}},\quad\mathbf{M}^{(N)}\in\mathbb{R}^{|\Sigma|^{N}\times d_{m}} (3) where \Sigma is the token alphabet (A,T,C,G,N) and d_{m} is the memory vector dimension. Intuitively, \mathbf{M}^{(N)} stores an embedding for each possible N -gram key; the key-space size is |\Sigma|^{N} . The mapping from an N -gram g\in\Sigma^{N} to a row index is defined by a hash function h_{N}(\cdot):\Sigma^{N}\rightarrow\{0,1,...,|\Sigma|^{N}-1\} (4) Since |\Sigma| is small, h_{N}(\cdot) can use a deterministic base- |\Sigma| encoding, giving efficient collision-free indexing. 3.3 Local Window and N-gram Enumeration For each position t, we define a causal window index set \mathcal{W}_{t}=\{t-W,t-W+1,\ldots,t-1\} (5) We scan N -grams over the local window \mathcal{W}_{t} to obtain a complete set of substrings, enabling the model to capture multi-scale motif information. The local window also reduces noise and supports efficient retrieval. Within this window, we enumerate all contiguous N -grams using a sliding window scan, as shown in Algorithm 1 : Algorithm 1 Sliding window search Retrieve at Position t Input: position t Output: \{m_{t}^{(N)}\}_{N\in\mathcal{N}} for N\in\mathcal{N} do j_{start}\leftarrow\max(1,\,t-W) j_{end}\leftarrow t-N sum\leftarrow\mathbf{0}_{d_{m}}