DNAZEN: Enhanced Gene Sequence Representations via Mixed Granularities of Coding Units

Lei Mao, Yuanhe Tian, Yan Song · 2025

Read DNAZEN: Enhanced Gene Sequence Representations via Mixed Granularities of Coding Units by Lei Mao, Yuanhe Tian, Yan Song (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-02206.txt.

Introduction

Lei Mao, Yuanhe Tian, and Yan Song build richer gene sequence representations in a 2025 paper. Genome models borrow language tools, treating DNA like text, but they usually fix one unit size. Real sequences carry meaning at several scales, from short blocks to grouped patterns the authors call G-grams.

Their framework, DNAZEN, mines G-grams from large genome collections without supervision and builds a vocabulary of them. During learning, matching G-grams from each sample are encoded by a dedicated Transformer and merged with the base unit encoder. A second training task sharpens the joint representation. The mixed scale design aims to catch both local motifs and longer range grammar in gene sequences.

Catalog record

CREATORLei Mao, Yuanhe Tian, Yan Song
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-02206

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DNAZEN: Enhanced Gene Sequence Representations via Mixed Granularities of Coding Units Lei Mao, Yuanhe Tian, Yan Song Abstract Genome modeling conventionally treats gene sequence as a language, reflecting its structured motifs and long-range dependencies analogous to linguistic units and organization principles such as words and syntax. Recent studies utilize advanced neural networks, ranging from convolutional and recurrent models to Transformer-based models, to capture contextual information of gene sequence, with the primary goal of obtaining effective gene sequence representations and thus enhance the models’ understanding of various running gene samples. However, these approaches often directly apply language modeling techniques to gene sequences and do not fully consider the intrinsic information organization in them, where they do not consider how units at different granularities contribute to representation. In this paper, we propose DNAZEN, an enhanced genomic representation framework designed to learn from various granularities in gene sequences, including small polymers and G-grams that are combinations of several contiguous polymers. Specifically, we extract the G-grams from large-scale genomic corpora through an unsupervised approach to construct the G-gram vocabulary, which is used to provide G-grams in the learning process of DNA sequences through dynamically matching from running gene samples. A Transformer-based G-gram encoder is also proposed and the matched G-grams are fed into it to compute their representations and integrated into the encoder for basic unit (E4BU) , which is responsible for encoding small units and maintaining the learning and inference process. To further enhance the learning process, we propose whole G-gram masking to train DNAZEN, where the model largely favors the selection of each entire G-gram to mask rather than an ordinary masking mechanism performed on basic units. Experiments on the genome understanding evaluation (GUE) benchmark show that DNAZEN outperforms representative models on various downstream tasks such as promoter detection, transcription factor prediction, splice site prediction, etc. The results confirm that G-grams with an appropriate learning process improve genomic representation through generalization ability to capture broader bioinformatic signals while maintaining high efficiency in genome modeling. 1 1 1 Code and models are available at https://github.com/oomics/dnazen . 1 1 footnotetext: Corresponding author. 1 Introduction Gene sequences offer rich information underpinning biological functions, and the analysis upon them supports breakthroughs in biology and medicine over years ( Gill et al., 1994 ; He et al., 2010 ; Corcoran and Chabner, 2018 ) . Similar to linear chain signals such as natural language, gene sequences share common characteristics to them so that genome modeling is able to perform such studies as if they are processing a special type of language ( Ji, 1999 ; Yoon et al., 2002 ; Searls, 2002 ) . To have an accurate understanding of the information carried by gene sequences, similar to text, it requires an appropriate processing method, where precise encoding of units at various granularities (e.g., nucleotide bases, polymers, oligonucleotides, etc) is critically important. Existing studies ( Ji et al., 2021 ; Nguyen et al., 2023 ; Liu et al., 2024 ; Schiff et al., 2024 ; Nguyen et al., 2024 ; Dalla-Torre et al., 2024 ) utilize various types of model structures, such as convolutional neural networks (CNNs) ( LeCun et al., 1998 ) , recurrent neural networks (RNNs) ( Rumelhart et al., 1986 ) , and Transformer ( Vaswani et al., 2017 ) , etc. Among different models, most existing studies adopt Transformer ( Ji et al., 2021 ; Zhang et al., 2023 ; Zhou et al., 2023 ; Li et al., 2023 ; Rozowsky et al., 2023 ; Choi and Lee, 2023 ) as their foundation models because it demonstrates its effectiveness in language modeling for various scenarios ( Devlin et al., 2019 ; Brown et al., 2020 ) . In language modeling, the basic units of language (tokens) are crucial, since variants such as characters, subwords, and words carry different semantic information and thus produce distinct learning outcomes. For example, in English, characters (letters) have poor semantic expressiveness and yield unsatisfactory learning results, while words struggle to cover novel terms and weaken generalization, so subwords, which both convey meaning independently and support generalization, become the dominant approach Devlin et al. (2019) ; Brown et al. (2020) . Similarly, for gene sequences, choosing the appropriate unit of representation for learning is equally important. Although it is straightforward to directly use single nucleotides (i.e., A , C , G , T ) as the molecular units, they lead to an issue of small alphabet, resulting in the problem of lacking structural modeling ability. Therefore, genome modeling approaches with different architectures often utilize a tokenizer to segment the sequence into small tokens, which in most cases are larger than single nucleotides and serve as the basic building blocks for the learning process of gene sequence representations ( Choi and Lee, 2023 ; Dahiya et al., 2023 ; Estabrook et al., 2023 ; Ma et al., 2023 ; Morimoto et al., 2023 ) . For example, “ k-mer ” is a widely used tokenization mechanism that uses a fixed number of nucleotide bases for each unit ( Wen et al., 2014 ) ; some studies ( Zhou et al., 2023 ) employ byte pair encoding (BPE) ( Sennrich et al., 2015 ) to segment DNA sequences into dynamically grouped nucleotide bases, and shows its superiority in learning gene sequence representations in many cases ( Zhou et al., 2023 ) . There are also studies that combine different tokenization strategies ( Wu et al., 2025 ) , leading to better performance on some downstream tasks than the models that rely on a single tokenization strategy. Although these studies achieve a promising effect in bioinformation tasks, they pay limited attention to larger coding units, which might play essential roles in understanding important information carried by gene sequences ( Peng et al., 1992 ) , because a sequence must be sufficiently long to encapsulate an independent and distinct biological function. Still, one may argue that genome evolution is driven by single-nucleotide mutations ( Lamason et al., 2005 ; Kamberov et al., 2013 ) , each of which effectively creates a new gene sequence with possible different functions, so that a particular coding scheme may be greatly affected by such mutations, thus leading to the inefficient encoding problem. Moreover, understanding biomolecular interactions potentially requires ultimate fine-grained sequence resolution, since critical functional sites may consist of only one or a few specific nucleotide bases ( Stamatoyannopoulos, 1972 ; Enattah et al., 2002 ) . Therefore, the encoding of gene sequences actually faces a dilemma, as it must simultaneously accommodate the processing of both large and small coding units while in view of the necessity of them. To this end, it is essential to develop an improved learning framework for gene sequence representation that effectively handles such coding scheme at multiple granularities, particularly by enhancing the processing of large units, so as to address the limitations of previous approaches that primarily focused on polymers or groups of a few nucleotide bases. Figure 1: The overview of DNAZEN. The left part presents the G-gram construction process, where the G-grams are obtained from large-scale gene corpora through an unsupervised approach named pointwise mutual information (PMI). The constructed G-gram vocabulary, the extracted G-grams, the matching matrix between the

Read the complete text