eccDNAMamba: A Pre-Trained Model for Ultra-Long eccDNA Sequence Analysis
Jien Li, Zhenke Liu, Ziqi Zhang · 2025
Read eccDNAMamba: A Pre-Trained Model for Ultra-Long eccDNA Sequence Analysis by Jien Li, Zhenke Liu, Ziqi Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-18940.txt.
Introduction
Circular DNA loops that sit outside chromosomes get a dedicated AI reader in a 2025 paper by Zhenke Liu and co-authors. These loops steer gene activity and can drive cancer growth when copied many times. Past models read only short or straight DNA, not full circles. Long loops also strain slow attention based tools.
The new encoder reads loops in both directions at once with linear time cost. A training trick keeps the circle shape intact for the model. Tests on two real datasets show strong sorting scores. The model handles loops up to two hundred thousand letters long. Shared code lets other labs apply it to cancer and genome studies.
Catalog record
| CREATOR | Jien Li, Zhenke Liu, Ziqi Zhang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-18940
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
eccDNAMamba: A Pre-Trained Model for Ultra-Long eccDNA Sequence Analysis Jien Li, Zhenke Liu, Ziqi Zhang Abstract Extrachromosomal circular DNA (eccDNA) plays key regulatory roles and contributes to oncogene overexpression in cancer through high-copy amplification and long-range interactions. Despite advances in modeling, no pre-trained models currently support full-length circular eccDNA for downstream analysis. Existing genomic models are either limited to single-nucleotide resolution or hindered by the inefficiency of the quadratic attention mechanism. Here, we introduce eccDNAMamba, the first bidirectional state-space encoder tailored for circular DNA sequences. It combines forward and reverse passes for full-context representation learning with linear-time complexity, and preserves circular structure through a novel augmentation strategy. Tested on two real-world datasets, eccDNAMamba achieves strong classification performance and scales to sequences up to 200 Kbp, offering a robust and efficient framework for modeling circular genomes. Our codes are available at https://github.com/zzq1zh/GenAI-Lab.git . † † affiliationnotice: Equal contribution 1 Introduction Extrachromosomal circular DNA (eccDNA) has emerged as a ubiquitous and functionally important element in diverse organisms and conditions, with a particularly pronounced relevance in cancer genomes. These circular DNA molecules, which range from a few hundred base pairs to over megabase, often carry oncogenes or distal regulatory elements and contribute to tumor evolution, therapeutic resistance, and intratumor heterogeneity ( Dong et al. 2023 ; Turner et al. 2017 ; Koche et al. 2020 ) . In biomedical applications, eccDNA has been implicated in diverse processes such as oncogene amplification, genome instability, and non-Mendelian inheritance of drug resistance. Recent studies suggest that understanding the structure and function of eccDNA can provide novel insights into cancer diagnostics and personalized treatment strategies ( Cao et al. 2021 ; Wu et al. 2019 ; Qiu et al. 2020 ) . Despite advances in eccDNA detection pipelines (e.g., eccDNA-Pipe ( Fang et al. 2024 ) ), analyzing the full-length sequence of eccDNA remains a significant challenge. The circular nature of eccDNA introduces unique modeling constraints: linearizing sequences at arbitrary breakpoints creates artificial boundaries, potentially disrupting biologically meaningful head–tail interactions ( Ling et al. 2021 ) . Additionally, many eccDNAs exceed 10,000 bp in length, which precludes standard Transformer architectures due to their quadratic complexity ( Zhou et al. 2023 ) . These structural and computational challenges hinder the development of principled models that reason over the full circular topology. Recent genomic foundation models demonstrate both promise and limitation. HyenaDNA ( Nguyen et al. 2023 ) leverages implicit convolutions for long sequences but remains unidirectional and topology-agnostic. DNABERT-2 ( Zhou et al. 2023 ) uses byte-pair encoding to capture motifs but retains a linear input assumption and standard Transformer layers. Moreover, state-space models such as Caduceus offers linear scalability ( Schiff et al. 2024 ) but focuses on single-nucleotide resolution and restrain their scalability on ultra-long eccDNA sequences for downstream applications. These limitations prevent modeling ultralong eccDNA sequences. To address these challenges, we propose eccDNAMamba, the first bidirectional state-space encoder tailored for circular DNA. Our design fuses forward and reverse passes of a Mamba-2 backbone to provide full contextual encoding while preserving linear time complexity. We introduce an augmentation strategy that preserves head–tail dependencies by appending 64 tokens from the start of each sequence to its end. Training is performed using a SpanBERT-style objective ( Joshi et al. 2020 ) , which masks contiguous spans to encourage motif-level reconstruction. Our proposed model, eccDNAMamba, demonstrates strong and stable performance across multiple classification tasks, including cancer-state prediction and distinguishing authentic eccDNAs from synthetic circular fragments, consistently outperforming domain-specific baselines. Notably, it generalizes to full-length sequences up to 200,000 base pairs without performance degradation, underscoring its scalability for ultra-long circular genome modeling. These results position eccDNAMamba as a robust foundation for downstream applications in eccDNA analysis. 2 Related Work 2.1 Genomic Foundation Models The past few years have seen the rise of genomic foundation models that adapt natural language processing pretraining paradigms to DNA. DNABERT-2 ( Zhou et al. 2023 ) introduced byte-pair encoding (BPE) tokenization to capture sequence motifs, while the Nucleotide Transformer ( Mendoza-Revilla et al. 2024 ) scaled pretraining to multiple genomes. There also exist other genomic foundation models that can model ultra-long sequences. HyenaDNA ( Nguyen et al. 2023 ) leverages implicit convolutions for long sequences. Caduceus offer linear scalability ( Schiff et al. 2024 ) . eccDNAMamba differs from this paradigm by combining the scalability of state-space models with a bidirectional encoder architecture and span masking training objective. This allows us to capture global dependencies while maintaining compatibility with span-masked objectives and motif-centric tokenization. 2.2 eccDNA Modeling Most computational efforts on eccDNA to date have focused on detection, breakpoint localization, and cataloging. Tools such as eccDNA-Pipe ( Fang et al. 2024 ) identify circular structures through split-read and discordant alignment signatures but do not attempt sequence-to-function modeling. Recent work has explored the use of deep learning for eccDNA classification. For instance, Li et al. ( Chang et al. 2023 ) proposed a CNN-based classifier that identifies eccDNA from short non-eccDNA sequence segments. In contrast, eccDNAMamba performs self-supervised pretraining over long circular sequences, capturing long-range dependencies and structural context across diverse eccDNA types. 2.3 Mamba and State-Space Models Mamba ( Gu & Dao 2023 ) is a recently proposed state-space model (SSM) that models input sequences through a discretized linear dynamical system as shown in Figure 2 : h_{t}=Ah_{t-1}+Bx_{t},\quad y_{t}=Ch_{t} (1) where x_{t}\in\mathbb{R}^{d} is the input at position t , h_{t}\in\mathbb{R}^{d} is the hidden state, y_{t}\in\mathbb{R}^{m} is the output, and A,B,C are learnable matrices. To improve expressiveness, Mamba introduces input-dependent gating and parameterizes output as: y_{t}=\sum_{j=1}^{t}K_{t-j}\odot(G_{j}\odot x_{j}) (2) where K is a learned convolutional kernel derived from the state-space dynamics, G_{j} is a gating function, and \odot denotes element-wise multiplication. This structure allows long-range dependency modeling while maintaining linear time and memory complexity. Mamba-2 ( Dao & Gu 2024 ) builds upon this by introducing Dynamic Block Sparse Selective Scans as illustrated in Figure 2 . It reformulates the SSM kernel using semi-separable matrices and applies block-wise sparsity, significantly improving runtime efficiency. We adopt Mamba-2 as the base encoder in our framework and extend it to support bidirectional processing. Unlike the original decoder-only formulation, our adaptation enables full-context encoding over circular DNA while preserving the model’s linear scalability. 3 Method 3.1 The eccDNAMamba Model As shown in Figure 1 , given a BPE-tokenized eccDNA input sequence \displaystyle x=(x_{1},x_{2},\dots,x_{L}),\quad x_{i}\in\mathcal{V} (3) Figure 1: eccDNAMamba initially embeds the input eccDNA sequence and incorporates circular augmentation to fit their periodic structure. The embedded representations are subsequently processed by Mamba blocks to capture and learn intricate patterns within the eccDNA sequences. where \mathcal{V} denotes the vocabulary constructed via Byte-Pair Encoding (BPE). The model appends a special [CLS] token to the front of the input sequence to provide a unified semantic representation of the entire input sequence. To perform circular augmentation, the first s=64 tokens are appended to the end of the sequence, resulting in: \displaystyle\tilde{x}=(\texttt{[CLS]},x_{1},x_{2},\dots,x_{L},x_{1},x_{2},\dots,x_{s}) (4) The embedded sequence is then fed into two directional Mamba-2 encoders to model bidirectional contextual information. Here, \mathbf{h} denotes the contextual embedding. We use \overset{\rightarrow}{(\cdot)} and \overset{\leftarrow}{(\cdot)} to denote the forward and backward directions, respectively: \displaystyle\overset{\rightarrow}{\mathbf{h}} \displaystyle=\overset{\rightarrow}{\text{Mamba}}(\overset{\rightarrow}{\mathbf{\tilde{x}}})=[\overset{\rightarrow}{h}_{cls},\overset{\rightarrow}{h}_{1},\overset{\rightarrow}{h}_{2},\ldots,\overset{\rightarrow}{h}_{L}] (5) \displaystyle\overset{\leftarrow}{\mathbf{h}} \displaystyle=\overset{\leftarrow}{\text{Mamba}}(\overset{\leftarrow}{\mathbf{\tilde{x}}})=[\overset{\leftarrow}{h}_{L},\overset{\leftarrow}{h}_{L-1},\ldots,\overset{\leftarrow}{h}_{1},\overset{\leftarrow}{h}_{cls}] (6) where \vec{h}_{\text{cls}} denotes the output corresponding to the special [CLS] token for global