DNA Language Models: An Assessment of Pre-Training for Fine-Tuning Tasks
Karpinsky Romain, Julien Mozziconacci, Mickaël Delcey · 2026
Read DNA Language Models: An Assessment of Pre-Training for Fine-Tuning Tasks by Karpinsky Romain, Julien Mozziconacci, Mickaël Delcey (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-30140.txt.
Description
Romain Karpinsky, Julien Mozziconacci, and Mickaël Delcey benchmark DNA language models against older convolutional rivals. Genomic foundation models borrow transformer designs from text AI. Models like DNABERT2 pretrain on billions of nucleotides at great cost. Whether that cost pays off on fine-tuning tasks is unclear. Tokenization choices for DNA are also hotly debated.
The authors pose three questions for their assessment. Do transformer models improve enough to justify heavy pretraining? What does pretraining itself actually contribute in this setting? How does byte pair encoding tokenization affect genomics task performance? DNABERT2 represents the transformer camp in their comparisons. ConvNova represents the convolutional approach to genomic sequences. Systematic benchmarks across these methods remain scarce in the literature. Their study aims to supply the missing head-to-head evidence.
Catalog record
| CREATOR | Karpinsky Romain, Julien Mozziconacci, Mickaël Delcey |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.30140