DNA Language Models: An Assessment of Pre-Training for Fine-Tuning Tasks

Karpinsky Romain, Julien Mozziconacci, Mickaël Delcey · 2026

Read DNA Language Models: An Assessment of Pre-Training for Fine-Tuning Tasks by Karpinsky Romain, Julien Mozziconacci, Mickaël Delcey (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-30140.txt.

Description

Romain Karpinsky, Julien Mozziconacci, and Mickaël Delcey benchmark DNA language models against older convolutional rivals. Genomic foundation models borrow transformer designs from text AI. Models like DNABERT2 pretrain on billions of nucleotides at great cost. Whether that cost pays off on fine-tuning tasks is unclear. Tokenization choices for DNA are also hotly debated.

The authors pose three questions for their assessment. Do transformer models improve enough to justify heavy pretraining? What does pretraining itself actually contribute in this setting? How does byte pair encoding tokenization affect genomics task performance? DNABERT2 represents the transformer camp in their comparisons. ConvNova represents the convolutional approach to genomic sequences. Systematic benchmarks across these methods remain scarce in the literature. Their study aims to supply the missing head-to-head evidence.

Catalog record

CREATORKarpinsky Romain, Julien Mozziconacci, Mickaël Delcey
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.30140

Browse the full library · About the Tower of Babel