Entropy, Disagreement, and the Limits of Foundation Models in Genomics
Maxime Rochkoulets, Lovro Vrček, Mile Šikić · 2026
Read Entropy, Disagreement, and the Limits of Foundation Models in Genomics by Maxime Rochkoulets, Lovro Vrček, Mile Šikić (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-04287.txt.
Introduction
Rochkoulets, Vrček and Šikić ask why foundation models in genomics lag their language counterparts. They train matched ensembles on text and DNA, comparing predictions, static embeddings and Fisher information flow. High entropy in genomic sequences, seen through next-token prediction, yields near-uniform outputs, disagreement across models and unstable embeddings, even with matched architecture, training and data. DNA-trained models pile Fisher information into embedding layers, missing links between tokens. The authors conclude that self-supervised training from sequences alone may not suit genomic data, shaking current assumptions.
Catalog record
| CREATOR | Maxime Rochkoulets, Lovro Vrček, Mile Šikić |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 18045 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.04287
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Entropy, Disagreement, and the Limits of Foundation Models in Genomics Maxime Rochkoulets, Lovro Vrček, Mile Šikić Abstract Foundation models in genomics have shown mixed success compared to their counterparts in natural language processing. Yet, the reasons for their limited effectiveness remain poorly understood. In this work, we investigate the role of entropy as a fundamental factor limiting the capacities of such models to learn from their training data and develop foundational capabilities. We train ensembles of models on text and DNA sequences and analyze their predictions, static embeddings, and empirical Fisher information flow. We show that the high entropy of genomic sequences—from the point of view of unseen token prediction—leads to near-uniform output distributions, disagreement across models, and unstable static embeddings, even for models that are matched in architecture, training and data. We then demonstrate that models trained on DNA concentrate Fisher information in embedding layers, seemingly failing to exploit inter-token relationships. Our results suggest that self-supervised training from sequences alone may not be applicable to genomic data, calling into question the assumptions underlying current methodologies for training genomic foundation models. 1 Introduction Genomic Foundation Models (GFMs) represent a significant part of current research in machine learning applied to biology. This approach consists in training large deep learning models on vast collections of DNA sequences, using techniques such as autoregressive or masked language modeling, essentially training the model to predict unseen tokens from their surrounding context. The goal is that the model will eventually be able to learn hidden relations or grammar encoded in genomic sequences, and develop general biological underlying knowledge. This knowledge, materialized in its embeddings, can then be applied to a variety of downstream tasks. In natural language processing, this approach has been very successful, with transformer models such as BERT ( 2 ) or RoBERTa ( 9 ) becoming widely used as a base for many NLP applications. Despite numerous papers reporting good results on a variety of downstream tasks ( 6 ; 11 ; 1 ) , the performance and usefulness of GFMs trained in self-supervised settings from sequences alone has recently been questioned ( 17 ) . In fact, it has been observed that specialized solutions, such as smaller models trained specifically for the task, often outperform GFMs embeddings ( 3 ) while being faster and cheaper to train and much more convenient to use. Having said that, no explanation for why these models perform poorly has been proposed so far. In this work, we argue that entropy provides a natural starting point, as human text and DNA sequences differ fundamentally from an information-theoretic perspective ( 15 ; 13 ) . We show that the low information content of DNA—from the point of view of unseen token prediction—prevents models from learning a confident and interpretable distribution, leads to disagreement between models, and causes models to not leverage inter-token relationships. 2 Methodology We train 3 ensembles each made of N transformer encoder BERT models ( 2 ) . The first ensemble is trained on English text with a byte-pair encoding tokenizer, the second on DNA sequences, also using BPE tokenization, ensuring meaningful comparison with the text models, and a third ensemble is also trained on DNA, but uses a k -mer non-overlapping tokenizer, a more widely used tokenization scheme for genomic language models in practice ( 1 ) . All 3N models use the exact same architecture and thus have an equal number of parameters (90M). They are trained on the same number of tokens (5B) and their vocabulary size is fixed to 4^{6} = 4096 , matching a 6-mers tokenization scheme. Models iterate over the data in the same deterministic order, but their weights are randomly initialized. We used N = 5 in this work, that we consider a good tradeoff between number of models and training costs. Details regarding training procedure, datasets and hyperparameters can be found in Appendix A.1 and A.2 . Matching hyperparameters across (data, tokenizer) pairs allows us to compare the models precisely, and to show that fundamental differences between natural language and genomic sequences yield very different models, even when matched in data, architecture and training. We argue that these differences are best explained by high training data entropy. 3 Ensemble Disagreement Naturally, we expect the high entropy of DNA to be reflected in the output distributions of our models. We set the stage by computing the Kullback–Leibler (KL) divergence with the uniform distribution \mathbb{E}_{{\textnormal{x}}\sim\mathcal{D}}\left[D_{\mathrm{KL}}\left(P\parallel\mathcal{U}\right)\right] , where P denotes a model’s discrete distribution over its vocabulary for predicting the masked token with context x sampled from dataset \mathcal{D} . We find that for DNA models using a BPE tokenizer, this quantity is roughly 3 bits, and less than 1 bit for k -mer models, while it is more than 10 bits for text models. This large KL difference already suggests that DNA models yield predictions that reflect greater uncertainty, whereas text models are highly confident in their predictions, diverging significantly from \mathcal{U} . In the following sections, we show that these introductory results translate into disagreement and instability across DNA models, especially when compared to their counterpart text models. 3.1 Output Distributions To compare the output distributions of two models, we use the Jensen-Shannon distance: d_{\operatorname{JS}}(P_{i},P_{j})=\sqrt{\operatorname{JSD}(P_{i}\parallel P_{j})}=\sqrt{\frac{1}{2}D_{\mathrm{KL}}(P_{i}\parallel M)+\frac{1}{2}D_{\mathrm{KL}}(P_{j}\parallel M)} where \operatorname{JSD}(P_{i}\parallel P_{j}) is the Jensen-Shannon divergence between two models i\neq j of the same family and M is the mixture distribution M=\left(P_{i}+P_{j}\right)/\,2 . The Jensen-Shannon distance defines a proper metric and is therefore symmetric. This makes it more suited than the KL divergence for measuring pairwise models agreement in terms of their output distributions. At first glance, we find that DNA models seem to agree in their output distributions over unseen tokens as much as text models. However, the expected pairwise distance between DNA models increases when applying nucleus sampling ( 5 ) , that is, reweighting the distribution by keeping only the top- p % of the probability mass. Figure 1: Jensen-Shannon distance \mathbb{E}_{{\textnormal{x}}\sim\mathcal{D}}\left[d_{\mathrm{JS}}(P_{i},P_{j})\right] between models of the same ensemble as a function of top- p mass kept. Values computed over 100\,000 samples unseen during training. This shows that small distances for high values of p are a consequence of the DNA models assigning roughly the same probability to each token, rather than agreeing over what the unseen token really is. Figure 1 shows that this apparent agreement breaks as p is lowered. Expected distance between text models, on the other hand, is barely impacted as a function of p , indicating stable convergence in their output distributions, and strong inter-model agreement. 3.2 Static Word Embeddings Having shown that DNA models, unlike text models, disagree in their distributions over masked tokens, we aim to go deeper and evaluate their levels of agreement in embedding space. In this work, we limit ourselves to the first embedding layer, also called word embedding layer or static embeddings, in BERT terminology 1 1 1 To avoid any confusion with static embeddings created from contextual ones, also common in the literature, we use the term static word embeddings to characterize this layer. . Although contextual representations are typically preferred when evaluating transformer encoder models, we show that our text models encode meaningful static relationships. This approach to word embeddings can be associated to