MetagenBERT: a Transformer-based Architecture using Foundational genomic Large Language Models for novel Metagenome Representation
Gaspar ROY, Eugeni BELDA, Baptiste HENNECART, Yann CHEVALEYRE, Edi PRIFTI, Jean-Daniel ZUCKER · 2026
Read MetagenBERT: a Transformer-based Architecture using Foundational genomic Large Language Models for novel Metagenome Representation by Gaspar ROY, Eugeni BELDA, Baptiste HENNECART, Yann CHEVALEYRE, Edi PRIFTI, Jean-Daniel ZUCKER (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-03295.txt.
Introduction
Gaspar Roy and five co-authors represent a metagenome, the full DNA of a microbial community, without first sorting its reads into species. Standard disease prediction from gut samples relies on species abundance tables. Those tables come from incomplete reference catalogs, which cap resolution and throw away information in the reads. MetagenBERT instead embeds raw DNA reads with foundation genomic language models, including one specialized for the microbiome.
The read embeddings are clustered, and each sample becomes a vector of cluster counts. On five benchmark gut datasets, covering cirrhosis, type 2 diabetes, obesity, bowel disease, and colorectal cancer, the method matched or beat species-based baselines on most tasks. Concatenating both representations improved prediction further. The two views of a sample complement each other.
Catalog record
| CREATOR | Gaspar ROY, Eugeni BELDA, Baptiste HENNECART, Yann CHEVALEYRE, Edi PRIFTI, Jean-Daniel ZUCKER |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 54 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-03295
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MetagenBERT: a Transformer-based Architecture using Foundational genomic Large Language Models for novel Metagenome Representation Gaspar ROY, Eugeni BELDA, Baptiste HENNECART, Yann CHEVALEYRE, Edi PRIFTI, Jean-Daniel ZUCKER Abstract Metagenomic disease prediction commonly relies on species-abundance tables derived from large and incomplete reference catalogs, a strategy that constrains resolution, increases computational costs, and discards valuable information contained in raw sequencing reads. To overcome these limitations, we introduce MetagenBERT, a Transformer-based framework that produces end-to-end metagenome embeddings directly from raw DNA sequences, without taxonomic or functional annotations. Individual reads are embedded using foundational genomic language models (DNABERT-2 and the microbiome-specialized DNABERT-MS), then aggregated through a scalable global clustering strategy based on FAISS-accelerated K-Means. Each metagenome is represented as a cluster-abundance vector summarizing the distribution of its embedded reads. We evaluate this approach on five benchmark gut microbiome datasets (Cirrhosis, T2D, Obesity, IBD, CRC) and on the large, phenotypically diverse MetaCardis cohort (n = 2,022). MetagenBERT achieves competitive or superior AUC performance relative to species-abundance baselines across most tasks. Concatenating species abundances with embedding-based cluster abundances further improves prediction, demonstrating complementarity between taxonomic and embedding-derived signals. Global clustering remains robust when applied to as little as 5–10% of reads, highlighting substantial redundancy in metagenomes and enabling major computational gains. We additionally introduce MetagenBERT-Glob-MCardis, a cross-cohort variant in which clusters are trained on MetaCardis and transferred to benchmark datasets. Despite reduced performance compared to dataset-specific clustering, the MetaCardis-trained clusters retain strong predictive signal—including for phenotypes absent from MetaCardis—indicating the feasibility of a foundation model for metagenome representation. Robustness analyses (PERMANOVA, PERMDISP, entropy metrics) show consistent separation of healthy and diseased states and stable cluster structures across subsamples. Overall, MetagenBERT provides a scalable, annotation-free, and interpretable representation of metagenomes, bridging foundational genomic language models with population-level microbiome variation. These results point toward future phenotype-aware metagenomic LLMs capable of generalizing across heterogeneous cohorts and sequencing technologies. \corresponding [email protected] \papername Roy et al. (2025) MetagenBERT: a Transformer Architecture using Foundational DNA Read Embedding Models to enhance Disease Classification keywords Metagenomics, Gut microbiome, Transformer models, DNA sequence embedding, Disease classification Introduction Traditional metagenome representations typically rely on abundance tables, which serve as baseline embeddings of microbial communities [ 30 ] . While effective for many tasks, such approaches face several limitations: they are sensitive to taxonomic resolution, incur high computational costs, and may oversimplify microbial diversity [ 5 ] . Furthermore, they depend on large yet incomplete reference catalogs, preventing them from serving as foundation models : as reference databases evolve, embeddings must be recalculated and models retrained, hindering scalability and long-term consistency. [ 35 ] Moreover, by reducing sequences to species counts, they lose valuable information about functional potential and ecological roles within the microbiome. [ 38 ] To this date, these representations are used by Machine Learning (ML) and Deep Learning (DL) models to classify gut metagenomic samples from patients with various diseases such as Cirrhosis, Colorectal Cancer or Type 2 Diabetes. [ 24 ] [ 29 ] [ 23 ] In the meantime, relying on the analogy between DNA and Natural Language, Large Language Models (LLM) architectures have been adapted to biological sequences, and in particular to genomic sequences, resulting in the new class of genomic Large language Models (gLLMs) [ 31 ] . Through self-supervised pretraining, these models learn regulatory grammar and sequence structures into mathematical representations named embeddings with models such as DNABERT-2. [ 42 ] They act as genomic foundation models and can be fine-tuned to produce representations adapted to various tasks such as gene annotations, variant predictions or taxonomic classification. [ 43 ] [ 10 ] However, these models still face several limitations. Benchmark studies show mixed performance across core genomic tasks, highlighting limitations in sequence alignment accuracy and reasoning without external reference integration. [ 39 ] More recent models focus on integrating longer sequences potentially hoarding more expressive power in order to produce more robust representations, with models such as Gene42 or EVO. [ 36 ] [ 22 ] Lastly, metagenomic-based LLMs have very recently emerged. Trained specifically for pathogen detection (METAGENE-1 [ 19 ] ) or metagenomic annotation (GENERanno [ 18 ] ), they open the way to specialized metagenomic-LLM. To address the challenges coming with species-based representations, we introduce MetagenBERT , a deep learning architecture that directly processes raw sequencing reads to produce end-to-end metagenome embeddings. Inspired by advances in Natural Language Processing, our approach integrates the gLLM models’ expressive power in metagenomic classification to extract rich biological information from sequence data and propose a new representation of metagenomic samples without relying on external annotations or reference databases. Materials and Methods Pipeline Overview Our method processes a metagenomic sample—typically a FASTA or FASTQ file containing millions of short, unordered DNA reads from multiple species—to produce a single embedding that represents the entire metagenome. Unlike approaches relying on taxonomic annotations, our goal is to generate these embeddings directly from raw reads in an end-to-end manner. Conceptually, a metagenome can be viewed as an unordered collection of short “sentences,” analogous to a massive, unstructured text. However, unlike natural language data, metagenomic reads lack sequential order, exhibit vast diversity across species, and may overlap or share conserved regions, posing unique computational challenges. Our pipeline comprises two main stages: (1) read embedding and (2) embedding aggregation. In the first stage, a large language model encodes each read into a vector representation. In the second, these read embeddings are aggregated into a single fixed-length vector summarizing the entire sample for downstream classification. Details of these components and the datasets used for evaluation are presented in later sections. The entire pipeline is displayed in Figure 1 . (a) Computing a Reference Latent Space (b) Projecting samples in the reference space Figure 1 : MetagenBERT-Glob: The Global Clustering Architecture. Panel (a). For each metagenome, all reads are embedded through the use of a gLLM. A subsample of each embedded metagenome is used to train a KMeans common to all the selected dataset. The centroids of the clusters obtained through K-means are retrieved, effectively partitioning the latent space. Panel (b). Every read embedding of each sample is then assigned to its cluster by nearest neighbor search against the centroids, thus creating a new abundance vector based on embeddings rather than species used for classification. Datasets To assess the performance of MetagenBERT in disease classification, we used 5 shotgun metagenomic datasets representing various gut microbiome associated diseases along with control samples that have been used as benchmarks in evaluating many different classification methods : following their use in MetAML [ 24 ] , they have also been used in testing DeepMicro [ 23 ] , PopPhy-CNN [ 29 ] or EnseDeepDP [ 33 ] . These datasets represent five diseases : Liver Cirrhosis (LC) [ 27 ] , Colorectal Cancer (CRC) [ 41 ] , Inflammatory Bowel Disease (IBD) [ 25 ] , Obesity (Obe) [ 16 ] and Type 2 Diabetes (T2D) [ 26 ] . Noteworthy, one sample is corresponding to one metagenome, hence composed of millions to tens of millions of short reads. The dataset is paired-end reads. Some information can be found in Table 1 Total Number Number of Number of Mean Number of reads of Samples Control Samples Case Samples per sample (stddev)