Embedding Is (Almost) All You Need: Retrieval-Augmented Inference for Generalizable Genomic Prediction Tasks

Nirjhor Datta, Swakkhar Shatabda, M Sohel Rahman · 2025

Read Embedding Is (Almost) All You Need: Retrieval-Augmented Inference for Generalizable Genomic Prediction Tasks by Nirjhor Datta, Swakkhar Shatabda, M Sohel Rahman (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-04757.txt.

Introduction

Nirjhor Datta, Swakkhar Shatabda, and Sohel Rahman published this paper in 2025. It challenges a costly habit in genomics AI: fine-tuning DNA language models for every prediction task. They ask whether the models' embeddings, the numeric fingerprints they assign to sequences, already carry enough signal.

Feeding frozen embeddings from three leading DNA models into light classifiers matched fine-tuning on several tasks, and beat it when test data came from a different distribution than the training data. Inference ran ten to twenty times faster, with carbon emissions cut by a similar factor. In one enhancer task the embedding route scored higher accuracy at a fraction of the footprint. Promoter classification landed near the fine-tuned score at twenty-two times lower emissions. The authors recommend embeddings with retrieval-style inference as the default baseline for genomic prediction, and fine-tuning only where it proves its extra cost.

Catalog record

CREATORNirjhor Datta, Swakkhar Shatabda, M Sohel Rahman
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-04757

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Embedding Is (Almost) All You Need: Retrieval-Augmented Inference for Generalizable Genomic Prediction Tasks Nirjhor Datta, Swakkhar Shatabda, M Sohel Rahman Abstract Large pre-trained DNA language models such as DNABERT-2, Nucleotide Transformer, and HyenaDNA have demonstrated strong performance on a wide range of genomic benchmarks. However, most applications rely on expensive fine-tuning, which benefits significantly from the similar distribution between training and test data. In this work, we investigate whether task-specific fine-tuning is always necessary. We show that simple embedding-based pipelines extracting fixed representations from these models and feeding them into lightweight classifiers can achieve competitive performance. Moreover, in independent evaluation settings with different data distributions, embedding-based methods outperform fine-tuning in several tasks while reducing inference time by 10×–20×. Our results suggest that embedding extraction is not only a strong baseline but also a more generalizable and efficient alternative to fine-tuning, particularly for deployment in diverse or unseen genomic contexts. Qualitatively, our retrieval-augmented pipeline combining fixed transformer embeddings (DNABERT-2, Nucleotide Transformer, HyenaDNA) with lightweight sequence features achieved competitive result across nine genomic tasks. Also, DNABERT-2 and Hyenadna embeddings with simple sequence features closely matches or exceeds fine-tuned transformer performance on two new independent test set, enhancer and promoter classification tasks, without any model retraining. Quantitatively, our embedding-based approaches consistently outperform fine-tuning in terms of carbon efficiency while maintaining competitive accuracy. For enhancer classification, embedding with zCurve achieves an accuracy of 0.68 using HyenaDNA (vs. 0.58 for fine-tuned HyenaDNA) with almost 88% reduction in inference time and over 8 × \times lower carbon emissions (0.02 kg vs. 0.17 kg CO 2 ). For non-TATA promoter classification, fixed embeddings from DNABERT-2 combined with zCurve, GC content, or AT/GC ratio reach 0.85 accuracy, compared to 0.89 for fine-tuning, but with a 22 × \times lower carbon footprint (0.02 kg vs. 0.44 kg CO 2 ). Similarly, HyenaDNA with embeddings achieves up to 0.84 accuracy while emitting just 0.01 kg CO 2 compared to 0.04 kg for fine-tuning. These results show embedding-based pipelines can offer over 10 × \times higher carbon efficiency while achieving similar or better predictive performance. The code is available here: https://github.com/NIRJHOR-DATTA/EMBEDDING-IS-ALMOST-ALL-YOU-NEED . 1 Introduction Genomic sequence analysis is one of the most promising areas of research in bioinformatics. The ability to understand the functional elements of the genome, such as promoters, enhancers, and splice sites, has vast implications for fields, such as gene regulation, personalized medicine, and disease prediction [ 1 ] . Traditional computational methods often rely on handcrafted features or pre-defined models, which, while useful, can struggle to capture the complexities inherent in genomic sequences [ 2 ] . Recent advances in transformer-based models, originally developed for natural language processing, have been successfully adapted to nucleotide sequences, yielding state-of-the-art performance on a variety of benchmarks [ 3 , 4 , 5 ] . Despite their empirical success, these transformer-based methods are computationally expensive to fine-tune and deploy, often requiring hundreds of GPU hours and substantial memory resources. Prior work has highlighted the environmental impact of large-scale model training, showing that the carbon footprint of training a single transformer can rival that of multiple automobiles over their lifetimes [ 6 , 7 ] . In genomics, where sequence lengths frequently exceed 10 Kb, the quadratic complexity of self-attention exacerbates these challenges, making full fine-tuning impractical in many research and clinical settings. To address these limitations, retrieval-augmented methods, originally proposed in the NLP domain offer a promising alternative. By indexing fixed transformer embeddings in FAISS (Facebook AI Similarity Search) and performing nearest-neighbor retrieval at inference time, one can sidestep full model fine-tuning while still leveraging rich contextual representations [ 8 , 9 ] . Recent works in gene interaction prediction has shown that retrieval-augmented generation can achieve better accuracy with substantially lower computational cost [ 10 ] . In parallel, hybrid approaches that combine pretrained embeddings with handcrafted genomic features (e.g., GC content, z-curve, pseudoKNC) have demonstrated marginal gains in classification accuracy while maintaining lightweight inference. So, there is a need for an approach which can levarage the power of large pre-trained models, without the need for fine-tuning, but still achieve state of the art results on various genomic classification tasks. This motivates the exploration of alternative methods that can provide high performance without the computational cost and complexities associated therewith. In this work, we propose a comprehensive, retrieval-augmented genome classification framework that: (1) employs pretrained transformer embeddings (DNABERT-2, Nucleotide Transformer, HyenaDNA) as fixed representations; (2) optionally augments these embeddings with biologically motivated features such as GC content, z-curve components, AT/GC ratio, cumulative skew, and pseudoKNC; and (3) uses FAISS L_{2} indexing for fast k-nearest-neighbor retrieval and weighted voting. We have evaluated our approach on nine diverse genomic benchmarks. The central focus of this research is to investigate whether fixed, pretrained embeddings can serve as a competitive alternative to full model fine-tuning, particularly in scenarios that demand strong generalizability. To this end, we conduct evaluations on a new, independent test set to assess how well the learned representations transfer beyond the training distribution. Our embedding-based approach achieves comparable or even superior performance to fine-tuned models, while reducing GPU fine-tuning time by over 70% and reducing carbon emission by upto 77.5x, thereby promoting a more sustainable and efficient paradigm aligned with Green AI principles [ 11 ] . The key contributions of this work are as follows. 1. Retrieval-Augmented Classification Pipeline: We propose a hybrid framework that combines pretrained transformer embeddings (DNABERT-2, Nucleotide Transformer, HyenaDNA) with handcrafted genomic features (e.g., pseudoKNC, GC-content, z-curve) and a lightweight kNN classifier. By indexing precomputed embeddings in FAISS and using majority/weighted voting, we avoid full fine-tuning of large models at inference time, drastically reducing GPU memory usage and compute cost. 2. Modular, Scalable, and Carbon-Efficient Architecture: The embedding and retrieval stages are decoupled, enabling plug-and-play substitution of any pretrained model or handcrafted feature set. This modularity allows flexible trade-offs between accuracy and resource usage, with our pipeline achieving up to 77.5 \times lower CO 2 emissions compared to full fine-tuning while maintaining competitive performance. 3. Comprehensive Empirical Evaluation: We benchmark our method on nine publicly available genomic datasets (e.g., Human Ensembl Regulatory, Drosophila Enhancers Stark, Human Non-TATA Promoters), comparing embedding-only, feature-augmented, and fine-tuned variants across three transformer models. We evaluated our approach on an independent test set across two genome classification tasks, using both a fine-tuned model trained on a genomic benchmark dataset and a retrieval-based method that leverages embeddings from the same dataset. 2 Related Work Transformer architectures have revolutionized sequence modeling across domains, including genomics. DNABERT [ 3 ] introduced a BERT-style model pre-trained on k-mers from the human genome, achieving strong performance on promoter and enhancer classification. However, limitations such as redundant k-mer tokenization, short context windows (512 tokens), and single-species training motivated DNABERT-2 [ 12 ] , which uses byte pair encoding, multi-species data, and longer input handling via gradient checkpointing. Nucleotide Transformer (NT) [ 4 ] extends this trend by pre-training large-scale transformer models (up to 2.5B parameters) on thousands of genomes. NT embeddings support strong performance across diverse tasks even in low-data settings, demonstrating the utility of context-aware nucleotide representations. HyenaDNA [ 5 ] , inspired by efficient implicit convolution mechanisms, offers longer context windows with lower time complexity. It

Read the complete text