GenomeQA: Benchmarking General Large Language Models for Genome Sequence Understanding

Weicai Long, Yusen Hou, Junning Feng, Houcheng Su, Shuo Yang, Donglin Xie, Yanlin Zhang · 2026

Read GenomeQA: Benchmarking General Large Language Models for Genome Sequence Understanding by Weicai Long, Yusen Hou, Junning Feng, Houcheng Su, Shuo Yang, Donglin Xie, Yanlin Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-05774.txt.

Introduction

Long and six colleagues test general language models directly on raw genome sequences with GenomeQA. Across 5,200 samples spanning six task families and lengths from 6 to 1,000 base pairs, models beat random baselines and exploit local signals like GC content. Performance fades on indirect multi-step inference, marking a diagnostic gap for sequence reasoning.

Catalog record

CREATORWeicai Long, Yusen Hou, Junning Feng, Houcheng Su, Shuo Yang, Donglin Xie, Yanlin Zhang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53165
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.05774

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

GenomeQA: Benchmarking General Large Language Models for Genome Sequence Understanding Weicai Long, Yusen Hou, Junning Feng, Houcheng Su, Shuo Yang, Donglin Xie, Yanlin Zhang Abstract Large Language Models (LLMs) are increasingly adopted as conversational assistants in genomics, where they are mainly used to reason over biological knowledge, annotations, and analysis outputs through natural language interfaces. However, existing benchmarks either focus on specialized DNA models trained for sequence prediction or evaluate biological knowledge using text-only questions, leaving the behavior of general-purpose LLMs when directly exposed to raw genome sequences underexplored. We introduce GenomeQA, a benchmark designed to provide a controlled evaluation setting for general-purpose LLMs on sequence-based genome inference tasks. GenomeQA comprises 5,200 samples drawn from multiple biological databases, with sequence lengths ranging from 6 to 1,000 base pairs (bp), spanning six task families: Enhancer and Promoter Identification, Splice Site Identification, Taxonomic Classification, Histone Mark Prediction, Transcription Factor Binding Site Prediction, and TF Motif Prediction. Across six frontier LLMs, we find that models consistently outperform random baselines and can exploit local sequence signals such as GC content and short motifs, while performance degrades on tasks that require more indirect or multi-step inference over sequence patterns. GenomeQA establishes a diagnostic benchmark for studying and improving the use of general-purpose LLMs on raw genomic sequences 1 1 1 Data and code are available at https://anonymous.4open.science/r/GenomeQA-E350 . $\dagger$ $\dagger$ footnotetext: Co-first authors 1 Introduction Figure 1: Overview of GenomeQA. The pipeline consists of Data collection, processing, construction and LLM evaluation. Genome analysis has long relied on specialized sequence models and task-specific pipelines. Recent years have seen rapid progress in DNA foundation models that are trained directly on nucleotide sequences, as well as emerging DNA-Text systems that couple a dedicated DNA encoder with a large language model ( 46 ; 11 ; 6 ; 39 ; 12 ; 14 ) . These approaches achieve strong supervised performance, but they typically require task-specific heads, probing, or additional adaptation to support downstream applications. In parallel, large language models have become widely used as conversational assistants across scientific domains, including chemistry ( 19 ; 27 ) , physics ( 3 ; 43 ) , and medicine ( 40 ; 45 ) . In genomics, their most common role is to reason over derived information such as annotations for genes and variants, functional summaries, experimental metadata, literature, and results from existing bioinformatics tools through natural language interfaces. However, direct interaction with raw genomic sequences poses challenges that differ fundamentally from standard natural language processing. DNA lacks human-interpretable semantic units such as words or grammar, exhibits long-range dependencies, and encodes biological signals in highly degenerate and context-dependent patterns ( 9 ; 5 ) . As a result, it remains unclear how general-purpose LLMs behave when they are directly exposed to nucleotide sequences, and whether their responses reflect non-trivial sequence-level cues or are driven primarily by superficial heuristics. Despite the increasing presence of LLMs in genomics studies, there is currently no standardized benchmark that evaluates how general-purpose language models perform when directly exposed to raw genomic sequences. Existing genome benchmarks ( 28 ; 18 ; 35 ) primarily target trainable DNA-specific models and assess representation quality under fine-tuning or probing. In contrast, exam-style benchmarks ( 7 ; 36 ; 44 ) evaluate biological knowledge using text-only questions without requiring sequence-level analysis. Consequently, a practically important evaluation setting remains underexplored: a general-purpose LLM receiving natural language questions together with raw nucleotide sequences and producing answers under a fixed instruction-following protocol. We take a step toward filling this gap by introducing GenomeQA, a question-answering benchmark designed to provide a controlled evaluation of general-purpose LLMs on sequence-based genome inference tasks. As shown in Figure 1 , GenomeQA comprises 5,200 samples across six representative task families: Enhancer and Promoter Identification, Splice Site Identification, Taxonomic Classification, Histone Mark Prediction, Transcription Factor Binding Site (TFBS) Prediction, and Transcription Factor (TF) Motif Prediction. Each instance is formatted as either a Binary Choice Question (BCQ) for validity judgment or a four-option Multiple Choice Question (MCQ). To ensure consistent evaluation across models and tasks, we use a single system prompt derived from a small pilot study; the prompt is fixed throughout all experiments and provides domain-relevant guidance for analyzing sequence signals such as motifs and base composition. Our contributions are summarized as follows: • We introduce GenomeQA, a benchmark that provides a controlled evaluation setting for assessing how general-purpose LLMs perform on sequence-based genome inference tasks. The benchmark comprises 5,200 curated samples spanning six biologically grounded task families. • We conduct a comprehensive evaluation of six frontier LLMs, establishing baseline performance on raw DNA sequences and showing that current models can exploit certain local sequence signals (e.g., GC content and short motifs) but struggle with tasks requiring more complex or indirect inference. • We present a fine-grained analysis of failure modes such as Sequence Motif Over-reliance and Character Fidelity Loss, highlighting systematic error patterns and directions for future research. 2 Related Work 2.1 Genome-Specific Models Recent DNA foundation models treat genome sequences as language. They pretrain on billions of nucleotides and add task-specific heads for downstream applications such as regulatory elements prediction and splice site identification. Typical models include DNABERT-2 ( 46 ) , Nucleotide Transformer ( 11 ) , HyenaDNA ( 32 ) , Genos ( 25 ) , GENA-LM ( 15 ) and Evo ( 31 ) . Recent multimodal approaches couple a pretrained DNA foundation model with a general-purpose large language model (LLM), such as ChatNT ( 12 ) , BioReason ( 14 ) and Omni-DNA ( 24 ) . Empirically, these models achieve strong performance and can match or even surpass state-of-the-art results on standard genome understanding benchmarks. Meanwhile, some researchers argue that an alternative route is to repurpose general LLMs into DNA-LLMs directly ( 8 ) , e.g., by adapting tokenization and training objectives so that the LLM can model DNA sequences without an explicit separate DNA encoder. Overall, despite their different interfaces, these methods share a key property: they rely on models that are specifically trained or substantially adapted on DNA sequences. 2.2 Genome Benchmarks Existing benchmarks for genome modeling primarily target models that are specifically designed or adapted for genome data. Sequence-based benchmarks, such as BEND ( 28 ) , DNALongBench ( 9 ) , DART-Eval ( 35 ) , and the Genomics Long-range Benchmark ( 22 ) , are mainly constructed to evaluate genome foundation models. These benchmarks typically consist of raw DNA sequences paired with predefined prediction tasks, and are intended to measure a model’s ability to learn transferable sequence representations for downstream biological applications. Another line of benchmarks focuses on biomedical or clinical knowledge assessment through natural language questions, such as CMExam ( 26 ) , Bio-Benchmark ( 20 ) , and EHRXQA ( 4 ) . While these benchmarks evaluate language understanding and domain knowledge, they do not require models to directly process or reason over raw genomic sequences. More recently, several benchmarks and evaluation protocols have emerged in the context of multimodal genome-language systems, including Lab-Bench ( 23 ) and the task suites used in ChatNT and BioReason. These benchmarks partially involve sequence-based question answering, but they rely on a dedicated genome encoder to transform DNA sequences into

Read the complete text