Beyond Benchmarks: A Novel Framework for Domain-Specific LLM Evaluation and Knowledge Mapping
Nitin Sharma, Thomas Wolfers, Çağatay Yıldız · 2025
Read Beyond Benchmarks: A Novel Framework for Domain-Specific LLM Evaluation and Knowledge Mapping by Nitin Sharma, Thomas Wolfers, Çağatay Yıldız (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-07658.txt.
Introduction
Nitin Sharma, Thomas Wolfers, and Cagatay Yildiz build a pipeline that turns raw domain texts into clean tests of what language models know, in a paper from 2025. Fields like law, medicine, and education need honest measures of model expertise. Existing tests are often leaked into training data and lean on multiple choice formats with known biases.
Their deterministic pipeline mines a corpus for domain keywords and related vocabulary. It then builds prompts whose missing word is a domain term, and scores models on completing them. No other model and no paid annotators are involved, so tests can be rebuilt cheaply as data grows. Scores lined up well with an expert made benchmark, and the method compares base and chat tuned models fairly.
Catalog record
| CREATOR | Nitin Sharma, Thomas Wolfers, Çağatay Yıldız |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 84 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-07658
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Beyond Benchmarks: A Novel Framework for Domain-Specific LLM Evaluation and Knowledge Mapping Nitin Sharma, Thomas Wolfers, Çağatay Yıldız Abstract The paper addresses two critical challenges in language model (LM) evaluation: creating reliable domain-specific benchmarks and understanding knowledge representation during domain adaptation. We introduce a deterministic pipeline that converts raw domain corpora into completion-type benchmarks without relying on LMs or human curation, eliminating benchmark contamination issues while enabling evaluation on the latest domain data. Our approach generates domain-specific keywords and related word lists using Term Frequency (TF) and Term Frequency-Inverse Document Frequency(TF-IDF) methods and constructs prompt-target pairs. We evaluate models by measuring their ability to complete these prompts with the correct domain-specific targets, providing a direct assessment of domain knowledge with low computational cost. Through comprehensive experiments across multiple models (GPT-2 medium/XL, Llama-2/3.1, OLMo-2, Qwen-2, Mistral) and domains, we demonstrate that our benchmark strongly correlates with expert-generated benchmarks while providing a more accurate measure of domain knowledge than traditional perplexity metrics. We reveal that domain adaptation happens rapidly in smaller models (within 500 steps) and illustrate a new approach to domain knowledge evaluation in base models during training for early stopping. By extending mechanistic analysis to domain adaptation, we discover that initial-to-mid layers are primarily responsible for attribute extraction, while later layers focus on next token prediction. Furthermore, we show that during adaptation, forgetting begins in the middle layers, where attribute extraction happens and is amplified in later layers. Our work provides both a practical evaluation methodology for domain-specific LMs and novel insights into knowledge representation during adaptation, with implications for more efficient fine-tuning strategies and targeted approaches to mitigate catastrophic forgetting. 1 Introduction In recent years, we have witnessed a rapid proliferation of application of language models (LMs) across virtually every domain. These models demonstrate remarkable capabilities in coding ( Nam et al., 2024 ) , mathematical reasoning ( Imani et al., 2023 ) , medical diagnosis ( Wang et al., 2024 ) , finance ( Li et al., 2023 ) , psychological evaluations ( Hu et al., 2024 ) , and many more ( Lai et al., 2024 ; Tian et al., 2023 ; Liu et al., 2024 ) . With the rise in applications, we have models available that vary in size ( Kaplan et al., 2020 ; Touvron et al., 2023 ) , architecture ( Gu and Dao, 2023 ; Dai et al., 2019 ) , context window size ( Ding et al., 2023 ; Chen et al., 2023 ) , and computational efficiency ( Dao et al., 2022 ; Frantar et al., 2023 ) , creating an ecosystem where practitioners have unprecedented options to find the model for their specific application. Amidst the abundance of these options, finding a balance between accuracy and computational cost that works best for your use case has become challenging. While benchmarking offers a potential solution to this unprecedented problem ( Gehrmann et al., 2021 ; Srivastava et al., 2022 ) , most practitioners prefer to fine-tune base models for their specific niche rather than relying on assistant chat models ( Han et al., 2023 ; Cui et al., 2025 ; Nguyen et al., 2023 ) . For these base models, perplexity has traditionally served as a performance evaluation metric ( Gururangan et al., 2021 ; Yıldız et al., 2024 ) . However, perplexity suffers from several critical limitations. Most notably, it fails to capture the actual knowledge and focuses mainly on general linguistic model capabilities ( Xu et al., 2024 ; Meister and Cotterell, 2021 ) . Recent research has dived deeper into understanding this issue at token level ( Öncel et al., 2024 ) and has discovered that lower perplexity scores might stem from domain-unrelated tasks and fail to reflect the actual domain-specific capabilities of the model. These limitations make the perplexity score an unreliable metric for measuring the model capability for a domain-specific task, mainly when focusing on evaluating the domain knowledge. Benchmarking offers a potential solution as it directly targets domain-specific knowledge rather than measuring the general language capability of the model ( Srivastava et al., 2022 ; Hendrycks et al., 2020 ) , but it also suffers from potential challenges. First, many existing benchmarks for LMs focus predominantly on question-answer formats rather than completion-type formats that would better evaluate base models ( Chang et al., 2024 ; Srivastava et al., 2022 ; Hendrycks et al., 2020 ) . While studies often use few-shot learning to adapt to this format, model performance in few-shot settings is influenced by factors beyond domain knowledge such as model size ( Brown et al., 2020 ) . Additionally, when benchmarks become well established, the data related to these benchmarks (including training sets, test prompts, or even test sets) may inadvertently be included in the pre-training corpora of new models—a phenomenon known as ’benchmark leakage’ or ’data contamination’ ( Zhou et al., 2023 ) . This issue has become increasingly common as models are trained on ever-larger datasets of public text, and can dramatically inflate evaluation results, creating unfair advantages when comparing different Large Language Models (LLMs) and leading to unreliable assessment of model performance. A potential solution is to employ LLMs to generate benchmarks ( Perez et al., 2023 ; Yuan et al., 2025 ) . This approach has limitations in terms of LLMs’ ability to effectively process and reason over long context lengths ( Li et al., 2024 ) , failing to comprehensively ground in the new data without introducing their own parametric knowledge ( Monea et al., 2023 ) especially in large models, and can be computational expensive depending on the size of input data ( Chavan et al., 2024 ) . To address these challenges, we have developed a deterministic pipeline that converts a raw corpus into a completion-type benchmark without relying on LLMs, ensuring consistent and unbiased assessment across domains. Our approach offers several advantages: (1) it processes raw corpus directly, eliminating data contamination problems present in LLM-generated benchmarks; (2) it can readily incorporate new sources such as recent arXiv papers or domain-specific raw data; (3) it particularly targets base models using completion style prompts while remaining applicable to assistant models; (4) unlike perplexity, it measures the domain knowledge rather than general linguistic capabilities, and more computational efficient as does not require evaluation on each token of a given corpus. We verified the benchmark created by using our approach through systematic experiments across multiple settings. First, we tested its efficiency on an intuitive task of domain adaptation in the GPT2-XL model. Second, we compared the results against perplexity and the completion task generated by the state-of-the-art model (Claude 3.7 Sonnet) for the same domain. Third, we validated our approach in a continual learning setting with a larger model like Llama to demonstrate scalability. Fourth, we evaluated knowledge acquisition during training by evaluating checkpoints for GPT2-XL and OLMo-2 models. Lastly, we conducted cross-model comparisons to determine the optimal model for specific domain applications. Beyond benchmarking, we extended our analysis to study how knowledge representation evolves during adaptive pre-training. Previous studies have investigated where factual knowledge is stored in LLMs ( Meng et al., 2022 ; Geva et al., 2020 ) and how subject-related attributes are encoded across model