Line Goes Up? Inherent Limitations of Benchmarks for Evaluating Large Language Models

James Fodor · 2025

Read Line Goes Up? Inherent Limitations of Benchmarks for Evaluating Large Language Models by James Fodor (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-14318.txt.

Introduction

James Fodor argues, in a 2025 paper, that rising benchmark scores are weak evidence that language models are gaining general competence. Models keep posting striking gains on language, knowledge, and reasoning tests. Commentators read the climb as broad cognitive progress that should carry into real work.

Fodor answers that the benchmarking habit itself is flawed, and that specific tests carry flaws of their own, so scores track test-taking more than transferable skill. Other probes tell a colder story: adversarial inputs and interpretability work show models failing simple reasoning and forming representations that do not support general inference. His verdict is blunt: benchmark lines going up should not be read as minds going up. Capability claims need harder evidence.

Catalog record

CREATORJames Fodor
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-14318

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Line Goes Up? Inherent Limitations of Benchmarks for Evaluating Large Language Models James Fodor Abstract Large language models (LLMs) regularly demonstrate new and impressive performance on a wide range of language, knowledge, and reasoning benchmarks. Such rapid progress has led many commentators to argue that LLM general cognitive capabilities have likewise rapidly improved, with the implication that such models are becoming progressively more capable on various real-world tasks. Here I summarise theoretical and empirical considerations to challenge this narrative. I argue that inherent limitations with the benchmarking paradigm, along with specific limitations of existing benchmarks, render benchmark performance highly unsuitable as a metric for generalisable competence over cognitive tasks. I also contend that alternative methods for assessing LLM capabilities, including adversarial stimuli and interpretability techniques, have shown that LLMs do not have robust competence in many language and reasoning tasks, and often fail to learn representations which facilitate generalisable inferences. I conclude that benchmark performance should not be used as a reliable indicator of general LLM cognitive capabilities. K eywords Benchmarking \cdot Large language models \cdot Artificial intelligence \cdot Interpretability 1 Introduction The recent development of large language models (LLMs) 1 1 1 A note on terminology. In this article I use the term ‘large language models’ (LLMs) to refer to models based on the transformer architecture trained on large corpus of text data to interact with the user via natural language. Examples of such models include the Llama series, the GPT series, and the Gemini series. Although these models are capable of tasks beyond language processing, such as mathematical and coding tasks, this is still performed by next token prediction after training on linear strings of text. I avoid the term ‘artificial intelligence’, as it has no accepted clear meaning and does not refer to any specific model architecture. based on the transformer architecture has led to extensive discussion as to how to best measure their capabilities. The most common way to assess LLMs is by their performance on standardised tests called benchmarks. It is often argued that recent substantial improvements in LLM benchmark scores, largely driven by state-of-the-art systems like GPT-4 and OpenAI’s o3, are indicative of large increases in the capability of LLMs. On this basis, it is often inferred that such models are becoming significantly more capable, potentially outstripping human-level performance, on a wide range of real-world tasks. In this article I contend that this argument is flawed in two key respects. First, I argue that inherent limitations with the benchmarking paradigm, along with specific limitations of existing benchmarks, combine to render benchmark performance highly unsuitable as a metric for generalisable competence across a range of tasks. Second, I argue that an alternative set of methods more suited for investigating the robustness and generalisability of LLM capabilities, namely adversarial stimuli and interpretability techniques, has yielded strong evidence that LLMs do not have robust competence in many language and reasoning tasks, and often fail to learn representations which facilitate generalisable inferences. 2 Problems with benchmarks A benchmark is a test designed to assess the ability of an LLM to perform a particular type of task. They are designed to interrogate the ability of LLMs to perform tasks such as grammatical judgements, commonsense reasoning, logical inference, answering science or math questions, and producing computer code. Hundreds of benchmarks have been developed over the past decade, and it is outside my scope to review them here. Interested readers can consult existing literature reviews [ 1 , 2 , 3 ] . Here my purpose is to highlight several major limitations inherent in the practise of using benchmarks to assess the real-world capabilities and competence of LLMs, along with specific limitations of existing popular benchmarks. Critically, benchmarks aim to assess LLM capabilities in performing certain types of tasks that are similar, but not identical, to those contained in the benchmark itself. A high benchmark score is therefore not meaningful unless we have reason to believe the benchmark provides a good estimate for performance on related real-world tasks. In this section, however, I will argue that this is rarely the case. 2.1 Over-fitting to the benchmarks A major problem with many benchmarks is that shortly after they are made publicly available, they become incorporated into the data used to train the very LLMs they are intended to assess. Modern LLMs are easily large enough to memorise large swaths of data, with numerous studies finding evidence of benchmark tasks having been learned during model pre-training or subsequent fine-tuning [ 4 , 5 , 6 ] . Further, even if LLMs do not directly memorise the answers to a task, exposing LLMs to example problems from a specific benchmark allows them to learn statistical regularities associated with that task. This can allow the LLM to achieve a higher score on that benchmark without any actual improvement in the LLM’s general reasoning capabilities [ 4 ] . More broadly, the development of LLMs has become guided by benchmark performance to the extent that the benchmarks lose much of their value in assessing LLM capabilities. This problem is an instance of Goodhart’s law, the adage that “when a measure becomes a target, it ceases to be a good measure”. Initially presented in the context of economics, it also applies in the context of evaluating LLMs [ 7 , 8 ] . Unfortunately, its importance in this context often goes unappreciated. For instance, recently the World Economic Forum erroneously inferred that LLMs have exceeded human performance on various cognitive tasks, on the basis that LLMs have exceeded human performance on benchmarks designed to test those task [ 9 ] . Such an inference is only valid if the benchmarks in question are unbiased and representative indicators of LLM performance for the corresponding task as a whole. This measurement function of benchmarks, however, has been supplanted by their use as targets. New LLMs are designed specifically to achieve ‘state-of-the-art performance’ on various benchmarks, with this being rewarded by conference papers, citations, and media attention. Such incentives result in model architectures, parameters, and training regimes being fitted specifically to these benchmarks, thereby reducing their relevance for assessing LLM performance [ 10 ] . Such over-fitting to the test is manifested in the rapid pace with which new benchmarks become saturated, even while fundamental limitations of LLM competence are still evident, thereby requiring still newer benchmarks to assess performance [ 11 , 7 ] . Similar problems have long been appreciated in the field of psychometrics [ 12 ] . When people practise or receive coaching on components of an IQ test, performance on that task improves [ 13 ] , but the test become less predictive of subsequent performance on other tasks [ 14 ] . This phenomenon results in recommendations to limit the number of times a given test is assigned to a particular individual. The problem is likely to be even worse in the case of LLMs, as compared to humans they can be much more thoroughly adapted to specific tasks through architectural design choices, hyperparameter selection, choice of training data, and fine-tuning paradigm. This further highlights the difficulty of using benchmarks as both research targets and as evaluation metrics. Supplementing these theoretical concerns, several recent

Read the complete text