Position: Ideas Should be the Center of Machine Learning Research

Jairo Diaz-Rodriguez · 2026

Read Position: Ideas Should be the Center of Machine Learning Research by Jairo Diaz-Rodriguez (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-15253.txt.

Introduction

Jairo Diaz-Rodriguez argues, in a position paper, that ideas should sit at the center of machine learning research. The field splits between benchmark-driven engineering that prizes metrics over understanding and idealized theory that usually fail to carry over. His Ideas First framework values ideas by the behavior patterns they predict in modern models, tested through tailored experiments rather than leaderboard wins. The shift also promotes equity by dropping the complexity premium that favors researchers with large resources.

Catalog record

CREATORJairo Diaz-Rodriguez
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39162
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.15253

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Position: Ideas Should be the Center of Machine Learning Research Jairo Diaz-Rodriguez Abstract Machine learning research increasingly bifurcates into two disconnected modes: benchmark-driven engineering that prioritizes metrics over understanding, and idealized theory that often fails to transfer to modern systems. In this position paper, we argue that the field focuses too heavily on these endpoints, neglecting the central scientific object: the idea. We propose an Ideas First framework in which ideas are valued for the behavioral signatures they predict in modern models, and these signatures are tested through tailored experiments designed to detect the relevant patterns rather than to win leaderboards. This shift not only bridges the gap between theory and practice but also promotes equity by removing the “complexity premium,” enabling rigorous scientific contributions from researchers with modest computational, financial, and human resources. Ultimately, we advocate for a research culture centered on ideas, treating benchmarks and theorems as instruments for testing mechanistic hypotheses rather than as ends in themselves. 1 Introduction Machine learning research has converged on two dominant modes. On the empirical side, shared benchmarks and leaderboards define success: contributions are judged primarily by their effect on a single number on a widely recognized test set ( 37 ; 16 ) . On the theoretical side, many results are evaluated by the strength of guarantees in highly idealized settings that only partially resemble contemporary overparameterized models ( 5 ; 65 ) . Both modes have delivered real progress, yet they also narrow what counts as a contribution and who can participate, particularly when large complex models, and extensive ablations become de facto requirements ( 1 ) . Our position. In this paper we argue that the central scientific object in machine learning is not a benchmark score or a theorem, but an idea : a hypothesis about how learning systems work, what kinds of structure they can exploit, or how they should be evaluated. We call our perspective Ideas First . Ideas gain scientific content when they give rise to signatures that we can look for in the behavior of modern models (Figure 1 ), such as specific patterns, characteristic failure modes, or qualitative changes in representation geometry. Experiments are then designed to search for these signatures and rule out competing explanations, rather than primarily to maximize performance on a fixed leaderboard. This reverses the usual order of justification: instead of starting from a large system, benchmark, or theory and asking what results can be extracted, we start from the idea and ask what behavior it predicts and how those predictions can be tested. On this view, simple models, small-scale experiments, and minimal theoretical analyses can be especially valuable when they isolate the mechanism of interest, while large benchmarks, complex systems, and sophisticated theorems remain important when they clarify, operationalize, or stress-test the idea. We are not against benchmarks or theory; we are in favor of placing ideas at the center so that strong ideas can later generate both informative benchmarks and illuminating theorems without being defined by them at birth. Equity and simplicity. This shift also puts equity and simplicity at its core. When publication standards implicitly demand large models, complex implementations, and exhaustive ablations, researchers with substantial computational and engineering resources are favored, while simple but sharp ideas that can be tested in modest settings struggle to be seen ( 1 ; 55 ; 50 ) . Large labs can distribute work across many people, combining teams that run large scale experiments with teams that derive sophisticated theoretical results. This capacity can and does lead to impressive contributions. The problem arises when the same expectations are used as a gatekeeper: an idea can be rejected because it lacks a strong theorem in a highly idealized setting whose assumptions limit its relevance, or because it lacks large benchmark results that may themselves be vulnerable to overfitting and design artifacts ( 16 ; 9 ) . History and sociology of science suggest that major advances often emerge from chains of small, conceptually clear steps rather than isolated breakthroughs ( 39 ; 35 ) . A culture that filters out such steps because they are not wrapped in heavy systems or frontier benchmarks risks discarding the very ideas that future theory and benchmarks could build on. Change of culture. Our position is a call to return to hypothesis driven inquiry in the sense used in empirical sciences such as physics and biology. This style of work already underlies much of the most insightful machine learning research (see Section 4 ), but it is rarely foregrounded in how contributions are framed, evaluated, and rewarded. We argue that the community should normalize the Ideas First perspective, especially for junior researchers, and align reviewing and publishing norms with it. Doing so would deliberately shift incentives away from a status quo that unconsciously mimics the resource intensive priorities of large industrial labs toward a culture in which clear ideas, explicit signatures, and accessible experiments are recognized as core scientific contributions. Goals. The objectives of this position paper are threefold: (i) We propose a framework that links ideas to predicted signatures to tailored experiments for modern neural systems. (ii) We develop a case in favor of idea centric evaluation, connecting it to work on benchmarks, theory, philosophy of science, and equity in AI. (iii) We provide concrete examples of this framework in practice, together with practical notions for evaluating ideas. Structure of the paper. Section 2 introduces our critique of benchmark driven engineering and idealized theory. Section 3 presents the Ideas First framework. Section 4 revisits several deep learning results through this lens, and Section 5 offers practical guidance for authors and reviewers. Section 6 gives a hypothetical case study that illustrates the full workflow. Section 7 develops the broader case for idea centric evaluation and its implications. Finally, we present alternative views in Section 8 . Figure 1 : (Top) Current practice often bifurcates into Benchmark-Driven Engineering (optimizing a single metric on large systems, often obscuring mechanism) and Idealized Theory (rigorous proofs in simplified settings that may not transfer to modern models). (Bottom) Our proposed framework links these worlds: An Idea generates a concrete Signature (an observable behavioral commitment ). A Tailored Experiment is then designed specifically to detect this signature, prioritizing mechanistic clarity and accessible experimental design over state-of-the-art leaderboard performance. 2 Standard modes Before introducing our framework, we briefly characterize two standard modes that structure much of contemporary work on neural systems. While we acknowledge that research methodologies are diverse, we argue that with sufficient flexibility, the incentives of the field tend to align the vast majority of current work with one of these two dominant archetypes. Mode A: Benchmark-driven engineering This mode improves comparative performance on standard tasks by scaling models, adjusting training recipes, and modifying architectures and data. The central artifact is a system together with transparent comparisons on leaderboards and ablation studies that indicate which choices influenced the metric. Success is a higher score at fixed resources or at clearly reported increases in resources. The unit of evidence is a demonstrated score gain supported by controlled ablations. The strength of this mode is that it maps where and how well a method works across tasks and scales. Its

Read the complete text