NeurIPS 2025 E2LM Competition : Early Training Evaluation of Language Models
Mouadh Yagoubi, Yasser Dahou, Billel Mokeddem, Younes Belkada, Basma El Amel Boussaha, Reda Alami, Jingwei Zuo, Mugariya Farooq, Mounia Lalmas, Patrick Gallinari, Philip Torr, Hakim Hacid · 2025
Read NeurIPS 2025 E2LM Competition : Early Training Evaluation of Language Models by Mouadh Yagoubi, Yasser Dahou, Billel Mokeddem, Younes Belkada, Basma El Amel Boussaha, Reda Alami, Jingwei Zuo, Mugariya Farooq, Mounia Lalmas, Patrick Gallinari, Philip Torr, Hakim Hacid (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-07731.txt.
Introduction
Mouadh Yagoubi, Yasser Dahou, and Billel Mokeddem lead a large team on this paper from two thousand twenty-five. It sets out the E2LM competition, run with NeurIPS in that year, on grading language models while they are still young in training.
Set benchmarks grade finished models well but give weak, blurry signals early in training, when small models shift fast. The contest asks entrants to build science tests tuned to that early phase. Three small pre-trained models and their mid-training saves are given, and all work can run on free cloud GPUs. Entries are judged on signal quality, on how well early ranks match the ranks at a far later stage, and on fit to science. The aim is benchmark-led training from day one.
Catalog record
| CREATOR | Mouadh Yagoubi, Yasser Dahou, Billel Mokeddem, Younes Belkada, Basma El Amel Boussaha, Reda Alami, Jingwei Zuo, Mugariya Farooq, Mounia Lalmas, Patrick Gallinari, Philip Torr, Hakim Hacid |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-07731
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
NeurIPS 2025 E2LM Competition : Early Training Evaluation of Language Models Thanks: California Institute of Technology, Pasadena, CA, USA Mouadh Yagoubi, Yasser Dahou, Billel Mokeddem, Younes Belkada, Basma El Amel Boussaha, Reda Alami, Jingwei Zuo, Mugariya Farooq, Mounia Lalmas, Patrick Gallinari, Philip Torr, Hakim Hacid Abstract Existing benchmarks have proven effective for assessing the performance of fully trained large language models. However, we find striking differences in the early training stages of small models, where benchmarks often fail to provide meaningful or discriminative signals. To explore how these differences arise, this competition tackles the challenge of designing scientific knowledge evaluation tasks specifically tailored for measuring early training progress of language models. Participants are invited to develop novel evaluation methodologies or adapt existing benchmarks to better capture performance differences among language models. To support this effort, we provide three pre-trained small models (0.5B, 1B, and 3B parameters), along with intermediate checkpoints sampled during training up to 200B tokens. All experiments and development work can be run on widely available free cloud-based GPU platforms, making participation accessible to researchers with limited computational resources. Submissions will be evaluated based on three criteria: the quality of the performance signal they produce, the consistency of model rankings at 1 trillion tokens of training, and their relevance to the scientific knowledge domain. By promoting the design of tailored evaluation strategies for early training, this competition aims to attract a broad range of participants from various disciplines, including those who may not be machine learning experts or have access to dedicated GPU resources. Ultimately, this initiative seeks to make foundational LLM research more systematic and benchmark-informed from the earliest phases of model development. Keywords Early training analysis, Evaluation benchmarks, Scientific knowledge, Low-resource ML research, Language Models. 1 Competition description 1.1 Background and impact Large Language Models (LLMs) have advanced at an unprecedented pace in recent years grattafiori2024llama ; yang2024qwen2 ; malartic2024falcon2 ; zhao2023survey . In response, the research community has increasingly focused on developing benchmarking frameworks to assess their capabilities in a variety of domains, including language, mathematics, and reasoning. Early evaluations primarily focused on measuring the language modeling skills zellers2019hellaswag ; sakaguchi2021winogrande , a fundamental competence that appears to be increasingly well-acquired by newer LLM variants saturating such benchmarks. Recently, attention has shifted towards evaluating scientific knowledge, code generation and reasoning capabilities, using benchmarks such as MMLU hendryckstest2021 and its enhanced version MMLU-pro wang2024mmlu , GPQA rein2024gpqa , GSM8K cobbe2021training , MATH hendrycks2021measuring , and LiveCodeBench jain2024livecodebench . A common theme across these benchmarks is the increasing complexity of tasks, aimed at testing the limits of frontier models. Despite these advances, Small Language Models (SLMs) with lower than 7B parameters consistently underperform relative to their larger counterparts on these challenging benchmarks. This performance gap is particularly problematic when training SLMs from scratch, as these benchmarks fail to offer discriminative signals during early training phases (typically between 0-200 billion tokens). Consequently, they provide limited guidance for training dynamics regarding: hyperparameter search, data mixture and architectural sweeps. Developing more granular and targeted evaluation benchmarks in the scientific knowledge domain would enable more systematic investigation of fundamental research questions during the early training stages of SLMs. Figure 1: Noisy results obtained with state-of-the-art benchmarks with different models sizes. Concretely, we provide three pre-trained SLMs (0.5B, 1B, and 3B) along with intermediate checkpoints up to 200 billion tokens. As shown in Figure 1 , current benchmarks are often noisy and fail to provide a clear signal to measure progress, raising several fundamental questions: Are SLMs not learning any scientific knowledge early in the training? Or are the benchmark questions too difficult at this stage? Is the noise due to a lack of acquired knowledge, or could it be related to how the questions are formatted from a linguistic standpoint? Are there ways to refine current benchmarks to better extract meaningful signals? The primary goal of this competition is to investigate these questions by inviting participants to design and develop novel evaluation tasks that enable a progressive assessment of scientific knowledge in SLMs, hence, enabling a progressive assessment of scientific knowledge in SLMs and producing more informative performance signals. Specifically, the expected impact of the challenge can be summarized as follows: • Enhancing SLMs research: By enabling more rigorous and conclusive studies on smaller models, this competition will open opportunities to develop stronger SLMs in scientific knowledge, code, and mathematics–ultimately helping to bridge the gap with larger models through meaningful analysis across a large number of small-scale experiments. • Transfer to large models: Lessons learned from studying SLMs can help inform better decisions when training larger models, particularly in terms of data mixtures, hyper-parameters, and architectural choices. We refer the reader to chen2024role ; yang2022tensor for a comprehensive overview of the role of small models in advancing LLMs. • Understanding the learning dynamics: Current benchmarks often do not fully reflect the convergence or complexity of the model in a meaningful way. Benchmark difficulty should ideally evolve with a model’s level of training. Tackling this problem may offer insights into how models learn over time. • Expand the reach of LLM research: We aim to encourage participation from researchers beyond the core machine learning community, inviting them to bring evaluation ideas and benchmarks rooted in their domain expertise. Such interdisciplinary collaboration has the potential to broaden the impact and relevance of LLM research. To this end, participants can either adapt existing benchmarks or design entirely new tasks from scratch. Moreover, we provide a baseline approach that has been successfully tested on the provided checkpoints (see Section 1.4 ). This baseline draws inspiration from the work of muennighoff2024olmoe , who proposed a variant of the well-known MMLU benchmark hendryckstest2021 , referred to as MMLU-var. In this variant, prompts are reformulated in a completion/close-style format rather than as multiple-choice questions. The motivation for this adaptation stems from the observation that LLMs typically gain proficiency in multiple-choice questions only in later training stages (e.g., after 1 trillion tokens), which already represents a substantial computational budget. Furthermore, the results on MMLU-var ( Figure 2 ) demonstrate how simple prompt modifications (see Appendix B ) can substantially improve the evaluation of SLMs. Grade-school-level benchmarks like ARC-easy allenai:arc and SciQ SciQ also provide useful signals (refer to Appendix A ). That said, benchmark complexity is inherently multi-dimensional; while MMLU-var addresses linguistic complexity in prompts, participants are encouraged to explore additional dimensions such as semantic depth and domain-specific reasoning in areas like mathematics or code generation. 1.2 Novelty Research on LLMs has been rapidly evolving across multiple fronts, including how to evaluate them effectively weidinger2025toward , how to improve their efficiency wan2023efficient , their training scaling laws kaplan2020scaling ; hoffmann2022training , and how to combine or adapt them for diverse use cases hadi2023survey . In recent NeurIPS editions, several competitions have been introduced to address the high costs associated with training and deploying them: the LLM Merging Challenge explored how to combine existing models to create more powerful ones without additional training tam2024llm , while the Edge LLM Challenge liu2024edge focused on developing efficient, optimized models capable of running on resource-constrained edge devices. A Single-GPU Fine-Tuning and Inference Challenge was also organized to promote the development of methods that