HardML: A Benchmark For Evaluating Data Science And Machine Learning knowledge and reasoning in AI

Tidor-Vlad Pricope Independent Machine Learning Engineer · 2025

Read HardML: A Benchmark For Evaluating Data Science And Machine Learning knowledge and reasoning in AI by Tidor-Vlad Pricope Independent Machine Learning Engineer (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-15627.txt.

Introduction

Tidor-Vlad Pricope builds a hard test for artificial intelligence in data science and machine learning. His 2025 test, called HardML, holds one hundred multiple-choice questions. He wrote them by hand over six months, drawing on the most used and most modern parts of the field. Most of the content is new, so models cannot pass by recalling leaked answers.

Even strong engineers would find the questions tough, Pricope notes. Top models miss about three in ten, a rate three times worse than on a well known older test. The test is small and not meant to push the field's edge. Still, it gives a sharp, modern yardstick for tracking how far the best models have come in this area.

Catalog record

CREATORTidor-Vlad Pricope Independent Machine Learning Engineer
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-15627

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

HardML: A Benchmark For Evaluating Data Science And Machine Learning knowledge and reasoning in AI Tidor-Vlad Pricope Independent Machine Learning Engineer Abstract . We present HardML, a benchmark designed to evaluate the knowledge and reasoning abilities in the fields of data science and machine learning. HardML comprises a diverse set of 100 challenging multiple-choice questions, handcrafted over a period of 6 months, covering the most popular and modern branches of data science and machine learning. These questions are challenging even for a typical Senior Machine Learning Engineer to answer correctly. To minimize the risk of data contamination, HardML uses mostly original content devised by the author. Current state-of-the-art AI models achieve a 30% error rate on this benchmark, which is about 3 times larger than the one achieved on the equivalent, well-known MMLU-ML. While HardML is limited in scope and not aiming to push the frontier—primarily due to its multiple-choice nature—it serves as a rigorous and modern testbed to quantify and track the progress of top AI. While plenty benchmarks and experimentation in LLM evaluation exist in other STEM fields like mathematics, physics and chemistry, the sub-fields of data science and machine learning remain fairly underexplored. 2020 Mathematics Subject Classification 68T50, 68T07, 68T05, 68T20 1. Introduction Recent advancements in large language models (LLMs) have led to significant progress in natural language processing tasks such as translation, summarization, question answering, and code generation [ 1 , 2 ] . These models have been extensively evaluated using benchmarks covering a wide range of subjects, providing valuable insights into their capabilities [ 3 , 4 ] . For instance, the Massive Multitask Language Understanding (MMLU) benchmark [ 5 ] assesses LLMs across diverse disciplines, including STEM fields like mathematics, physics, and chemistry [ 6 , 7 , 8 ] . However, data science (DS) and machine learning (ML) have received relatively little attention in benchmarking efforts. The MMLU test set contains only 112 machine learning questions. Moreover, in the few instances where these domains have been explored, state-of-the-art AI models achieve near-saturation performance, rendering existing benchmarks less effective for distinguishing model capabilities. It is imperative to devise novel benchmarks that keep up with the rapid advancements in LLMs. This necessity is exemplified in the FrontierMath benchmark [ 15 ] , which introduces a future-proof evaluation for mathematics by presenting problems that remain unsolved by over 98% of current AI models. Such benchmarks are crucial for continuing to challenge and develop advanced AI systems. Data science and machine learning are foundational to modern artificial intelligence, driving advancements in everything from healthcare to finance [ 9 , 10 ] . Mastery in these fields requires not only theoretical understanding but also practical skills in applying algorithms, statistical methods, and computational techniques to solve complex, real-world problems [ 11 , 12 ] . As AI systems become increasingly involved in DS and ML tasks—ranging from automated model training to data analysis—it is crucial to assess their proficiency and reasoning abilities in these areas. However, as of January 2025, benchmarks in this domain are very limited. The most notable examples include the test ML subsection of MMLU (MMLU-ML) [ 5 ] , which consists of 112 multiple-choice questions, and MLE-bench [ 16 ] , introduced by OpenAI, which evaluates practical ML engineering skills using a collection of 75 coding questions modeled after Kaggle competitions. To address this gap, we propose HardML, a benchmark specifically designed to evaluate the knowledge and reasoning capabilities of AI models in data science and machine learning. HardML employs the same testing framework as MMLU, comprising multiple-choice questions, with the primary difference being that more than one answer can be correct. It differs in scope from MLE-bench, as it does not test coding capabilities but focuses on theoretical understanding and reasoning skills based on theoretical concepts in DS and ML. HardML uses 100 challenging multiple-choice questions, meticulously crafted over six months to cover the most relevant and contemporary topics in DS and ML. The questions are designed to be difficult even for experienced professionals, such as senior machine learning engineers, thereby ensuring that the benchmark assesses advanced understanding and critical problem-solving skills. A key aspect of HardML is its emphasis on originality and contemporary relevance, featuring questions that reflect the latest advancements in machine learning from the past two years. To minimize the risk of data contamination—where models might have been trained on benchmark content, leading to artificially inflated performance [ 13 ] —we developed primarily original questions. By ”original,” we mean that while the core concepts required to solve these questions may be known (similar to foundational theorems in mathematics), the specific applications and reasoning required are unique. These questions span topics including natural language processing, computer vision, statistics and statistical modeling, classical machine learning algorithms, and more. In this paper, we also introduce EasyML, a benchmark of 85 multiple-choice questions designed to provide a more accessible and slightly easier set of questions than MMLU-ML for evaluating smaller language models, such as GPT4o-mini [ 21 ] and LLaMA models with fewer than 70 billion parameters [ 22 ] . Our evaluation of state-of-the-art AI models (o1) [ 20 ] , reveals a substantial performance gap compared to existing benchmarks [Figure 1 ]. Specifically, these models exhibit an error rate of approximately 30% on HardML, which is significantly higher than the error rate on the machine learning section of MMLU (MMLU-ML) [ 5 ] . This disparity highlights the challenges that current AI models face in mastering the complexities of DS and ML, particularly in understanding nuanced concepts and applying them to non-trivial problems. The initial motivation behind constructing this benchmark was to generate a comprehensive set of interview-preparation questions for individuals seeking positions in machine learning at leading technology companies (FAANG). However, the interesting results obtained during large language model evaluation, purely out of curiosity, led to the development of this paper. Given the relative scarcity of specialized benchmarks in these fields compared to others like mathematics and physics, we believe HardML fills an important gap and provides a foundation for future research and development. Figure 1. Comparison of error rate across 3 DS&ML benchmarks. While existing benchmarks are approaching saturation, HardML keeps an average level above saturation, in line with benchmarks from other fields like MathVista [ 23 ] or AIME [ 24 ] despite the multiple-choice nature 2. Data collection The data collection involved a multi-step process spanned over 6 months. As mentioned in the last paragraph of the introduction, the initial purpose of this project was to form a set of question-answer for ML interview assessment for entrance of the top tech companies. These are to be used on the platform neuraprep.com, which is a website similar to leetcode.com for interview preparation. Therefore, the dataset construction wasn’t biased towards building problems that the LLMs wouldn’t be able to solve, they were fully intended for human use. 2.1. The collection pipeline The collection pipeline for the development of HardML and EasyML involved a meticulous 4-step process: (1) Raw data collection and scraping. We have sourced approximately 400 questions from public platforms such as Glassdoor, Blind, Quora, Stack Exchanges, YouTube, as well as from papers and books —including those by Bishop

Read the complete text