Towards Adaptive ML Benchmarks: Web-Agent-Driven Construction, Domain Expansion, and Metric Optimization

Hangyi Jia Yuxi Qian Hanwen Tong Xinhui Wu Lin Chen Feng Wei · 2025

Read Towards Adaptive ML Benchmarks: Web-Agent-Driven Construction, Domain Expansion, and Metric Optimization by Hangyi Jia Yuxi Qian Hanwen Tong Xinhui Wu Lin Chen Feng Wei (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-09321.txt.

Introduction

Hangyi Jia and 5 co-authors build TAM Bench, a benchmark for machine learning agents, in a 2025 paper. Agents now run full ML workflows end to end, from data prep to model training. Old benchmarks lack the task range, domain spread, and rigor to judge them fairly.

A web agent plus language models harvest and structure tasks from sites such as Kaggle. Leaderboard counts and score spread set each task's difficulty grade. Scoring checks task score, format, rule adherence, and transfer to new tasks. One hundred fifty tasks form Lite, Medium, and Full sets. The Lite set, with balanced modes and grades, suits daily use.

Catalog record

CREATORHangyi Jia Yuxi Qian Hanwen Tong Xinhui Wu Lin Chen Feng Wei
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-09321

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Towards Adaptive ML Benchmarks: Web-Agent-Driven Construction, Domain Expansion, and Metric Optimization Hangyi Jia Yuxi Qian Hanwen Tong Xinhui Wu Lin Chen Feng Wei Abstract Recent advances in large language models (LLMs) have enabled the emergence of general-purpose agents for automating end-to-end machine learning (ML) workflows, including data analysis, feature engineering, model training, and competition solving. However, existing benchmarks remain limited in task coverage, domain diversity, difficulty modeling, and evaluation rigor, failing to capture the full capabilities of such agents in realistic settings. We present TAM Bench, a diverse, realistic, and structured benchmark for evaluating LLM-based agents on end-to-end ML tasks. TAM Bench features three key innovations: (1) A browser automation and LLM-based task acquisition system that automatically collects and structures ML challenges from platforms such as Kaggle, AIcrowd, and Biendata, spanning multiple task types and data modalities (e.g., tabular, text, image, graph, audio); (2) A leaderboard-driven difficulty modeling mechanism that estimates task complexity using participant counts and score dispersion, enabling scalable and objective task calibration; (3) A multi-dimensional evaluation framework incorporating performance, format compliance, constraint adherence, and task generalization. Based on 150 curated AutoML tasks, we construct three benchmark subsets of different sizes—Lite, Medium, and Full—designed for varying evaluation scenarios. The Lite version, with 18 tasks and balanced coverage across modalities and difficulty levels, serves as a practical testbed for daily benchmarking and comparative studies. Introduction With the successful application of large language models (LLMs) in areas like code generation and task planning, an increasing number of researchers have begun exploring their potential in automating end-to-end machine learning (ML) tasks. These tasks include model training, feature engineering, data analysis, and competition problem solving. This emerging direction has led to the development of intelligent agent systems for the data science pipeline, such as AIDE ( Jiang et al. 2025 ) AutoMind ( Ou et al. 2025 ) ML-Agent ( Liu et al. 2025b ) and ML-Master ( Liu et al. 2025a ) . These systems aim to build general-purpose agents capable of task understanding, tool invocation, and workflow execution to fully automate the ML pipeline. However, the current evaluation of LLMs’ comprehensive capabilities in ML tasks remains insufficient. Existing benchmarks suffer from the following major issues: 1) High Manual Cost of Data Collection,Traditional benchmarks rely heavily on manual efforts for competition collection, content extraction, and task filtering. While this ensures data quality, it is inefficient, costly, and difficult to scale.2) Imbalanced Distribution of Task Types and Application Domains, Existing benchmarks are biased in both task types and application scenarios. For example, MLEBench ( Chan et al. 2025 ) includes 36 image/text classification tasks but only sparsely covers tasks like object detection or image-text generation. CALM ( Feng et al. 2024 ) focuses exclusively on tabular modeling in the finance domain. Common real-world applications such as e-commerce recommendation and educational analytics are missing, which limits the representativeness and generalizability of evaluations.3) Unreasonable Task Difficulty Modeling, In a diverse task set, reasonable difficulty grading is essential to assess agents fairly. It helps highlight performance gaps across complexity levels and understand generalization boundaries. Currently, only MLEBench ( Chan et al. 2025 ) attempts difficulty modeling via expert-annotated time estimates, which is subjective, costly, and not scalable. 4) Single-Dimensional Evaluation Metrics, Most benchmarks rely on single metrics, e.g., medals of MLEBench ( Chan et al. 2025 ) or performance improvement over baseline of MLAgentBench ( Huang et al. 2024 ) . This oversimplifies agent assessment and may cause reward hacking, where agents prioritize scores while ignoring format or business constraints. To address these limitations, we propose TAM-Bench—a structured, diverse, and realistic benchmark to evaluate the comprehensive capabilities of LLMs in ML tasks. Key innovations include: • Automated Task Collection and Standardization across Domains: Building on MLEBench ( Chan et al. 2025 ) , we identify key ML task types and domains. Inspired by MCP ( Hou et al. 2025 ) and Browser-Use, we develop a web-agent-based system to scrape, extract, and schema-unify tasks from Kaggle, AIcrowd, Biendata, etc., enabling scalable cross-domain task acquisition and reducing manual overhead. • Leaderboard-Based Difficulty Modeling: Using real competition data (e.g., participant count and score distribution), we design an automated and extensible task difficulty scoring system, reducing subjectivity and improving scalability. • Multi-Dimensional Evaluation Framework: We incorporate performance, format compliance, constraint adherence, and generalization to provide a holistic agent capability assessment. Our design prevents reward hacking by ensuring improvements follow real-world constraints. TAM-Bench aims to enable robust evaluation and development of LLM-powered agents in complex, realistic ML workflows, establishing a key standard for assessing data science capabilities. Figure 1: An overall illustration of TAM-Bench Related Work Benchmarks for End-to-End ML Tasks With LLMs widely adopted in data science and ML automation, several benchmark frameworks have emerged to evaluate agents’ end-to-end ML task performance. MLE-Bench ( Chan et al. 2025 ) is a benchmark built from Kaggle competitions with 75 offline challenges spanning various data types and complexity levels. It features standardized difficulty tiers (Easy/Medium/Hard) and enables fine-grained agent evaluation. MLAgentBench ( Huang et al. 2024 ) focuses on LLM agents’ experimental performance across 13 tasks, including image classification (e.g., CIFAR-10), NLP, and BabyLM. It supports file I/O and code execution in a unified environment. CALM ( Feng et al. 2024 ) targets credit risk assessment. It introduces an instruction-tuned LLM trained on over 45,000 samples from nine datasets, emphasizing reproducibility and real-world application in finance. These benchmarks standardize evaluation and foster agent development, but each has scope limitations. Agent Systems for ML Automation To tackle benchmark challenges, various LLM agent systems have been proposed to automate ML pipelines from problem understanding to interpretation. AIDE ( Jiang et al. 2025 ) (Jiang et al., 2025) and OpenHands ( Wang et al. 2025 ) (Wang et al., 2025) are two key systems on MLE-Bench. AIDE uses modular RL-LLM integration for Kaggle-style tasks. OpenHands, a general-purpose software agent platform, allows agents to operate OS environments in human-like ways, enabling complex workflows. These systems combine LLM reasoning with structured pipelines but still face challenges in long-term dependencies, code quality, and generalization. Future research will focus on improving autonomy, transferability, and interpretability. Method This section describes the core design and implementation of TAM-Bench, including automated pipeline, task schema, task diversity strategy, difficulty modeling, and multi-metric evaluation. Automated benchmark construction Traditional benchmark construction typically relies on manual processes such as competition selection, content extraction, and task filtering. Although this ensures data quality, it suffers from low efficiency and high labor costs. To address these limitations, we propose a fully automated benchmark construction method that leverages large language models (LLMs) and browser automation technologies, significantly reducing repetitive human labor while improving scalability and efficiency.To achieve this, we design a technical pipeline that automatically retrieves tasks and constructs evaluation samples from major competition platforms such as Kaggle, AIcrowd, and Biendata.Inspired by the core idea of Model Context Protocol (MCP) ( Hou et al. 2025 ) —which connects heterogeneous data sources and tools to LLMs via a unified interface—we recognize that automated benchmark construction faces similar challenges in accessing and processing multi-format, cross-platform task data. Drawing upon the design philosophy of ”unified access interface + agent-driven tool collaboration,” we implement an automated pipeline using the open-source WebAgent framework, Browser-Use

Read the complete text