TML-bench: Benchmark for Data Science Agents on Tabular ML Tasks
Mykola Pinchuk, PhD · 2026
Read TML-bench: Benchmark for Data Science Agents on Tabular ML Tasks by Mykola Pinchuk, PhD (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-05764.txt.
Introduction
Pinchuk introduces TML-Bench for data science agents on Kaggle-style tasks. Autonomous coding agents produce strong baselines quickly, but value depends on correctness. The paper evaluates 10 open-source models on four Kaggle competitions at 240, 600, and 1200 seconds. Each model runs five times per task and budget. A run succeeds with a valid submission plus a private-holdout score. Results report median performance, success rates, and variability. MiniMax-M2.1 achieves the best aggregate score. Average performance improves with larger budgets, though scaling is noisy.
Catalog record
| CREATOR | Mykola Pinchuk, PhD |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 16017 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.05764
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
TML-bench: Benchmark for Data Science Agents on Tabular ML Tasks Mykola Pinchuk, PhD Abstract Autonomous coding agents can produce strong tabular baselines quickly on Kaggle-style tasks. Practical value depends on end-to-end correctness and reliability under time limits. This paper introduces TML-bench, a tabular benchmark for data science agents on Kaggle-style tasks. This paper evaluates 10 OSS LLMs on four Kaggle competitions and three time budgets (240s, 600s, and 1200s). Each model is run five times per task and budget. A run is successful if it produces a valid submission and a private-holdout score on hidden labels that are not accessible to the agent. This paper reports median performance, success rates, and run-to-run variability. MiniMax-M2.1-TEE achieves the best aggregate performance score on all four competitions under the paper’s primary aggregation. Average performance improves with larger time budgets. Scaling is noisy for some individual models at the current run count. Code and materials are available at https://github.com/MykolaPinchuk/TML-bench/tree/master . 1 Introduction Tabular machine learning remains a common practical workload. The workflow includes data loading, feature preparation, model training, evaluation, iteration, and producing a correctly formatted submission artifact. Benchmarks that only test isolated coding tasks miss important failure modes and trade-offs. This paper evaluates data science agents on a strict tabular benchmark with private-holdout scoring. The target is an auditable leaderboard built from repeatable runs. The benchmark considers score distributions across runs rather than a single best attempt. For practitioners, this framing matters for two reasons. First, a good agent must be reliable. Capability on a single lucky run is insufficient. Second, comparisons should remain meaningful when tasks use different metrics and when runs are time-bounded. Contributions This paper makes the following contributions: • This paper introduces a strict benchmark protocol for Kaggle-style tabular tasks with deterministic preparation, strict submission validation, and private-holdout scoring. • This paper uses a repeatable reporting policy with fixed agent instructions, a fixed suite, and median-of-five aggregation with explicit coverage requirements. • This paper adopts contamination controls and a low-cost evaluation setup intended to be runnable by individual practitioners (internet-off execution and pre-competition knowledge-cutoff model selection). • This paper provides a reproducible evaluation protocol and supporting materials to regenerate figures and tables. • This paper reports results and analysis that highlight performance, cross-competition consistency, reliability, and scaling with time budget. Related work Several existing benchmarks evaluate coding-capable models and autonomous coding systems on programming and data workflows, but they differ in scope and evaluation philosophy. SWE-bench focuses on real-world software engineering tasks derived from GitHub issues [7]. It evaluates whether a system can produce code changes that resolve the issue under a test suite. MLE-bench evaluates autonomous systems on ML engineering via Kaggle-style competitions curated from Kaggle [8]. DSBench evaluates data science systems on a broader set of tasks that includes both data analysis and data modeling, and it introduces a metric designed to normalize heterogeneous task metrics [9]. MLAgentBench evaluates autonomous coding systems on a suite of ML experimentation tasks in a controlled environment [10]. Relative to MLE-bench, TML-bench makes a different set of trade-offs. MLE-bench optimizes breadth across many competitions and modalities. TML-bench instead constrains scope to tabular tasks and a fixed small suite, which reduces cross-modality confounding and makes run-to-run comparisons easier to interpret. Constraining to relatively small tabular datasets also keeps evaluation accessible: the marginal cost to run the full suite in the setup used for this paper was on the order of $10. TML-bench also emphasizes repeatability with fixed instruction sets, explicit complete-coverage requirements, and repeated runs per setting. In addition, TML-bench uses private-holdout scoring outside the system workspace, keeps internet access disabled during runs, and selects evaluated models so their knowledge cutoff predates the start date of the tested Kaggle competitions. These design choices report reliability side by side with performance and reduce contamination risk from post-competition information. 2 Benchmark and protocol 2.1 Suite and evaluation grid This paper evaluates model-assisted tabular ML work over a four-competition suite and three time budgets (240s, 600s, 1200s). The 1200s configuration uses an XGBoost-focused instruction set. Each time budget uses a fixed instruction set, so the 240s/600s/1200s settings differ in both available time and the instructions given to the agent. Reporting is restricted to models that reached complete five-run coverage across all 12 task \times budget settings (4 competitions \times 3 time budgets). Under that rule, 10 models are included in the main tables. 2.2 Prompt strategy and aggregation rule A fixed instruction template is used for all runs in this paper. For each (competition, model, budget) setting, the reported value is the median of the earliest 5 successful runs ordered by created_at . 2.3 Evaluation logging and reproducibility The evidence in this paper is derived from logged run outcomes (status, score, runtime, and run configuration). I focus only on models with complete coverage: five successful runs for every task \times budget setting included in the benchmark. 2.4 Agent harness (Kilo Code) Each run is executed in a clean, per-run workspace managed by the Kilo Code harness. The harness enforces the time budget, validates the submission format, and scores submissions on hidden holdout labels (not accessible to the agent). Appendix D describes what Kilo Code is and why this paper standardizes on it. 2.5 Contamination controls This paper uses two controls to reduce contamination risk from external information. First, internet access is disabled during benchmark runs. Second, evaluated models are selected so that their pretraining knowledge cutoff predates the start date of each tested Kaggle competition. In this suite, the earliest competition release is in late October 2025, and all listed cutoff dates in Appendix A precede that date. 2.6 Metrics and normalization Each competition has a task-defined metric. This paper reports score_raw in the task’s native direction (for example, AUC where higher is better, and RMSE where lower is better). Raw metric values are not comparable across competitions because they have different scales and different directions. To build aggregate leaderboards and scaling plots, this paper uses a within-setting min-max normalization: • For each (competition, budget) setting, compute each model’s five-run median score_raw . • Convert to a common “higher is better” value: value_hib = score_raw for higher-is-better metrics, and value_hib = -score_raw for lower-is-better metrics. • Min-max normalize within that setting so the best model gets 1.0 and the worst gets 0.0: \mathrm{score}=\frac{\mathrm{value\_hib}-\min(\mathrm{value\_hib})}{\max(\mathrm{value\_hib})-\min(\mathrm{value\_hib})}. Appendix C defines the normalization and the primary aggregation precisely. 2.7 Time budgets This paper evaluates three wall-clock time budgets per competition: 240 seconds, 600 seconds, and 1200 seconds. A time budget represents the total time available to the agent to read the task, train, iterate, and produce a final submission file. Implementation details (what “240s” means in practice): • The time budget applies to the agent interaction stage, enforced as a hard timeout by the Kilo Code harness. • Runtime is measured from run start to the end of the agent stage (or timeout). Validation and private-holdout scoring occur after this stage and do not count against the time budget. • If the harness observes that the agent exceeded the time budget (with a small grace window for timing granularity), the run is marked as a timeout even if a submission file exists. Time budgets are included because “fast baseline” and