AssistedDS: Benchmarking How External Domain Knowledge Assists LLMs in Automated Data Science
Xuan Bi, Zirui Liu, Jiawei Zhou, Jayanth Srinivasa, Ashish Kundu, Charles Fleming, Mingyi Hong, Jie Ding · 2025
Read AssistedDS: Benchmarking How External Domain Knowledge Assists LLMs in Automated Data Science by Xuan Bi, Zirui Liu, Jiawei Zhou, Jayanth Srinivasa, Ashish Kundu, Charles Fleming, Mingyi Hong, Jie Ding (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-13992.txt.
Description
An Luo and co-authors benchmark whether language models can use outside domain knowledge in data science work, in a 2025 paper. Automated data tools have grown capable, yet real analysts lean on field knowledge. Whether models can judge such knowledge critically was untested. Their benchmark, AssistedDS, pairs prediction tasks with bundles of documents.
Some documents help and others mislead on purpose. Tasks span synthetic data with known ground truth and real contest datasets. Top models were scored on telling good guidance from bad. They were also scored on applying the good parts to cleaning, features, and model choice. The test targets a skill that separates a tool user from a true analyst.
Catalog record
| CREATOR | Xuan Bi, Zirui Liu, Jiawei Zhou, Jayanth Srinivasa, Ashish Kundu, Charles Fleming, Mingyi Hong, Jie Ding |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 105 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-13992