Ambig-DS: A Benchmark for Task-Framing Ambiguity in Data-Science Agents

Josefa Lia Stoisser, Marc Boubnovski Martell, Sidsel Boldsen, Kaspar Märtens, Robert Kitchen, [3.5ex] Novo NordiskLondon, UK · 2026

Read Ambig-DS: A Benchmark for Task-Framing Ambiguity in Data-Science Agents by Josefa Lia Stoisser, Marc Boubnovski Martell, Sidsel Boldsen, Kaspar Märtens, Robert Kitchen, [3.5ex] Novo NordiskLondon, UK (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-09698.txt.

Introduction

Stoisser and four colleagues built a test for a quiet failure mode in data-science AI agents: committing to the wrong task without noticing. Ambig-DS pairs each fully specified task with an ambiguous twin, one suite of 51 tasks about unclear prediction targets and one of 61 about unclear scoring goals. Each variant was checked to admit several plausible readings with real consequences for the answer.

Across five agents, failures were silent commitments, wrong targets or wrong scoring choices, rather than crashes. Letting an agent ask one clarifying question recovered much of the loss under ideal conditions. Yet agents could not judge when to ask: loose prompts made them over-ask on clear tasks, while strict prompts made them silently guess on ambiguous ones. Spotting underspecification, not running code, is the missing skill.

Catalog record

CREATORJosefa Lia Stoisser, Marc Boubnovski Martell, Sidsel Boldsen, Kaspar Märtens, Robert Kitchen, [3.5ex] Novo NordiskLondon, UK
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE112 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.09698

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Ambig-DS: A Benchmark for Task-Framing Ambiguity in Data-Science Agents Josefa Lia Stoisser, Marc Boubnovski Martell, Sidsel Boldsen, Kaspar Märtens, Robert Kitchen, [3.5ex] Novo NordiskLondon, UK Abstract As data-science agents shift from co-pilots to auto-pilots, silent misframing becomes a critical failure mode. Agents quietly commit to plausible but unintended task framings, producing clean, executable artifacts that hide their incorrect assessment of the task. Existing benchmarks score whether the pipeline runs, ignoring whether the agent recognized the task was underspecified. We introduce Ambig-DS, two diagnostic suites — one for prediction-target ambiguity (Ambig-DS-Target, 51 tasks built on DSBench, a tabular modeling benchmark) and one for evaluation-objective ambiguity (Ambig-DS-Objective, 61 tasks built on MLE-bench, a Kaggle-style ML competition benchmark) — constructed so that scoring uses each source benchmark’s original evaluator. For every task we pair the original, fully specified version with an ambiguous variant produced by controlled edits; a human-and-LLM verification pipeline confirms each variant admits multiple plausible interpretations with decision-relevant consequences. The suites are analyzed independently and ambiguity lowers performance in both. Across five agents spanning efficient to frontier-class models, we find in our controlled diagnostic setting: (i) failures are silent commitments — wrong-target submissions on Target, wrong-metric or non-committal baseline submissions on Objective — rather than execution errors; (ii) allowing the agent to ask one clarifying question recovers much of the loss under idealized conditions, suggesting missing framing information drives a substantial part of the observed degradation; but (iii) agents cannot reliably tell when to use it: permissive prompts induce over-asking on clear tasks, while conservative prompts induce silent defaulting on ambiguous ones. Recognizing target and objective underspecification, not pipeline execution, is the bottleneck missing from standard DS-agent evaluations. 1 Introduction The five frontier code-based agents we test, given a Kaggle-style prompt and the underlying CSV, commit to the wrong prediction target on 39 – 63\% of ambiguous tasks — and do not flag the ambiguity in any of them. The pipeline executes, the model trains, the submission is well-formed, and the source-benchmark evaluator accepts the artifact as valid. We call this failure mode unflagged misframing : a coherent, executable artifact that silently solves a plausible-but-unintended task, with no signal for a downstream consumer to catch (illustrated in Appendix Figure 3 ). Recent code-agent benchmarks ( 13 ; 8 ; 7 ; 9 ; 3 ; 38 ; 24 ; 36 ) measure pipeline execution under prompts that already fix the prediction target, output form, and evaluation metric. Real data-science work routinely lacks these commitments ( 26 ) : stakeholders supply a goal and a dataset, and the modeler decides what counts as the target and what counts as success. Whether agents can recognize when those decisions are underdetermined — and ask before acting — is not measured by current benchmarks, and is the competence we evaluate. We introduce Ambig-DS , a paired diagnostic for framing recognition : the competence of noticing that the prompt–data observation does not justify a unique commitment, and asking before acting. Ambig-DS holds the source-benchmark evaluator fixed and intervenes only on the prompt–data pair. Ambig-DS-Target (51 paired tasks on DSBench) isolates prediction-target ambiguity; Ambig-DS-Objective (61 paired tasks on MLE-bench) isolates evaluation-metric ambiguity. The intended framing is used only for scoring; the desired behavior is not to recover it from an underdetermined observation, but to recognize the underdetermination and ask. Each ambiguous variant is human-reviewed against a four-item retention checklist (plausible alternatives, ambiguity preservation, decision relevance, task preservation), with cross-family LLM-verifier audits confirming the labels. We analyze each suite independently and do not pool results or compare effect magnitudes across suites. Across five code-based agents, ambiguity induces unflagged misframing on 39 – 63\% of Ambig-DS-Target runs and silent failure on 16 – 62\% of Ambig-DS-Objective runs — failures of framing recognition rather than execution. An idealized one-shot clarification oracle recovers much of the loss when invoked, but no model–prompt-policy combination we test achieves both calibrated asking on ambiguous tasks and restraint on fully-specified ones: agents can use missing framing information when offered but cannot reliably detect when to ask. Data-science agent evaluation should therefore test framing recognition before pipeline execution. Contributions. We (i) formalize task framing and isolate two of its core variables (prediction target and evaluation objective); (ii) release Ambig-DS , two independently analyzed diagnostic suites (51 DSBench + 61 MLE-bench paired tasks) built via controlled prompt–data interventions verified by humans and cross-family LLM verifiers; and (iii) quantify the resulting misframing and ask–act miscalibration across five frontier agents as above. 2 Related Work Data Science / ML Benchmarks. Recent benchmarks evaluate LLMs and agents on increasingly realistic data-science and ML workflows. DS-1000 ( 13 ) and DA-Code ( 8 ) focus on executable data-science code generation, while MLAgentBench ( 7 ) , DSBench ( 9 ) , MLE-bench ( 3 ) , DataSciBench ( 38 ) , MLGym ( 24 ) , and RE-Bench ( 36 ) evaluate broader agentic workflows involving data preparation, modeling, experimentation, and iterative improvement, and serve as substrates for recent code-DS agent systems ( 23 ; 27 ; 29 ) . These benchmarks make DS/ML-agent evaluation more realistic, but still assume a specified task: the agent is told what to predict, optimize, and measure. Ambig-DS instead evaluates whether agents can recognize and resolve underspecified task framings before executing a pipeline, motivated by work showing that problem formulation choices are discretionary and consequential ( 26 ) . Ambiguity, Clarification, and Ask-Before-Act Agents. Recognizing underspecification 1 1 1 We use underspecification in the task-framing sense: the prompt–data observation does not pin down a unique supervised-learning problem. This is distinct from the model-level sense of 5 . has been studied across a sequence of progressively more action-laden settings. In open-domain QA and information seeking, prior work measures and trains for clarification of ambiguous questions, information needs, and entity references ( 20 ; 12 ; 42 ; 14 ; 25 ; 32 ) ; in dialogue, work studies when and what to ask, and how clarification reshapes downstream answers ( 40 ; 41 ; 4 ; 46 ; 19 ) ; for tool-using and planning agents, clarification is studied over unclear instructions, missing arguments, and uncertain tool calls ( 43 ; 21 ; 35 ; 44 ; 33 ; 18 ; 45 ; 31 ) . A common arc holds across these subfields: the community first measures silent failure under ambiguity, then builds ask-before-act benchmarks, then trains for calibrated asking. For executable predictive data-science workflows — where the unit of ambiguity is the supervised-learning framing rather than a question, a tool argument, or a plan step — the first step is missing. Ambig-DS supplies it. Executable and Data-Centric Underspecification. Closest to our setting are works where ambiguity affects executable artifacts. Code-generation and software-engineering work studies ambiguous requirements, clarification-aware training, and interactive clarification before implementation ( 22 ; 37 ; 15 ; 16 ) , including on real software-engineering tickets ( 34 ) , establishing ask–act evaluation as a mature setting for code and software agents. Data-centric work studies ambiguity in tabular analysis and natural-language table queries ( 17 ; 6 ) , while text-to-SQL work considers cases where one natural-language query maps to multiple plausible SQL programs ( 2 ; 30 ; 28 ) . Across

Read the complete text