Grading the Grader: Lessons from Evaluating an Agentic Data Analysis System

Tian Zheng, Kai-Tai Hsu · 2026

Read Grading the Grader: Lessons from Evaluating an Agentic Data Analysis System by Tian Zheng, Kai-Tai Hsu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-24839.txt.

Introduction

Tian Zheng and Kai-Tai Hsu study how reliably automated graders judge an agentic data analysis system. Such agents produce code, numbers, and verbal diagnostics, so grading them is harder than grading a single chat reply. The testbed is LAMBDA, a multi agent analysis system, run on 153 numerical tasks from DSGym. The authors build a three layer grading cascade and measure each layer against human labels.

The cascade runs strict pattern matching first, then a lenient language model grader, then human inspection of snippets. Both automated graders show perfect observed precision, with no false positives in seventy cases. The lenient grader's recall reaches 97 percent. A keyword anchored extraction step lifts the strict grader's recall by sixty points over a naive last number rule. A nudging mechanism raises grading run success from 36 to 97 percent. Re adding the original question to the nudge, they find, gives no extra benefit.

Catalog record

CREATORTian Zheng, Kai-Tai Hsu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.24839

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Grading the Grader: Lessons from Evaluating an Agentic Data Analysis System Thanks: Code and data available at GitHub: https://github.com/TZstats-Columbia/STAI-X-Grade-The-Grader . Tian Zheng, Kai-Tai Hsu Abstract Agentic data analysis systems produce rich outputs, including code, numerical results, and verbal diagnostics. This makes them more challenging to evaluate than single-turn LLM responses. It is therefore necessary to distinguish genuine disagreement between an agent’s output and a ground-truth answer from grading artifacts. We investigate how reliably automated graders assess such a system and what strategies improve grading quality by applying LAMBDA, a multi-agent data-analysis system, on 153 153 numerical QRData tasks from DSGym. We develop and evaluate a three-layer human-AI grading cascade: strict regex matching, LLM-based lenient grading, and snippet-based human inspection, which combines non-GenAI and GenAI strategies with different failure profiles. Both automated graders achieve 100 % 100\% observed precision (0/70 false positives) . The lenient grader’s recall is 97 % 97\% against human labels. A keyword-anchored extraction pipeline raises the strict grader’s recall by 60 60 percentage points over a last-number heuristic; the lenient grader is architecturally parser-independent. An iterative nudge mechanism raises grading run success from 36 % 36\% to 97 % 97\% and lenient-pass rates from 16 % 16\% to 46 % 46\% ; comparing nudging with and without original-question re-injection shows that re-injection offers no benefit, confirming the nudge as an answer template cue. We further observe in this case study that variable type is the task metadata field most consistently associated with grading pipeline dynamics and observed outcome grades. 1 Introduction Multi-agent LLM systems are increasingly deployed to automate data analysis workflows, and benchmarks such as DSGym ( 4 ) have made real progress toward systematic evaluation. One challenge is that, in addition to task completion, data analysis agents must also be evaluated on the correctness and rigor of their analytical results, while the graders used to make this determination are themselves error-prone. An agentic system like LAMBDA ( 6 ) generates rich, multi-step outputs, including code, execution logs, intermediate statistics, diagnostics, which are further interleaved with verbal comments. Extracting the intended answer from this output is non-trivial. In his discussion of LAMBDA, 1 calls for “rigorous, reproducible benchmarks” with scientific validity. More broadly, the veridical data science framework ( 9 ; 8 ) argues that any data-analytic pipeline must be evaluated along the axes of predictability, computability, and stability (PCS); applying these principles to agentic workflows ( 5 ) sharpens the question of what “correct output” means when the pipeline itself is stochastic and multi-step. In this paper, we study how reliably automated graders can assess agentic outputs and identify strategies to improve grading quality, focusing first on tasks with scalar ground truth. For example, DSGym’s default scorer, exact_match , treats "0.2" and "0.20" as distinct, producing false negatives and posing grading challenges ( 4 ) . This creates a structural difficulty for evaluating agentic AI on quantitative tasks: the richer the output, the harder it is to grade. We systematically address this auto-grading problem for agentic data-analysis outcomes by examining agent outputs and their grades, quantifying how often graders fail even when agents agree with the ground truth, and developing practical strategies to improve grading quality and reliability. The contributions of this paper are as follows. 1. Evaluation framework with nudging. A two-loop wrapper with per-turn instrumentation and iterative nudges adapts LAMBDA’s conversational design to single-shot benchmarks, raising grading success from 36\% to 97\% and lenient-pass rates from 16\% to 46\% . 2. Three-layer grading with human calibration. A cascade of strict regex, LLM-based lenient grading, and snippet-based human checks combines non-GenAI and GenAI graders. Both automated graders reach 100\% observed precision; the lenient grader’s recall is 97\% ( 70/72 ). Human audits of 100-character snippets confirm 89% of the auto-grading results and help spot cases that require further full review. 3. Parsing-grading interaction. A keyword-anchored extraction pipeline improves strict-grader recall by 60 pp over a last-number heuristic parser. The LLM-based lenient grader, run on full agent output, is parser-independent. 4. Task covariates as interpretive signals. To help interpret the observed grading dynamics, we analyze how they associate with simple task metadata (e.g., variable type) in this case study. Variable type (categorical, continuous, or mixed) is the only task covariate that consistently aligns with grading dynamics and outcomes. In this case study, categorical tasks are harder to extract but exhibit a higher rate of agreement with the ground truth; continuous tasks are easier to grade but tend to produce more discrepancies between the agent’s outputs and the single ground truth. 5. Negative result: prompt re-injection. Comparing nudge modes (answer-format only vs. plus original question) shows no benefit from re-injection on these short tasks: the nudge mainly cues answer format; the agent retains the answer internally. 2 Background and Related Work LAMBDA. LAMBDA ( 6 ) is a multi-agent data-science assistant built around two core agents: the programmer , which generates and executes code in response to user instructions, and the inspector , which checks execution results for errors and issues revision suggestions when errors occur. This programmer-inspector loop iterates until the code runs without error or a maximum number of attempts is reached. Communication between agents is exclusively textual. Sun et al. benchmarked LAMBDA on classical tabular datasets for classification (accuracy) and regression (mean squared error), using five-fold cross-validation with results compared against manual R implementations; they also evaluated the system on high-dimensional, missing-data, image, and text datasets. We chose LAMBDA as our case study because it is open-source, light-weight, versatile, requires no fine-tuning, and has a key-value knowledge base that injects domain-specific code via in-context learning ( 6 ) . It has demonstrated strong benchmark performance, making it a suitable testbed for studying grading methodology on a capable system. DSGym and QRData. DSGym ( 4 ) provides a unified evaluation harness over multiple data-science benchmarks. We use the QRData split, which contains 251 problems; of these, 153 have numerical (scalar) ground-truth answers. We carry out our investigation using these 153 numerical tasks, each comprising a natural-language question, a tabular data file, and a single ground-truth scalar answer. 3 Methodology Figure 1 gives an overview of our evaluation pipeline. All experiments were conducted in a GitHub Codespace using 8 cores for task evaluation and 2 cores for result processing (no GPU); prior computational experiments utilized a MacBook Pro with M1 Max (no GPU). LAMBDA uses gpt-5-mini and gpt-4.1-mini via the OpenAI API; the lenient grader employed GPT-4o-mini (preliminary experiments with LLaMA-3 via Ollama produced similar results). For transparency and reproducibility, we provide the wrapper-authored prompt templates used in Figure 1 (nudges and lenient grader) in Supplement S.3 . The total pipeline time was {\sim}120 minutes per full run. LAMBDA Answer extraction & grading QRData task question + data file Programmer generates & runs code Inspector checks for errors execute revise suggestion Nudge option: re-inject question Answer detected? no \leq\!2 nudge/prompt modes task anchoring + code actions Instrumentation Answer extraction keyword-anchored parsing yes verbal–exec scalars Strict grader DSGym exact_match Strict pass? Strict pass yes Lenient grader LLM, 3% tol. no strict outcome lenient outcome Outcome taxonomy rescued Human inspection human label Figure 1: Evaluation pipeline. Blue: LAMBDA’s programmer–inspector loop. Purple (dashed): per-turn instrumentation probes. Orange: nudge

Read the complete text