Vesta : Visual Exploration with Statistical Tool Agents
William Rudman, Abhishek Divekar, Kanishk Jain, Sebastian Joseph, Stella S. R. Offner, Matthew Lease, Kyle Mahowald, Greg Durrett, Junyi Jessy Li, [0.5em] The University of Texas at Austin · 2026
Read Vesta : Visual Exploration with Statistical Tool Agents by William Rudman, Abhishek Divekar, Kanishk Jain, Sebastian Joseph, Stella S. R. Offner, Matthew Lease, Kyle Mahowald, Greg Durrett, Junyi Jessy Li, [0.5em] The University of Texas at Austin (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-00384.txt.
Introduction
Rudman and co-authors give sight-word models a tool kit that grows as they probe data. Fit of a math form to data is a core step in lab work. It is also one of the least auto-bound steps. Past agent tools draft and hone forms by talk loops, yet stall on hard tasks.
Their frame, VESTA, adds stats tools to the loop in a live way. Tools reshape data, draw tests of a claim, and run firm checks. Tools made on the fly pile up for use in rounds to come. A new bench, DAWN, spans form fits and time-line tasks in tiers. It ends with real sky tasks, like star mass forms and wave chirps. Live-made tools beat past pipes most on hard tasks.
Catalog record
| CREATOR | William Rudman, Abhishek Divekar, Kanishk Jain, Sebastian Joseph, Stella S. R. Offner, Matthew Lease, Kyle Mahowald, Greg Durrett, Junyi Jessy Li, [0.5em] The University of Texas at Austin |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 102 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.00384
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Vesta : Visual Exploration with Statistical Tool Agents William Rudman, Abhishek Divekar, Kanishk Jain, Sebastian Joseph, Stella S. R. Offner, Matthew Lease, Kyle Mahowald, Greg Durrett, Junyi Jessy Li, [0.5em] The University of Texas at Austin Abstract Fitting quantitative models to data is a central step in scientific workflows, yet it remains one of the least automated. Recent agent-based systems leverage language and vision-language models (VLMs) to iteratively propose and refine statistical models, but these systems struggle on more challenging modeling tasks. To address these limitations, we introduce Vesta ( V isual E xploration with S tatistical T ool A gents), a framework that equips VLMs with a dynamically growing exploration toolkit to guide model refinement through data transformations, hypothesis-driven visualizations, and robust statistical tests. Unlike prior systems that rely on iterative critique alone, Vesta actively explores data before and during refinement by selecting or creating diagnostic tools, which accumulate in the model’s context and can be reused later. We evaluate Vesta against established baselines in three toolkit configurations: no tools, static expert-written tools, and dynamic model-written tools. To support this evaluation, we introduce Dawn ( D ataset for A utomated W orkflows and N umerical Modeling), a benchmark targeting distribution fitting and time series modeling with varying difficulty tiers, and culminating in real-world astronomy tasks including modeling initial mass functions and gravitational-wave chirp signals. We find that Vesta ’s dynamic tool creation outperforms prior agentic pipelines, with the largest gains on complex and domain-specific tasks. We further show that dynamically generated tools are substantially more sophisticated than those produced by existing visual tool-creation systems, covering more diagnostic categories per function and strongly preferring visual outputs that the VLM critic can reason over directly. 1 Introduction Large Language Model (LLM) and Vision-Language Model (VLM) agents have demonstrated remarkable promise in scientific discovery, with systems capable of forming hypotheses, designing experiments, and synthesizing results. However, failures at different steps impede these models’ abilities to truly accelerate science. This work focuses on the task of fitting quantitative models to data . This step is complex, requiring priors about the domain in question and careful statistical analysis, while being less studied than other quantitative analysis tasks like ML engineering and data science [ 13 , 10 ] . Addressing this gap is particularly important in scientific disciplines like astronomy where models struggle to effectively use domain conventions [ 23 ] . We introduce Vesta 1 1 1 VESTA is the second most massive body in the main asteroid belt [ 33 ] . ( V isual E xploration with S tatistical T ool A gents), a framework that proposes, refines and evaluates statistical models by analyzing visual representations of data. Vesta operates over models defined in PyMC [ 1 ] , a Bayesian probabilistic programming framework that enables model definition and fitting to data using an efficient Markov Chain Monte Carlo (MCMC) method. Model-building with PyMC code is compositional, allowing for complex distributions to be constructed by combining simpler components. Vesta instantiates a loop of proposing models, selecting the best one, then iterating, broadly similar to existing autoresearch workflows [ 40 , 8 ] . However, unlike prior work, Vesta utilizes multimodal feedback by augmenting this basic loop with a new type of visual tools . Visual tools allow Vesta to inspect model fits using a VLM, giving it leverage in a modality distinct from past work on tool use. Figure 1 shows how a model can use a tool to compute residuals, produce a visualization, and then inspect that visualization to identify areas where the predicted model does not fit the data well and propose a new statistical model that better fits the data. For the diversity of statistical model fitting tasks we explore here, it is infeasible to enumerate all necessary tools up front; dynamic tool instantiation is required to cover the full breadth of cases. Figure 1: Overview of Vesta . By effectively using and creating tools, Vesta produces a probabilistic PyMC program that models the input data. To support advancement on challenging model fitting problems where current approaches fail, we introduce a new benchmark we call Dawn ( D ataset for A utomated W orkflows and N umerical Modeling). 2 2 2 DAWN was NASA’s first truly interplanetary spaceship that explored VESTA [ 32 ] . Dawn problems cover two domains central to both general data science and scientific research: fitting probability distributions to discrete data and constructing models for time-series analysis. In contrast to existing benchmarks which often contain a small number of distributions with a limited number of points [ 15 , 29 ] , Dawn consists of synthetic problems spanning two levels of difficulty ( Easy and Hard ), as well as two fundamental problems from astronomy ( Astro ): fitting stellar initial mass functions (IMFs), and detecting and characterizing chirp signals embedded in gravitational-wave data. Using Dawn , we benchmark Vesta against prior frameworks specialized for Bayesian model and dynamic VLM-based tool-generation systems. While existing approaches perform competitively on our Easy splits, performance degrades substantially as data complexity increases and tasks shift toward domain-specific scientific phenomena. This degradation reveals a systematic gap in current agents’ capacity for iterative, evidence-driven model refinement. We show that this is effectively addressed by Vesta ’s visual tools, with which the agent is capable of exploring multiple hypotheses, visualizing them, and generating new hypotheses based on its tool’s visual analyses. Furthermore, we find that Vesta not only successfully recovers the expert tools at a higher rate than existing baselines, but also creates sophisticated tools that often composes several expert tools in order to test multiple hypotheses at once. While using dynamically generated visual tools does not outperform expert-written tools, likely due to limitations in agents’ ability to process complex diagrams, Vesta shows great promise for creating and utilizing statistical tools. 2 Related Work Automated Model Fitting Past work evaluated the capabilities of LLMs and VLMs to automate statistical analysis [ 42 , 22 ] , time-series modeling [ 52 , 47 , 16 , 56 , 34 ] and Bayesian model proposals [ 37 , 28 ] . 22 and 42 survey how LLM-based agents can automate statistical workflows from data cleaning and model selection. However, 59 find that even state-of-the-art LLMs achieve only modest accuracy on benchmarks for selecting appropriate statistical methods, revealing significant limitations in LLMs’ ability to reason over complex statistical tasks. While much of the focus in the literature has used LLMs / VLMs to predict a continuation of time series or detect anomalies, our work seeks to represent underlying model that generates the observed data using a probabilistic Bayesian model (PyMC). Previous work in generating PyMC code to model data has been successful in scientific settings, but this work often relies on human-written kernels for Gaussian process, limiting the use cases of such methods [ 5 , 6 , 12 ] . Recent works addresses these key limitation by using LLMs to automatically perform “Box’s Loop” of iteratively proposing new and critiquing existing models written in PyMC code until an adequate fit is achieved [ 27 ] . Despite some promising results using LLMs to propose probabilistic models, 15 find LLMs struggle on more challenging