AutoStan: Autonomous Bayesian Model Improvement via Predictive Feedback
Oliver Dürr · 2026
Read AutoStan: Autonomous Bayesian Model Improvement via Predictive Feedback by Oliver Dürr (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-27766.txt.
Introduction
Dürr lets a coding agent improve Bayesian models on its own. AutoStan loops through write, sample, judge, and revert steps. Stan model files are edited by a command-line agent. Held-out predictive scores guide each keep-or-drop call. Sampler warnings add a second feedback channel. On outlier data, the agent finds robust t-based forms. It matches a top black-box method while staying readable. Four more tasks yield hierarchical and sports models.
Catalog record
| CREATOR | Oliver Dürr |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41183 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.27766
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
AutoStan: Autonomous Bayesian Model Improvement via Predictive Feedback Oliver Dürr Abstract We present AutoStan, a framework in which a command-line interface (CLI) coding agent autonomously builds and iteratively improves Bayesian models written in Stan. The agent operates in a loop, writing a Stan model file, executing MCMC sampling, then deciding whether to keep or revert each change based on two complementary feedback signals: the negative log predictive density (NLPD) on held-out data and the sampler’s own diagnostics (divergences, R-hat, effective sample size). We evaluate AutoStan on five datasets with diverse modeling structures. On a synthetic regression dataset with outliers, the agent progresses from naive linear regression to a model with Student- t t robustness, nonlinear heteroscedastic structure, and an explicit contamination mixture, matching or outperforming TabPFN, a state-of-the-art black-box method, while remaining fully interpretable. Across four additional experiments, the same mechanism discovers hierarchical partial pooling, varying-slope models with correlated random effects, and a Poisson attack/defense model for soccer. No search algorithm, critic module, or domain-specific instructions are needed. This is, to our knowledge, the first demonstration that a CLI coding agent can autonomously write and iteratively improve Stan code for diverse Bayesian modeling problems. 1 Introduction Bayesian modeling is, in principle, beautifully simple: you state what you believe about the data, condition on observations, and let probability theory do the rest. In practice, this simplicity was historically locked behind intractable integrals—until probabilistic programming languages like BUGS ( 15 ) , JAGS ( 19 ) , and Stan ( 21 ) made arbitrary generative models accessible via MCMC. Yet even with Stan, the practitioner must still closely monitor and ”babysit” the inference: diagnose divergences, check R-hat and effective sample sizes, reparameterize funnels, tune priors, and iterate, a workflow that textbooks like 7 and 16 have made accessible but not less laborious. The question we tackle in this paper is whether modern CLI coding agents can automate parts of this tedious cycle. Karpathy’s autoresearch ( 12 ) demonstrated that a coding agent with a single scalar reward (validation bits-per-byte) can autonomously improve a neural network training script overnight. We translate this idea to Bayesian modeling. Supervised Bayesian inference is a natural fit for autonomous optimization because it admits a principled, model-agnostic evaluation metric like the negative log predictive density (NLPD) or any other strictly proper scoring rule ( 8 ) . NLPD rewards calibrated predictive distributions and cannot be gamed by over- or under-confident predictions, making it an ideal hands-off reward signal. The agent edits a Stan model file, runs MCMC, observes NLPD on held-out data, and decides whether to keep or revert. No additional orchestration beyond the CLI agent is needed. We demonstrate AutoStan on five datasets spanning four statistical paradigms: regression with outliers, hierarchical partial pooling, varying-slope models, and sports modeling. Each reveals qualitatively different agent behaviors: from immediately writing the correct model (hierarchical) to discovering Student- t robustness for outlier handling and ultimately an explicit contamination mixture (regression). To our knowledge, this is the first demonstration that a CLI coding agent can autonomously write and improve Stan code. 2 Method 2.1 CLI Coding Agents A CLI coding agent is a terminal-based AI assistant that can read and edit files, execute shell commands, and reason about their output in a read–edit–execute loop. In our case, the agent edits Stan model files and executes MCMC via cmdstanpy ( 4 ) . Examples of CLI coding agents include Claude Code ( 1 ) , Gemini CLI ( 9 ) , Codex CLI ( 17 ) , and opencode ( 18 ) , the last of which is open-source. AutoStan relies on this capability but is not tied to any specific agent: any CLI coding agent that can edit files and execute shell commands can implement the optimization loop. All experiments reported here use Claude Code (v2.1.86) with Sonnet 4.6 (Model ID claude-sonnet-4-6) and a context window of 200,000 tokens ( 1 ) . The framework, however, is agent-agnostic and should work with any modern CLI coding agent. 2.2 Workflow The agent is launched with a single prompt: Read program.md for your instructions. Your dataset is datasets/<name>. Two files define the task: program.md contains domain-agnostic Bayesian workflow instructions ( {\sim} 56 lines; full text in Appendix E ): read the data, inspect the training set, edit model.stan , run evaluation, interpret NLPD and diagnostics, decide whether to keep or revert, repeat. The instructions mention general strategies (non-centered parameterization, prior tuning, alternative likelihoods) but contain no dataset-specific guidance. dataset.md is a short dataset description: column names, data format, the Stan data interface, and the dataset-specific evaluation command ( python evaluate.py ), which the agent calls to compile the model, run MCMC, and receive the resulting NLPD. For synthetic datasets, variable names are anonymized; the true generative process is never disclosed. Examples are given in Appendix D and at https://github.com/tidit-ch/autostan . 2.3 Agent Autonomy The agent is free to explore the data before writing code. In practice, it often computes summary statistics, checks for outliers, examines group structure, or inspects value ranges on its own initiative, behavior that emerges from the agent’s statistical reasoning rather than from any script. The agent then writes a model.stan file with complete freedom over priors, likelihood, parameterization, and model structure. The only contract: the model must output a log_lik vector in the generated quantities block which is needed for the evaluation script to compute the NLPD. 2.4 Evaluation and Reward Each dataset has an evaluate.py script (executable but not readable by the agent) that compiles the model via cmdstanpy, runs MCMC (4 chains, 1 000 post-warmup draws for most datasets; 30 000 for the large 1D regression, to reduce Monte Carlo noise), computes NLPD from the log_lik vector (provided in the stan file). Test data is protected via filesystem permissions and never visible to the agent. The primary reward is NLPD on held-out data: \mathrm{NLPD}=-\frac{1}{N_{\mathrm{test}}}\sum_{n=1}^{N_{\mathrm{test}}}\log\!\left(\frac{1}{S}\sum_{s=1}^{S}p\!\left(y_{n}^{\mathrm{test}}\mid\theta^{(s)}\right)\right)\!. (1) As a strictly proper scoring rule ( 8 ) , the NLPD is uniquely minimized by the true predictive distribution. In contrast to metrics such as accuracy or RMSE, it penalizes both errors in the predictive mean and miscalibrated uncertainty. In principle, any other strictly proper scoring rule (e.g., CRPS) could be used. The agent halts after 3 consecutive non-improving iterations or 20 total. 3 Case Study: Regression with Outliers We examine this case study in detail, as it clearly illustrates the core principles of our approach. Additional examples are provided in Section 4 . 3.1 Setup The agent receives a dataset.md file (see Appendix D ) containing the column names ( predictor , response ), the Stan data interface, and the evaluation command, plus the one-line dataset description: “Observations of a continuous predictor and a continuous response.” The true generative process, which is unknown to the agent, involves three independent sources of difficulty: \displaystyle f(x) \displaystyle=2\sin(1.2x)+0.3x, (2) \displaystyle\sigma(x) \displaystyle=0.3+0.8\exp\!\Bigl[-\tfrac{1}{2}\Bigl(\tfrac{x-3}{1.5}\Bigr)^{2}\Bigr], (3) and {\sim}6\% of training observations shifted \pm 10 – 15 units as extreme outliers. The test set is drawn from the clean generative process (no outliers), so the oracle NLPD is simply the negative log-density under the true f(x) and \sigma(x) ; no mixture component is needed. This oracle represents a lower bound that cannot be reached: beyond the usual parameter uncertainty