RoboPhD: Self-Improving Text-to-SQL Through Autonomous Agent Evolution

Andrew Borthwick & Stephen Ash · 2026

Read RoboPhD: Self-Improving Text-to-SQL Through Autonomous Agent Evolution by Andrew Borthwick & Stephen Ash (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-01126.txt.

Introduction

Andrew Borthwick and Stephen Ash present RoboPhD, a system in which AI agents improve themselves at turning plain questions into database queries. Two agents work in a loop. One writes the query code and its instructions. The other studies the results and designs the next version, with a rating system picking the winners in a survival-of-the-fittest cycle.

Starting from a naive seventy-line seed, the system grew its best agent to fifteen hundred lines over eighteen rounds, with no outside hints about the task. It found tricks on its own, such as adjusting how deeply it studies a database based on that database's size. Gains were largest on cheaper models, letting a small tuned model beat a bigger untuned one. The best agent scored about seventy-four percent on the BIRD test set. The work suggests AI can build strong agent systems from a trivial starting point.

Catalog record

CREATORAndrew Borthwick & Stephen Ash
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-01126

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

RoboPhD: Self-Improving Text-to-SQL Through Autonomous Agent Evolution Andrew Borthwick & Stephen Ash Abstract We present RoboPhD, a system where AI agents autonomously conduct research to improve Text-to-SQL performance. RoboPhD implements a closed-loop evolution cycle with two coordinated components: a SQL Generation agent composed of a database analysis script and SQL generation instructions, and an Evolution agent that designs new versions based on performance feedback. Central to the framework is an ELO-based selection mechanism enabling survival-of-the-fittest dynamics while handling non-transitivity in performance. Starting from a naive 70-line baseline, RoboPhD evolves agents through iterative cross-pollination, discovering effective techniques without any external guidance on the Text-to-SQL domain. Our best agent, evolved to 1500 lines over 18 iterations, autonomously discovered strategies such as size-adaptive database analysis that adjusts depth based on schema complexity and SQL generation patterns for column selection, evidence interpretation, and aggregation. Evolution provides the largest gains on cheaper models: while we improve by 2.3 points over a strong Claude Opus 4.5 naive baseline, we show an improvement of 8.9 points over the weaker Claude Haiku model. This enables ”skip a tier” deployment: evolved Haiku exceeds naive Sonnet accuracy, and evolved Sonnet exceeds naive Opus—both at lower cost. The full system achieves 73.67% accuracy on the BIRD test set, demonstrating that AI can autonomously build a strong agentic system with only a trivial human-provided starting point. 1 Introduction Text-to-SQL, the task of translating natural language questions into executable SQL queries, remains a fundamental challenge in natural language processing with significant real-world applications. Recent advances have shown that large language models (LLMs) can achieve impressive performance on benchmarks like Spider ( 25 ) and BIRD ( 10 ) , but these approaches typically require extensive manual prompt engineering, careful model selection, and domain expertise. The ability of AI systems to conduct AI research autonomously has long been identified as a crucial capability milestone, with the potential to create a discontinuity in technological development ( 5 ; 1 ) . Recent work has begun to explore this frontier, with systems demonstrating increasing autonomy in scientific discovery ( 19 ) , machine learning research ( 14 ) , and algorithm development ( 4 ) . While well short of a full self-improving system, our work demonstrates a concrete instantiation of AI agents autonomously conducting the research cycle of hypothesis generation, experimentation, and iterative refinement to improve performance on text-to-SQL generation. RoboPhD evolves AI agents composed of two artifacts—a database analysis tool and SQL generation instructions—separating offline analysis from online inference, a pattern that generalizes to other tasks with similar structure. This work makes the following key contributions: • Autonomous AI Research Framework : We demonstrate AI agents conducting systematic research on text-to-SQL through a closed-loop evolution cycle, autonomously discovering effective optimization strategies without human intervention. We open-source our code at https://github.com/andborth/RoboPhD . • ELO-Based Evolutionary Selection : First application of ELO ratings for evolutionary prompt/agent optimization, effectively handling asynchronous agent entry and non-transitivity in relative agent performance across different databases. • Evolution of a complete Text-to-SQL system : RoboPhD simultaneously evolves a set of Python database analysis tools and SQL-generation instructions to improve end-to-end performance. The evolved agent yields a fixed prompt for a given database, allowing easy deployment in an industrial setting. • Inverse Scaling of Evolutionary Benefit : We show that evolution provides larger gains on cheaper models (+8.9 points for Haiku vs +2.3 for Opus), enabling cost-effective deployment where evolved cheaper models exceed naive expensive models. • Self-Directed Improvement without Author-Supplied Domain Expertise : The system improves without author-provided Text-to-SQL knowledge from a trivial baseline; gains arise from latent LLM capabilities and iterative error-driven learning across iterations. 2 Related Work 2.1 Text-to-SQL Systems The BIRD benchmark ( 10 ) has driven significant advances in Text-to-SQL, with top systems achieving (as of mid-September 2025) 71-77% accuracy through diverse approaches. AskData ( 21 ) achieves 77.14% using GPT-4o with data analysis and automatic metadata extraction. CHASE-SQL ( 17 ) reaches 76.02% through a multi-agent framework employing divide-and-conquer query decomposition, chain-of-thought reasoning based on query execution plans, and instance-aware synthetic few-shot demonstration generation, with a fine-tuned binary selection LLM for candidate ranking. XiYan-SQL ( 12 ) achieves 75.63% using a multi-generator ensemble strategy that combines in-context learning with supervised fine-tuning, a novel M-Schema representation for database structures, and a fine-tuned selection model to choose among diverse candidates. CSC-SQL ( 20 ) reaches 73.67% by integrating self-consistency and self-correction, using GRPO-based reinforcement learning to train both SQL generation and merge-revision models that reconcile conflicting candidates. Reasoning-SQL ( 18 ) achieves 72.78% through reinforcement learning with SQL-tailored partial rewards encompassing execution accuracy, syntax validity, and n-gram matching. OpenSearch-SQL ( 23 ) achieves 72.28% using a multi-agent framework with consistency alignment and a novel SQL-Like intermediate language which together reduce instruction-following failures and hallucinations. OmniSQL ( 9 ) reaches similar accuracy, 72.06%, by leveraging SynSQL-2.5M, the first million-scale synthetic text-to-SQL dataset containing 2.5 million samples with chain-of-thought solutions across 16,000 databases. GenaSQL ( 2 ) achieves 72.28% through ‘N-rep consistency’, a cost-efficient approach that generates diverse SQL candidates using multiple schema representations without chain-of-thought reasoning or fine-tuning, reducing per-query costs to $0.039. Distillery ( 15 ) reaches 71.83% by fitting the full schema in the input context window, using augmentation, selection, and correction techniques. CHESS ( 22 ) achieves 71.1% using a four-agent architecture comprising an Information Retriever, Schema Selector, Candidate Generator, and Unit Tester. Our work uniquely focuses on autonomous discovery of these strategies rather than manual design. While prior systems require human experts to engineer prompts and architectures, RoboPhD discovers optimization techniques autonomously. Note that we have experimented with using all of the above papers (with the exception of the BIRD benchmark paper ( 10 ) ) as inputs to the evolution agent in the research-driven evolution strategy mentioned in Section 5.2 . Prior work on prompt optimization includes APE ( 29 ) , OPRO ( 24 ) , DSPy ( 7 ) , TextGrad ( 26 ) , and ACE ( 27 ) . While these systems optimize prompts, our approach evolves complete agent artifacts including database analysis tools and SQL generation instructions. Crucially, our system conducts research autonomously without human intervention in the optimization loop. One key aspect of our approach is allowing the Evolutionary agent access to the errors made from previous candidate agents. This is similar to the idea of text gradients explored in the TextGrad paper ( 26 ) , where the LLM produces a (metaphorical) gradient to describe how to address a particular error. In our system, the error gradients are available for the agent to choose how to incorporate them into the next candidate agent. Evolutionary algorithms have been applied to neural architecture search ( 16 ) and hyperparameter optimization ( 13 ; 6 ) . We extend this to evolving complete AI agents with natural language specifications and tool usage, creating a fully autonomous research system. The ELO rating system ( 3 ) , originally developed for chess, has found recent applications in AI evaluation. The Chatbot Arena ( 28 ) uses ELO ratings to create a dynamic leaderboard of LLMs based on human preferences, demonstrating ELO’s effectiveness for model comparison. While these systems use ELO for passive

Read the complete text