Theory Under Construction: Orchestrating Language Models for Research Software Where the Specification Evolves
Halley Young, Nikolaj Bjørner · 2026
Read Theory Under Construction: Orchestrating Language Models for Research Software Where the Specification Evolves by Halley Young, Nikolaj Bjørner (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-27209.txt.
Introduction
Young and Björner examine research software where the math, the code, the benchmarks, and the public claims must mature together. Often they drift apart. Language models can now write large codebases and draft papers. Yet two failures recur. First, hallucination accumulation, where claims outrun what code or theory supports. Second, desynchronization, where code, theory, and the model's world model fall out of step. Their Comet-H system couples ideation, building, testing, grounding, and writing in one workspace.
Catalog record
| CREATOR | Halley Young, Nikolaj Bjørner |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 85005 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.27209
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Theory Under Construction: Orchestrating Language Models for Research Software Where the Specification Evolves Halley Young, Nikolaj Bjørner Abstract Large language models can now generate substantial code and draft research text. We are here aiming at a combination of code, empirical evaluation, and research text. In this pursuit, the hardest research-software projects fail while co-evolving the connected artifacts: the mathematical thesis, the executable system, the benchmark surface, and the public claims must mature together, and they routinely drift apart. We identify two LM-specific failure modes— hallucination accumulation , where claims exceed what code or theory supports and unsupported assertions propagate across sessions, and desynchronization , where code, theory, or the LM’s own world model fall out of alignment with one another. We propose Comet-H : an iterative prompt automaton that orchestrates ideation, implementation, evaluation, grounding, and paper-writing as coupled coordinates of a single workspace state. At each step the controller picks the next prompt by scoring it against what the workspace currently lacks, carries unfinished follow-up work forward with a half-life so old items fade unless they are addressed, and re-checks the paper and README against the underlying code and benchmarks whenever the documentation changes. We view this prompt-selection step as a small contextual bandit problem over prompt families—prompts are arms, workspace deficits are the context, and the score is a plain linear function whose weights are fixed by hand—and argue that pairing such a transparent scorer with a fading record of unfinished work is itself a useful design point: it keeps long-horizon follow-ups bounded, requires no learned policy, and makes every prompt choice easy to read off the workspace. We created a full portfolio spanning 46 research-software repositories across two dozen domains. We study a3 in depth: a Python static-analysis tool built entirely within the loop that reaches F 1 = 0.768 F_{1}=0.768 on a 90-case benchmark against a next-best baseline of 0.364 0.364 , with monotone ablation gains from each added theory/practice coupling layer. We report observations on LM behavior across ∼ 400 {\sim}400 commits of orchestrated development, including the finding that audit-and-contraction passes dominate the later phases of every successful trajectory. 1 Introduction Language models are now capable code producers. Systems like SWE-agent [ 35 ] , Devin [ 8 ] , and OpenHands [ 34 ] can resolve GitHub issues, pass SWE-bench tasks, and generate multi-file repositories from specifications. FunSearch [ 30 ] and AlphaEvolve [ 23 ] show that LM-guided search can discover novel algorithms. The AI Scientist [ 22 ] automates the full cycle from ideation through paper writing. Mainstream coding agents Claude [ 2 ] , Codex [ 25 ] , Co-Pilot [ 13 ] , Gemini [ 15 ] , Kiro [ 1 ] are rapidly democratizing the capabilities of the earlier systems. But they cannot be trusted naively—hallucinations, plausible garbage, and point solutions that address symptoms instead of root causes are the norm. Much effort has been spent trying to “ask the right questions” to LMs to avoid bad outcomes. Our approach is different: instead of crafting better prompts, we design a process architecture that (i) determines when to ask what , (ii) determines when to trust or test the answers, (iii) allows goals, claims, theory, code, and evidence to evolve together, and (iv) scales useful compute trajectories to days, not minutes. Figure 1: High-level Comet-H loop: the system cycles through ideation, generation, hardening, and audit until a batch of repositories reaches quality thresholds. The controller that implements this loop is described in Section 4 . The hardest research-software projects do not fail at generation. They fail based on drift: when the mathematical thesis, the executable system, the benchmark surface, and the public claims drift apart. When these coupling edges are ignored, two characteristic failure modes emerge: hallucination accumulation , where claims exceed what the code or theory actually supports and unsupported assertions propagate; and desynchronization , where code or theory advances past the claims, or where even pairwise consistency between theory, code, and claims is insufficient to guarantee global coherence. This co-evolution problem is invisible to existing evaluation frameworks because those frameworks assume a fixed external task. Our Comet-H process creates research artifacts—code and papers—using a state machine transitioning along states illustrated in Figure 1 . It starts with an initial seed, exploring a research direction; then through reactive iteration it produces research artifacts with the Comet-H process ensuring that theory co-evolves with empirical feedback. Research software development occupies a regime (Table 1 ) that requires both mathematical ambition and repository scale, but where the specification itself is under construction. Prior systems either assume a fixed evaluation frame (FunSearch [ 30 ] , AlphaEvolve [ 23 ] ), operate on shorter horizons with fixed tasks (SWE-agent [ 35 ] , Devin [ 8 ] ), or automate ideation without a durable repository loop (AI Scientist [ 22 ] ). Recent long-horizon systems such as Ralph-style persistent engineering loops [ 17 ] and AutoResearchClaw-style staged research pipelines [ 21 ] move closer to this regime, but still center engineering closure or paper-production pipelines rather than explicit control of cross-artifact co-evolution. None addresses the full co-evolution problem. This paper studies Comet-H , an iterative prompt automaton for this regime (detailed in § 3 –§ 4 ; see Figure 4 ). The standard pipeline generates batches of ten repositories per run; across multiple runs, it has produced 46 research-software repositories spanning 12+ domains. Each repository contains source code, a tool paper, a machine-readable grounding ledger, benchmark harnesses, and a README. 1 1 1 The experiments leading up to the Comet-H process, and the subsequent creation process for the portfolio involved running hundreds of sub-agents over several days. It currently requires a very generous budget for token consumption. The key design principles are: • Theory is mutable. The underlying mathematical thesis of the research software is a state variable subject to revision under empirical pressure. This makes sense because the right formalization is rarely known at the outset: implementation reveals gaps, benchmarks expose false assumptions, and the correct abstraction only becomes clear once evidence accumulates. • Artifacts and prompts are coupled. Every prompt stage is keyed to the artifact surfaces it touches, ensuring no surface drifts unattended. Because prompts are generated from and evaluated against live artifacts, the system cannot silently diverge from what has actually been built. • Artifact changes are audited. Public-facing artifact changes force an immediate grounding-and-audit pass, bounding hallucination propagation to one step. Without this check, a single erroneous claim in a paper or README can cascade into downstream code changes and benchmark modifications before any inconsistency is detected. • Drift is controlled by adjacency. The research question may evolve, but only through adjacent moves that preserve the current evaluation surface. Unconstrained drift would allow the system to silently redefine the problem, rendering prior benchmarks irrelevant and making longitudinal comparison impossible. Table 1: Comparison of Comet-H against adjacent LM-based construction settings. Comet-H is the only system where the metric, specification, and evaluation surface are all mutable. System Scale Duration Metric fixed? Multi-session? Governance FunSearch function mins yes no none AlphaEvolve patch hours yes no none AI Scientist paper+code 4–8 h yes no none SWE-agent patch mins yes no none ChatDev prototype