Glite ARF: Verifier-Driven Research with Parallel LLM Coding Agents
Vassili Philippov, Pavel Katunin, Dmitry Andreev, Igor Ostanin, Anton Nikolaev · 2026
Read Glite ARF: Verifier-Driven Research with Parallel LLM Coding Agents by Vassili Philippov, Pavel Katunin, Dmitry Andreev, Igor Ostanin, Anton Nikolaev (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-27416.txt.
Introduction
Philippov leads the team behind Glite ARF, a framework for letting language model coding agents run empirical research without wrecking it. Delegating experiments to agents is tempting, but small instruction lapses compound into fabricated citations, contaminated data splits, and irreproducible artefacts. Glite ARF splits the work three ways: a human picks hypotheses, agents implement isolated tasks, and deterministic verifier scripts enforce the rules. Task isolation, immutability of finished work, and a corrections overlay live in code that fails noisily when violated in this research software paper from 2026.
The proving ground was a shared task on vocabulary difficulty, where the team took first place in the closed track and the runner-up spot in the open track across three languages. Provenance records caught four target-leaking feature sets before they poisoned the results. Structural overhead added only about one percent of wall-clock time across three campaigns.
Catalog record
| CREATOR | Vassili Philippov, Pavel Katunin, Dmitry Andreev, Igor Ostanin, Anton Nikolaev |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43 KB |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.27416
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Glite ARF: Verifier-Driven Research with Parallel LLM Coding Agents Vassili Philippov, Pavel Katunin, Dmitry Andreev, Igor Ostanin, Anton Nikolaev Abstract LLM coding agents make it tempting to automate empirical research by delegating experiments to them directly, but naive delegation does not scale to large projects: low-rate instruction lapses compound into broken, irreproducible artefacts. To address this problem, we present Glite ARF , an open-source Python framework for running many LLM coding agents in parallel on a research repository without sacrificing reproducibility or auditability. The framework defines a three-role stack : a human researcher chooses which hypotheses to test, coding agents (Claude Code, Codex CLI) implement individual tasks under a fixed structure, and deterministic Python verifier scripts enforce task isolation, immutability of completed work, a corrections overlay, and a materialised project overview. We call this verifier-driven research : the rules of the research process live in code that fails loudly when violated, not in prose that agents are merely asked to follow. Using Glite ARF, we developed our submission to the BEA 2026 vocabulary-difficulty shared task, placing first in the closed track and second in the open track on all three target languages (Spanish, German, Mandarin) and reducing the official baseline RMSE by 29.9% (closed) and 35.9% (open). The campaign comprised 273 tracked tasks (146 experiment runs) across 129 feature sets, run by up to twelve parallel agents orchestrated from a single laptop — with some model training on rented A100s — at ∼ $ 450 {\sim}\$450 in LLM API spend ( ∼ $ 498 {\sim}\$498 total third-party cost), and structured per-fold provenance let us catch and strip four target-leaking feature sets, correcting an implausible 0.609 RMSE to 0.802. Across three campaigns in three domains, the framework’s structural machinery adds only ∼ 1 % {\sim}1\% of wall-clock time. Framework and a public demo project accompany this paper. 1 Introduction LLM coding agents make it natural to imagine research as many delegated experiments running in parallel. Naive automation, however, does not scale to large projects: agents follow most instructions, but the few they skip compound into fabricated citations, contaminated splits, stale summaries, out-of-scope edits, and irreproducible provenance. Glite ARF addresses this failure mode by making the research process itself executable. A human researcher proposes hypotheses, agents implement isolated tasks, and deterministic Python verifiers enforce task structure, immutability, corrections, and a materialised project overview (Figure 1 ). We used Glite ARF to develop our submission to the BEA 2026 vocabulary-difficulty shared task ( Felice and Skidmore 2026 ) , detailed in our case study (§ 4 ). The resulting system placed first in the closed track and second in the open track on all three target languages; we developed it across 273 tracked tasks (146 experiment runs) inside the Python framework released with this paper. The campaign covered 129 feature sets at approximately $450 in LLM API spend for feature engineering ($498 total third-party cost, including rented compute). We have since run the framework across three campaigns in three domains; its structural machinery adds only {\sim}1\% of wall-clock time (§ 5 ). Human Coding agents Python scripts suggestion task artefact suggests produces materialised overview/ feeds back Figure 1: Glite ARF’s three-role stack. The human researcher writes suggestions; coding agents execute tasks; deterministic Python scripts verify artefacts and materialise the canonical view that the human reads back. Each role works at a different unit of granularity. The failure mode we target is visible once campaigns involve hundreds of runs. Independent evaluation of fully autonomous research systems has shown how the failure rate compounds: Beel et al. 2025 found that 42% of experiments generated by The AI Scientist v1 ( Lu et al. 2024 ) failed due to coding errors. Our own audit of the BEA 2026 campaign repository catalogues thirteen incidents of similar character, ranging from train-data corruption affecting 38 feature sets to four feature sets that leaked the target variable into LLM prompts (Appendix D ). The natural response is to write better prompts. We tried. Across thousands of agent invocations the failure rate of prompt-encoded rules does not go to zero — it stays in the low single digits, which compounds to dozens of broken artefacts across a multi-week campaign. We argue that rules in agentic research must live in deterministic code: scripts that refuse to advance a step until an artefact conforms to a versioned specification, scripts that detect when files outside an agent’s task folder have been modified, scripts that materialise the canonical view of cumulative results so no agent ever needs to write a manual summary. The companion BEA 2026 system paper describes the same mechanism in its framework-overview section ( Philippov et al. 2026 ) . Glite ARF defines a three-role stack for autonomous research (Figure 1 ): the human researcher chooses which hypotheses to test, coding agents implement individual tasks under fixed structure, and deterministic Python scripts enforce task isolation, immutability, a corrections overlay, and a materialised project overview. We call this verifier-driven research , by analogy with test-driven development: the rules of the research process live in scripts that fail loudly when violated, not in prose that agents are merely asked to follow. Our contribution is the framework, used at scale across three domains, with a refereed external shared task as one anchor and measured campaign evidence (§ 5 ) as another. Section 2 positions Glite ARF in the autoresearch and AI-for-science landscape. Section 3 describes the framework’s seven structural principles and the verifier layer that enforces them. Section 4 reports our BEA 2026 case study, including an example where structured per-fold provenance let us catch and strip target leakage. Section 5 reports measured evidence from running the framework across three campaigns. Section 6 discusses lessons and limits — most importantly, we are explicit that role 1 (hypothesis selection) remains human-driven by design. Section 7 covers availability. 2 Related Work Multi-agent orchestration. AutoGen ( Wu et al. 2023 ) , MetaGPT ( Hong et al. 2024 ) , CAMEL ( Li et al. 2023 ) , smolagents ( Roucher et al. 2025 ) , and CrewAI ( CrewAI Inc. 2023 ) provide intra-task agent coordination and may run inside an ARF task; ARF is orthogonal, structuring the campaign -level lifecycle that wraps any of them. Coding agents. Coding agents like Claude Code ( Anthropic 2025 ) , OpenAI Codex CLI ( OpenAI 2025 ) , Aider ( Gauthier 2023 ) , OpenHands ( Wang et al. 2024 ) , and SWE-agent ( Yang et al. 2024 ) execute the code-writing inside an ARF task. SWE-bench ( Jimenez et al. 2024 ) is the closest adjacent benchmark, evaluating single-issue resolution rather than research-campaign integrity. Autoresearch (practitioner side). A practitioner community has formed around Karpathy’s autoresearch pattern ( Karpathy 2026 ) — short, single-metric optimisation loops applied to one training script. Recent work makes this approach more rigorous: CORAL ( Qu et al. 2026 ) introduces multi-agent evolution with isolated worktrees and shared persistent memory; EvoSkill ( Alzubi et al. 2026 ) discovers reusable agent skills through failure-driven evolution with each agent program represented as a git branch; autoreason ( SHL0MS and Hermes Agent 2026 ) addresses iterative document refinement by treating “do nothing” as a first-class option.