Feedback Over Form: Why Execution Feedback Matters More Than Pipeline Topology in 1-3B Code Generation
Charles Junichi McAndrews · 2026
Read Feedback Over Form: Why Execution Feedback Matters More Than Pipeline Topology in 1-3B Code Generation by Charles Junichi McAndrews (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-21950.txt.
Introduction
McAndrews asks whether composing small language models into pipelines can recover capability on code tasks. He studies code pipelines with execution feedback. He tests whether complex pipeline structure helps beyond a simple refinement loop. On two benchmarks, run locally on one laptop, self-refinement with execution feedback improves generation by more than 4 standard deviations. Refinement fixes many runtime errors but rarely fixes logic errors. Generator matters less than refiner capability: a 1.5 billion generator with a 3 billion refiner matches a 3 billion model doing both.
Catalog record
| CREATOR | Charles Junichi McAndrews |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40903 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.21950
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Feedback Over Form: Why Execution Feedback Matters More Than Pipeline Topology in 1-3B Code Generation Charles Junichi McAndrews Abstract Small language models (1-3B) are practical to run locally, but individually limited on harder code generation tasks. We ask whether composing them into pipelines can recover some of that lost capability. We study code generation pipelines built from 1-3B models with execution feedback, and use a NEAT-inspired evolutionary search to test whether more complex pipeline structure helps beyond a simple refinement loop. We evaluate on HumanEval (164 problems) and sanitized MBPP (427 problems), all with local inference on a single laptop. Self-refinement with execution feedback improves code generation by > 4 σ >4\sigma on both benchmarks. The gains are narrow in mechanism: refinement fixes many runtime errors (especially NameError and SyntaxError), but rarely fixes logic errors such as AssertionError. Within our tested general-purpose model pool, generator identity mattered less than refiner capability: a 1.5B generator paired with a 3B refiner matched a 3B model doing both roles. Early stopping is essential — without it, every iteration is net-negative. The code-specialized models outperform every general-purpose pipeline configuration, suggesting model specialization matters more than pipeline architecture. Preliminary text-only pipeline experiments without execution feedback did not show gains at this scale. In our constrained search space, evolutionary search mostly rediscovered the same simple generate → \rightarrow execute → \rightarrow refine loop we found manually, with no clearly significant gain from added topology. Single-evaluation fitness inflates results by 5–7%, selecting lucky genomes over good ones. On these benchmarks at 1-3B scale, execution feedback mattered more than added pipeline complexity in determining whether composition helped. 1 Introduction Small language models (1-3B parameters) are becoming increasingly relevant. These models are small enough for on-device deployment, edge computing, privacy-sensitive applications, and resource-constrained environments. Individually these models are extremely limited and cannot match larger models on complex tasks. This raises a natural question: can composing multiple small models into pipelines exceed what any single small model achieves? This question is well-studied at larger scales (70B+, API Models) where prior work (e.g., MoA, DSPy, AgentConductor) has shown gains from composition. At small scale, this answer is not as clear. Prior work like Self-MoA [ 9 ] suggests mixing weak models degrades quality. However, CYCLE [ 7 ] shows self-refinement with execution feedback works. Where exactly is the boundary between composition that helps and composition that hurts at small scale? Our main claim is that execution feedback is the mechanism that makes small-model composition work for code generation. A secondary claim is that, at this scale, more complex topology adds little over a simple generate \rightarrow execute \rightarrow refine loop. We systematically tested small-model composition on code generation tasks using two standard benchmarks: HumanEval [ 6 ] , Sanitized MBPP [ 4 ] . We built pipelines that compose multiple 1-3B models with code execution feedback: a model generates code, an executor runs it, and a refiner attempts to fix errors based on the traceback. We used NEAT [ 13 ] to search over pipeline configurations, such as which models fill which roles, how many refinement stages, whether to include an analyzer node, iteration counts, and temperatures. All of these experiments were run locally on a single laptop, demonstrating that meaningful research on model composition is accessible without cloud infrastructure. We found that execution feedback is the load-bearing mechanism. Self-refinement with execution feedback improved code generation by more than 4 standard deviations on both benchmarks. But the improvement is narrow: refinement reliably fixes runtime errors such as NameError and SyntaxError, while logic errors remain mostly unfixable. The traceback provides a usable signal when the failure is explicit, and much weaker signal when it only reports a wrong answer. We also found that the generator mattered less than the refiner in the general-purpose model pool we tested. A 1.5B generator paired with a 3B refiner matched a 3B model doing both jobs. Early stopping was essential: without it, every extra refinement pass was net-negative because the refiner often broke code that was already correct. Our evolutionary search over constrained linear pipelines mostly converged to the same simple generate \rightarrow execute \rightarrow refine loop that manual experimentation also found. One completed run added an analyzer node, but it did not produce a clearly significant gain over the simpler baseline. What mattered more was the capability of the model doing the refinement step. Code-specialized models remained stronger than every general-purpose pipeline configuration we tested. We also ran exploratory experiments on text-only pipelines, heuristic fitness optimization, and small-model fine-tuning. These mostly produced negative results, but they are secondary to the main code-generation story and we discuss them later as supporting context rather than central evidence. • We show that execution feedback significantly improves small-model code generation on HumanEval and MBPP, with gains above 4 standard deviations over solo generation. • We provide a mechanism-level analysis showing that refinement mainly fixes runtime errors, while logic errors remain mostly unfixable. • We show that early stopping is load-bearing: without it, every refinement iteration is net-negative because the refiner often breaks already-correct code. • We show that, within the tested general-purpose model pool, refiner capability mattered more than generator identity. • We apply a NEAT-inspired evolutionary search over constrained pipelines and find that it mostly rediscovered simple human-designed topologies rather than yielding a clearly significant improvement over them. • We quantify evaluation noise as a confounder for evolutionary search over stochastic LLM pipelines, showing that single-evaluation fitness inflates results by 5–7 2 Related Work 2.1 Self-Refinement with Execution Feedback The closest literature to this work is found in CYCLE [ 7 ] . This paper featured same-model self-refinement with execution feedback on code generation benchmarks. CYCLE showed that feeding tracebacks back to the same model improves code generation. To our knowledge, no prior work systematically compares cross-model composition against self-refinement at 1-3B scale. Our work extends CYCLE in two ways: cross-model composition (different generator and refiner) and evolutionary search over pipeline configurations. We also provide mechanistic analysis CYCLE doesn’t: Exactly which error types are fixable and why, the early stopping paradox, and regression rates. 2.2 Pipeline and Workflow Optimization AgentConductor [ 15 ] uses reinforcement learning (GRPO) to evolve multi-agent topologies with homogeneous workers which required training. Our approach uses NEAT with heterogeneous models and no training. AFlow [ 1 ] and EvoFlow [ 2 ] were both workflow optimizations but at the scale of 70B+ and API models. We operate at 1-3B with local inference, which is a fundamentally different resource regime. DSPy [ 8 ] is a framework for programmatic prompt optimization for LLM pipelines, but did not search over topology. The key distinction between these works is that they optimized pipelines of large capable models. We’re asking whether composition can compensate for individual model weakness at small scale. 2.3 Model Composition and Ensembles Mixture-of-Agents [ 14 ] aggregates outputs from multiple LLMs to improve quality. Self-MoA [ 9 ] showed that mixing weak models degrades rather than improves quality, which is directionally consistent with our exploratory text-pipeline results. Our experiments suggest a possible boundary at this scale: composition did not show gains on text tasks without verification, but did help on code tasks with