Understanding the Challenges in Iterative Generative Optimization with LLMs

Allen Nie, Xavier Daull, Zhiyi Kuang, Abhinav Akkiraju, Anish Chaudhuri, Max Piasevoli, Ryan Rong, YuCheng Yuan, Prerit Choudhary, Shannon Xiao, Rasool Fakoor, Adith Swaminathan, Ching-An Cheng · 2026

Read Understanding the Challenges in Iterative Generative Optimization with LLMs by Allen Nie, Xavier Daull, Zhiyi Kuang, Abhinav Akkiraju, Anish Chaudhuri, Max Piasevoli, Ryan Rong, YuCheng Yuan, Prerit Choudhary, Shannon Xiao, Rasool Fakoor, Adith Swaminathan, Ching-An Cheng (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-23994.txt.

Introduction

Nie and co-authors study why iterative generative optimization with language models often fails in practice. Such loops improve code, workflows, or prompts from execution feedback, yet only 9 percent of surveyed agents used automated optimization. The authors blame hidden design choices: the starting artifact, the credit horizon, and how trials are batched as evidence. Case studies in MLAgentBench, Atari, and BigBench Extra Hard show each choice can decide success. They give practical guidance for setting up these learning loops.

Catalog record

CREATORAllen Nie, Xavier Daull, Zhiyi Kuang, Abhinav Akkiraju, Anish Chaudhuri, Max Piasevoli, Ryan Rong, YuCheng Yuan, Prerit Choudhary, Shannon Xiao, Rasool Fakoor, Adith Swaminathan, Ching-An Cheng
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE114206
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.23994

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Understanding the Challenges in Iterative Generative Optimization with LLMs Allen Nie, Xavier Daull, Zhiyi Kuang, Abhinav Akkiraju, Anish Chaudhuri, Max Piasevoli, Ryan Rong, YuCheng Yuan, Prerit Choudhary, Shannon Xiao, Rasool Fakoor, Adith Swaminathan, Ching-An Cheng Abstract Generative optimization uses large language models (LLMs) to iteratively improve artifacts (such as code, workflows or prompts) using execution feedback. It is a promising approach to building self-improving agents, yet in practice remains brittle: despite active research, only 9 % 9\% of surveyed agents used any automated optimization. We argue that this brittleness arises because, to set up a learning loop , an engineer must make “hidden” design choices: What can the optimizer edit and what is the “right” learning evidence to provide at each update? We investigate three factors that affect most applications: the starting artifact, the credit horizon for execution traces, and batching trials and errors into learning evidence. Through case studies in MLAgentBench, Atari, and BigBench Extra Hard, we find that these design decisions can determine whether generative optimization succeeds, yet they are rarely made explicit in prior work. Different starting artifacts determine which solutions are reachable in MLAgentBench, truncated traces can still improve Atari agents, and larger minibatches do not monotonically improve generalization on BBEH. We conclude that the lack of a simple, universal way to set up learning loops across domains is a major hurdle for productionization and adoption. We provide practical guidance for making these choices. 1 Introduction Rapid advances in the capabilities of large language models (LLMs) have enabled a proliferation of software systems with the ability to perceive, plan, and reflect ( Kwa et al. 2025 ) . Recent work has shown that LLMs have the ability to generate and revise program workflows to optimize an objective ( Yang et al. 2024 ) , such as increasing compute throughput or decreasing latency of hardware accelerator kernels ( Ouyang et al. 2025a ; Lange et al. 2025b ; Wei et al. 2025b ; Zhang et al. 2025c ) , design novel algorithms for search and matrix multiplication ( Wei et al. 2025a ; Press et al. 2025 ; Novikov et al. 2025 ) , exploiting security vulnerabilities ( Zhang et al. 2025a ; Chaudhuri et al. 2025 ) , and propose therapeutic candidates for diseases ( Ghareeb et al. 2025 ) . This ability to optimize objectives, combined with LLMs continuing to approach human-level performance in producing complex programs ( Jimenez et al. 2024 ; Wang et al. 2025a ; El-Kishky et al. 2025 ; Wei et al. 2025c ) , gives rise to an emerging class of software that automatically changes its own behavior to achieve a desired outcome. Automated generation and optimization with LLMs have been adopted broadly in two types of applications. The first is to use the LLM to repeatedly modify a software program to improve on a metric, such as writing kernels that have low compute latency ( Baronio et al. 2025 ; Wei et al. 2025b ; Zhang et al. 2025c ) , creating automated ML pipelines to train models that achieve high test accuracy ( Huang et al. 2024 ; Chan et al. 2025 ; Toledo et al. 2025 ) , and writing a script that can exploit security vulnerabilities ( Zhang et al. 2025a ) . The second type uses LLMs to modify another LLM system to achieve desired behavior, also using metrics like success rate, through prompt tuning and direct code revisions ( Khattab et al. 2024 ; Yuksekgonul et al. 2025 ; Cheng et al. 2024 ; Wang et al. 2024 ; Zhang et al. 2025b ) . The underlying mechanism in both types of applications is the same: construction of an LLM-based generative optimization process to ingest feedback and modify an existing system, which we describe as a learning loop in Section 2 . However, despite the popularity in research, LLM-based generative optimization has not been widely adopted in production. Pan et al. 2025b report that the current development of agentic systems remains largely human-driven, with only 9% of surveyed systems employing any form of automated design, including simple LLM-assisted prompt tuning. By contrast, using LLMs to optimize programs has been more successful in specialized domains ( Toledo et al. 2025 ; Novikov et al. 2025 ) . In a field where end-to-end automation that scales with compute is a highly sought-after objective ( Sutton 2019 ; Hendrycks et al. 2025 ) , the lack of wider adoption is puzzling, pointing to a potential gap between the high ideal of an automated LLM optimizer and reality, especially for optimizing LLM agentic systems. This lack of adoption is not a consequence of inadequate infrastructure support or insufficient software abstraction. On the contrary, over the past two years, a rich ecosystem of powerful libraries for building agentic systems has emerged ( Khattab et al. 2024 ; Wu et al. 2024 ; LangChain 2024 ; Cheng et al. 2024 ; Li et al. 2025 ) . Many of these libraries offer mechanisms for automatically optimizing different targets, ranging from tuning prompts ( Khattab et al. 2024 ; Yuksekgonul et al. 2025 ) to program synthesis ( Cheng et al. 2024 ) . Most of them have received considerable attention from agent engineers, suggesting that the low adoption rate cannot be attributed solely to limited awareness or inadequate software. In this paper, we hypothesize that the low adoption stems from the hidden difficulty of setting up the learning loop . Our experiments show that getting the learning loop right requires substantial engineering effort and/or guesswork. This setup burden is a major hurdle for productionization, which requires simple, universal solutions across application domains. We first introduce two core concepts that impact the learning loop in Section 2 : system initialization and learning context construction. Then we examine three case studies to isolate and highlight the challenges related to these core concepts, and show how different design decisions can impact the final performance. In Section 4 , we show how different system architectures and initializations impact the final model quality of an ML training pipeline. In Section 5 , using the example of an Atari game-playing program, we show that engineers can specify a credit horizon that is shorter than the full gameplay trajectory and still learn programs that obtain high reward on a full playthrough. But updating the system too frequently with too short a credit horizon leads to worse performance. Finally, in Section 6 , we show that the number of examples placed in the learning context matters for optimizing the prompt of an LLM call. Interestingly, many of the challenges in setting up a learning loop for LLM-based optimization parallel well-studied concepts in machine learning. The starting artifact problem resembles neural network architecture ( Zoph & Le 2017 ) and weights initialization ( Glorot & Bengio 2010 ) , where different starting points determine which solutions are reachable. The credit horizon problem mirrors horizon debates in episodic reinforcement learning (i.e. deciding how many timesteps to include before computing returns ( Arjona-Medina et al. 2019 ) ) and truncated back-propagation through time ( Tallec & Ollivier 2017 ; Shaban et al. 2019 ) . The experience batching problem parallels batch

Read the complete text