Resource-Efficient Iterative LLM-Based NAS with Feedback Memory
Xiaojie Gu, Dmitry Ignatov, Radu Timofte · 2026
Read Resource-Efficient Iterative LLM-Based NAS with Feedback Memory by Xiaojie Gu, Dmitry Ignatov, Radu Timofte (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-12091.txt.
Introduction
Gu, Ignatov, and Timofte search for neural network designs with LLMs in a tight feedback loop. The loop runs on one consumer GPU, with no fine-tuning of the language models. A sliding memory keeps the last five improvement tries, each stored as a problem, a fix, and an outcome. Failed runs count as first-class learning signals here, not waste. One model writes PyTorch code while a second handles diagnostic reasoning. Three frozen models improved sharply on CIFAR tests over long searches.
Catalog record
| CREATOR | Xiaojie Gu, Dmitry Ignatov, Radu Timofte |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37581 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.12091
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Resource-Efficient Iterative LLM-Based NAS with Feedback Memory Xiaojie Gu, Dmitry Ignatov, Radu Timofte Abstract Neural Architecture Search (NAS) automates network design, but conventional methods demand substantial computational resources. We propose a closed-loop pipeline leveraging large language models (LLMs) to iteratively generate, evaluate, and refine convolutional neural network architectures for image classification on a single consumer-grade GPU without LLM fine-tuning. Central to our approach is a historical feedback memory inspired by Markov chains: a sliding window of K = 5 K{=}5 recent improvement attempts keeps context size constant while providing sufficient signal for iterative learning. Unlike prior LLM optimizers that discard failure trajectories, each history entry is a structured diagnostic triple—recording the identified problem, suggested modification, and resulting outcome—treating code execution failures as first-class learning signals. A dual-LLM specialization reduces per-call cognitive load: a Code Generator produces executable PyTorch architectures while a Prompt Improver handles diagnostic reasoning. Since both the LLM and architecture training share limited VRAM, the search implicitly favors compact, hardware-efficient models suited to edge deployment. We evaluate three frozen instruction-tuned LLMs ( ≤ 7 {\leq}7 B parameters) across up to 2000 iterations in an unconstrained open code space, using one-epoch proxy accuracy on CIFAR-10, CIFAR-100, and ImageNette as a fast ranking signal. On CIFAR-10, DeepSeek-Coder-6.7B improves from 28.2% to 69.2%, Qwen2.5-7B from 50.0% to 71.5%, and GLM-5 from 43.2% to 62.0%. A full 2000-iteration search completes in ≈ 18 {\approx}18 GPU hours on a single RTX 4090, establishing a low-budget, reproducible, and hardware-aware paradigm for LLM-driven NAS without cloud infrastructure. Code Generator (LLM) Validator & Evaluator Prompt Improver (LLM) Historical Feedback Memory candidate code accuracy or error suggestions history + best code Figure 1 : Overview of the iterative NAS pipeline. The Code Generator produces a candidate architecture as executable PyTorch code. The Evaluator validates and trains it using one-epoch proxy evaluation. The Prompt Improver analyzes results with historical feedback memory to generate targeted improvement suggestions for the next iteration. 1 Introduction (a) DeepSeek-Coder (CIFAR-10) (b) Qwen2.5 (CIFAR-10) (c) GLM-5 (CIFAR-10) (d) DeepSeek-Coder (CIFAR-100) (e) Qwen2.5 (CIFAR-100) (f) GLM-5 (CIFAR-100) (g) DeepSeek-Coder (ImageNette) (h) Qwen2.5 (ImageNette) (i) GLM-5 (ImageNette) Figure 2 : One-epoch proxy accuracy on CIFAR-10 (top row, a–c), CIFAR-100 (middle row, d–f), and ImageNette (bottom row, g–i) across all iterations. Light curves show per-iteration accuracy (the accuracy of iterations with errors fall back to previous value), dashed lines show the smoothed trend (window w{=}15 ), and bold lines show the best-so-far trajectory. All models exhibit clear upward trends. For DeepSeek-Coder on ImageNette, only the first 30 iterations are plotted because all subsequent iterations resulted in errors. Neural Architecture Search (NAS) has emerged as a powerful paradigm for automating the design of deep neural networks, achieving competitive or superior performance to hand-crafted architectures across diverse tasks 39 ; 24 ; 20 . However, early conventional NAS methods are notoriously resource-intensive: early reinforcement learning approaches required up to 22,400 GPU-days 39 , evolutionary algorithms demanded thousands of GPU-days 24 , and even efficient differentiable relaxations 20 depend on heavy supernet training. This severe computational cost has motivated data-efficient and training-free approaches that use proxy metrics to rank architectures without full training 21 ; 18 , yet these methods still operate within constrained, predefined search spaces (e.g., cell-based structures). Recent advances in large language models (LLMs) have opened a fundamentally different avenue: using LLMs as architecture generators that produce executable neural network code directly 16 ; 14 ; 31 . Unlike traditional NAS, LLM-based approaches operate in the unconstrained open code space of object-oriented programs rather than fixed cell-based encodings, enabling more flexible and expressive architectural invention. This efficiency-oriented perspective is central to our work: by operating on a single consumer-grade GPU with frozen {\leq}7 B LLMs and no fine-tuning, our pipeline makes architecture search accessible in resource-constrained environments and implicitly favors compact models suited to deployment on hardware-limited devices. Prior work within the NNGPT framework 16 and related efforts 12 ; 22 ; 29 have demonstrated that LLMs can generate functional vision models, but predominantly through single-shot or few-shot generation without iterative self-improvement. A critical limitation of single-shot LLM generation is that it treats architecture design as a one-pass prediction, discarding the evaluation signal entirely. In contrast, human engineers iterate: they build, test, analyze failures, and refine their designs based on accumulated experience. This observation motivates our central question: Can a small LLM iteratively improve neural network architectures by learning from a structured history of its own attempts? We propose a closed-loop pipeline comprising three components: (1) a Code Generator that prompts an instruction-tuned LLM to produce PyTorch model implementations, (2) an Evaluator that trains each generated model for a single epoch on CIFAR-10, CIFAR-100 17 , or ImageNette 11 as a fast proxy for architecture quality, and (3) a Prompt Improver that analyzes evaluation results alongside a sliding window of recent iteration history—a mechanism we term historical feedback memory —to produce targeted improvement suggestions for the next iteration. This design draws on the Markov property: the improvement decision at each step depends on the current best architecture and a bounded window of recent transitions, rather than the full trajectory. Our main contributions are: • A closed-loop, iterative NAS pipeline driven by LLMs that progressively discovers better architectures through code generation, evaluation, and prompt refinement. • A historical feedback memory mechanism that maintains a sliding window of past improvement attempts, enabling the LLM to mitigate repeating failed strategies and build on successful ones. • An empirical demonstration across three LLMs of different origins and specializations showing that the pipeline consistently improves architecture quality—from 28.2% to 69.2% (DeepSeek-Coder-6.7B-Instruct), 50.0% to 71.5% (Qwen2.5-7B), and 43.2% to 62.0% (GLM-5) on CIFAR-10—requiring only \sim 18 GPU hours on a single consumer-grade 24GB GPU, establishing a low-budget approach to NAS. 2 Related Work Neural Architecture Search. NAS methods aim to automate the design of neural network architectures. Early approaches employed reinforcement learning 39 and evolutionary algorithms 24 to explore architecture search spaces, but required thousands of GPU hours per search. Parameter-sharing methods such as ENAS 23 and differentiable approaches like DARTS 20 substantially reduced search cost by amortizing evaluation across architectures. More recently, zero-shot or training-free NAS methods 21 ; 18 bypass training entirely by scoring architectures at initialization using proxy metrics such as activation overlap 21 or gradient-based indicators. While effective within their respective search spaces, these methods remain constrained to predefined, discrete architecture parameterizations (e.g., cell-based structures) and cannot generate truly novel architectural patterns. LLMs for Code Generation and AutoML. Large language models have demonstrated strong capabilities in code generation 4 , and their application to machine learning automation is an active research area. The NNGPT framework 16 uses LLMs as neural network generators, showing that language models can produce functional vision models from textual prompts. Subsequent work explored fractal-inspired architectures 22 , few-shot prompting strategies 31 , novel architecture creativity 14 , and non-standard channel priors 29 . LLMs have also been applied to hyperparameter optimization 15 , data transformation design 28 , and collaborative vision-language pipelines 27 ; 9 ; 13 . These efforts are supported by curated datasets of neural network architectures 10 ; 30 and deployment pipelines 6 . However, most existing approaches treat LLM-based architecture generation