Convergence Theory for Iterative LLM-Based Neural Architecture Search: A Parametric Cross-Entropy Framework with Closed-Form Proxy Reliability

Santosh Premi Adhikari, Radu Timofte, Dmitry Ignatov · 2026

Read Convergence Theory for Iterative LLM-Based Neural Architecture Search: A Parametric Cross-Entropy Framework with Closed-Form Proxy Reliability by Santosh Premi Adhikari, Radu Timofte, Dmitry Ignatov (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-30103.txt.

Introduction

Santosh Premi Adhikari, Radu Timofte and Dmitry Ignatov build a convergence theory for iterative LLM-based neural architecture search. They model the search as a parametric cross-entropy method over programs and prove six results. Elite fine-tuning matches the cross-entropy update, expected quality never decreases, and elite-set probability converges geometrically. Further results favour delta-based generation, show a novelty filter blocks mode collapse, and give a closed-form proxy reliability formula. A 22-cycle, three-model, six-dataset experiment confirms two predictions and two in direction.

Catalog record

CREATORSantosh Premi Adhikari, Radu Timofte, Dmitry Ignatov
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38217
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.30103

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Convergence Theory for Iterative LLM-Based Neural Architecture Search: A Parametric Cross-Entropy Framework with Closed-Form Proxy Reliability Santosh Premi Adhikari, Radu Timofte, Dmitry Ignatov Abstract Large language models (LLMs) are increasingly used as generators in iterative neural architecture search (NAS), yet no formal convergence theory exists for this class of algorithms. We model iterative LLM-NAS as a parametric Cross-Entropy (CE) method over executable programs and prove six results: (1) iterative LLM fine-tuning on elite architectures is equivalent to the CE update restricted to the LLM parametric family; (2) expected architecture quality is monotonically non-decreasing across cycles; (3) elite-set probability converges to a fixed point at a geometric rate C t ≥ 1 − ( 1 − ρ 0 ) t C_{t}\geq 1-(1-\rho_{0})^{t} ; (4) delta-based generation achieves a strictly higher valid-generation rate than full-code generation under a first-order Markov token-error model, replacing the independent-token assumption of prior work; (5) the MinHash-Jaccard novelty filter prevents mode collapse; (6) proxy reliability admits the closed-form ρ S = 6 π ​ arcsin ⁡ ( ρ P ​ ( SNR ) / 2 ) \rho_{\mathrm{S}}=\tfrac{6}{\pi}\arcsin\bigl(\rho_{\mathrm{P}}(\mathrm{SNR})/2\bigr) , yielding the practical diagnostic σ arch 2 ≫ σ noise 2 \sigma^{2}_{\mathrm{arch}}\gg\sigma^{2}_{\mathrm{noise}} as a necessary condition for trustworthy proxy-based rankings. Testing against a 22-cycle, three-LLM, six-dataset experiment with 3,300 generated architectures: two predictions are consistent quantitatively (monotone quality trend, persistent novel admissions), two at direction-of-effect level (elite-concentration plateau converging to a LoRA ceiling; DeepSeek non-significance under low SNR), and two depart from calibrated values (delta-rate ratio 1.41 vs. predicted 2.23; Qwen Spearman 0.635 vs. predicted 0.10 due to ceiling truncation). Proxy-reliability validation confirms the predicted LLM ordering (Mistral, Qwen, DeepSeek) with a coherent ordinal pattern matching the SNR ranking, Mistral the sole result surviving Bonferroni correction ( p = 7.7 × 10 − 7 p{=}7.7\times 10^{-7} ), and explains the proxy-reliability ceiling effect previously reported empirically but left unexplained. 1 Introduction Neural Architecture Search (NAS) has evolved from early reinforcement-learning controllers operating within predefined cell topologies ( 31 ; 20 ) to gradient-based differentiable search ( 15 ) and one-shot super-network methods ( 19 ; 2 ; 7 ) . A recent paradigm shift replaces hand-crafted search spaces entirely, using large language models as open-ended generators of executable torch.nn.Module programs. This approach was pioneered by 11 and 6 , who showed empirically that iteratively fine-tuning an LLM on its own successful generations progressively improves both generation reliability and architecture quality. 1 further demonstrated that generating unified diffs (deltas) rather than complete implementations reduces output length by 75–85% while improving the valid-generation rate from 50.6% to 66–75%. Despite these empirical successes, fundamental theoretical questions remain unanswered. Does iterative LLM fine-tuning provably improve architecture quality? Under what conditions does the process converge? Why does delta generation achieve higher valid generation rates? What role does novelty filtering play in preventing degenerate solutions? We address all four questions by establishing a formal mathematical framework. Our main contribution is to show that iterative LLM-NAS is a parametric instance of the Cross-Entropy (CE) method ( 22 ) —a well-studied optimisation algorithm with known convergence properties—and to derive consequences specific to the LLM-NAS setting. The paper is organised as follows. Section 2 surveys related work. Section 3 presents the mathematical framework. Section 4 states and proves the five main convergence theorems. Section 5 develops proxy-reliability theory. Section 6 presents an illustrative case study. Section 7 discusses limitations and implications. 2 Related Work Neural Architecture Search. NAS methods span three broad families. Reinforcement-learning methods ( 31 ; 20 ; 26 ) use a controller to propose architectures evaluated on a proxy task, but are computationally prohibitive. One-shot methods ( 19 ; 2 ; 7 ; 29 ) train a weight-sharing super-network and inherit weights for sub-architectures. Differentiable methods such as DARTS ( 15 ) relax the discrete search space to a continuous one and apply gradient descent. All three operate within predefined cell or operation search spaces; LLM-based NAS is distinguished by its use of open-ended program synthesis. LLMs as Optimisers. 30 showed that LLMs can serve as optimisers by iteratively refining candidate solutions expressed in natural language. 16 introduced Self-Refine, a feedback-driven iterative prompting strategy. 21 used LLMs inside an evolutionary loop to discover novel mathematical functions. The present work provides the first convergence guarantees for such iterative LLM-based search procedures. Cross-Entropy Method and Estimation of Distribution Algorithms. The CE method was introduced by 22 for rare-event simulation and extended to combinatorial optimisation by 4 . Its application to reinforcement learning was demonstrated by 25 . Estimation of Distribution Algorithms (EDAs) ( 14 ; 18 ; 17 ) share the same update structure: fit a parametric distribution to elite samples and resample. Convergence of EDAs to global optima is established under conditions including finite search spaces and sufficient model expressiveness ( 23 ) . Our work establishes an analogous convergence result for the LLM parametric family, subject to explicit assumptions that we discuss in detail. LoRA and Parameter-Efficient Fine-Tuning. 8 introduced LoRA, which restricts fine-tuning updates to a low-rank subspace of the weight matrices. While LoRA dramatically reduces computational cost, it introduces a model-expressiveness constraint that is central to our analysis. We explicitly characterise the projection error introduced by this restriction and treat it as an open problem for future work, rather than absorbing it silently into the theory. 3 Mathematical Framework 3.1 Architecture Space and Quality Function Definition 1 (Architecture Space) . Let \mathcal{A} denote the set of all syntactically valid, executable Python programs that define a torch.nn.Module subclass. For delta generation with baseline b\in\mathcal{A} , define \mathcal{A}_{\delta}(b)\subseteq\mathcal{A} as the set of all architectures obtainable by applying a valid unified diff to b . Definition 2 (Quality Function) . A quality function q:\mathcal{A}\to[0,1] maps each architecture to its first-epoch validation accuracy on a fixed dataset. The elite set at threshold \tau\in(0,1) is \mathcal{E}_{\tau}=\{a\in\mathcal{A}:q(a)\geq\tau\} . The choice of 1 epoch as the proxy length follows the convention of 11 ; 1 ; varying this parameter would rescale both \sigma^{2}_{\mathrm{arch}} and \sigma^{2}_{\mathrm{noise}} in Section 5 , leaving the \mathrm{SNR} -driven predictions of Theorem 8 qualitatively unchanged but changing absolute correlation magnitudes. Section 5.1 reports the actual SNR values that result from this convention. 3.2 The Iterative LLM-NAS Process Let \Theta be the parameter space of a pre-trained LLM with LoRA adaptation ( 8 ) , and let p_{\theta}:\mathcal{A}\to[0,1] denote the generation distribution induced by parameters \theta . The parametric family is \mathcal{F}=\{p_{\theta}:\theta\in\Theta\} . Definition 3 (Iterative LLM-NAS) . Given initial parameter \theta_{0} and static corpus \mathcal{L} (the LEMUR dataset ( 5 ; 27 ) of \sim 626 hand-curated, peer-reviewed torch.nn.Module architectures across image-classification, segmentation and regression tasks; in the experiments of 1 this corpus is restricted to its image-classification subset which serves as the warm-start prior), the process proceeds for cycles t=0,1,\ldots,T : 1. Generate: Sample N candidates a_{1},\ldots,a_{N}\sim p_{\theta_{t}} . 2. Evaluate: Compute q(a_{i}) for each i . 3. Filter: Form \mathcal{A}^{+}_{t}=\{a_{i}:q(a_{i})\geq\tau\}\cap\mathrm{Novel}_{t} , where \mathrm{Novel}_{t} denotes architectures passing the MinHash–Jaccard novelty filter against the current corpus. 4. Update corpus: S_{t}=S_{t-1}\cup\mathcal{A}^{+}_{t} . 5. Fine-tune: \theta_{t+1}=\arg\min_{\theta\in\Theta}-\sum_{a\in S_{t}\cup\mathcal{L}}\log p_{\theta}(a) . 3.3 CE Method Background The CE method for combinatorial optimisation ( 4 ) maintains a distribution p_{t} over solutions and iterates: p_{t+1}=\arg\min_{p\in\mathcal{F}}\mathrm{KL}\!\left(\hat{p}^{\tau\text{-elite}}_{t}\,\|\,p\right), where \mathrm{KL}(\cdot\,\|\,\cdot) denotes the Kullback–Leibler divergence (

Read the complete text