Delta-Based Neural Architecture Search: LLM Fine-Tuning via Code Diffs
Santosh Premi Adhikari, Radu Timofte, Dmitry Ignatov · 2026
Read Delta-Based Neural Architecture Search: LLM Fine-Tuning via Code Diffs by Santosh Premi Adhikari, Radu Timofte, Dmitry Ignatov (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-04903.txt.
Description
Santosh Premi Adhikari, Radu Timofte, and Dmitry Ignatov propose generating new neural network designs as small code changes rather than whole programs. Language models can invent architectures, but writing a full model each time is costly and produces long, verbose code. Their delta approach instead asks a fine-tuned model to output a compact diff against a baseline design.
The pipeline fine-tunes models in cycles on curated architectures, filtering for structural novelty along the way. Three coder models of similar size all beat full generation on validity and first-epoch accuracy across six image datasets. Their outputs ran far shorter, and a longer training study confirmed early accuracy predicted later rankings well.
Catalog record
| CREATOR | Santosh Premi Adhikari, Radu Timofte, Dmitry Ignatov |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 80 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.04903