Delta-Based Neural Architecture Search: LLM Fine-Tuning via Code Diffs

Santosh Premi Adhikari, Radu Timofte, Dmitry Ignatov · 2026

Read Delta-Based Neural Architecture Search: LLM Fine-Tuning via Code Diffs by Santosh Premi Adhikari, Radu Timofte, Dmitry Ignatov (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-04903.txt.

Description

Santosh Premi Adhikari, Radu Timofte, and Dmitry Ignatov propose generating new neural network designs as small code changes rather than whole programs. Language models can invent architectures, but writing a full model each time is costly and produces long, verbose code. Their delta approach instead asks a fine-tuned model to output a compact diff against a baseline design.

The pipeline fine-tunes models in cycles on curated architectures, filtering for structural novelty along the way. Three coder models of similar size all beat full generation on validity and first-epoch accuracy across six image datasets. Their outputs ran far shorter, and a longer training study confirmed early accuracy predicted later rankings well.

Catalog record

CREATORSantosh Premi Adhikari, Radu Timofte, Dmitry Ignatov
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE80 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.04903

Browse the full library · About the Tower of Babel