From Brute Force to Semantic Insight: Performance-Guided Data Transformation Design with LLMs

Usha Shrestha, Dmitry Ignatov, Radu Timofte · 2026

Read From Brute Force to Semantic Insight: Performance-Guided Data Transformation Design with LLMs by Usha Shrestha, Dmitry Ignatov, Radu Timofte (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-03808.txt.

Introduction

Usha Shrestha, Dmitry Ignatov, and Radu Timofte teach language models to design data transformations for image training by learning from measured results instead of rules of thumb. Data augmentation is usually tuned by heuristics or brute force search. Their closed-loop system, built inside the NNGPT project ecosystem, fine-tunes models on a new repository of more than six thousand PyTorch augmentation functions, each labeled only by the downstream accuracy it produced.

Training compares better and worse transformations in pairs, aligning the model through empirical feedback without reinforcement learning or reward models. The result needs up to six hundred times fewer evaluated candidates than brute-force discovery while keeping peak accuracy competitive. Generation shifts from random synthesis toward informed design, which could cut the compute cost of squeezing accuracy out of vision models.

Catalog record

CREATORUsha Shrestha, Dmitry Ignatov, Radu Timofte
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-03808

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

From Brute Force to Semantic Insight: Performance-Guided Data Transformation Design with LLMs Usha Shrestha, Dmitry Ignatov, Radu Timofte Abstract Large language models (LLMs) have achieved notable performance in code synthesis; however, data-aware augmentation remains a limiting factor, handled via heuristic design or brute-force approaches. We introduce a performance-aware, closed-loop solution in the NNGPT ecosystem of projects that enables LLMs to autonomously engineer optimal transformations by internalizing empirical performance cues. We fine-tune LLMs with Low-Rank Adaptation on a novel repository of 6,000+ empirically evaluated PyTorch augmentation functions, each annotated solely by downstream model accuracy. Training uses pairwise performance ordering (better–worse transformations), enabling alignment through empirical feedback without reinforcement learning, reward models, or symbolic objectives. This reduces the need for exhaustive search, achieving up to 600 × \times fewer evaluated candidates than brute-force discovery while maintaining competitive peak accuracy and shifting generation from random synthesis to task-aligned design. Ablation studies show that structured Chain-of-Thought prompting introduces syntactic noise and degrades performance, whereas direct prompting ensures stable optimization in performance-critical code tasks. Qualitative and quantitative analyses demonstrate that the model internalizes semantic performance cues rather than memorizing syntax. These results show that LLMs can exhibit task-level reasoning through non-textual feedback loops, bypassing explicit symbolic rewards. 1 Introduction A neural network’s performance depends on both its architecture and how the data is preprocessed. Effective data preprocessing and augmentation are essential for model generalization and convergence. Numerous studies have investigated optimal strategies for data transformations and the development of novel methods. For instance, AutoAugment ( 6 ) applies reinforcement learning to identify effective augmentation policies, while Meta Learning ( 3 ) approach uses a predictive meta-model to suggest data transformations for a specific classification algorithm. As generative artificial intelligence becomes increasingly prevalent, recent studies have begun to explore code generation using large language models (LLMs), leveraging their generative capabilities to propose and assess complex, data-aware solutions. The NNGPT framework ( 18 ; 19 ) previously established a methodology for synthesizing neural network architectures using LLMs. Extending this work, we automate the generation and evaluation of data augmentation functions within the NNGPT ecosystem. This study also addresses the limited diversity of data augmentations in the LEMUR dataset, which comprises a broad range of high-capacity and edge-optimized neural network models ( 11 ; 37 ; 8 ) and serves as the knowledge base for the NNGPT. Motivated by recent advances in LLM applications across multiple domains ( 10 ; 34 ; 16 ) and prior NNGPT experiments ( 15 ; 38 ; 27 ; 17 ) , we curate a diverse set of PyTorch data transformation functions and systematically evaluate their effects, producing performance-annotated metadata that links each code snippet to its impact on model training. This metadata is used to fine-tune the language model, enhancing its understanding of data variability and performance effects. We further implement a system that generates PyTorch data transformation functions and iteratively refines the generator through supervised fine-tuning. 2 Related Works The automated generation of data augmentations ( 39 ) has evolved significantly. The process began with AutoAugment ( 6 ) , which uses reinforcement learning to search a fixed list of operations. This approach defines a discrete search space of 14 to 16 standard image processing methods, such as Rotate, ShearX, and ShearY. In this approach, generation refers to finding an optimal policy composed of sub-policies that specify two sequential transformation operations, the probability of applying each operation, and the magnitude of each operation. This method transforms the problem into a large-scale discrete search challenge, with AutoAugment’s search space containing approximately 10^{32} possible policies. The high computational cost of automated search in AutoAugment led to approaches that simplified the generation process. RandAugment ( 7 ) demonstrated that a complex search algorithm is unnecessary. Instead, it automated generation by reducing the search space to two interpretable hyperparameters: N (the number of transformations to apply) and M (a single, global magnitude for all transformations). RandAugment randomly samples N transformations from a predefined list and applies them with magnitude M. This generation method matched the performance of AutoAugment, indicating that the diversity of the transformation space is more critical than the complexity of the generation algorithm. A key limitation of both AutoAugment and RandAugment is that they generate a policy that is applied to every image. However, a policy that is good for one image may be harmful to another ( 1 ) . This led to the development of automated generation frameworks that are instance-specific and creates a unique augmentation policy for each individual image ( 26 ) . For each image, a Deep Q-Network (DQN) iteratively generates a policy by selecting an action from a list of transformations or a "Stop" action. This process generates a unique, optimal chain of transformations for every sample in the dataset. This is a far more granular method of "generating a large number of transforms," as it generates one policy per instance rather than one policy per dataset. Later research aimed to generate more effective and diverse transforms by expanding the space of possible transformations ( 29 ) . This expansion of the generation space occurred in two ways: through learned transformations and generative models. This approach enables the model to learn the transformation function itself. For example, Spatial Transformer Networks (STNs) ( 29 ) can be integrated into a model to learn optimal affine transformations directly from the data. The latest paradigm in automated generation utilizes Large Language Models (LLMs) as the primary generation engine ( 28 , 9 ). Recent approaches to adapting LLMs for specific tasks have shifted toward Instruction Tuning and Supervised Fine-Tuning (SFT) ( 31 , 5 ). 30 showed that fine-tuning models on human-written instructions aligns them more closely with user intent than simply increasing model size. This method is also effective in specialized domains, such as Python programming ( 2 ) . In code generation, where models must address complex and functional requirements beyond basic text completion, such alignment is crucial. 4 further confirmed this by demonstrating that optimizing general-purpose LLMs on code corpora enhances performance on functional correctness benchmarks. Recent research in LLM adaptation also highlights the significance of data quality and training strategies over sheer data quantity. 24 identified that task balancing and enriching training data, such as by inverting input-output pairs, are essential for improving generalization. Their results indicate that combining zero-shot, few-shot, and Chain-of-Thought (CoT) data during fine-tuning leads to better performance across evaluation settings. The structure of input prompts and the training data have a significant impact on the quality of generated outputs, even though SFT updates model weights. A systematic survey by 35 categorizes advanced prompting techniques, including Chain-of-Thought (CoT) and decomposed prompting, which are essential for guiding models through multi-step logical tasks such as complex data augmentation. Building on these findings, 20 demonstrated that LLMs function as effective "zero-shot reasoners" and can perform complex task by simply appending the prompt "Let’s think step by step." This approach enables the creation of reasoning-dense training data without the need for manual annotation. In the field of code generation, AceCoder ( 23 ) addresses the challenge of requirement understanding through a guided code generation mechanism. This approach prompts the model to produce intermediate outputs, such

Read the complete text