Transducer Tuning: Efficient Model Adaptation for Software Tasks Using Code Property Graphs
Imam Nur Bani Yusuf, Lingxiao Jiang · 2024
Read Transducer Tuning: Efficient Model Adaptation for Software Tasks Using Code Property Graphs by Imam Nur Bani Yusuf, Lingxiao Jiang (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-13467.txt.
Introduction
Imam Nur Bani Yusuf and Lingxiao Jiang adapt big code models cheaply in this 2024 paper. Fine-tuning such models for software tasks eats memory as the models grow. Teams with modest hardware get locked out.
Their method adds a small part, a transducer, beside the frozen model. It reads code property graphs, maps of a program's structure and links, and blends what it learns into the model's code vectors. A vector engine builds the graph data; an attention layer does the blending. Only the small part trains, so memory stays low. The work targets code tasks where structure, not just text, carries the meaning.
Catalog record
| CREATOR | Imam Nur Bani Yusuf, Lingxiao Jiang |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-13467
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Transducer Tuning: Efficient Model Adaptation for Software Tasks Using Code Property Graphs Imam Nur Bani Yusuf, Lingxiao Jiang Abstract Large language models have demonstrated promising performance across various software engineering tasks. While fine-tuning is a common practice to adapt these models for downstream tasks, it becomes challenging in resource-constrained environments due to increased memory requirements from growing trainable parameters in increasingly large language models. We introduce Transducer Tuning, a technique to adapt large models for downstream code tasks using Code Property Graphs (CPGs). Our approach introduces a modular component called Transducer that enriches code embeddings with structural and dependency information from CPGs. The Transducer comprises two key components: Graph Vectorization Engine (GVE) and Attention-Based Fusion Layer (ABFL). GVE extracts CPGs from input source code and transforms them into graph feature vectors. ABFL then fuses those graph feature vectors with initial code embeddings from a large language model. By optimizing these transducers for different downstream tasks, our approach enhances the models without the need to fine-tune them for specific tasks. We have evaluated Transducer Tuning on three downstream tasks: code summarization, assert generation, and code translation. Our results demonstrate competitive performance compared to full parameter fine-tuning while reducing up to 99% trainable parameters to save memory. Transducer Tuning also remains competitive against other fine-tuning approaches (e.g., LoRA, Prompt-Tuning, Prefix-Tuning) while using only 1.5%-80% of their trainable parameters. Our findings show that integrating structural and dependency information through Transducer Tuning enables more efficient model adaptation, making it easier for users to adapt large models in resource-constrained settings. keywords Model Adaptation, Fine-Tuning, Code Property Graph, Large Language Model, Graph Neural Network 1 Introduction Large language models have demonstrated promising performance across various software engineering tasks, such as code generation ( Zan \BOthers. \APACyear2023 ) , code summarization ( Z. Zheng \BOthers. \APACyear2023 ; Wan \BOthers. \APACyear2024 ) , and code repair ( Xia \BOthers. \APACyear2023 ; Jin \BOthers. \APACyear2023 ) . ”Pretrain, then fine-tune” is a common practice to adapt the models for downstream tasks ( Devlin \BOthers. \APACyear2019 ; Feng \BOthers. \APACyear2020 ; Guo \BOthers. \APACyear2021 ; Ahmad \BOthers. \APACyear2021 ; Wang \BOthers. \APACyear2023 ) . These models are often pretrained by large organizations on huge corpora, learning code syntax, semantics, and patterns ( Shi \BOthers. \APACyear2023 ; Z. Han \BOthers. \APACyear2024 ) . Users can then adapt the pretrained models to their specific needs through fine-tuning on domain-specific data. Model fine-tuning often works by iteratively updating the models’ parameters through gradient descent ( Rumelhart \BOthers. \APACyear1986 ) , using input-output examples for specific downstream tasks. Through this optimization process, the model’s parameters gradually align with the patterns and objectives of the downstream task, leading to improved task-specific performance. One significant challenge in fine-tuning large language models is the substantial GPU memory required to store gradients and optimizer states during parameter updates. The memory demand increases with the models’ parameter counts. For example, in our preliminary experiment, we fine-tuned two variants of CodeT5+ ( Wang \BOthers. \APACyear2023 ) on the assert generation training dataset ( Watson \BOthers. \APACyear2020 ) (approximately 5K examples) using identical settings and compared their peak memory consumption: The 220-million parameter variant required 12.1GB of GPU memory, while the larger 770-million parameter variant consumed 37.7GB. Such memory demand challenge is increasingly pronounced as recent models ( T. Zheng \BOthers. \APACyear2024 ; Luo \BOthers. \APACyear2024 ; Muennighoff \BOthers. \APACyear2024 ) continue to grow, often containing orders of magnitude more parameters than their predecessors from just a few years ago ( Feng \BOthers. \APACyear2020 ; Guo \BOthers. \APACyear2021 ; Devlin \BOthers. \APACyear2019 ) . Prior studies have proposed various efficient adaptation techniques, such as Adapter-based ( Bapna \BBA Firat \APACyear2019 ; Houlsby \BOthers. \APACyear2019 ; Pfeiffer, Vulic\BCBL \BOthers. \APACyear2020 ; Pfeiffer, Rücklé\BCBL \BOthers. \APACyear2020 ; Pfeiffer \BOthers. \APACyear2021 ; H. Liu \BOthers. \APACyear2022 ; E.J. Hu \BOthers. \APACyear2022 ; Hyeon-Woo \BOthers. \APACyear2022 ; Yeh \BOthers. \APACyear2024 ; Ponti \BOthers. \APACyear2022 ; Kopiczko \BOthers. \APACyear2024 ) and Prompt-based ( Li \BBA Liang \APACyear2021 ; Lester \BOthers. \APACyear2021 ; X. Liu \BOthers. \APACyear2022 ) methods to address the memory requirement challenge. Adapter-based methods introduce additional trainable parameters into a model and update only these parameters during the fine-tuning stage rather than the entire model. On the other hand, embedding-based methods modify the output of the embedding layer before feeding it to the initial encoder/decoder layer of the model. The existing efficient fine-tuning methods for code-related tasks in software engineering face two limitations. First, they involve an inherent trade-off between parameter efficiency and model performance, where reducing trainable parameters can degrade performance compared to full parameter fine-tuning ( J. Liu, Sha\BCBL \BBA Peng \APACyear2023 ) . Second, these methods fail to leverage inherent structural and dependency information in source code as they rely solely on the sequential representation of code. Prior studies have demonstrated that incorporating structural and dependency information can improve model performance compared to learning from code sequences alone ( J. Zhang \BOthers. \APACyear2023 ; J. Liu, Zeng\BCBL \BOthers. \APACyear2023 ; Mi \BOthers. \APACyear2023 ; Allamanis \BOthers. \APACyear2018 ; K. Zhang \BOthers. \APACyear2022 ) . Unlike natural language texts, source code contains well-defined structures, dependencies, and control flows that can be explicitly represented using graphs. These graph-based representations can naturally better capture long-range relationships between program elements than code sequences. For example, connecting variable declarations with their uses or representing control flow relationships that span across many lines. Building on these insights, we propose Transducer Tuning, a new model adaptation and fine-tuning method for our main goal for code-related tasks: to achieve reasonable performance compared to full parameter fine-tuning by leveraging rich code structural and dependency information using much fewer trainable parameters compared to the other fine-tuning methods. Transducer Tuning efficiently adapts large models by leveraging modular neural network layers that minimize trainable parameters while maintaining strong performance. At the core of Transducer Tuning is the Transducer, which enriches model inputs using Code Property Graphs (CPGs) ( Yamaguchi \BOthers. \APACyear2014 ) that capture rich code properties including syntactic structures, control flows, and data dependencies. The Transducer comprises two key components: Graph Vectorization Engine (GVE) and Attention-based Fusion Layer (ABFL). GVE extracts CPGs from input source code and converts dependency information into graph feature vectors. ABFL then fuses these vectors with initial code embeddings from an existing model, enriching them with structural and dependency information. Transducer Tuning optimizes the Transducer for various downstream tasks, improving the input embeddings for the model without fine-tuning the model itself. We have evaluated Transducer Tuning on three downstream tasks: code summarization, assert generation, and code translation. The results demonstrate that Transducer Tuning achieves comparable performance to full fine-tuning while reducing up to 99% trainable parameters to save GPU memory. Also, Transducer Tuning delivers competitive results against efficient fine-tuning methods like LoRA ( E.J. Hu \BOthers. \APACyear2022 ) , Prompt-Tuning ( Lester \BOthers. \APACyear2021 ) , and Prefix-Tuning ( Li \BBA Liang \APACyear2021 ) , while using only 1.5%-80% of their trainable parameters. Our key contributions are as follows: • A novel adaptation method, Transducer Tuning, that effectively adapts large language models for downstream code-related tasks using Code Property Graphs (CPGs) while minimizing trainable parameters and maintaining strong performance. We make the code available at https://github.com/imamnurby/Transducer-Tuning . •