Selective Shot Learning for Code Explanation
Paheli Bhattacharya, Rishabh Gupta · 2024
Read Selective Shot Learning for Code Explanation by Paheli Bhattacharya, Rishabh Gupta (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-12852.txt.
Introduction
Paheli Bhattacharya and Rishabh Gupta compare in 2024 methods for choosing few-shot examples to help language models explain code. Good examples boost explanation quality, but selection methods had been tested mainly on closed models. Syntax of the programming language was also often ignored in past work.
They benchmark open-source code models across two data sets. Their new method selects examples using named entity information from the code. Results show it beats leading token and embedding methods. The study claims to be the first systematic benchmark of open code models for this selection task.
Catalog record
| CREATOR | Paheli Bhattacharya, Rishabh Gupta |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-12852
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Selective Shot Learning for Code Explanation Paheli Bhattacharya, Rishabh Gupta Abstract Code explanation plays a crucial role in the software engineering domain, aiding developers in grasping code functionality efficiently. Recent work shows that the performance of LLMs for code explanation improves in a few-shot setting, especially when the few-shot examples are selected intelligently. State-of-the-art approaches for such Selective Shot Learning (SSL) include token-based and embedding-based methods Geng et al. (2024) . However, these SSL approaches have been evaluated on proprietary LLMs, without much exploration on open-source Code-LLMs. Additionally, these methods lack consideration for programming language syntax. To bridge these gaps, we present a comparative study and propose a novel SSL method ( S S L n e r SSL_{ner} ) that utilizes entity information for few-shot example selection. We present several insights and show the effectiveness of S S L n e r SSL_{ner} approach over state-of-the-art methods across two datasets. To the best of our knowledge, this is the first systematic benchmarking of open-source Code-LLMs while assessing the performances of the various few-shot examples selection approaches for the code explanation task. 1 Introduction Code understanding and explanation MacNeil et al. (2023) , also known as code summarization Ahmed and Devanbu (2022) ; Iyer et al. (2016) and code comment generation Hu et al. (2018) ; Sharma et al. (2022) , is an important problem in the domain of software engineering. It involves generating concise and informative explanations for pieces of source code. This provides the developers with a quick understanding of its functionality aiding in code maintenance, search and retrieval Ye et al. (2020) . For programmers new to a particular programming language, code summaries serve as valuable documentation to familiarize them with the new environment efficiently MacNeil et al. (2023) . Automating the task of code documentation through comments and explanations can therefore prove beneficial in many ways. Large Language Models (LLMs) have proven their efficiency in a variety of NLP tasks. LLMs have shown promising results in several software engineering tasks like code generation Li et al. (2023) ; Yin et al. (2023) , translation Huang et al. (2023) , test case generation Schäfer et al. (2023) and code explanation Geng et al. (2024) ; Ahmed and Devanbu (2022) ; MacNeil et al. (2023) ; Bhattacharya et al. (2023) ; Ahmed et al. (2024) . While using LLMs for the code explanation task, It has been shown that few-shot prompting achieves better results than zero-shot prompting Geng et al. (2024) ; Ahmed et al. (2024) . Hence, selecting the demonstration examples for few-shot learning is an important design criteria. We use the term Selective Shot Learning (SSL) when examples for few-shot prompting are chosen intelligently, instead of being random. SSL approaches for code explanation include token-based and embedding-based methods Geng et al. (2024) without taking into account the language syntax. Recent work in the area of code explanation have only considered proprietary LLMs like Codex Geng et al. (2024) ; MacNeil et al. (2023) , Code-davinci-002 Ahmed and Devanbu (2022) , Text-Davinci-003 Ahmed et al. (2024) , GPT-3 MacNeil et al. (2023) and GPT-3.5-turbo Ahmed et al. (2024) . However there is a huge gap in proper benchmarking and performance evaluation of several competing, open-source Code-LLMs like CodeLlama Rozière et al. (2023) , StarCoder Li (2023) for the code explanation task. To this end, the contributions of the paper are: \bullet We explore several open-source Code-LLMs for the task of code explanation, across two datasets covering different levels of code explanations (inline and method-level). To the best of our knowledge, this is the first systematic attempt at benchmarking the task. We make the dataset and code publicly available at https://github.boschdevcloud.com/HXT2KOR/code-explanation . \bullet We assess the performance of several selective-shot learning approaches, including token-based and embedding-based approaches. Additionally we propose a novel Selective-shot Learning method using NER ( SSL_{ner} ) that includes code-based entity information for examples selection. \bullet We draw several interesting insights – for e.g., we find that the performance of the medium-sized LLMs (StarCoder 15B) increase more rapidly compared to the larger-sized LLM (CodeLlama 34B) and SSL_{ner} to be the best performing SSL approach and being interpretable. 2 Related Work The Code Explanation MacNeil et al. (2023) task is a well studied problem in the domain of software engineering Haiduc et al. (2010) ; Moreno et al. (2013) ; Eddy et al. (2013) . With the advent of deep learning, methods combining neural architectures Cai et al. (2020) ; Ahmad et al. (2020) ; Sharma et al. (2022) along with software engineering approaches like AST trees Hu et al. (2018) have been proposed. Large Language Models have shown exceptional performance in a plethora of NLG tasks Yang et al. (2023) . The zero-shot and few-shot capabilities of these model make them highly adaptable to many NLP tasks. Generic, open-source LLMs like LLama-2 Touvron et al. (2023) , Alpaca Taori et al. (2023) are trained on open internet datasets. CodeLLMs such as StarCoder Li (2023) , CodeUp Jiang and Kim (2023) , CodeLlama Rozière et al. (2023) and Llama-2-Coder Manuel Romero (2023) have been either trained or fine-tuned on code-specific datasets containing source codes covering around 80 + programming languages. The Large Language Models , when used for the Code explanation task, has shown some encouraging results. The recent approaches MacNeil et al. (2023) ; Geng et al. (2024) ; Ahmed and Devanbu (2022) ; Ahmed et al. (2024) demonstrate that the LLMs performs better in the few-shot setup when good examples of the task are provided. Hence, deciding the relevant examples is an important design criteria while using LLMs for the code explanation task. Existing approaches involve token-based, embedding-based Geng et al. (2024) and BM-25 along with repository information, data flow graph, AST tree etc. Ahmed et al. (2024) . However, these methods do not explore the efficacy of CodeLLMs. There has been systematic evaluations of transformer models (CodeT5 and CodeBERT) on code summarization Mondal et al. (2023) , LLMs on code search Diera et al. (2023) and non-CodeLLMs like GPT, Bard for code documentation generation Dvivedi et al. (2024) . To the best of our knowledge, this is the first work that performs an exhaustive analysis of four open-source CodeLLMs across two datasets and three SSL approaches for the task of code explanation. The SSL approaches explored in the paper do not incorporate secondary statistical analysis tools like AST tree, data-flow graph etc. like Ahmed et al. (2024) 3 Dataset Table 1: Statistics of the two datasets – CoNaLa and TLC – experimented within this paper. CoNaLa contains inline level codes written in Python. TLC contains function level codes written in Java. TLC is further subdivided into 5 different subdomains (code intents). CoNaLa contains shorter codes compared to TLC. The average length of the comments are comparable for the two datasets. Code Level Language Dataset Sub-domain # Samples Average length train test train test Code Comment Code Comment Inline Python CoNaLa – 1666 350 13.92 14.68 14.35 14.06 Function Java TLC How-to-use 838 37 75.14 12.75 65.41 12.97 Property 5,016 292 69.96 12.86 73.5 12.59 Why 5,935 297 82.29 12.47 83.38 12.34 How-it-is-done 11,478 507 89.5