In-Context Learning as an Effective Estimator of Functional Correctness of LLM-Generated Code Conference: Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 13–18, 2025; Padua, Italy Proceedings of the 48th International ACM SIGIR Co
Susmita Das, Madhusudan Ghosh, Priyanka Swami, Debasis Ganguly, Gul Calikli · 2025
Read In-Context Learning as an Effective Estimator of Functional Correctness of LLM-Generated Code Conference: Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 13–18, 2025; Padua, Italy Proceedings of the 48th International ACM SIGIR Co by Susmita Das, Madhusudan Ghosh, Priyanka Swami, Debasis Ganguly, Gul Calikli (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-05200.txt.
Introduction
Susmita Das and co-authors ask whether a language model can judge its own computer code without running any tests, in a 2025 paper. When software teams generate code with AI help, they often get several candidate solutions and no test suite to check them against. Someone still has to pick the best one. The authors borrow an idea from search science: predicting which results are good before anyone clicks.
Their trick is to show the model a few examples of correct code as part of the prompt, a method called in-context learning. Those examples sharpen the model's sense of what working code looks like. In their tests, the few-shot setup improved older prediction methods and a bare zero-shot judge alike. The result points to cheaper quality control for AI-written software, where ranking candidates well matters as much as writing them.
Catalog record
| CREATOR | Susmita Das, Madhusudan Ghosh, Priyanka Swami, Debasis Ganguly, Gul Calikli |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 20 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-05200
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
In-Context Learning as an Effective Estimator of Functional Correctness of LLM-Generated Code Conference: Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 13–18, 2025; Padua, Italy Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’25), July 13–18, 2025, Padua, Italy DOI: 10.1145/3726302.3730212 ISBN: 979-8-4007-1592-1/2025/07 CCS: Information systems Information retrieval CCS: Computing methodologies Machine learning CCS: Computing methodologies Natural language processing Susmita Das, Madhusudan Ghosh, Priyanka Swami, Debasis Ganguly, Gul Calikli Abstract . When applying LLM-based code generation to software development projects that follow a feature-driven or rapid application development approach, it becomes necessary to estimate the functional correctness of the generated code in the absence of test cases. Just as a user selects a relevant document from a ranked list of retrieved ones, a software generation workflow requires a developer to choose (and potentially refine) a generated solution from a ranked list of alternative solutions, ordered by their posterior likelihoods. This implies that estimating the quality of a ranked list – akin to estimating “relevance” for query performance prediction (QPP) in IR – is also crucial for generative software development, where quality is defined in terms of “functional correctness”. In this paper, we propose an in-context learning (ICL) based approach for code quality estimation. Our findings demonstrate that providing few-shot examples of functionally correct code from a training set enhances the performance of existing QPP approaches as well as a zero-shot-based approach for code quality estimation. † † cc-license: by 0 0 footnotetext: Copyright is held by the owner/author(s). Publication rights licensed to ACM. This is the author’s version of the work. It is posted here for your personal use. Not for redistribution. The definitive version was published in Proceedings of SIGIR ’25, https://doi.org/10.1145/3726302.3730212 . 1. Introduction Large language models (LLMs) can model semantics in an abstract and general manner without any task-specific training ( Radford et al. 2018 ; Brown et al. 2020 ; Arora et al. 2022 ; Weidinger et al. 2022 ) , making them effective zero-shot predictors for a wide range of different downstream tasks, such as assessing reviews ( Mysore et al. 2023 ) , answering questions ( Li et al. 2023 ) , recommending relevant documents ( Pradeep et al. 2023 ) etc. Figure 1 . Schematic workflow of our proposed few-shot code quality estimation: ① and ② show how training data is prepared offline by first generating solution lists for each problem and then executing test cases (TCs) on these solutions to construct two separate indexes for correct and incorrect solutions. ③ shows how problem-solution pairs that are most similar to the current input are retrieved and then included as ICL examples to facilitate prediction of code quality. In the context of software development, LLMs have been shown to be a valuable resource for a wide range of software development tasks, such as generating a code snippet from a natural language problem description ( Athiwaratkun et al. 2023 ; Yin et al. 2022 ) , or complete a partially written software code to accomplish a specific task ( Chen et al. 2021 ; Wu et al. 2024a ; Guo et al. 2023 ) , or even predict the quality of the generated code ( Zhuo 2024 ) . Such automatically generated code by LLMs can prove to be particularly beneficial under a “human-in-the-loop” setting, e.g., inexperienced software developers may find it easier to post-edit the code generated for a particular functionality without writing code from scratch ( Wu et al. 2024a ) , or this may help to reduce the code review time among software development teams ( Wu et al. 2024b ; Dakhel et al. 2024 ; Lin et al. 2023 ) . On the flip side, LLM-generated software code may lead to risks, such as errors in terms of functional correctness ( Dinh et al. 2023 ) , security vulnerabilities ( Asare et al. 2024 ) , and integration issues due to their limited understanding of a programming context ( Rognerud and Hannay 2009 ) . While unit tests can be applied in test-driven development (TDD) ( Karac et al. 2019 ) to check the functional correctness of code, one of the restrictions is that the test-cases need to be written before the actual implementation of a problem description. In such situations, estimating the functional correctness of code without the presence of any unit test-cases can potentially ensure an effective and efficient software development life-cycle ( Schäfer et al. 2023 ) . You are an experienced software engineer. Your task is to check the functional correctness of code for the given problem statement. Generate ‘yes’ if the code is functionally correct (i.e., code meets the problem’s requirements), otherwise generate ‘no’. Figure 2 . Prompt instruction for code quality estimation with a sample input problem-solution pair. A common workflow in LLM-based software code generation involves a user (software developer) prompting an LLM to generate a number of alternative solutions ranked by the posterior likelihoods of generation. As a subsequent step, a solution from the list is eventually selected by a manual choice of what appears to be correct . Drawing an analogy between “ functional correctness ” and “ relevance ”, this is similar to the a user’s task of finding the first relevant document by scanning a ranked list of documents. Obviously, as per the analogy, the tasks become more efficient if relevant documents are retrieved towards the top ranks, or correct solutions appear towards the top of the list - or in other words - an LLM’s generation posteriors correlate well with correctness. A code quality estimator should, therefore, seek to estimate the correlation between the posteriors of the solutions and the correctness of each, i.e., estimate if correct solutions appear before the incorrect ones in a generated list of solutions. Again, drawing the IR analogy, this task is somewhat similar to the standard task of query performance prediction (QPP) ( Shtok et al. 2010 ; Shtok et al. 2012 ; Ganguly et al. 2022 ; Datta et al. 2023 ) in IR - which involves estimating the quality of a ranked list of documents retrieved for a query - a high quality indicating that a relevant document is retrieved at a better rank than a non-relevant one. This paper leverages the semantic capabilities of LLMs for the automatic detection of code quality . While prior research ( Zhuo 2024 ) has demonstrated the effectiveness of zero-shot LLM inference as a code quality estimator, a key limitation of this approach is that it does not utilize labeled examples. We hypothesize that incorporating labeled examples can enhance predictive performance, as has been widely observed in other natural language processing (NLP) tasks, where in-context learning (i.e., the inclusion of labeled examples) consistently outperforms zero-shot inference ( Li et al. 2022 ; Li et al. 2023 ; Tang et al. 2023 ; Mysore et al. 2023 ; Pornprasit and Tantithamthavorn 2024 ) . 2. Proposed Few-shot Code Quality Estimator Figure 1 shows a schematic overview of our proposed approach. In this section, we describe the details of each step involved. Code Generation . The input to a code