When Benchmarks Talk: Re-Evaluating Code LLMs with Interactive Feedback

Jane Pan, Ryan Shar, Jacob Pfau, Ameet Talwalkar, He He, Valerie Chen · 2025

Read When Benchmarks Talk: Re-Evaluating Code LLMs with Interactive Feedback by Jane Pan, Ryan Shar, Jacob Pfau, Ameet Talwalkar, He He, Valerie Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-18413.txt.

Introduction

Coding assistants are usually graded on fixed problem sets, yet real programming is a back-and-forth with a user. A six-author team builds an interactive test in 2025. Problems are altered so the model must ask a simulated user for missing details before it can solve them. They run ten models across three datasets and compare the new rankings with the static ones.

Rankings often shift once interaction enters, even though models cope fairly well with feedback that contains mistakes. The type of feedback also changes behavior, steering models toward polished code or working code. The authors argue that judging assistants through dialogue comes closer to real use.

Catalog record

CREATORJane Pan, Ryan Shar, Jacob Pfau, Ameet Talwalkar, He He, Valerie Chen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE64 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-18413

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

When Benchmarks Talk: Re-Evaluating Code LLMs with Interactive Feedback Jane Pan, Ryan Shar, Jacob Pfau, Ameet Talwalkar, He He, Valerie Chen Abstract Programming is a fundamentally interactive process, yet coding assistants are often evaluated using static benchmarks that fail to measure how well models collaborate with users. We introduce an interactive evaluation pipeline to examine how LLMs incorporate different types of feedback in a collaborative setting. Specifically, we perturb static coding benchmarks so that the code model must interact with a simulated user to retrieve key information about the problem. We find that interaction significantly affects model performance, as the relative rankings of 10 10 models across 3 3 datasets often vary between static and interactive settings, despite models being fairly robust to feedback that contains errors. We also observe that even when different feedback types are equally effective with respect to performance, they can impact model behaviors such as (1) how models respond to higher- vs. lower-quality feedback and (2) whether models prioritize aesthetic vs. functional edits. Our work aims to “re-evaluate” model coding capabilities through an interactive lens toward bridging the gap between existing evaluations and real-world usage. Figure 1: While most existing benchmarks statically evaluate LLM coding capabilities, code LLMs are used interactively in practice. We introduce an evaluation pipeline that evaluates code models in an interactive setting (top). Across three datasets, such as LiveCodeBench (bottom), we find that interactively evaluating models with different feedback types ( Code Feedback , Query Rephrasing , Paragraph , and Sentence ) leads to different rankings when compared to static evaluation. 1 Introduction Figure 2: Overview of our interactive pipeline for coding evaluation. (A) We obfuscate the input of existing fully specified datasets to reflect how programmers tend to underspecify requests to LLMs (e.g., via docstrings or comments) in practice. (B) As developers may interact with models in a variety of ways, we explore 4 different feedback types and introduce a pipeline that mimics the iterative refinement loop that programmers often use with chat models, (C) where the code model generates a solution using feedback on its previous solution, (D) and the user provides updated feedback to the code model. Programming with a language model is a highly collaborative process, where developers interact with code models to provide updated information about initially underspecified requests or critique the output of the code model. Thus, giving and receiving feedback are critical elements of the process in which programmers use code models Chidambaram et al. (2024) . For example, chat interfaces like ChatGPT ( OpenAI, 2022 ) or the chat panel of Github Copilot Github (2022) facilitate multi-turn conversations in which programmers can iteratively refine a piece of code by providing additional context and details to the LLM ( Kalla et al., 2023 ; Xiao et al., 2023 ) . Despite the popularity of these tools, existing static benchmarks that measure task performance often rely on a simple input-output configuration, where the question is well defined and the model is asked to generate the whole completion in one shot ( Chen et al., 2021 ; Austin et al., 2021 ; Jain et al., 2024 ; White et al., 2024 ) . While this relatively simplistic setting is scalable and enables efficient evaluation, it does not capture how developers realistically use models to write code ( Mozannar et al., 2024 ) . While recent benchmarks have begun to explore how interactive settings can lead to performance gains in coding applications Wang et al. (2023b) , they assume a single form of natural language feedback. In practice, developers provide many forms of feedback when implementing code ( Chidambaram et al., 2024 ) , which can range from binary feedback on the correctness of the code to suggesting direct changes to the code. Building evaluations of programming assistants that more closely mimic this setting enables a better understanding of model behavior and potential pitfalls in the interactive setting, such as model capability with respect to processing feedback, the effects of different feedback types, or model robustness. We bridge the gap between existing evaluations and real-world use cases by benchmarking how different feedback types impact model behavior in a simulated interactive setting (Figure 1 , top). We propose an evaluation method that transforms a static coding benchmark into an interactive, collaborative one (Figure 2 ). The pipeline components include input obfuscation to create underspecified problems to induce collaboration, a simulated user for scalability, and multiple types of feedback ( Code Feedback , Query Rephrasing , Paragraph , and Sentence ). Across 10 coding models ( 6 open and 4 closed) and 3 coding benchmarks, we find that relative performance between models often changes between static and interactive settings (Figure 1 , bottom), suggesting that models perform differently in the static vs. interactive settings. Beyond performance-based metrics, we analyze important components of model-user interactions, including feedback quality and code model steerability. We use these insights to investigate the effects of different feedback types. For example, we find that Paragraph and Code Feedback tend to lead to the highest performance boost compared to other feedback types (e.g., Sentence or Query Rephrasing ). However, when considering the effect of feedback quality on performance, we find that unlike Paragraph , Code Feedback ’s higher-quality feedback makes output worse more frequently than its lower-quality counterpart for stronger models. Furthermore, Paragraph leads to more surface-level edits than Code Feedback , whereas users may prefer variation in model behavior to be robust to changes in feedback type. Our work provides a new approach to investigating the downstream ramifications of different interactive programming settings and their effects on model behavior. We open-source our evaluation pipeline 1 1 1 Our code is publicly available at https://github.com/janepan9917/WhenBenchmarksTalk/ . , which makes it easy to add static benchmarks and turn them into interactive ones, to facilitate the evaluation of more models and datasets. 2 Methodology Figure 2 provides an overview of our pipeline for interactive evalution, which is driven by an iterative refinement loop in which the user model and code model interact. At each step of the loop, the code model is given the obfuscated programming question in natural language, the code from the previous attempt, and the user model’s feedback on the previous attempt. Each iterative refinement loop lasts 5 steps for each question, terminating early if a correct solution is reached in less than 5 steps. 2.1 Dataset Transformation Our pipeline is designed to transform static coding questions into interactive ones. We describe our selection criteria for the datasets used in this work, as well as the input obfuscation protocol that enables collaboration between the user and code model. Input obfuscation. To ensure that the user and code model collaborate, we obfuscate the input given to the code model to remove critical information from the input (Figure 2 A). This induces an information asymmetry, akin to that which might exist in practice, that forces the code model to rely on feedback from the user to recover key details about the full problem specification. We underspecify the input to the code model by using Sonnet-3.5 to summarize the original questions, which often contain significant detail about desired behaviors and potential edge cases, into one-sentence summaries. To illustrate

Read the complete text