What Drives Interactive Improvement from Feedback?

Bartłomiej Cupiał 1,2,*, Jan Łojek 1, Mikołaj Garstecki 1, Alicja Ziarko 1,2,3, Piotr Miłoś 1,4,5 · 2026

Read What Drives Interactive Improvement from Feedback? by Bartłomiej Cupiał 1,2,*, Jan Łojek 1, Mikołaj Garstecki 1, Alicja Ziarko 1,2,3, Piotr Miłoś 1,4,5 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-30774.txt.

Introduction

Bartłomiej Cupiał and four colleagues ask when feedback makes a language agent better, rather than just busier. In multi-turn settings, accuracy often rises over turns, but the rise might come from resampling, format fixes or extra computation rather than from the feedback itself. Their controlled student-teacher protocol separates these effects across four benchmarks, from Olympiad maths to coding contests.

Thirteen open-weight models played student and teacher roles. The team compared external feedback, self-feedback and unguided self-refinement, varying history, difficulty and the teacher's access to privileged information. Multi-turn gains often turned out not to be feedback use at all: self-generated feedback added little beyond unguided retrying. Strong external teachers produced much larger feedback-specific gains. Interaction matrices showed the student's ability to act on feedback mattered more than the teacher's identity. The authors urge evaluating feedback agents against repeated-attempt baselines.

Catalog record

CREATORBartłomiej Cupiał 1,2,*, Jan Łojek 1, Mikołaj Garstecki 1, Alicja Ziarko 1,2,3, Piotr Miłoś 1,4,5
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.30774

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

What Drives Interactive Improvement from Feedback? Bartłomiej Cupiał 1,2,*, Jan Łojek 1, Mikołaj Garstecki 1, Alicja Ziarko 1,2,3, Piotr Miłoś 1,4,5 Abstract We study when natural-language feedback produces improvement beyond the gains obtainable from repeated attempts alone. In multi-turn language agent setting, higher final accuracy can reflect useful feedback, but it can also arise from resampling, format correction, or additional test-time computation. To separate these effects, we introduce a controlled student-teacher protocol across Omni-MATH, Codeforces, BBEH Linguini, and ARC-AGI1, evaluating thirteen open-weight models in both student and teacher roles. We compare external feedback, self-feedback, and unguided self-refinement, while varying interaction history, task difficulty, and teacher access to privileged task information. Across settings, we find that multi-turn improvement is often not evidence of feedback use: self-generated feedback adds little beyond unguided self-refinement, whereas the strongest external teachers produce substantially larger feedback-specific gains, suggesting that useful feedback must provide guidance beyond generic retry. Dense student-teacher interaction matrices further show that interactive gains are driven more by the student’s ability to use feedback than by the teacher’s identity, although teacher choice remains important for a fixed student. These results suggest that feedback-based agents should be evaluated against repeated-attempt baselines, and that ability to act on feedback, not merely feedback availability, is a central bottleneck for interactive improvement. We release our controlled student-teacher evaluation framework at https://j-lojek.github.io/feedback-generation-is-a-bottleneck/. 0 0 footnotetext: Accepted to “RLxF: Reinforcement Learning from World Feedback”. 0 0 footnotetext: * Corresponding author: [email protected] . 1 University of Warsaw 2 AKCES NCBR 3 Princeton University 4 Mistral AI 5 Institute of Mathematics, Polish Academy of Sciences 1 Introduction Figure 1 : Multi-turn trajectories separate mostly by student. While stronger teachers like Gemma4-31B provide noticeable boosts for highly capable students, the dominant vertical separation between the curves is driven by student rather than teacher. Language models (LMs) are increasingly deployed as agents that act, observe, and revise rather than as systems that produce a single isolated answer. Coding agents react to tests and error traces, computer-use agents observe changing interface states, and assistants revise plans in response to user corrections or clarifications. Across these settings, feedback can do more than mark success or failure: it can explain what went wrong, identify missing constraints, and suggest a path to recovery. Improving from feedback is therefore becoming an important capability in its own right: an agent must recognize which parts of the feedback are actionable, preserve the parts of its attempt that are already correct, and revise the parts that caused failure. Studying improvement from feedback requires separating several mechanisms that are usually entangled. A model may improve over turns because it receives useful corrective information, but it may also improve simply because it samples another attempt with more computation. When feedback does help, the limiting factor may lie on either side of the interaction: the teacher may fail to identify the relevant error, or the student may fail to translate a correct diagnosis into a better solution. We therefore distinguish feedback generation from improving from feedback by evaluating separate student and teacher models under controlled interaction histories, teacher-information conditions, and repeated-attempt baselines. This lets us ask not only whether feedback improves performance, but when it improves performance beyond self-refinement, and which part of the feedback system constrains that improvement. We answer these questions in a student-teacher protocol on verifiable tasks. In each episode, a student model attempts a problem. If the attempt is incorrect, a teacher model provides natural-language feedback, and the student revises its answer. The verifier makes every attempt measurable, while the student-teacher split lets us vary the student, the teacher, the interaction budget, the visible history, and the teacher’s access to privileged task information. We instantiate this evaluation across four verifiable reasoning environments: Omni-MATH, Codeforces, BBEH Linguini, and ARC-AGI1, and run student-teacher interaction grids in which each model acts in both roles. Across these settings, we find that multi-turn improvement is often not evidence of feedback use: repeated attempts explain a large share of the gain, self-feedback improves over generic retry inconsistently, the strongest feedback-specific improvements come from high-quality external teachers, and the student’s ability to use feedback explains more variation than the teacher’s identity. In summary, our contributions are as follows: • We introduce a controlled zero-shot student-teacher evaluation for natural-language feedback across four verifiable reasoning environments. • We show that multi-turn improvement is not necessarily feedback use: self-feedback adds little beyond self-refinement, while the best external teachers add substantial feedback-specific gains. • We find that interactive performance is primarily student-dependent: student identity explains substantially more gain variation than teacher identity, although teacher choice can matter for a fixed student. • We show that longer histories and privileged teacher information are selective interventions rather than default improvements, with benefits depending on task and model capability. 2 Related Work Self-refinement and repeated sampling. Large language models are increasingly used as interactive agents that solve tasks over multiple turns rather than through a single generation ( 25 ) . In such settings, models must follow sequential instructions and incorporate new feedback across turns ( 14 ) . However, multi-turn improvement does not necessarily imply that the model has used feedback: later attempts also receive more test-time computation and additional samples. This distinction is especially important because additional computation is known to improve performance through mechanisms such as chain-of-thought prompting ( 27 ) , and Best-of- N sampling ( 26 ) . Related agent systems such as Reflexion use verbal feedback or memories of past failures to improve future attempts ( 22 ) . Our work builds on this line of work by explicitly comparing feedback-conditioned improvement against unguided self-refinement, allowing us to estimate how much of the observed gain is feedback-specific rather than a consequence of repeated attempts. Post-training LMs with teacher feedback. Recent post-training methods increasingly frame model improvement as a student-teacher problem, where a student model learns from signals produced by a stronger teacher, verifier, or environment. Standard approaches include supervised fine-tuning, preference optimization, reinforcement learning from AI feedback, and reinforcement learning with verifiable rewards ( 19 ; 15 ; 20 ; 8 ; 21 ) . These methods are effective, but their feedback signals are often demonstrations, pairwise preferences, scalar rewards, or outcome-level verification. Such signals can improve behavior, but they provide limited information about which part of a student’s attempted solution Figure 2 : (Top) Single Didactic Interaction. We transform single-turn problems into multi-turn didactic interactions. A teacher model, conditioned on optionally privileged information (e.g., the ground-truth solution), provides natural language feedback to a student model without revealing the final answer, guiding it to correct its errors. (Bottom) Multi-turn feedback protocol. Each failed student attempt is converted into teacher feedback for the next attempt. This cycle repeats until the predefined maximum number of turns is reached or the student correctly solves the problem. caused failure. This limitation has motivated recent work on denser teacher supervision, including on-policy distillation, where trajectories are sampled from the student and a teacher provides supervision on the student-generated states ( 17 ) . It has also motivated work on synthetic and interactive data generation, especially as high-quality human-generated training data becomes increasingly limited ( 24 ; 12 ) . In verifiable domains, natural-language feedback

Read the complete text