Building Coding Agents via Entropy-Enhanced Multi-Turn Preference Optimization

Jiahao Yu, Zelei Cheng, Xian Wu, Xinyu Xing · 2025

Read Building Coding Agents via Entropy-Enhanced Multi-Turn Preference Optimization by Jiahao Yu, Zelei Cheng, Xian Wu, Xinyu Xing (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-12434.txt.

Introduction

Jiahao Yu and 3 co-authors propose EntroPO in a 2025 paper on building coding agents. Software tasks require reasoning over large code bases and coordinated tool use across many steps. Benchmarks such as SWE-bench show that current language models still struggle to resolve real-world issues.

Test-time scaling, generating many attempts, can help, but only if the model's outputs are diverse. Standard preference optimization methods can reduce that diversity while aligning outputs with human preferences, and they are usually designed for single-turn tasks. EntroPO adds an entropy focus, preserving variety, and adapts preference optimization to multi-turn, tool-using settings. The goal is a coding agent that both follows preferences and keeps enough diverse attempts for scaling to work.

Catalog record

CREATORJiahao Yu, Zelei Cheng, Xian Wu, Xinyu Xing
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE64 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-12434

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Building Coding Agents via Entropy-Enhanced Multi-Turn Preference Optimization Jiahao Yu, Zelei Cheng, Xian Wu, Xinyu Xing Abstract Test-time scaling (TTS) offers a promising path for solving complex software engineering tasks like SWEBench , yet its efficacy is often bottlenecked by the diversity collapse inherent in standard alignment methods like DPO. To address this, we introduce EntroPO , an entropy-enhanced preference optimization framework tailored for multi-turn, tool-using agents. We formulate the problem as an entropy-regularized MDP to derive the EntroPO -DPO and EntroPO -KTO objectives. Theoretically, we show that this approach promotes diversity specifically among correct solutions, preventing the policy from converging on a narrow set of reasoning paths. To maximize TTS efficiency, we pair this training with a hybrid best-trajectory selection scheme. We validate EntroPO on SWEBench leaderboard, where it achieves state-of-the-art results among open-weight models. Notably, our 30B model ranks 1st on SWEBench-Lite and 4th on SWEBench-Verified , suppressed only by models that are over 10 × \times larger. These results highlight the importance of preserving diversity for effective test-time scaling and establish EntroPO as a robust method for building powerful, interactive coding agents. 1 Introduction Large Language Models (LLMs) have achieved impressive breadth across language understanding, coding assistance, and planning. Yet, they still struggle on complex, multi-step software engineering (SWE) tasks that demand reasoning over large codebases and coordinated tool use ( e.g., search, execution, and patching) ( Yang et al., 2024b ; Wang et al., 2025b ; Xia et al., 2025 ; Antoniades et al., 2025 ; Zhang et al., 2024 ) . A promising line of work that improves performance is test-time scaling (TTS)—sampling more trajectories, searching deeper, and verifying candidates, which can uncover higher-quality solutions on challenging instances ( Snell et al., 2024 ; Beeching et al., ; Yao et al., 2023 ; Xu et al., 2024a ) . However, TTS only helps if the model produces sufficiently diverse candidates to explore meaningfully different solution modes. Recent alignment methods, including Reinforcement Learning from Human Feedback (RLHF) ( Ouyang et al., 2022 ; Ziegler et al., 2019 ; Bai et al., 2022 ) and Direct Preference Optimization (DPO) ( Rafailov et al., 2023 ) , have been observed to inadvertently reduce generation diversity ( Kirk et al., 2024 ; Padmakumar and He, 2024 ; Kim et al., 2025 ; O’Mahony et al., 2024 ; Murthy et al., 2025 ) . This diversity collapse limits the returns of TTS: when a model concentrates probability mass on a narrow set of responses for a given prompt, additional samples become redundant, and deeper search yields diminishing marginal gains. While recent work has attempted to adapt preference optimization to multi-turn tool trajectories ( e.g., M-DPO ( Xiong et al., 2025 ) ), these methods often fail to counteract the “winner-take-all” dynamics of preference learning, leading to collapsed policies that hinder the exploration required for complex problem solving. Prior efforts to preserve diversity typically target single-turn settings ( Slocum et al., 2025 ; Li et al., 2025 ; Wang et al., 2024 ; Lanchantin et al., 2025 ) or adjust decoding temperatures at inference time ( Renze, 2024 ) . These approaches do not directly address multi-turn, tool-using workflows, where diversity must be maintained throughout the trajectory to encourage exploration of a sequence of tool calls and partial hypotheses. To address this limitation, we introduce EntroPO , an entropy-enhanced preference optimization method for multi-turn SWE agents. Our approach augments the standard preference optimization objective with an entropy regularization term to preserve policy diversity and formulate it as an entropy-regularized Markov Decision Process framework. Based on this framework, we formally derive the EntroPO -DPO and EntroPO -KTO loss functions in the multi-turn setting. Crucially, our theoretical analysis reveals that this entropy-regularized objective promotes diversity specifically among correct solutions: it assigns comparatively larger updates to high-utility trajectories that are underrepresented by the reference policy, effectively counteracting mode collapse. By optimizing over multi-turn interactions, EntroPO aligns the learning process with the sequential nature of coding tasks, teaching the model to build diverse and valid reasoning paths. To maximize the performance gain of TTS, we pair EntroPO with a hybrid best-trajectory selection scheme. We combine (i) a learned verifier model that scores trajectories with (ii) model-free approaches that favor high-quality trajectories ( e.g., passing tests, trajectory steps). This hybrid selector improves sampling effectiveness and amplifies the gains from parallel rollouts. We empirically validate EntroPO across a diverse suite of models from different families and sizes (up to 106B parameters) on SWEBench-Verified ( Chowdhury et al., 2024 ) and SWEBench-Lite ( Jimenez et al., 2024 ) . Our approach achieves state-of-the-art results among open-weight models, with our 30B model ranking 1st on SWEBench-Lite and 4th on SWEBench-Verified (surpassed only by models over 350B, which are 10x larger). Across all evaluations, EntroPO significantly outperforms standard DPO and KTO in the TTS setting. We also empirically demonstrate that our method effectively prevents the diversity collapse of the policy during preference learning, maintaining higher trajectory diversity compared to baselines. These results confirm that the entropy-preserving term is critical for maximizing the returns of test-time compute. Our contributions are threefold: • We propose EntroPO , an entropy-enhanced multi-turn preference optimization method that preserves policy diversity, and derive its corresponding DPO and KTO loss functions. • We theoretically analyze our objective and show that our method promotes diversity over correct solutions, avoiding the convergence on a narrow set of “winning” trajectories common in standard preference learning. • We present state-of-the-art results among open-weight models on SWEBench-Verified and SWEBench-Lite , where TTS yields substantial gains through the diversity of our generated solutions. By addressing the critical challenge of preserving diversity in multi-turn agents, our work paves the way for developing more powerful LLM-based tools capable of tackling real-world software engineering tasks. 2 Related Work LLM Post-training. RLHF has become the standard approach for aligning LLMs with human preferences ( Ouyang et al., 2022 ; Ziegler et al., 2019 ; Bai et al., 2022 ; Schulman et al., 2017 ) , but the PPO-style online approach is computationally intensive, as it requires numerous interactions with a reward model or live environment to generate samples during training ( Xu et al., 2024b ; Wei et al., 2025 ) . To reduce the compute cost, preference learning methods such as DPO ( Rafailov et al., 2023 ) and KTO ( Ethayarajh et al., 2024 ) replace explicit reward modeling and online RL with simpler, reward-free objectives that require far less compute. However, most preference optimization methods remain limited to single-turn interactions. While recent extensions such as M-DPO ( Xiong et al., 2025 ) adapt these objectives for multi-turn tool trajectories, they primarily target mathematical reasoning. In long-context coding tasks, we observe that M-DPO suffers from diversity collapse , hindering the exploration required for complex problem solving ( Golubev et al., 2025 ; Gao et al., 2025 ) . While recent methods ( Slocum et al., 2025 ; Wang et al., 2024 ; Lanchantin et al., 2025 ) have attempted to modify divergences, decouple KL components, or construct diversity-aware preference pairs to better control diversity, these approaches primarily focus on single-turn settings . How to effectively extend these diversity-preserving mechanisms to complex, multi-turn tool-using workflows remains

Read the complete text