TurnWise : The Gap between Single- and Multi-turn Language Model Capabilities
Victoria Graf, Valentina Pyatkin, Nouha Dziri, Nathan Lambert, Hannaneh Hajishirzi · 2026
Read TurnWise : The Gap between Single- and Multi-turn Language Model Capabilities by Victoria Graf, Valentina Pyatkin, Nouha Dziri, Nathan Lambert, Hannaneh Hajishirzi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-16759.txt.
Introduction
Graf and co-authors study how well language models handle long, multi-turn chats. Most training and test data cover single turns only. They build TurnWiseEval, a benchmark that compares multi-turn skill against matched single-turn tasks. They also build TurnWiseData, a pipeline that makes synthetic multi-turn training data at scale. In tests with Olmo 3, multi-turn training data proved vital for strong chat results. The authors report that just 10,000 such chats in post-training lifted scores by 12 percent.
Catalog record
| CREATOR | Victoria Graf, Valentina Pyatkin, Nouha Dziri, Nathan Lambert, Hannaneh Hajishirzi |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 27290 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.16759
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
TurnWise : The Gap between Single- and Multi-turn Language Model Capabilities Victoria Graf, Valentina Pyatkin, Nouha Dziri, Nathan Lambert, Hannaneh Hajishirzi Abstract Multi-turn conversations are a common and critical mode of language model interaction. However, current open training and evaluation data focus on single-turn settings, failing to capture the additional dimension of these longer interactions. To understand this multi-/single-turn gap, we first introduce a new benchmark, TurnWiseEval , for multi-turn capabilities that is directly comparable to single-turn chat evaluation. Our evaluation isolates multi-turn specific conversational ability through pairwise comparison to equivalent single-turn settings. We additionally introduce our synthetic multi-turn data pipeline TurnWiseData which allows the scalable generation of multi-turn training data. Our experiments with Olmo 3 show that training with multi-turn data is vital to achieving strong multi-turn chat performance, and that including as little as 10k multi-turn conversations during post-training can lead to a 12% improvement on TurnWiseEval . 1 Introduction Large language models (LLMs) are increasingly deployed to real-world users for free-form, multi-turn conversations. However, many training and evaluation methodologies continue to rely primarily on single-turn data, creating a mismatch between real-world usage and current paradigms. Open language modeling recipes 7 ; 11 focus on and are primarily comprised of single-turn data. Where multi-turn data is included in these recipes, it is added without systematic consideration of its contribution to conversational abilities. Moreover, collecting scalable multi-turn conversational data proves difficult due to the challenges of user simulation 10 and the expensive nature of large-scale manual collection of user interactions 18 . While gaps in multi-turn conversational ability have been documented 6 ; 15 , current measurements of this gap are incomplete and insufficiently informative. Existing multi-turn dialogue benchmarks (e.g., MTBench ( 19 ) , MTEval ( 5 ) ) are being saturated, making it difficult to assess what remains of the multi-/single-turn performance gap. Additionally, many conversational benchmarks do not distinguish whether low model performance in multi-turn conversations stems from multi-turn specific performance gaps or from general model ability for similar tasks 19 ; 1 . To improve evaluation of the multi-turn conversational abilities of LLMs, we first introduce a new benchmark, TurnWiseEval (Section 2 ), which isolates multi-turn abilities from other confounding signals. Unlike previous multi-turn evaluations, our approach grounds evaluation in pairwise comparisons between responses to multi-turn and equivalent single-turn conversations (Figure 1 ). This methodology isolates multi-turn performance beyond task difficulty, minimizing sensitivity to other LLM capabilities such as factuality and knowledge and providing a more focused assessment of conversational abilities. Additionally, we use controlled conversational context to further separate multi-turn ability from dependencies on conversational state and to enable better comparisons of results between models. We show that even frontier models like GPT-5 Chat can underperform on TurnWiseEval and that open model recipes lag behind. Figure 1: TurnWiseEval -Absolute example. Model responses to a multi-turn conversation are compared to a strong model’s response on an equivalent single-turn prompt. Our evaluation metric is the win-rate of the evaluated model as determined by an LM judge. To address this gap between chat performance of LLMs in single- versus multi-turn conversations, we introduce a scalable method, TurnWiseData , for creating synthetic multi-turn context on arbitrary single-turn conversations (Section 3 ). We use data generated with our methodology to analyze how incorporating multi-turn data in training can improve conversational performance with Olmo 3 11 (Section 4 ). Through these experiments, we show that adding even a small number of synthetic multi-turn conversations in training can significantly reduce the gap with single-turn performance by up to 12%, demonstrating the importance of multi-turn conversations in training. 2 The TurnWiseEval Benchmark In this section we detail the motivation, design, and resulting data inside the TurnWiseEval benchmark. We then show that TurnWiseEval measures multi-turn conversational abilities for models with a range of single-turn abilities and that multi-turn conversational ability is a distinct dimension of evaluation to consider beyond single-turn chat. 2.1 Evaluation Design To isolate multi-turn conversational ability beyond single-turn task abilities, we use pairwise comparisons of responses to single- and multi-turn conversations. We consider two comparison settings: • TurnWiseEval -Absolute. In our first evaluation setting, we measure “absolute” multi-turn performance by comparing an evaluated model’s completions of multi-turn conversations to responses from a strong model, specifically GPT-4 ( 1106-preview ), on equivalent single-turn prompts (Figure 1 ). These reference responses represent a strong baseline response to the single-turn setting. This setting uses length-controlled win rate as the primary metric and is intended for easy interpretation of TurnWiseEval scores across models. • TurnWiseEval -Self. Our second setting further isolates multi-turn ability by comparing an evaluated model’s responses to multi-turn conversations to its own responses to corresponding single-turn conversations. This setting uses discrete win rate as the primary metric. The single-turn response here represents the model’s own baseline ability for the task instruction. Win rates below 50% imply that a model’s responses to multi-turn conversations are consistently worse than their single-turn counterparts, indicating degraded performance in the multi-turn setting beyond what can be attributed to general task ability. In best-case scenarios, models can leverage context from previous turns to improve their final response, leading to win rates above 50%. For pairwise judgments in TurnWiseEval , we use GPT-4.1 as judge with the judgment prompt from AlpacaEval (Appendix A ). 2.2 Multi-turn Conversation Curation Our evaluation captures a common pattern in real-world interactions where users explore related topics across multiple turns. Our data curation is informed by manual inspection of multiple real-world data sources. To ground our design, we read and analyzed multi-turn interactions from the Ai2 Playground model hosting platform (Appendix B ). Sourcing Prompts. To build our TurnWiseEval dataset, we create multi-turn conversations of up to eight user turns where user turns pose related questions to each seed instruction drawn from AlpacaEval. Each user turn is synthetically generated and contains a self-contained, topically-related request, allowing the assistant to optionally leverage earlier turns for richer responses without removing any necessary contextual information from the single-turn comparison conversation. To control the information seen by the evaluated model in the conversational context so far, we include assistant turns generated by the model used for the reference single-turn completions in TurnWiseEval -Absolute to create conversational states for each evaluation sample. We choose not to go beyond eight user turns to avoid confounding our evaluation with varying long-context abilities 1 1 1 Conversations to be a maximum of \sim 4.7k tokens . We manually ensured conversational coherence and topical consistency for generated user turns in TurnWiseEval and verified a random subset of 100 conversations (Appendix B ). Comparison to AlpacaEval. For reliable comparison to widely-used chat evaluation settings, our final evaluation instructions are sourced from AlpacaEval 9 with additional context created as earlier turns in the conversations. Thus, there exists a single-turn complement to each TurnWiseEval conversation in AlpacaEval, making comparison of these metrics 1:1. This relation with single-turn evaluation allows direct comparison of a model’s multi- and single-turn abilities. Comparison between TurnWiseEval -Absolute and AlpacaEval performance can isolate difference in performance due to the multi-turn setting rather than general model abilities. 2 2 2 All AlpacaEval results reported here refer to length-controlled AlpacaEval with GPT 4.1 judge (gpt-4.1-2025-04-14). This matches the judge used for TurnWiseEval and