CoReflect: Conversational Evaluation via Co-Evolutionary Simulation and Reflective Rubric Refinement
Yunzhe Li, Richie Yueqi Feng, Tianxin Wei, Chin-Chia Hsu · 2026
Read CoReflect: Conversational Evaluation via Co-Evolutionary Simulation and Reflective Rubric Refinement by Yunzhe Li, Richie Yueqi Feng, Tianxin Wei, Chin-Chia Hsu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-12208.txt.
Introduction
Yunzhe Li, Richie Yueqi Feng, Tianxin Wei, and Chin-Chia Hsu grade chat systems in multi-turn talks via co-evolution. Past grading uses fixed rubrics and fixed chat context made by hand. That static form misses the diverse acts that dialogue models show in real talks.
Their frame, called CoReflect in the work, joins simulation and grading in one loop. Chats grow harder as rubrics refine, and rubrics refine as chats reveal gaps. An adaptive loop can catch faults that fixed tests miss. The work aims for more true grades of chat skill.
Catalog record
| CREATOR | Yunzhe Li, Richie Yueqi Feng, Tianxin Wei, Chin-Chia Hsu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-12208
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CoReflect: Conversational Evaluation via Co-Evolutionary Simulation and Reflective Rubric Refinement Yunzhe Li, Richie Yueqi Feng, Tianxin Wei, Chin-Chia Hsu Abstract Evaluating conversational systems in multi-turn settings remains a fundamental challenge. Conventional pipelines typically rely on manually defined rubrics and fixed conversational context—a static approach that limits coverage and fails to capture the diverse, emergent behaviors of dialogue models. To address this, we introduce CoReflect (Conversational Evaluation via Co-Evolutionary Simulation and Reflective Rubric Refinement), which unifies dialogue simulation and evaluation into an adaptive, iterative process. CoReflect employs a conversation planner that generates structured templates to guide a user simulator through diverse, goal-directed dialogues. Subsequently, a reflective analyzer processes these dialogues to identify systematic behavioral patterns and automatically refine the evaluation rubrics. Crucially, the insights from the conversation analysis are fed back into the planner to update conversation templates for subsequent iterations. This co-evolution loop ensures that the complexity of test cases and the diagnostic precision of rubrics improve in tandem. By minimizing human intervention, CoReflect provides a scalable and self-refining methodology that allows evaluation protocols to adapt alongside the rapidly advancing capabilities of dialogue models. \reportnumber keywords Conversational Evaluation, LLMs, User Simulation, LLM-as-a-Judge, Personalization 1 Introduction Large language models (LLMs) increasingly power multi-turn applications such as personalized tutors and customer service agents ( 5 ; 14 ) . In these dynamic settings, users demand more than linguistic fluency; they expect coherent, adaptive, and contextually grounded dialogues. Evaluating such conversational capabilities, however, remains a challenging open problem. To date, the most reliable approach to conversational evaluation relies on open-ended human assessment against predefined rubrics, as human raters can dynamically probe nuanced behaviors and value alignment over extended interactions. However, this method is costly, time-consuming, and prone to subjective bias. While automated alternatives, such as “LLM-as-a-judge” frameworks, offer efficiency and have shown promising alignment with human judgments ( 12 ; 28 ; 27 ) , they often rely on static, pre-designed conversational contexts that fail to capture authentic model behavior. Furthermore, LLM-based simulators, while capable of producing fluent and natural conversational interactions, often struggle to systematically elicit challenging or adversarial behaviors, and may exhibit shallow exploration or mode collapse in self-driven interactions ( 16 ; 19 ; 6 ) . More fundamentally, both human and automated evaluation approaches can be constrained by the evaluation architect’s subjective focus and limited foresight. Overly narrow rubrics may fail to capture emergent model capabilities or unforeseen behaviors that manifest during subsequent deployment, leading to costly redesign and re-execution of the evaluation process. To address this limitation, we propose CoReflect , a co-evolutionary and reflective evaluation framework that moves beyond static benchmarks and fixed rubrics. By introducing an adaptive, iterative process, CoReflect ensures that both dialogue simulation and evaluation criteria evolve in tandem to capture model behaviors. Instead of relying on static conversational scripts, the framework utilizes a conversation planner as the central engine that orchestrates the entire interaction under specific user persona and scenario. Specifically, the planner generates structured, goal-directed templates that prescribe specific turn-level instructions for a user simulator on how to interact with test models. Given these planner-guided interactions with the test models, an LLM-as-a-judge evaluator then assesses the resulting conversations using a structured three-step protocol—analyzing turns, synthesizing observations, and assigning ratings against the current rubrics. Following the evaluation results, a reflective analyzer processes these conversation assessments to cluster behavioral patterns and systematic failures, extracting insights to refine the evaluation rubrics automatically. These insights are also fed back to the planner, which updates its conversation templates to target identified weaknesses with greater precision in subsequent iterations. Our main contributions are presented as follows: • Co-evolutionary conversational evaluation framework. We propose CoReflect, a novel co-evolutionary evaluation framework in which conversation templates and evaluation rubrics are jointly refined through an iterative feedback loop. This co-evolution mechanism is the core innovation of our work, allowing the user simulation and the diagnostic precision of the rating criteria improve in tandem. • Autonomous synthesis of evaluation rubrics. CoReflect introduces a reflective analyzer as the primary mechanism for translating model behavioral data into structured evaluation logic, effectively alleviating the human burden of manual design. By autonomously synthesizing patterns from simulated interactions, it mitigates subjective human bias and enables the system to dynamically expand its scope beyond static rubrics. This data-driven approach allows CoReflect to capture emergent model behaviors that might otherwise remain hidden, ensuring the evaluation framework evolves in lockstep with the model’s complexity. • Comprehensive empirical validation. We conduct extensive automatic and human-in-the-loop experiments across diverse personas, scenarios, and model families, demonstrating the ability of CoReflect to support comprehensive and behavior-sensitive evaluation of state-of-the-art LLMs in multi-turn conversations. 2 Methodology Figure 1: Overview of the CoReflect framework. Initially, evaluation instances are generated from persona-scenario pairs and validated through a consistency check to ensure contextual coherence. Then the core co-evolution loop proceeds through two parts: (i) Conversation simulation, where a planner generates structured conversation templates to guide a user simulator through goal-directed dialogues across diverse persona-scenario pairs; and (ii) Reflective rubric refinement, where an LLM-as-a-judge evaluates simulated interactions and a reflective analyzer extracts insights from clustered behavioral patterns to refine evaluation rubrics. The insights are fed back to the planner to update conversation templates for subsequent iterations. The evaluation process is initialized by a human architect who manually defines a set of coarse, high-level rubrics to serve as the foundational criteria for assessing the conversational quality of the test models. In each iteration, each model interacts with a user simulator across various persona–scenario pairs, guided by a conversation planner . This planner generates goal-directed, persona-grounded conversation templates that instructs the user simulator on how to conduct each user turn, thereby precisely challenging the models against the given rubrics. Following these interactions, an evaluator assesses the conversations based on the established rubrics, providing quantitative ratings and qualitative rationales. These results are then sampled and processed by a reflective analyzer to characterize model behaviors ad limitations, offering insights that drive an iterative refinement loop; in this loop, the rubrics are tuned and the planner accordingly updates its templates to better challenge the models in subsequent rounds. Figure 1 illustrates this interplay between the planner, evaluator, and reflective analyzer. This process proceeds through time steps t=1,2,\dots T , where t=1 represents the initial state, and terminates after T iterations. We provide the specifics of the framework and the process as follows. 2.1 Personas, Scenarios, and Test Models Each evaluation instance in this study comprises two components: (i) a user, who is manifested through a persona profile, (ii) a scenario, which defines a context, task, and objective of the conversation. Every test model is evaluated within these personas and scenarios pairings, with the goal of fulfilling the user’s intent throughout the conversation. Below, we elaborate the construction and the pairing process of personas and scenarios to generate the evaluation set. Personas. To capture user behaviors in our evaluation, we create structured personas that specify how users express themselves and how they expect the system to interact with them. Specifically, each persona comprises two parts: (1) User traits : the user’s attributes manifested in their communication, such as tone, verbosity, and conversational nuances. We denote user traits by x . (2) Response preferences : the desired