Figure 10:

Judge prompt for Pairwise Indistinguishability Metric · 2026

Read Figure 10: by Judge prompt for Pairwise Indistinguishability Metric (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-08118.txt.

Introduction

Large language models now act as stand-in users in tests of chat systems and in training data creation. A 2026 arXiv study warns that simple act-as-a-user prompting yields wordy, unrealistic speech. That gap creates a need for principled checks on these user proxy agents. The paper introduces a pairwise indistinguishability metric, judged with a dedicated prompt shown in its tenth figure. The goal is to test whether proxy users sound like real users side by side. Better proxies could make system evaluation cheaper and more trustworthy.

Catalog record

CREATORJudge prompt for Pairwise Indistinguishability Metric
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE75 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-08118

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Figure 10: Judge prompt for Pairwise Indistinguishability Metric Abstract Large language models (LLMs) are increasingly used as human simulators, both for evaluating conversational systems and for generating fine-tuning data. However, naive “act-as-a-user" prompting often yields verbose, unrealistic utterances, underscoring the need for principled evaluation of so-called user proxy agents . We present MirrorBench , a reproducible, extensible benchmarking framework that evaluates user proxies solely on their ability to produce human-like user utterances across diverse conversational tasks, explicitly decoupled from downstream task success. MirrorBench features a modular execution engine with typed interfaces, metadata-driven registries, multi-backend support, caching, and robust observability. The system supports pluggable user proxies, datasets, tasks, and metrics, enabling researchers to evaluate arbitrary simulators under a uniform, variance-aware harness. We include three lexical-diversity metrics ( MATTR , Yule’s K K , and HD-D ) and three LLM-judge–based metrics ( GTEval , Pairwise Indistinguishability , and Rubric-and-Reason ). Across four open datasets, MirrorBench yields variance-aware results and reveals systematic gaps between user proxies and real human users. The framework is open source and includes a simple command-line interface for running experiments, managing configurations and caching, and generating reports. The framework can be accessed at https://github.com/SAP/mirrorbench marginparsep has been altered. topmargin has been altered. marginparwidth has been altered. marginparpush has been altered. The page layout violates the ICML style. Please do not change the page layout, or include packages like geometry, savetrees, or fullpage, which change it for you. We’re not able to reliably undo arbitrary changes to the style. Please remove the offending package(s), or layout-changing commands and try again. MirrorBench: An Extensible Framework to Evaluate User-Proxy Agents for Human-Likeness Ashutosh Hathidara 1 Julien Yu 1 Vaishali Senthil 1 Sebastian Schreiber 1 Anil Babu Ankisettipalli 1 † † footnotetext: 1 SAP Labs. Correspondence to: Ashutosh Hathidara <[email protected]>. 1 Introduction & Related Work The idea of simulating users to train and evaluate dialogue agents has a long history in NLP 5 . Early user simulators were typically goal-driven or rule-based, following predefined agendas of dialogue acts to mimic human behavior 28 . Recent work has moved towards large language model (LLM) based user proxies that generate more realistic, open-ended interactions. For example, 31 introduced a User Simulator with Implicit Profiles (USP) that infers latent user traits to produce personalized, authentic dialogues, addressing issues of role confusion and limited diversity in naive LLM role-play. Likewise, DialogueForge 35 bootstraps AI-to-AI conversations from real chat logs as seed prompts, leveraging powerful models (e.g. GPT-4) as the user to yield multi-turn dialogues. It demonstrates that while larger LLMs produce the most human-like exchanges, fine-tuned smaller models can approximate realism with customization, though maintaining long-range coherence remains a challenge. Collectively, these simulators underscore the potential of LLM-driven user agents for producing high-fidelity dialogues. Parallel to improvements in user simulation, researchers have begun using LLMs as judges to evaluate dialogue quality. These “LLM-as-a-judge" evaluators prompt a strong model to rate or compare chatbot responses, bypassing the need for reference answers or extensive human annotation 20 , 19 . With careful prompt design (e.g. asking the model to explain its criteria via chain-of-thought reasoning), such LLM-based evaluation metrics can flexibly assess custom quality criteria 29 . Notably, LLM evaluators have enabled large-scale dialogue assessments by providing fast, consistent scoring of responses. However, existing LLM-as-judge systems mainly focus on rating the assistant’s outputs rather than evaluating the simulators themselves. Deploying and improving conversational AI hinges on evaluation under realistic user interactions. Expert human annotation, while authoritative, is costly and difficult to scale for systematic evaluation. At the same time, the instruction-tuning of models for production systems demands large volumes of authentic user–assistant exchanges. To meet both needs, practitioners increasingly rely on user proxies , namely LLMs prompted to emulate specified human personas, to synthesize user utterances for automated testing and post-training at scale. For example, recent work shows that LLM simulator can be used to evaluate an AI assistant over hundreds of interactive conversations 22 , 1 , 10 . Yet the naive prompting of a single LLM to act as user simulator often yields verbose, overly cooperative behavior that diverges from real user speech patterns 34 . Recent studies further note that assistant LLMs can lose coherence in extended dialogues, in part because they fail to capture or elicit the subtleties of human behavior 18 . Scalable, practical evaluation of conversational AI therefore depends on realistic user simulators, which must themselves be rigorously evaluated for how closely they resemble human users across diverse scenarios. This leads to the central question: How can we rigorously measure the human-likeness or user-likeness of a user proxy, independently of downstream task success? Here, we define a user as a human interlocutor participating in a specific conversational or task-oriented dialogue. In this work, we introduce MirrorBench , a reproducible and extensible framework for benchmarking user-proxy agents on human-likeness metrics using multiple human reference datasets across diverse conversational settings. MirrorBench evaluates proxies solely on human-likeness and deliberately separates this from task completion. The framework comprises strongly typed domain models and metadata-rich registries, a compatibility-aware planner that validates dataset and metric compatibility while producing replayable manifests for experiment reproducibility, and a flexible execution engine supporting multi-turn interactions with synchronous, asynchronous, and distributed backends such as Ray 24 . It also includes a caching layer for LLM outputs and dataset statistics, and a command-line interface (CLI) for validating, dry-running, executing, and reporting evaluation experiments. Together, these components form a modular harness in which proxies, datasets, tasks, and metrics are pluggable yet rigorously validated. MirrorBench is designed for extensibility and controlled variation. Researchers can register custom LLM clients, user-proxy adapters, datasets, evaluation metrics, task drivers, and execution back-ends through a unified API. Component capabilities and compatibility constraints are declared in metadata (while registering) and validated by the planner, allowing modules to be swapped or extended without modifying the core. This separation of concerns enables fair, apples-to-apples comparisons across proxies and judges under consistent orchestration and logging, and ensures seamless integration into the evaluation pipeline. To evaluate the realism of user–proxy agents against real-world dialogues, MirrorBench packages four open-source corpora preprocessed for user-proxy evaluation: QULAC 3 , ClariQ 2 , OASST1 16 , and ChatbotArena 33 . To quantify alignment in lexical diversity with human users, we implement Moving-Average Type–Token Ratio ( MATTR ) 8 , Yule’s K 30 , and a Hypergeometric Distribution Diversity ( HD-D ) 6 . To assess behavioral indistinguishability, we employ a family of LLM-judge metrics, including GTEval 35 , Pairwise Indistinguishability ( PI ) 33 , and Rubric-and-Reason ( RNR ) 21 . Taken together, these metrics quantify the realism of the user side of the dialogue, explicitly decoupled from assistant capabilities and task success. MirrorBench is not merely a metric suite; it is a full-fledged evaluation harness. It validates inputs against strongly typed schemas, persists every turn, score, and judge invocation to a run database, and caches LLM outputs for reproducibility and cost control. The engine logs fine-grained telemetry (latency, token counts, cost) at each turn and supports synchronous, asynchronous, and distributed evaluation. A compatibility-aware planner expands grids over proxies, datasets, metrics, and random seeds, and emits a replayable manifest that precisely reproduces an evaluation. The command-line interface supports quick dry-runs

Read the complete text