ChatBench: From Static Benchmarks to Human-AI Evaluation

Serina Chang, Ashton Anderson, Jake M. Hofman · 2025

Read ChatBench: From Static Benchmarks to Human-AI Evaluation by Serina Chang, Ashton Anderson, Jake M. Hofman (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-07114.txt.

Introduction

Serina Chang, Ashton Anderson, and Jake Hofman test, in 2025, what happens when people answer hard questions together with a chatbot. Standard benchmarks grade the model alone, yet real use is collaborative. The team turned questions from a major benchmark into conversations, seeding users with a question and letting them consult a model.

The released dataset holds thousands of conversations and well over one hundred thousand answers across two models. Solo model accuracy failed to predict team accuracy, with gaps varying by subject. Fine-tuning a user simulator on part of the data sharply improved its forecasts of team performance, pointing toward scalable ways to evaluate human-AI teamwork.

Catalog record

CREATORSerina Chang, Ashton Anderson, Jake M. Hofman
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE80 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-07114

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ChatBench: From Static Benchmarks to Human-AI Evaluation Serina Chang, Ashton Anderson, Jake M. Hofman Abstract With the rapid adoption of LLM-based chatbots, there is a pressing need to evaluate what humans and LLMs can achieve together. However, standard benchmarks, such as MMLU, measure LLM capabilities in isolation (i.e., “AI-alone”). Here, we design and conduct a user study to convert MMLU questions into user-AI conversations, by seeding the user with the question and having them carry out a conversation with the LLM to answer their question. We release ChatBench, a new dataset with AI-alone, user-alone, and user-AI data for 396 questions and two LLMs, including 144K answers and 7,336 user-AI conversations. We find that AI-alone accuracy fails to predict user-AI accuracy, with significant differences across multiple subjects (math, physics, and moral reasoning), and we analyze the user-AI conversations to provide insight into how they diverge from AI-alone benchmarks. Finally, we show that fine-tuning a user simulator on a subset of ChatBench improves its ability to estimate user-AI accuracies, increasing correlation on held-out questions by more than 20 points, creating possibilities for scaling interactive evaluation. 1 1 1 Our dataset ChatBench is available at https://huggingface.co/datasets/microsoft/ChatBench . 1 Introduction In 2024, nearly 40% of US adults reported using generative AI in their everyday lives, an unprecedented rate of adoption for a new technology Bick et al. (2024) . As these models, particularly large language models (LLMs), become more integrated into our lives, it becomes increasingly important to evaluate not only their capabilities in isolation, but also on how effectively they support people in solving a variety of tasks. However, there is a large gap between human interactions and how standard benchmarks, such as Massive Multitask Language Understanding (MMLU), evaluate models Hendrycks et al. (2021) . These benchmarks test models on a fixed set of questions, and for each question, they prompt the model with the entire question text and often constrain it to respond with a single multiple choice option as its answer. In contrast, interactions with human users are far more variable, open-ended, and subject to ambiguity. Even conditioned on the same underlying intent, users may phrase their prompts differently, leave out information in their early prompts, or rely on context in later prompts. Robust AI models must be designed to handle user interactions in these contexts to provide accurate information and complement human expertise. Recently, there have been efforts to evaluate how humans interact with LLMs, such as examining real-world conversations using a strong LLM as a judge Lin et al. (2024) ; Li et al. (2024c) . However, these new evaluations have been largely disconnected from standard benchmarks, which are widely used; for example, every LLM released by OpenAI, Google, and Meta, inter alia, has reported its performance on MMLU OpenAI (2023) ; Gemini Team Google (2023) ; Llama Team, AI@Meta (2024) . This disconnect is due to a large distribution shift between benchmark questions and questions asked by real-world users, missing the user’s true intent, and missing ground-truth labels to judge the interaction, necessitating techniques like LLM-as-judge. As a result, it is difficult to directly compare results from standard benchmarks to real-world interactions or to understand how incorporating interactions changes evaluation insights. Here, we seek to bring these lines of research closer together by directly converting benchmarks into user-AI conversations. We focus on MMLU, as one of the most widely used benchmarks, and design a user study where we seed users with an MMLU question and have them carry out a conversation with an LLM with the intent of answering that question. For each question, we test the LLM in isolation (i.e., “AI-alone”) and evaluate the accuracy of a user interacting with the LLM (i.e., “user-AI”); furthermore, we also gather “user-alone” data per question to understand how much users improve with the LLM. This parallel data has two advantages: first, we can now conduct an apples-to-apples comparison of AI-alone performance, as reported in most papers, vs. user-AI performance on the same questions, so that we can isolate the effects of incorporating interaction into evaluation. Second, recent works have explored the possibility of simulating the user in user-AI conversations Li et al. (2024a) but lack sufficient data for training and testing. Our approach of “seeding” users with a question corresponds naturally to a new way to initialize user simulators, and the large-scale data we collect enables fine-tuning and validating a user simulator on this task, improving the trustworthiness of simulations for AI evaluation. Our resulting dataset ChatBench, which we release publicly, consists of AI-alone, user-alone, and user-AI data for 396 questions and two LLMs (GPT-4o and Llama-3.1-8b), with 144K answers and 7,336 user-AI conversations. Our study design also includes two user-AI conditions—where the user attempts the question first on their own vs. uses AI directly—to explore nuances in user behavior. Our study reveals that AI-alone accuracy fails to predict user-AI accuracy, with significant differences across multiple subjects (math, physics, and moral reasoning). We also analyze the user-AI conversations to understand where user-AI interactions are diverging from AI-alone benchmarks. Our contributions are summarized as follows: • We design and conduct a user study to convert MMLU questions into user-AI conversations and release a large-scale dataset ChatBench. • We show that AI-alone accuracy fails to predict user-AI accuracy, across subjects, models, AI-alone methods, and user-AI conditions, and we analyze user-AI conversations to understand where AI-alone and user-AI diverge. • We develop a new user simulator that mimics our user study task and show that fine-tuning our simulator on ChatBench improves its correlation with real user-AI accuracies by 22-26 points and outperforms baselines. All together, our work helps to reconcile two vital lines of research in AI evaluation, revealing how interactions change evaluation insights and paving the way towards scalable interactive evaluation. 2 Related Work Benchmarks. In this work, we focus on MMLU as one of the most commonly used LLM benchmarks Hendrycks et al. (2021) . MMLU is a question-answering (QA) dataset, consisting of multiple choice questions across 57 subjects (which we discuss in detail in Section 3.2 ). We also draw on the efforts of MMLU-Redux Gema et al. (2024) , where authors noted some quality concerns in the original MMLU, so they sampled a large number of MMLU questions and manually annotated them for errors. While we conduct our user study on MMLU, our approach of converting QA benchmarks to a user-AI conversation is general, and could be applied to other QA benchmarks, such as HotPotQA Yang et al. (2018) or GSM8K Cobbe et al. (2021) , as well as adapted to non-QA tasks. Evaluating human-AI interactions. Recently, there have been growing efforts to evaluate AI models based on their interactions with humans. For example, some works gather real-world interactions (e.g., WildChat Zhao et al. (2024) , ChatbotArena Chiang et al. (2024) ) and evaluate the interactions (e.g., WildBench Lin et al. (2024) , ArenaHard Li et al. (2024c) , MT-Bench Zheng et al. (2023) , LMSYS-Chat-1M Zheng et al. (2024) ), typically using a strong LLM as a judge. However, as discussed before, it is difficult to directly compare these evaluation results to standard benchmarks, due to the lack of ground-truth user intents and interaction labels, distribution shift in

Read the complete text