Beyond Benchmarks: How Users Evaluate AI Chat Assistants

2026

Read Beyond Benchmarks: How Users Evaluate AI Chat Assistants (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-25220.txt.

Introduction

Awan, Noor, and Munaf ask users, not benchmarks, how AI chat tools stack up. They surveyed 388 active users across seven platforms, from ChatGPT to Llama. The top three earn satisfaction scores too close to split, despite vast gaps in funds and test scores. Most users juggle two or more tools, and switching costs near nothing. Each tool wins fans for its own reason: interface, answer quality, word of mouth, or loose content rules. False answers and filters frustrate users most. The market looks plural, not winner-take-all.

Catalog record

YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61672
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.25220

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Beyond Benchmarks: How Users Evaluate AI Chat Assistants Abstract Automated benchmarks dominate the evaluation of large language models, yet no systematic study has compared user satisfaction, adoption motivations, and frustrations across competing platforms using a consistent instrument. We address this gap with a cross-platform survey of 388 active AI chat users, comparing satisfaction, adoption drivers, use case performance, and qualitative frustrations across seven major platforms: ChatGPT, Claude, Gemini, DeepSeek, Grok, Mistral, and Llama. Three broad findings emerge. First, the top three platforms (Claude, ChatGPT, and DeepSeek) receive statistically indistinguishable satisfaction ratings despite vast differences in funding, team size, and benchmark performance. Second, users treat these tools as interchangeable utilities rather than sticky ecosystems: over 80% use two or more platforms, and switching costs are negligible. Third, each platform attracts users for different reasons: ChatGPT for its interface, Claude for answer quality, DeepSeek through word-of-mouth, and Grok for its content policy, suggesting that specialization, not generalist dominance, sustains competition. Hallucination and content filtering remain the most common frustrations across all platforms. These findings offer an early empirical baseline for a market that benchmarks alone cannot characterize, and point toward competitive plurality rather than winner-take-all consolidation among engaged users. Moiz Sadiq Awan Independent Researcher [email protected] Muhammad Salman Munaf Independent Researcher [email protected] Muhammad Haris Noor Independent Researcher [email protected] Keywords: large language models, user satisfaction, AI chatbots, consumer preferences, multi-platform adoption, switching behavior, survey methodology 1 Introduction Three years after ChatGPT reached 100 million users in record time [ 13 ] , the AI chat assistant market has not followed the trajectory that early predictions suggested. Rather than a single dominant platform, users now choose among at least seven major competitors: OpenAI’s ChatGPT, Anthropic’s Claude, Google’s Gemini, Meta’s Llama, xAI’s Grok, DeepSeek, and Mistral’s Le Chat. Hundreds of millions of people interact with these systems daily [ 24 ] . Yet for all the capital, compute, and engineering talent flowing into this market, a basic empirical question remains open: what do users think of these platforms, and why do they choose one over another? Despite the enormous scale of adoption, empirical understanding of user preferences has not kept pace. The AI research community has developed sophisticated automated benchmarks: MMLU [ 12 ] assesses multi-task understanding across 57 academic subjects, HumanEval [ 8 ] evaluates code generation, and the LMSYS Chatbot Arena [ 9 ] provides crowd-sourced pairwise preference rankings in controlled settings. These benchmarks are useful for tracking technical progress, but they leave critical questions unanswered. What factors beyond raw capability drive adoption and retention? How do users allocate their usage across competing platforms? What pain points persist despite rapid technical improvement? These questions have practical consequences. For product teams, knowing which attributes users actually value can redirect development effort away from benchmark-chasing toward features that affect daily experience. For market analysts, the degree of platform switching and multi-homing determines whether the AI assistant market is consolidating or sustaining genuine competition. For regulators, user satisfaction data bear on questions of market concentration and consumer welfare. For evaluation researchers, understanding what drives real-world platform choice can inform the design of more ecologically valid assessment protocols. Yet companies continue to invest billions in model development guided primarily by benchmark rankings, and policy discussions about AI market dominance have proceeded with little evidence about how consumers actually experience competing products. This paper presents results from a cross-platform survey ( N=388 ) conducted in late 2025, examining preferences across seven major AI chat platforms. While existing user studies have focused on individual platforms (predominantly ChatGPT), comparative data across multiple competing platforms remains scarce. Because the sample is drawn from technology-oriented communities, our findings characterize the preferences of active, engaged AI chat users rather than the general population. Our contributions are as follows: • We document pervasive multi-platform usage and ChatGPT-centric switching patterns, providing systematic evidence that AI chat tools function as interchangeable utilities rather than sticky ecosystems. • We establish empirical baselines for user satisfaction across seven platforms, finding reported satisfaction parity among the top three (Claude, ChatGPT, and DeepSeek) despite vast resource asymmetries. • We identify distinct adoption profiles and domain specializations that differentiate each platform’s competitive positioning, suggesting that specialization rather than generalist dominance sustains competition. • We uncover a large first-mover anchoring effect: ChatGPT users who adopted it as their first AI tool rate satisfaction 1.34 points higher than those who arrived from a competitor, highlighting the role of habit and default effects. • We present a thematic analysis of 329 open-ended responses identifying hallucination and content moderation as the two most persistent user frustrations, representing a direct engineering tradeoff. 2 Related Work 2.1 Automated LLM Evaluation The evaluation of large language models has been dominated by automated benchmarks targeting specific capabilities. MMLU [ 12 ] tests multi-task understanding across 57 academic subjects, providing a broad measure of factual knowledge and reasoning. HumanEval [ 8 ] and MBPP [ 2 ] evaluate functional code generation by testing whether model outputs pass unit tests. GSM8K [ 10 ] tests mathematical reasoning through grade-school word problems, while TruthfulQA [ 19 ] probes factual accuracy and resistance to common misconceptions. More comprehensive suites such as HELM [ 18 ] and BIG-Bench [ 26 ] attempt broad-coverage assessment across dozens of tasks simultaneously, though they remain bounded by what can be automatically scored. A significant advance in ecological validity came with the LMSYS Chatbot Arena [ 9 ] , which collects human preference judgments through blind pairwise comparisons of model outputs in real conversational settings, producing Elo-style rankings that have become a widely cited industry reference. Zheng et al. [ 28 ] further proposed MT-Bench, a multi-turn benchmark using strong LLMs as judges, which correlates well with human preferences but still operates at the response level rather than the platform experience level. However, both Arena and MT-Bench capture isolated response quality in single-turn or short multi-turn exchanges. They do not measure the broader user experience that encompasses interface design, pricing, ecosystem integration, content policy, reliability over extended use, and the accumulated satisfaction that drives real-world platform choice. A shared limitation is that these approaches treat model quality as a single scalar, when in practice users weigh multiple factors simultaneously (accuracy, speed, style, policy permissiveness, and cost), and these factors interact in complex, context-dependent ways. 2.2 Technology Adoption and Platform Economics The Technology Acceptance Model (TAM; Davis [ 11 ] ) and its successor UTAUT [ 27 ] identify perceived usefulness, ease of use, social influence, and facilitating conditions as key determinants of technology adoption. The Expectation-Confirmation Model [ 4 ] extends this framework to continued usage, demonstrating that post-adoption satisfaction depends on the gap between experienced performance and initial expectations. In the context of AI tools, Jo [ 15 ] adapted TAM to study ChatGPT adoption among South Korean users, finding that output quality and ease of use were the strongest predictors of continued use intention. The concept of multi-homing, in which consumers simultaneously maintain active usage across competing platforms, has been studied extensively in platform economics. Rochet and Tirole [ 22 ] and Armstrong [ 1 ] established foundational models of two-sided market competition, showing that differentiation and switching costs jointly determine market structure. Shapiro and Varian [

Read the complete text