Understand User Opinions of Large Language Models via LLM-Powered In-the-Moment User Experience Interviews

Mengqiao Liu, Tevin Wang, Cassandra A. Cohen, Sarah Li, Chenyan Xiong · 2025

Read Understand User Opinions of Large Language Models via LLM-Powered In-the-Moment User Experience Interviews by Mengqiao Liu, Tevin Wang, Cassandra A. Cohen, Sarah Li, Chenyan Xiong (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-15226.txt.

Introduction

Mengqiao Liu and co-authors interview chatbot users in 2025 using an AI interviewer. Benchmarks rank language models, but what users actually think is murkier. Their system chats with users right after they try a target model, then mines thousands of interview logs for opinions.

The study recruits thousands of people to test mainstream models and be interviewed immediately. The interviews surface sharp, split views, for example on one model's visible reasoning traces, and strong demands for fresher facts and richer media. The authors release code and data, arguing that in-the-moment automated interviews capture user sentiment that scores alone miss.

Catalog record

CREATORMengqiao Liu, Tevin Wang, Cassandra A. Cohen, Sarah Li, Chenyan Xiong
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-15226

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Understand User Opinions of Large Language Models via LLM-Powered In-the-Moment User Experience Interviews Mengqiao Liu, Tevin Wang, Cassandra A. Cohen, Sarah Li, Chenyan Xiong Abstract Which large language model (LLM) is better? Every evaluation tells a story, but what do users really think about current LLMs? This paper presents CLUE , an LLM-powered interviewer that conducts in-the-moment user experience interviews, right after users interacted with LLMs, and automatically gathers insights about user opinions from massive interview logs. We conduct a study with thousands of users to understand user opinions on mainstream LLMs, recruiting users to first chat with a target LLM and then interviewed by CLUE . Our experiments demonstrate that CLUE captures interesting user opinions, for example, the bipolar views on the displayed reasoning process of DeepSeek-R1 and demands for information freshness and multi-modality. Our collected chat-and-interview logs will be released. * * footnotetext: These authors contributed equally to this work. † † footnotetext: This work does not relate to their position at Amazon. 1 Introduction The foundation and large language model (LLM) revolution is redefining the way users interact with the digital world. Billions of users now chat with LLMs regularly through one unified interface and consume information through generative contents for their information, entertainment, and task assistance needs Duarte 2025 . Figure 1: An example interview sequence about user experience with their previous LLM interaction. Understanding user opinions about LLMs is, however, a challenging task. It is difficult to design one evaluation task to reflect the rich experiences provided by LLMs. The user experiences powered through generative contents are also hard to characterize by metrics like accuracy and BLEU. The prone to data contamination during pretraining further complicates the understanding of LLM performances Schaeffer 2023 ; Singh et al. 2024 . The community often relies on coarse user preference ratings to understand LLMs performances Chiang et al. 2024 . This paper presents CLUE , Contextualized LLM-powered User Experience understanding, a new framework to gather fine-grained user opinions using massive in-the-moment user experience (UX) interviews. Following interview principles Wilson 2013 , we develop CLUE -Interviewer on top of an LLM to automatically conduct semi-structured UX interviews: asking user opinions on target dimensions and probing for deeper insights (Figure 1 ). As an automatic interviewer, it can chat with users right after their interactions with a product, enabling massive in-the-moment interviews. We then build CLUE -Insighter to automatically analyze CLUE interview logs to generate high-level insights of user opinions. Approved by our university Institutional Review Board (IRB), we recruited thousands of crowd source workers to chat with an LLM, randomly selected from six LLMs from the GPT, Claude, Gemini, LLaMA, and DeepSeek families. Then CLUE -Interviewer interviews participants right after their chat sessions. We obtained 1206 chat-and-interview sessions on diverse topics initiated by crowd source workers, ranging from travel destinations to programming developments, that we will make publicly available. Our human evaluation demonstrates the effectiveness of CLUE -Interviewer in conducting semi-structured and in-the-moment UX interviewers. It covers 74% of the pre-defined dimensions to evaluate target LLMs and actively asks follow-up questions to gain deeper insights. The interviews are in-the-moment, with frequent references to previous chat interactions, effectively eliciting in-the-moment user opinions. The evaluation also confirms that CLUE -Insighter generates interesting insights from mass interview logs. Our user study reflects interesting user opinions of LLMs. Users expressed reserved opinions about LLMs in their text responses to CLUE -Interviewer, considering all mainstream LLMs mediocre, but gave a near-perfect rating when asked explicitly. Users considered LLMs decent at understanding their needs but not as effective in meeting them. Users also expressed new feature requests in the interviews, for example, visual and multimedia capabilities, fresh knowledge access, personalized responses, and flexible access to reasoning and explanations processes, providing motivations and evidences for future LLM development. The main contributions of this paper are: 1. We present a new methodology to gather user opinions on LLMs via in-the-moment UX interviews. 2. We develop an LLM-powered interview framework that enables automatic in-the-moment interviews at a massive scale. 3. We collect thousands of publicly available chats and interviews that reflect interesting user opinions of LLMs. 2 Related Work Many benchmarks have been developed to evaluate predictive effectiveness of LLMs. Notable examples include GLUE Wang 2018 and SuperGLUE Wang et al. 2019 , both including a suite of natural language understanding tasks. The suite of language tasks quickly grows to hundreds, such as in FLAN Wei et al. 2021 and BIG-Bench bench authors 2023 benchmarks. The community also keeps increasing task difficulty, for example, from MMLU Hendrycks et al. 2020 to “Humanity’s Last Exam” Phan et al. 2025 , to test the boundary of LLM intelligence. These evaluations are effective in reflecting LLMs’ ability to predict the right label, which aligns closely with some real-world applications such as question answering. Evaluating generated content is challenging, as two text sequences with high n-gram overlap Papineni et al. 2002 or semantic similarities Zhang et al. 2019 may not lead to the same user experience Hanna and Bojar 2021 . Recent approaches switch to model-based evaluation and employ LLM-as-a-judge for predefined dimensions Zheng et al. 2023 . Model-based evaluation has become a common practice in evaluating generated content, albeit various challenges such as self-biases and inaccuracies Li et al. 2024a ; Ye et al. 2024 ; Wei et al. 2024 . The ultimate verdict of an AI model is how it serves its users. For established applications like search engines, understanding model performances based on noisy and coarse user feedback is a long lasting research topic Chuklin et al. 2022 . There are various efforts to collect user feedback on new LLM-powered scenarios. One notable effort is Chatbot Arena Chiang et al. 2024 . It asks users for preferences on side-by-side LLM chats and computes arena scores based on that. Many view the arena score a reliable reflection of user preferences on LLMs Hart 2024 . UX interview is a standard approach to gather in-depth insights about user opinions Rubin and Chisnell 2011 . It is widely recognized as an effective tool to guide product developments toward increased adoption, consumer loyalty, and overall product success Hartson and Pyla 2012 . Effective UX interviews often require interview experts Ahrend 2025 and are too expensive to scale up. Many explored the potential of AI-powered interview bots, for example, to conduct job interviews Li et al. 2017 and conversational surveys Xiao et al. 2020 . Recently, Li et al. 2024b built an LLM-powered interview system and demonstrated their effectiveness in evaluating student experiences in AI-assisted classroom. 3 Methodology CLUE leverages LLMs to conduct in-the-moment UX interviews and understand user opinions. It includes two components: CLUE -Interviewer and CLUE -Insighter. 3.1 CLUE -Interviewer CLUE -Interviewer conducts in-the-moment UX interviews right after users interacted with a product, which in this study is chatting with a mainstream LLM. We develop CLUE -Interviewer by implementing standard user interview practices Hartson and Pyla 2012 into LLMs through carefully designed prompts. Interview Design. CLUE -Interviewer is designed to conduct semi-structured interviews ( Wilson 2013 ) , starting with a set of predefined interview dimensions and probing user for deeper insights, with the flexibility to explore topics emerged from user responses.

Read the complete text