The Decision to Verify: How Warmth and User Characteristics Shape Reliance on Conversational Agents for Information Search Journal: Computers in Human Behavior

Mert Yazan, Frederik Bungaran Ishak Situmeang, Suzan Verberne · 2026

Read The Decision to Verify: How Warmth and User Characteristics Shape Reliance on Conversational Agents for Information Search Journal: Computers in Human Behavior by Mert Yazan, Frederik Bungaran Ishak Situmeang, Suzan Verberne (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-28498.txt.

Introduction

Mert Yazan, Frederik Bungaran Ishak Situmeang, and Suzan Verberne study whether pairing chatbots with web search curbs overreliance on AI answers. In a mixed-subjects question-answering experiment, participants used either a warm or a neutral chatbot alongside search. Reliance persisted despite fact-checking access, and the choice to verify tracked prior trust in chatbots more than answer features. Warmth acted indirectly, raising agreement with the chatbot even when it was wrong, while consulting extra AI sources predicted higher accuracy and plain web search did not.

Catalog record

CREATORMert Yazan, Frederik Bungaran Ishak Situmeang, Suzan Verberne
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE94045
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.28498

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The Decision to Verify: How Warmth and User Characteristics Shape Reliance on Conversational Agents for Information Search Journal: Computers in Human Behavior Mert Yazan, Frederik Bungaran Ishak Situmeang, Suzan Verberne Abstract Conversational artificial intelligence (AI) provides an efficient and convenient gateway to information access. However, it can cause overreliance when users blindly trust AI and accept its answers without fact-checking. Information search increasingly follows a hybrid interaction paradigm that combines conversational AI with web search, making fact-checking easier. In this paper, we examine whether this interaction paradigm is effective in curbing reliance. We further investigate the underlying factors (e.g., digital literacy and conversation warmth) that drive users to verify AI answers. We conduct a mixed-subjects question-answering experiment where participants interact with either a warm or a neutral chatbot. Our findings reveal that reliance persists despite users having access to both conversational and web search. The decision to verify is driven primarily by existing user perceptions (e.g., prior trust in chatbots) rather than answer properties, with some users fact-checking regardless of the context and others trusting chatbots by default. Warm conversational style has an indirect yet critical influence on reliance by increasing agreement with the chatbot when it is incorrect. Consulting additional AI sources predicts higher accuracy, while traditional web search does not. Our study extends overreliance research by: (a) demonstrating its persistence despite access to fact-checking, (b) identifying verification behavior as user-dependent, and (c) revealing conversational warmth’s indirect effect on overreliance with implications for designing trustworthy conversational search systems. † † graphicalabstract: † † highlights: Reliance persists in conversational search despite access to fact-checking tools. The decision to verify chatbot answers is driven by user characteristics. Some users always fact-check; others never do. Warmth leads to reliance by increasing agreement with incorrect chatbot answers. Consulting additional AI sources predicts higher accuracy. 1 Introduction Information search has seen a rapid change in recent years, becoming more conversational as the popularity of Large Language Models (LLMs) has grown ( 9 ) . Approximately 10% of the world’s adult population uses ChatGPT ( 35 ) at least weekly, with one in four queries involving information seeking ( 9 ) . Compared to traditional search interfaces, such as search engines, conversational agents offer better accessibility, are easier to use, and require less effort ( 29 ; 40 ; 26 ) . Despite the advantages, there exists a major downside of conversational agents: hallucinations. Humans rely on AI outputs even when they are incorrect, showing overreliance ( 28 ) . Overreliance leads people to accept the first answer they receive from an LLM without further inquiries in challenging tasks ( 40 ; 27 ) . Furthermore, chatbots give an inflated sense of trust and confidence, even when they perform worse than search engines ( 44 ; 30 ) . LLMs hallucinate responses that seem plausible at first glance, albeit being nonfactual ( 21 ) , making overreliance a critical issue for conversational agents ( 44 ; 40 ) . In controlled experiments, participants are generally constrained to interact with either a chatbot or with a search engine ( 40 ; 29 ) . However, in reality, users can access both: Recent developments have introduced a new search paradigm that provides users with simultaneous access to conversational agents (e.g., chatbots) and web search ( 37 ) . Due to this new paradigm, hallucinations are not regarded as a major issue for most users, since chatbot responses can be easily verified by external sources ( 39 ; 32 ) . It can be argued that the new paradigm solves overreliance by combining the strengths of both conversational and traditional approaches. However, there is evidence showing that the mere presence of AI can induce reliance ( 27 ) . Participants seek external confirmation less frequently when interacting with AI and tend to agree more with AI responses regardless of accuracy ( 27 ) . Moreover, chatbots still hallucinate and give incorrect answers, even with web search ( 34 ) . These findings suggest that the new search paradigm might not be the definitive answer against overreliance. Despite verification tools being readily available with the new paradigm, the decision to fact-check or trust an AI answer is up to the user. Almost 90% of the utterances contain check-worthy claims in information-seeking tasks ( 23 ) , making it unrealistic for users to verify all responses. Users have to make a conscious decision about which answers need further verification. However, users fail to fact-check results that confirm existing beliefs ( 11 ) or become anchored to the first information source they encounter ( 4 ) . Consequently, tool availability alone cannot ensure a solution to reliance, as the ultimate barrier remains the human decision-making process itself. Previous experience, AI expertise, and digital literacy shape fact-checking behavior by reflecting users’ awareness of LLM limitations ( 39 ; 42 ) . Similarly, trust in AI systems determines whether users accept AI outputs as is ( 28 ; 18 ) . Beyond user background, chatbot conversational style, especially its warmth, influences reliance by shaping risk perceptions and emotional trust ( 43 ; 24 ; 41 ) . In summary, overreliance appears to persist despite the emerging trend of integrating conversational AI with traditional search. Existing studies cannot conclusively explain why and how overreliance manifests when users have simultaneous access to both modalities ( 40 ; 29 ) . Addressing this gap requires a deeper understanding of the psychological and contextual factors that shape users’ verification decisions. In this paper, we investigate reliance on conversational agents for the new paradigm of information search with a mixed-subjects study (n=199). We designed an experiment based on question-answering where participants were given access to a helper chatbot. The chatbot’s answer correctness was randomized, with each participant seeing correct answers half of the time. Participants were randomly assigned to a warm or neutral chatbot and were free to use other sources beyond the chatbot, simulating a realistic search experience. Previous work has treated external source usage as a self-reported binary variable (used, not used), leading to noisy results ( 27 ) . We track participant interactions with a custom browser extension that records visited URLs, enabling us to measure both verification frequency (number of URLs) and verification type (consulting another AI source or web search). We look into not only overreliance but also underreliance by examining participant agreement (with the chatbot) and accuracy, given the chatbot’s correctness. Furthermore, we conduct behavioral analysis to understand how reliance manifests across different user groups. With this study, we answer the following research questions: 1. RQ1: How do overreliance and underreliance manifest in conversational search when users have access to fact-checking tools? 2. RQ2: What factors drive users’ decisions to verify or accept a chatbot’s answer? 3. RQ3: What is the influence of conversation warmth on reliance? Our findings confirm the existence of both types of reliance despite access to fact-checking tools. Some participants exhibited underreliance by always fact-checking, incorrectly assuming that the questions were too difficult for the chatbot. On the other side, other participants showed overreliance and never felt it necessary to question the chatbot responses. Overreliance also manifested subconsciously through conversational warmth: participants agreed more with the warm chatbot

Read the complete text