Not Too Short, Not Too Long: How LLM Response Length Shapes People’s Critical Thinking in Error Detection Conference: IUI MIRAGE 2026 workshop held at 31st International Conference on Intelligent User Interfaces; March 23, 2026; Paphos, Cyprus IUI MIRAGE 2026 workshop held at 31st International Conference on Intelligent User Interfaces Conference: Joint Proceedings of the ACM IUI Workshops 2026, Misleading Impacts Resulting from AI Generated Explanations; March 23–26, 2026; New York City, NY, USA

Natalie Friedman, Adelaide Nyanyo, Kevin Weatherwax, Lifei Wang, Chengchao Zhu, Zeshu Zhu, S. Joy Mountford · 2026

Read Not Too Short, Not Too Long: How LLM Response Length Shapes People’s Critical Thinking in Error Detection Conference: IUI MIRAGE 2026 workshop held at 31st International Conference on Intelligent User Interfaces; March 23, 2026; Paphos, Cyprus IUI MIRAGE 2026 workshop held at 31st International Conference on Intelligent User Interfaces Conference: Joint Proceedings of the ACM IUI Workshops 2026, Misleading Impacts Resulting from AI Generated Explanations; March 23–26, 2026; New York City, NY, USA by Natalie Friedman, Adelaide Nyanyo, Kevin Weatherwax, Lifei Wang, Chengchao Zhu, Zeshu Zhu, S. Joy Mountford (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-06878.txt.

Introduction

Friedman and colleagues test whether long or short LLM answers change how well readers spot reasoning errors. They ran a repeated-measures experiment on thinking tasks, giving each person Watson-Glaser items paired with answers that varied in length and in truth. The design isolates length from truth. Analysis with mixed-effects logistic regression found a strong, reliable truth effect, with length examined only in interaction with truth.

Catalog record

CREATORNatalie Friedman, Adelaide Nyanyo, Kevin Weatherwax, Lifei Wang, Chengchao Zhu, Zeshu Zhu, S. Joy Mountford
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30837
LICENSEhttp://creativecommons.org/publicdomain/zero/1.0/
RIGHTSCC0
SOURCEarXiv

Source: https://arxiv.org/abs/2603.06878

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Not Too Short, Not Too Long: How LLM Response Length Shapes People’s Critical Thinking in Error Detection Conference: IUI MIRAGE 2026 workshop held at 31st International Conference on Intelligent User Interfaces; March 23, 2026; Paphos, Cyprus IUI MIRAGE 2026 workshop held at 31st International Conference on Intelligent User Interfaces Conference: Joint Proceedings of the ACM IUI Workshops 2026, Misleading Impacts Resulting from AI Generated Explanations; March 23–26, 2026; New York City, NY, USA Natalie Friedman, Adelaide Nyanyo, Kevin Weatherwax, Lifei Wang, Chengchao Zhu, Zeshu Zhu, S. Joy Mountford Abstract . Large language models (LLMs) have become common decision-support tools across educational and professional contexts, raising questions about how their outputs shape human critical thinking. Prior work suggests that the amount of AI assistance can influence cognitive engagement, yet little is known about how specific properties of LLM outputs (e.g., response length) impacts users’ critical evaluation of information. In this study, we examine whether the length of LLM responses shapes users’ accuracy in evaluating LLM-generated reasoning on critical thinking tasks, particularly in interaction with the correctness of the LLM’s reasoning. To begin evaluating this, we conducted a within-subjects experiment with 24 participants who completed 15 modified Watson–Glaser critical thinking items, each accompanied by an LLM-generated explanation that varied in length and correctness. Mixed-effects logistic regression revealed a strong and statistically reliable effect of LLM output correctness on participant accuracy, with participants more likely to answer correctly when the LLM’s explanation was correct. Response length appeared to moderated this effect: when the LLM output was incorrect, medium-length explanations were associated with higher participant accuracy than either shorter or longer explanations, whereas accuracy remained high across lengths when the LLM output was correct. Together, these findings suggest that response length alone may be insufficient to support critical thinking, and that how reasoning is presented—including a potential advantage of mid-length explanations under some conditions—points to design opportunities for LLM-based decision-support systems that emphasize transparent reasoning and calibrated expressions of certainty. † † cc-license: by 1. Introduction Usage of large language model (LLM) tools has fast become ubiquitous in assisting with learning, decision making, and a wide variety of tasks in personal, scholastic, and professional settings. This rapid expansion of LLM use has outpaced our understanding of how these tools affect human cognition and judgment, even as they are increasingly embedded within everyday workflows. While LLMs are effective at quickly generating solutions and suggestions, people still must evaluate the quality of those suggestions and estimate appropriate trust to decide how, or whether, to act on them. As a result, LLMs do not replace human reasoning so much as reshape when, where, and how cognitive effort is applied. However, we do not yet fully understand how exposure to large volumes of confidently presented text, particularly when that text contains errors, influences human judgment and decision making. LLM outputs may appear coherent and authoritative at first glance, even when they are incomplete, misleading, or incorrect. This raises broader questions about how LLM-assisted interactions shape critical thinking. More specifically, does the presence of LLM assistance affect how people encode information, allocate cognitive effort, and evaluate decisions? Prior research on LLMs has largely focused on properties of the models themselves, including hallucinations, bias, error, misinformation, and trustworthiness ( 2 ) , ( 15 ) , ( 8 ) , ( 4 ) . Although this work is essential, it often treats users as passive recipients of LLM output rather than active evaluators of information. Comparatively little research has examined how characteristics of LLM responses themselves, such as information structure (e.g., paragraphs versus bullet points), reasoning depth, or response length, shape information assimilation and critical engagement. Many contemporary LLMs present an answer followed by an explanation and then restate the answer, with explanations commonly framed as mechanisms for improving transparency, trust, and understanding. Although explanations may support comprehension, they can also introduce risks, including inflated trust driven by overconfident reasoning, distorted mental models, and reduced critical scrutiny. Such risks may be especially pronounced when explanations are lengthy, fluent, or incorrect. In LLM interfaces, answers are often accompanied by explanations. These explanations are intended to improve transparency and support appropriate trust. However, explanation properties such as length may also influence user reasoning in unintended ways. In this work, we examine LLM output features that may exacerbate explainability pitfalls, focusing on LLM output length and correctness. We foreground how these LLM output characteristics influence human reasoning. By doing so, this work aims to contribute to research on human–LLM interaction and has implications for how LLMs are used in practice, as well as for the design of LLM output architectures and user interfaces. 2. Background Understanding how people think has become increasingly important as AI systems play a larger role in everyday reasoning. In this section, we outline foundational theories and measurements of critical thinking and describe how AI assistance can both support and hinder critical thought. Finally, we survey current AI-driven decision-support tools in enterprise settings to illustrate how large language models (LLMs) are redesigning the presentation of information and, in turn, influencing human judgment. 2.1. Critical Thinking: Conception & Measurement Critical thinking is a core facet of human performance in nearly all scholastic and vocational settings ( 3 ; 11 ; 6 ) . Critical thinking is often misunderstood and used as a catch-all to describe intelligence. This is because it is an umbrella term—or summation—of peoples ability to practice deductive reasoning, analyze arguments, understand likelihoods, make decisions, problem solve, and engage in creativity ( 6 ) . For this reason, people with strong critical thinking skills are highly sought after in most professional settings ( 1 ; 6 ) . Yet critical thinking itself is difficult to teach ( 14 ) . For a working definition, critical thinking is an outcome of a variety of meta-thinking or meta-cognitive skills and strategies ( 14 ; 6 ) . However, it requires more than just “thinking about thinking” ( 6 ) . In effect, critical thinking requires people to consider, carefully, not just a decision or conclusion that is being made, but the steps by which people arrive at it and the logic, reason, or facts that support it ( 6 ) . More specifically, is about moving beyond retention of rote information and learning how to masterfully apply said knowledge ( 6 ) . It is understanding not just how you arrived at a thought but how others may arrive at your same perspective, as well as alternate or even opposing perspectives. A common example in espousing the value of critical thinking stems from legal professions (e.g., trial lawyers) where the success often derives from someones ability to carefully map out, and then lead others, to very specific points of view while also predicting likely deviations of thought which could lead people (e.g., a jury or judge) to undesirable outcomes of position (e.g., finding someone guilty who is, in fact innocent). Similarly, in business settings it can be applied towards predicting likely customer or client behavior or considering the next moves of market competitors. Within educational and professional research, critical thinking has been examined closely, at length, and for many decades ( 6 ) . The first, and still foremost, measurement

Read the complete text