Investigating the Effects of LLM Use on Critical Thinking Under Time Constraints: Access Timing and Time Availability Conference: Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems; April 13–17, 2026; Barcelona, Spain Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems (CHI ’26), April 13–17, 2026, Barcelona, Spain DOI: 10.1145/3772318.3791796 ISBN: 979-8-4007-2278-3/2026/04 CCS: Human-centered computing Empirical studies in HCI CCS: Human-centered computing Laboratory experiments CCS: Applied computing Education
Jiayin Zhi, Harsh Kumar, Mina Lee · 2026
Read Investigating the Effects of LLM Use on Critical Thinking Under Time Constraints: Access Timing and Time Availability Conference: Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems; April 13–17, 2026; Barcelona, Spain Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems (CHI ’26), April 13–17, 2026, Barcelona, Spain DOI: 10.1145/3772318.3791796 ISBN: 979-8-4007-2278-3/2026/04 CCS: Human-centered computing Empirical studies in HCI CCS: Human-centered computing Laboratory experiments CCS: Applied computing Education by Jiayin Zhi, Harsh Kumar, Mina Lee (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-08849.txt.
Introduction
Jiayin Zhi and colleagues ask when language-model help helps or harms critical thinking. In a 393-person test, models were offered early, throughout, late, or never, under short or ample time. Help from the start lifted scores under pressure but hurt with ample time, while starting alone showed the mirror pattern, making timing central to human-model teamwork.
Catalog record
| CREATOR | Jiayin Zhi, Harsh Kumar, Mina Lee |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 96259 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.08849
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Investigating the Effects of LLM Use on Critical Thinking Under Time Constraints: Access Timing and Time Availability Conference: Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems; April 13–17, 2026; Barcelona, Spain Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems (CHI ’26), April 13–17, 2026, Barcelona, Spain DOI: 10.1145/3772318.3791796 ISBN: 979-8-4007-2278-3/2026/04 CCS: Human-centered computing Empirical studies in HCI CCS: Human-centered computing Laboratory experiments CCS: Applied computing Education Jiayin Zhi, Harsh Kumar, Mina Lee Abstract . The impact of large language models (LLMs) on critical thinking has provoked growing attention, yet this impact on actual performance may not be uniformly negative or positive. Particularly, the role of time—the temporal context under which an LLM is provided—remains overlooked. In a between-subjects experiment (n=393), we examined two types of time constraints for a critical thinking task requiring participants to make a reasoned decision for a real-world scenario based on diverse documents: (1) LLM access timing—an LLM available only at the beginning (early), throughout (continuous), near the end (late), or not at all (no LLM), and (2) time availability—insufficient or sufficient time for the task. We found a temporal reversal: LLM access from the start (early, continuous) improved performance under time pressure but impaired it with sufficient time, whereas beginning the task independently (late, no LLM) showed the opposite pattern. These findings demonstrate that time constraints fundamentally shape whether an LLM augments or undermines critical thinking, making time a central consideration when designing LLM support and evaluating human-AI collaboration in cognitive tasks. Figure 1 . This 4\times 2 experimental framework tests the effects of LLM use on critical thinking under two types of time constraints: access timing (x-axis) and time availability (y-axis). Participants, working under time pressure ( Insufficient ) or not ( Sufficient ), were randomized to receive LLM access only at the beginning of the task ( Early LLM access), throughout ( Continuous LLM access), near the end ( Late LLM access), or not at all ( No LLM access). † † cc-license: by 1. Introduction Critical thinking—the ability to reason through diverse and sometimes conflicting information by analysis, evaluation, and synthesis to reach reasoned decisions—is fundamental for our work and life ( 74 ; 7 ; 46 ) . In online information environments, whether navigating conflicting news, or making judgments based on scattered documents, people always face situations that require critical thinking and apply the reasoned conclusions to various purposes (e.g., solve a problem, decide on an action, or answer a given question), which involves distilling argumentative information from relevant sources, analyzing source credibility, avoiding bias, and communicating reasoning succinctly ( 25 ; 74 ; 7 ; 46 ) . As generative AI (GenAI), especially large language models (LLMs), have been increasingly integrated into information technologies, critical thinking is highly exposed to them. In fact, critical thinking is the most frequently required capability in tasks that users bring to popular LLM-powered chatbots ( 31 ) . Recent user research provides valuable initial insights on the connection between AI use and critical thinking. A survey collected examples of how knowledge workers enact critical thinking when using GenAI for different work tasks ( 50 ) . A mixed-methods study investigating how general users perceive their critical thinking abilities found that using AI tools can reduce their cognitive effort ( 25 ) . For example, ready-made AI summaries eliminate the need to comprehend sources or evaluate evidence ( 25 ) . However, these methods fall short in establishing causal relationships and objectively measuring performance, leaving the actual performance consequences unclear. Beyond user research and self-reports, experiments using tasks that assess critical thinking performance are needed to determine how the impact of LLMs actually manifests. Moreover, the impact of LLMs on critical thinking may not be uniformly negative or positive, but depend on the temporal context in which the LLM is provided. Two types of time constraints emerge as particularly crucial for human-AI collaboration in cognitive tasks requiring extended reasoning. First, the timing of LLM access. Recent studies show the benefits of having LLM access in the late time period for solving math problems ( 48 ) and creative writing ( 69 ) . However, for critical thinking, there are also reasons to hope that early access to an LLM could be beneficial. It could serve as initial scaffolding and handle preliminary groundwork—such as gathering available information and grasping basic concepts—potentially freeing cognitive resources for deep deliberation ( 46 ; 76 ; 36 ; 50 ) . Second, time availability. In practice, people often face time pressure for the tasks they do, whether from explicit deadlines or implicit expectations ( 93 ; 4 ; 42 ; 50 ; 33 ) . Research suggests that time pressure can shape cognitive performance, shifting people from deliberative reasoning to heuristic processing ( 44 ; 57 ; 42 ) . These two temporal dimensions inherently interact in reality, yet previous research has examined a subset of conditions within each temporal dimension in isolation across disparate domains. This fragmented picture leaves us with critical gaps in understanding how time constraints shape LLMs’ impact on critical thinking under realistic conditions where both temporal dimensions operate simultaneously. In this work, we investigated how LLMs influence participants’ critical thinking task performance under two types of time constraints: (1) LLM access timing and (2) time availability. We tested four LLM access timings: Early , Continuous , Late , and No LLM access, where an LLM was available only at the beginning of the task, throughout, near the end, or not at all, respectively. For time availability, participants had either Insufficient or Sufficient time for the task. Based on these time constraints, we asked: RQ1.: How does LLM access timing affect critical thinking task performance under different time availability? RQ2.: How does time availability affect critical thinking task performance when having the same LLM access timing? To answer these questions, we conducted a preregistered 1 1 1 https://aspredicted.org/q6yh-mrqg.pdf 4\times 2 between-subjects experiment ( n=393 ). We employed the iPAL (International Performance Assessment of Learning) framework ( 74 ; 7 ; 46 ) developed for assessing the authentic performance of critical thinking. A critical thinking performance assessment task ( 7 ; 18 ) ask participants to make a reasoned decision for a real-world scenario in civic life based on a set of documents of varying characteristics. Participants need to navigate these documents, analyze, and draw inferences from evidence, evaluate the sources, and synthesize conflicting perspectives into a reasoned written decision—capturing the non-linear, reciprocal mental processes that characterize critical thinking. Under Insufficient or Sufficient time, participants were randomly assigned to one of the four LLM access timings, and completed the task. Performance was primarily evaluated based on the Essay written as part of the task. Then, we additionally measured participants’ performance of Recall (remembering provided documents), Evaluation (evaluating the characteristics of source documents), and Comprehension (understanding documents by evidence-based inference), to capture cognitive activities that could influence critical thinking. After the task, participants completed a critical thinking self-assessment ( 66 ; 85 ) . Overall, our results revealed that the effect of LLM use on critical thinking task performance fundamentally depends on LLM access timing and time availability (Section 4