Counterargument for Critical Thinking as Judged by AI and Humans Journal: Open
Tosin Adewumi * , Marcus Liwicki, Foteini Simistira Liwicki, Lama Alkhaled, Hamam Mokayed, Esra Sümer-Arpak Machine Learning Group, EISLAB, Luleå University of Technology, Sweden. [email protected] · 2026
Read Counterargument for Critical Thinking as Judged by AI and Humans Journal: Open by Tosin Adewumi * , Marcus Liwicki, Foteini Simistira Liwicki, Lama Alkhaled, Hamam Mokayed, Esra Sümer-Arpak Machine Learning Group, EISLAB, Luleå University of Technology, Sweden. [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-05353.txt.
Introduction
Tosin Adewumi and five co-authors test whether asking students to write counterarguments against AI-generated text builds critical thinking, and whether AI judges can grade that work fairly. Thirty-six students in one university course chose among four debated thesis statements and wrote counterarguments, amid wider worries about cheating and cognitive offloading with generative tools. Each qualified essay received three human assessments using six established rubrics.
Six frontier language models then graded the same essays with the same rubrics and guidelines. The students' own counterarguments showed logic and other markers of critical thinking, the authors report. Model assessments generally aligned with human ones, suggesting rubric-guided AI grading can work at scale for this kind of writing task.
Catalog record
| CREATOR | Tosin Adewumi * , Marcus Liwicki, Foteini Simistira Liwicki, Lama Alkhaled, Hamam Mokayed, Esra Sümer-Arpak Machine Learning Group, EISLAB, Luleå University of Technology, Sweden. [email protected] |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.05353
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Counterargument for Critical Thinking as Judged by AI and Humans Journal: Open Tosin Adewumi * , Marcus Liwicki, Foteini Simistira Liwicki, Lama Alkhaled, Hamam Mokayed, Esra Sümer-Arpak Machine Learning Group, EISLAB, Luleå University of Technology, Sweden. [email protected] Abstract This intervention study investigates the use of counterarguments in writing for critical thinking by students in the context of Generative AI (GenAI) . This is especially as risks of cheating and cognitive offloading exist with the use of GenAI . We presented 36 students in a particular university course with 4 carefully selected thesis statements (from a set of popular debates) to write about anyone of them. We used six established rubrics (focus, logic, content, style, correctness and reference) to conduct three human assessments (two student peer-reviews and one experienced teacher) per writeup on a 5-point Likert scale for all the qualified samples ( n ) of 35 submissions (after disqualifying one for irregularity). Using the same rubrics and guidelines, we also assessed the submissions using six frontier LLM s as judges. Our mixed-method design included qualitative open-ended feedback per assessment and quantitative methods. The results reveal that (1) the students’ self-written counterarguments to AI -generated content contains logic, among other things, which is a key component of critical thinking, and (2) GenAI can be successfully used at scale to assess students’ written work, based on clear rubrics, and these assessments generally align with human assessments as shown with Gwets AC2 inter-rater reliability values of 0.33 for all the models except one. 1 Introduction Education is evolving rapidly, particularly its pedagogy ( Zou et al. 2025 ) , the engagement and evaluation of students ( Ennis and Weir 1985 ; Halpern 2006 ) , and the end-to-end planning of learning ( Romiszowski 2016 ) . This is thanks to Generative AI (GenAI) , particularly large language model s ( LLM s), which have many benefits ( Adewumi et al. 2025d ; Mulaudzi and Hamilton 2025 ) . Recent studies, however, have shown that people who used ChatGPT or similar tools to write essays demonstrated less brain activity in areas linked with cognitive processing or reported negative impact to their skills ( Kosmyna et al. 2025a ; Helal et al. 2025 ) . This behaviour of shifting cognitive tasks to GenAI is termed cognitive offloading and may lead to cognitive atrophy, where one’s abilities become worse ( Kosmyna et al. 2025a ; Gerlich 2025a ) . Given that GenAI is most likely here to stay, it is essential and beneficial to find ways that students can engage with it such that it encourages critical thinking, thereby boosting their skills and learning. Critical thinking is a purposeful, reflective meta-cognitive process that increases the possibility of forming a robust and logical conclusion to an argument or finding a solution to a problem ( Facione 1990 ; Ku 2009 ; Lai 2011 ; Dwyer et al. 2014 ) . It emphasizes moving beyond passive information intake to examining evidence, identifying assumptions, comparing perspectives, and using logic to determine the strength of a position ( Kuhn 1991 ; Sinfield and Burns 2023 ) . Meanwhile, a counterargument is a logical argument that offers a different viewpoint to an existing one, thereby constituting a debate ( Fulan et al. 2025 ) . While there are differences of opinion on what the components of critical thinking are ( Dwyer et al. 2014 ) , logic is widely seen as a fundamental ingredient ( Adewumi et al. 2025a ; Dwyer et al. 2014 ) . Assessing critical thinking through open-ended formats, such as with written counterarguments, is favored by many researchers because the format seeks to evaluate the logic in an existing content. One example of an open-ended format is the Ennis–Weir Critical Thinking Essay Test (EWCTET) ( Ennis and Weir 1985 ; Lai 2011 ) . However, objectively assessing students’ writing for critical thinking can be a challenging task for many reasons, including the differences of opinion on the details of critical thinking ( Dwyer et al. 2014 ) . The Likert scale offers a possible solution and has been used by other researchers ( Joshi et al. 2015 ; Alkharusi 2022 ; Gerlich 2025b ) . In this work, our motivation is to answer two research questions : (1) Do counterarguments to AI -generated arguments promote students’ critical thinking in writing, in terms of logic and relevant rubrics? and (2) How similar (or dissimilar) are GenAI systems in judging counterarguments compared to humans, given the same rubrics? This is more so that the topic of how student-written counterarguments to AI -generated arguments across diverse topics promote critical thinking in students is understudied in the literature, since argument-based learning appears under-utilized in pedagogy compared to traditional transmissive teaching methods ( González et al. 2026 ) . We used a mixed research design involving qualitative and quantitative methods, with a sample size ( n ) of 35 submissions and the 5-point Likert scale for the rubrics. We publicly release the data artifacts. 1 1 1 github.com/LTU-Machine-Learning/counterargument_ai Our key contributions are: 1. We show that students’ self-written counterarguments to AI -generated content contains logic, among other things, which is a key component of critical thinking. 2. We show that GenAI can be successfully used to assess students’ written work (particularly counterarguments) based on clear rubrics and its assessments generally align with human assessments, including expert review and students’ peer review. The rest of this paper is organised as follows. In Section 2 , we describe the theoretical framework in the literature. In Section 3 , we fully describe the method of this study, including the research design, participants and the checks for AI -generated counterarguments. In Section 4 , we present our findings using multiple charts. In Section 5 , we discuss the implications of this study in connection to theory. Finally, in Section 6 , we provide concluding remarks. 2 Theoretical Framework We conduct a relatively thorough review of the literature on some of the theoretical underpinnings of critical thinking, counterarguments, and GenAI assessment. 2.1 Models of Critical Thinking Bloom et al. 1956 ’s taxonomy of educational objectives is one of many models of thinking applicable to critical thinking because it contains the analysis of knowledge ( Dwyer et al. 2014 ) . The model has influenced many others, including Anderson and Krathwohl 2001 ’s revised taxonomy for learning, teaching, and assessing, Duron et al. 2006 ’s 5-step framework, and Romiszowski 2016 ’s course design using a systems approach. In the first layer of the model by Bloom et al. 1956 is knowledge , which not only relates to the specifics and terminologies of a content but knowledge about the ways of dealing with those specifics. In the second layer is comprehension , involving explaining and summarizing learned information, while in the third layer is application . In the fourth layer is analysis of elements and how they relate to one another. In the fifth layer is synthesis , which involves the production of a plan or new communication, while in the final, sixth, layer is evaluation . The Delphi panel (of 46 experts in critical thinking) agreed that analysis, evaluation and inference are core skills for critical thinking (