GLAT: The Generative AI Literacy Assessment Test

Yueqiao Jin, Roberto Martinez-Maldonado, Dragan Gašević, Lixiang Yan · 2024

Read GLAT: The Generative AI Literacy Assessment Test by Yueqiao Jin, Roberto Martinez-Maldonado, Dragan Gašević, Lixiang Yan (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-00283.txt.

Description

Abstract The rapid integration of generative artificial intelligence (GenAI) technology into education necessitates precise measurement of GenAI literacy to ensure that learners and educators possess the skills to engage with and critically evaluate this transformative technology effectively. Existing instruments often rely on self-reports, which may be biased.

Catalog record

CREATORYueqiao Jin, Roberto Martinez-Maldonado, Dragan Gašević, Lixiang Yan
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE71 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2411-00283

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

GLAT: The Generative AI Literacy Assessment Test Yueqiao Jin, Roberto Martinez-Maldonado, Dragan Gašević, Lixiang Yan Abstract The rapid integration of generative artificial intelligence (GenAI) technology into education necessitates precise measurement of GenAI literacy to ensure that learners and educators possess the skills to engage with and critically evaluate this transformative technology effectively. Existing instruments often rely on self-reports, which may be biased. In this study, we present the GenAI Literacy Assessment Test (GLAT), a 20-item multiple-choice instrument developed following established procedures in psychological and educational measurement. Structural validity and reliability were confirmed with responses from 355 higher education students using classical test theory and item response theory, resulting in a reliable 2-parameter logistic (2PL) model (Cronbach’s alpha = 0.80; omega total = 0.81) with a robust factor structure (RMSEA = 0.03; CFI = 0.97). Critically, GLAT scores were found to be significant predictors of learners’ performance in GenAI-supported tasks, outperforming self-reported measures such as perceived ChatGPT proficiency and demonstrating external validity. These results suggest that GLAT offers a reliable and valid method for assessing GenAI literacy, with the potential to inform educational practices and policy decisions that aim to enhance learners’ and educators’ GenAI literacy, ultimately equipping them to navigate an AI-enhanced future. [orcid=0009-0003-7309-4984][orcid=0000-0002-8375-1816] [orcid=0000-0001-9265-1908] [orcid=0000-0003-3818-045X] keywords Generative AI ,AI Literacy ,Item Response Theory ,Classical Test Theory ,Higher Education ,Validity and Reliability ,Learning Performance † † corresponding: Corresponding author 1 Introduction Generative artificial intelligence (GenAI) has rapidly emerged as a transformative force in higher education, challenging traditional pedagogical frameworks while simultaneously presenting novel opportunities for teaching, learning, and assessment. Tools like OpenAI’s ChatGPT, Google’s Gemini, and Anthropic’s Claude hold the potential to transform how personalised tutoring service can be delivered, how instructional materials can be generated, how lectures can be transcribed for accessibility, and how creativity can be nurtured through multimedia content generation ( Yan et al. 2024a ; Khosravi et al. 2023 ) . However, the integration of these technologies is accompanied by complex challenges, including ethical considerations, risks of misinformation from model ”hallucinations,” and concerns regarding academic integrity ( Ji et al. 2023 ; McDonald et al. 2024 ) . Such complexities necessitate a deeper focus on fostering AI literacy, particularly GenAI literacy, among both educators and learners to fully harness GenAI’s benefits and mitigate its associated risks ( Ng et al. 2021b ; Zhao et al. 2024 ) . AI literacy refers to the set of competencies that enable individuals to effectively interact with AI technologies, encompassing understanding fundamental AI concepts, engaging in critical evaluation, and using AI tools ethically in diverse contexts ( Long and Magerko 2020 ; Ng et al. 2021b ) . Within this broad framework, GenAI literacy emerges as a specialised subset, focusing on skills required to engage with GenAI systems that can autonomously produce text, visuals, or other forms of media ( Yan et al. 2024a ; Annapureddy et al. 2024 ) . Developing GenAI literacy involves more than just foundational knowledge; it requires proficiency in crafting prompts, interpreting AI-generated outputs, and understanding the socio-ethical implications of using such tools ( Zhao et al. 2024 ; Bozkurt 2024a ) . As GenAI becomes increasingly embedded in educational systems, it is imperative for learners and educators to acquire these competencies to effectively leverage the technology while minimising potential pitfalls such as biases or inaccuracies ( Lyu et al. 2024 ; Chiu 2024 ) . Numerous instruments have been developed to assess AI literacy, reflecting the diversity of competencies that individuals need to navigate AI technologies. Conventional AI literacy assessments often rely on self-reported surveys, which are effective in capturing perceived knowledge but may lack the reliability needed to accurately measure actual competencies, especially given the tendency for individuals to overestimate their understanding ( Lintner 2024 ; Laupichler et al. 2023b ) . Most existing instruments address general AI literacy, focusing on technical knowledge, awareness, and ethical considerations, but fail to adequately capture the unique skills required for GenAI ( Koch et al. 2024 ; Zhao et al. 2024 ) . There is a growing demand for more nuanced and context-specific instruments to evaluate GenAI literacy, particularly as generative tools become integral to both physical and digital learning environments ( Koch et al. 2024 ; Zhao et al. 2024 ; Yan et al. 2024b ) . Current AI literacy assessments can be broadly categorised into two types: self-reported and performance-based measures. Self-reported instruments, while commonly used, provide insights into individuals’ perceived abilities but may introduce biases that obscure a more reliable measure of literacy levels ( Ng et al. 2021b ; Lintner 2024 ) . In contrast, performance-based assessments evaluate actual competencies through direct engagement, offering a more reliable measure of skills. This distinction is especially pertinent for GenAI literacy, where there is often a gap between learners’ perceived understanding and their real ability to effectively utilise generative tools ( Lyu et al. 2024 ) . GenAI technologies necessitate iterative, context-specific interactions that require both sophisticated prompting skills and the ability to critically assess AI outputs, areas where self-reports may fall short ( Chiu 2024 ; Bozkurt 2024a ) . Therefore, developing performance-based instruments is essential to provide a reliable assessment of individuals’ abilities to engage with these advanced technologies in educational settings. However, to the best of our knowledge, there are still limited performance-based tools for measuring students’ GenAI literacy in higher education, particularly those that have been rigorously developed and validated according to established psychological and educational measurement standards ( Thorndike et al. 1991 ; American Educational Research Association et al. 2014 ) . The current study contributes to the field of AI in education and AI literacy by introducing the GenAI Literacy Assessment Test (GLAT), a performance-based instrument specifically designed to evaluate GenAI literacy within higher education contexts. The GLAT aims to fill a critical gap in existing assessment tools by providing a more reliable, comprehensive evaluation of the key competencies required to interact with GenAI tools. Unlike existing assessments that focus predominantly on general AI skills, GLAT targets the unique skills necessary for effective engagement with generative technologies, including technical proficiency, ethical awareness, and the capacity for critical evaluation of GenAI-generated outputs. This instrument is grounded in rigorous methodologies from psychological and educational measurement ( Thorndike et al. 1991 ; American Educational Research Association et al. 2014 ) , ensuring both validity and reliability. By focusing on performance-based metrics, GLAT provides educators and researchers with a reliable tool to assess how well students and educators understand and can leverage GenAI technologies, ultimately informing targeted interventions that can enhance these competencies. 2 Background 2.1 AI and GenAI Literacy The rapid advancements in AI technologies have accentuated the importance of AI literacy, especially in educational research. Researchers have endeavoured to define and conceptualise ”AI literacy,” with the definition by Long and Magerko 2020 being frequently cited: ”a set of competencies that enables individuals to critically evaluate AI technologies; communicate and collaborate effectively with AI; and use AI as a tool online, at home, and in the workplace.” Building upon this definition, other researchers have explored essential aspects of AI literacy. For instance, Kandlhofer et al. 2016 and Burgsteiner et al. 2016 concentrated on the comprehension of fundamental AI concepts present in various

Read the complete text