Understanding the LLM-ification of CHI: Unpacking the Impact of LLMs at CHI through a Systematic Literature Review DOI: XXXXXXX.XXXXXXX Conference: Proceedings of the CHI Conference on Human Factors in Computing Systems; April 26–May 1, 2025; Yokohama, Japan CHI Conference on Human Factors in Comput

Rock Yuren Pang, Hope Schroeder, Kynnedy Simone Smith, Solon Barocas, Ziang Xiao, Emily Tseng, Danielle Bragg · 2025

Read Understanding the LLM-ification of CHI: Unpacking the Impact of LLMs at CHI through a Systematic Literature Review DOI: XXXXXXX.XXXXXXX Conference: Proceedings of the CHI Conference on Human Factors in Computing Systems; April 26–May 1, 2025; Yokohama, Japan CHI Conference on Human Factors in Comput by Rock Yuren Pang, Hope Schroeder, Kynnedy Simone Smith, Solon Barocas, Ziang Xiao, Emily Tseng, Danielle Bragg (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-12557.txt.

Introduction

Rock Yuren Pang and co-authors review how large language models have spread through a top human-computer interaction conference, CHI. The models reshape both what researchers study and how they work, yet their uptake had not been mapped. The team read 153 CHI papers from 2020 to 2024 that engage with such models.

The papers span ten domains, with most adding hands-on tests or built tools. Models played five roles, from research aid to stand-in user. Authors often worried about whether results hold up or can be rerun, and most studied closed models they cannot inspect. The review closes with questions to help researchers judge when model use is sound.

Catalog record

CREATORRock Yuren Pang, Hope Schroeder, Kynnedy Simone Smith, Solon Barocas, Ziang Xiao, Emily Tseng, Danielle Bragg
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE88 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-12557

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Understanding the LLM-ification of CHI: Unpacking the Impact of LLMs at CHI through a Systematic Literature Review DOI: XXXXXXX.XXXXXXX Conference: Proceedings of the CHI Conference on Human Factors in Computing Systems; April 26–May 1, 2025; Yokohama, Japan CHI Conference on Human Factors in Computing Systems (CHI ’25), April 26–May 1, 2025, Yokohama, Japan 3320 CCS: Human-centered computing HCI theory, concepts and models Rock Yuren Pang, Hope Schroeder, Kynnedy Simone Smith, Solon Barocas, Ziang Xiao, Emily Tseng, Danielle Bragg Abstract . Large language models (LLMs) have been positioned to revolutionize HCI, by reshaping not only the interfaces, design patterns, and sociotechnical systems that we study, but also the research practices we use. To-date, however, there has been little understanding of LLMs’ uptake in HCI. We address this gap via a systematic literature review of 153 CHI papers from 2020-24 that engage with LLMs. We taxonomize: (1) domains where LLMs are applied; (2) roles of LLMs in HCI projects; (3) contribution types; and (4) acknowledged limitations and risks. We find LLM work in 10 diverse domains, primarily via empirical and artifact contributions. Authors use LLMs in five distinct roles, including as research tools or simulated users. Still, authors often raise validity and reproducibility concerns, and overwhelmingly study closed models. We outline opportunities to improve HCI research with and on LLMs, and provide guiding questions for researchers to consider the validity and appropriateness of LLM-related work. 1. Introduction Large language models (LLMs) are poised to transform the landscape of Human-Computer Interaction (HCI) research. Already, researchers have been using LLMs across the HCI research pipeline, from ideation and system development to data analysis and paper-writing ( Kapania et al. 2024 ) . Past work has shown rapid growth in the raw count of LLM-focused paper preprints, especially in HCI topics ( Movva et al. 2024 ) . The explosion of LLM-related research has also led to rising discourse in HCI on the opportunities and challenges of LLM usage, including interview and survey studies with researchers to understand their practices ( Kapania et al. 2024 ) , and workshops ( Aubin Le Quéré et al. 2024 ; Prpa et al. 2024 ) and social media commentary ( Hullman 2024 ) in which scholars debate how the field ought to respond. The surge in LLM-related papers and discussions indicates a growing need to support scholars in understanding the potential and pitfalls of LLMs in HCI. Such inquiry is consequential not only for HCI, but also for the broader landscape of computing research. On one hand, scholars in natural language processing (NLP) and machine learning (ML) increasingly look to incorporate human evaluation in LLM architectures, via techniques like reinforcement learning for human feedback (RLHF) that draw upon HCI methodologies ( Lee et al. 2022b ; Liao and Xiao 2023 ; Xiao et al. 2024 ; Heuer and Buschek 2021 ) . On the other hand, researchers across various communities, such as science and technology studies (STS), computer-supported cooperative work (CSCW), and fairness, accountability, and transparency (FAccT) have called for reflection on the potential negative impacts ( Smith et al. 2022 ; Hecht et al. 2021 ; Shen et al. 2023 ) , including a rising chorus of scholars exploring the societal implications of LLM development and the need for responsible AI practices ( Wang et al. 2024c ; Do* et al. 2023 ; Amershi et al. 2019 ) . As various research communities increasingly pursue human-centered methods and questions, there emerges an urgent need for we as the HCI community to scrutinize our own field, and to develop standards for researchers using LLMs and asking HCI-oriented questions. This work is motivated by the growing need to scrutinize how HCI methodologies are shaping and being shaped by LLM development, ensuring that their influence aligns with scientific rigor and societal benefit. To this end, we contribute a systematic literature review of the 153 LLM-related papers in the last five years of CHI proceedings (2020-2024). Our key research questions include: Where have LLMs been applied at CHI? How have researchers used LLMs in their papers? What contributions have LLM-related scholarship made to HCI? What concerns around LLMs do authors articulate? Our intended audience includes both HCI researchers exploring LLM integration in HCI work and LLM practitioners seeking to understand the current best practices around using LLMs to interact with humans in various domains. We identify that LLMs have been taken up in 10 diverse application domains, including Communication & Writing, Augmenting Capabilities, Education, Responsible Computing, Programming, Reliability & Validity of LLMs, Well-being & Health, Design, Accessibility & Aging, and Creativity. Applying Wobbrock and Kientz 2016 ’s framework of research contributions in HCI, we found that LLMs were overwhelmingly used for empirical and artifact contributions, with limited work in theoretical or methodological advances. To characterize how LLMs are affecting the lifecycles of HCI projects, we identified five roles that LLMs play in research projects: LLMs as system engines, LLMs as research tools, LLMs as participants or users, LLMs as objects of study (e.g., through audits), and users’ perceptions of LLMs (e.g., through interview studies of LLM users’ experiences). We also identified 22 common limitations and risks that authors acknowledged, ranging from qualms around LLMs’ performance to concerns around research validity, resource constraints, and potential consequences. We found that authors often raise validity and reproducibility concerns around LLM research, despite overwhelmingly studying closed-source LLMs. Overall, this work presents an in-depth investigation of the current landscape of how HCI applies and studies LLMs. Towards more rigorous research and responsible design with LLMs, we outline directions for future HCI research at the LLM frontier, and provide actionable recommendations to researchers and practitioners. In summary, we contribute: • a systematic literature review of 153 LLM-related papers from CHI proceedings 2020-2024, resulting in 10 domains where LLMs have been applied, 5 roles that LLMs play in HCI projects, and 29 limitations described by authors; • opportunities for HCI research to leverage LLMs, including under-researched application domains, contribution types, and methodological gaps; • guiding questions for HCI researchers to consider the validity and appropriateness of a proposed LLM-related study; • an open-source dataset of 153 sampled papers from CHI 2020-2024 with our qualitative codes and paper metadata, publicly available at https://github.com/rrrrrrockpang/llm-chi . 2. Related Work 2.1. Literature Reviews in HCI HCI has a rich tradition of using systematic literature reviews to identify patterns, trends, and limitations of a research area ( Stefanidi et al. 2023 ) . Such reviews provide conceptual frameworks for shared understanding across the field. Many prior works qualitatively analyzed their paper samples to surface high-level themes. For example, Mack et al. 2021 examined 836 accessibility papers over 26 years, coding for common contribution types, communities of focus, and methods. Stefanidi et al. 2023 annotated 189 HCI literature surveys 1982-2022 to explain current contributions and topics within HCI. Similarly, Dell and Kumar 2016 manually reviewed 259 HCI4D publications to provide an overview of the space. Caine 2016 synthesized standards for sample sizes at CHI by manually extracting data from each CHI2014 manuscript . Quantitative methods have also been employed to provide broader perspectives on HCI research trends. Liu et al. 2014 used hierarchical clustering, strategic analysis, and network analysis to map the evolution of

Read the complete text