Moving Beyond LDA: A Comparison of Unsupervised Topic Modelling Techniques for Qualitative Data Analysis of Online Communities DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation emai; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/

Amandeep Kaur, James R. Wallace · 2024

Read Moving Beyond LDA: A Comparison of Unsupervised Topic Modelling Techniques for Qualitative Data Analysis of Online Communities DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation emai; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/ by Amandeep Kaur, James R. Wallace (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-14486.txt.

Introduction

Amandeep Kaur and James Wallace compare topic modeling tools for studying online communities in this 2024 paper. Social media overflows with text that qualitative researchers want to read closely, but few of them code. Software that clusters posts into topics could help, if the topics make sense to a human reader.

Twelve qualitative researchers tried three methods hands on: the classic LDA, a matrix method called NMF, and BERTopic, which rides on large language models. Eight of the twelve favored BERTopic for detailed, coherent clusters that supported deeper reading and action. Across methods, users prized relevance, tidy organization, and the power to surface surprise links. The authors see language model tools lowering the barrier for qualitative work.

Catalog record

CREATORAmandeep Kaur, James R. Wallace
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE76 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-14486

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Moving Beyond LDA: A Comparison of Unsupervised Topic Modelling Techniques for Qualitative Data Analysis of Online Communities DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation emai; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/18/06 CCS: Human Computer Interaction Generate the Correct Terms for Your Paper CCS: Qualitative Analysis Generate the Correct Terms for Your Paper CCS: Topic modelling Generate the Correct Terms for Your Paper CCS: Large Language Model Generate the Correct Terms for Your Paper Amandeep Kaur, James R. Wallace Abstract . Social media constitutes a rich and influential source of information for qualitative researchers. Although computational techniques like topic modelling assist with managing the volume and diversity of social media content, qualitative researcher’s lack of programming expertise creates a significant barrier to their adoption. In this paper we explore how BERTopic, an advanced Large Language Model (LLM)-based topic modelling technique, can support qualitative data analysis of social media. We conducted interviews and hands-on evaluations in which qualitative researchers compared topics from three modelling techniques: LDA, NMF, and BERTopic. BERTopic was favoured by 8 of 12 participants for its ability to provide detailed, coherent clusters for deeper understanding and actionable insights. Participants also prioritised topic relevance, logical organisation, and the capacity to reveal unexpected relationships within the data. Our findings underscore the potential of LLM-based techniques for supporting qualitative analysis. 1. Introduction For qualitative researchers, social media serves as a rich and influential source of information ( Andreotta et al. 2019 ) . Reddit, in particular, has emerged as a crucial platform for gathering diverse and localized content; offering opinions, advice, and recommendations from trusted sources ( Ryan et al. 2017 ; Ahn and Shin 2013 ; Allen et al. 2014 ) . This geographically specific and up-to-date information is invaluable for aiding research, synthesis, and presentation of data ( Dugan et al. 2008 ; Skeels and Grudin 2009 ; Steinfield et al. 2009 ; Morris et al. 2010 ) . Researchers leverage social media to uncover nuanced insights in fields such as mental health, politics, consumer behavior, health and fitness, technology, gaming, education, and social movements ( Andreotta et al. 2019 ; Ahn and Shin 2013 ; Kapoor et al. 2018 ) . For instance, public health researchers use Reddit to study health-related topics and understand community perspectives and communication norms, which aids in developing more effective interventions and policies ( Eysenbach 2005 ; Heldman et al. 2013 ; Gauthier et al. 2022 ; Rotolo et al. 2022 ; Gauthier et al. 2023 ) . Despite the availability of social media data, the sheer volume, diversity, noise, dynamic nature, and contextual nuances of this data present significant challenges ( Andreotta et al. 2019 ) . Computational techniques like topic modelling are thus necessary to effectively analyze this extensive content ( Rana et al. 2016 ) . However, traditional topic modelling methods like Latent Dirichlet Allocation (LDA), Biterm Topic Model (BTM), and Non-Negative Matrix Factorization (NMF) often fail to capture nuanced meanings and contextual usage and require extensive data cleaning and pre-processing, making them labor-intensive ( Yan et al. 2013 ; Mazarura and De Waal 2016 ; Zou and Song 2016 ) . In contrast, Large Language Model (LLM) based techniques like BERTopic, Top2Vec and GPT-3 offer substantial improvements. They provide advanced understanding of text with minimal pre-processing, adapting efficiently to various contexts ( Brown et al. 2020 ; Devlin et al. 2019 ; Vaswani et al. 2017 ) . Further, qualitative researchers face significant challenges when integrating computational techniques like topic modelling into their qualitative workflows due to a lack of programming expertise ( Lochmiller 2021 ; Baden et al. 2022 ) . They also have concerns about losing control over the nuanced aspects of their research and doubts about the accuracy of computer-generated insights ( Abram et al. 2020 ; Jiang et al. 2021 ; Feuston and Brubaker 2021 ) . Ethical and privacy concerns further contribute to this reluctance ( Siiman et al. 2023 ) . Nevertheless, AI tools can significantly aid in analyzing large volumes of social media data. These tools efficiently process and analyze data, identifying trends, sentiments, and patterns that would be challenging to uncover manually ( Batrinca and Treleaven 2015 ) . Researchers could utilize AI’s capability for deep data analysis, providing a comprehensive understanding of social media interactions ( Fan and Gordon 2014 ) . In particular, topic modelling is increasingly being integrated into qualitative research to manage and interpret large datasets, offering deeper and more objective insights ( Gillies et al. 2022 ) . Our research is motivated by the desire to assist qualitative researchers who often face challenges with manual, time-consuming, and potentially biased data analysis methods due to limited programming skills. Recognizing these challenges, we aim to explore the potential of Large Language Models (LLMs) for unsupervised topic modelling, offering a more efficient and objective approach to data analysis. By understanding how qualitative researchers currently use topic modelling and identifying the specific difficulties they encounter, we can develop better tools to support their work. We integrated BERTopic into the Computational Thematic Analysis (CTA) toolkit ( Gauthier and Wallace 2022 ) . Integrating BERTopic into the CTA Toolkit required architectural changes to accommodate the model’s advanced requirements which uses word embeddings as compared to the original bag of words methodology, and leverages transformer-based processing. This included incorporating GPU utilization to meet BERTopic’s high computational demands and ensuring efficient processing of large datasets. Additionally, changes to the data filtering methods were necessary to ensure improved data processing and data integrity. We then conducted interviews with qualitative researchers to discuss the challenges they face with existing topic modelling tools. Following these discussions, researchers engaged with the CTA toolkit, applying it to their own datasets and models to identify their preferred options. This hands-on exploration allowed them to evaluate the tools in the context of their specific needs. The choices and reasons for their preferences were then collaboratively analyzed, ensuring that the selected topic modelling techniques aligned well with the researchers’ needs and preferences. Participants prioritized topic relevance, logical organization, and the capacity to reveal unexpected yet significant relationships within the data when evaluating topic modelling techniques. They desired detailed, coherent clusters that clearly separated and grouped significant topics, which facilitated deeper understanding and actionable insights, which were provided by BERTopic. Despite some visualization shortcomings, they valued BERTopic’s ability to uncover hidden connections, emphasizing the need for meaningful, comprehensive analysis tools that support their research objectives and enhance data interpretation. BERTopic was ranked first by 8 out of 12 participants (67%), LDA by 3 out of 12 participants (25%), and NMF by 1 out of 12 participants (8%). In summary, we make three contributions: (1) We integrated BERTopic into the Computational Thematic Analysis Toolkit. (2) We conducted interviews with qualitative researchers to discuss the challenges they face when using topic modelling tools to analyse social media data. (3) We discuss the findings on researchers’ priorities in topic modelling techniques. 2. Related Work The HCI community is increasingly exploring the use of social media to gain insights into human behavior ( Gauthier 2020 ; Gauthier et al. 2023 ; Andalibi et al. 2016 ; Ammari et al.

Read the complete text