Prompt Engineering for Large Language Model-assisted Inductive Thematic Analysis
Muhammad Talal Khalid, Ann-Perry Witmer · 2025
Read Prompt Engineering for Large Language Model-assisted Inductive Thematic Analysis by Muhammad Talal Khalid, Ann-Perry Witmer (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-22978.txt.
Introduction
Muhammad Talal Khalid and Ann-Perry Witmer study how to guide language models in sorting text themes in this 2025 paper. Sorting themes by hand from interviews or open text takes much time and money. Models can help, but many teams use them in ad hoc ways that are hard to check or repeat.
The authors review past work on model help for this task, with a focus on how prompts are written. From that review they set out four key steps for sound prompt design. They also discuss newer prompt methods that can help at each step. Their aim is more open, steady, and trusted use of models in theme sorting.
Catalog record
| CREATOR | Muhammad Talal Khalid, Ann-Perry Witmer |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-22978
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Prompt Engineering for Large Language Model-assisted Inductive Thematic Analysis Muhammad Talal Khalid, Ann-Perry Witmer Abstract The potential of large language models (LLMs) to mitigate the time- and cost- related challenges associated with inductive thematic analysis (ITA) has been extensively explored in the literature. However, the use of LLMs to support ITA has often been opportunistic, relying on ad hoc prompt engineering (PE) approaches, thereby undermining the reliability, transparency, and replicability of the analysis. The goal of this study is to develop a structured approach to PE in LLM-assisted ITA. To this end, a comprehensive review of the existing literature is conducted to examine how ITA researchers integrate LLMs into their workflows and, in particular, how PE is utilized to support the analytical process. Built on the insights generated from this review, four key steps for effective PE in LLM-assisted ITA are identified and extensively outlined. Furthermore, the study explores state-of-the-art PE techniques that can enhance the execution of these steps, providing ITA researchers with practical strategies to improve their analyses. In conclusion, the main contributions of this paper include: (i) it maps the existing research on LLM-assisted ITA to enable a better understanding of the rapidly developing field, (ii) it outlines a structured four-step PE process to enhance methodological rigor, (iii) it discusses the application of advanced PE techniques to support the execution of these steps, and (iv) it highlights key directions for future research. I Introduction The advent of generative artificial intelligence (AI), particularly large language models (LLMs), has profoundly transformed various research domains. From healthcare to engineering design, and from law to politics, it has impacted the day-to-day operations of society in ways that were unheard of before [ 1 ] . The more recent advancements of these models have made them cost-effective and even more accessible and available [ 2 ] . Following this curve, inductive thematic analysis (ITA) researchers have also begun to explore the application of LLMs to support their analyses. From cautious employment of LLMs as a research assistant to the proposal and development of stand-alone computer applications that can (potentially) conduct ITA with a few clicks [ 3 ] , the integration of these models into the ITA research is becoming pervasive. LLMs are specialized generative AI models that process natural language and are trained on ‘large’ amounts of textual data. They use advanced machine learning techniques (e.g., deep learning) to learn from data, understand context, identify complex patterns, and respond to user queries accordingly [ 1 ] . These capabilities enable LLMs to engage in meaningful conversations and assist in tasks that previously required exclusively human intervention [ 4 ] . There are, however, limitations that constrain the effective use of LLMs. The proprietary nature of advanced LLMs has met criticism on ethical grounds, focusing particularly on concerns related to data privacy and security [ 5 ] . In addition, the completeness, recency, and breadth of the data used to train these models also casts doubt on the truthfulness of the model responses and can lead to biases towards specific social groups and viewpoints [ 6 ] . Moreover, the LLMs’ notorious power of expression and suggestibility render them prone to hallucinations, i.e., to generate fabricated data and arguments that are misrepresented as the truth [ 5 ] . Even with all these limitations, LLMs offer complementary analytic power that qualitative researchers can leverage to critically evaluate and engage with their data. In this domain, ITA is a highly flexible and widely used systematic data analysis method to identify, analyze, and interpret patterns of meaning within qualitative data [ 7 ] . It is a flexible tool often employed by a wide range of qualitative researchers to understand experiences, thoughts, or behaviors represented in a dataset [ 8 ] . Unlike deductive approaches of qualitative inquiry in which the data are labeled according to pre-specified themes, ITA’s bottom-up focus employs the data itself to identify emerging and interesting common ways in which topics in the data are represented. Often ITA researchers follow Braun and Clarke’s six-step process to conduct their analyses [ 9 ] . According to this process, researchers first familiarize themselves with the data through repeated reading. They then generate initial codes and identify potential themes in the data. These themes are reviewed, refined, defined, named, renamed, and are finally integrated into a structured report to present the findings. As a result, performance of ITA is both costly and time-consuming, particularly when it is used to analyze large datasets. In addition, establishment of credibility requires the involvement of at least two trained researchers with relevant expertise, who must engage in multiple rounds of discussions. This labor-intensive process further increases the overall cost [ 10 ] . In this regard, the ability of LLMs to synthesize large volumes of data efficiently, enabled by their high processing speed, makes them well-suited to address some limitations of ITA. Although the integration of LLMs in ITA is still in a nascent state, there is a growing interest among researchers in their use [ 11 ] . Multiple studies have compared LLM-aided ITA to the traditional counterparts to assess reliability and establish confidence in their use (see Section III ). The similarity between the two analyses reported in these studies underscores the promising contributions that LLMs can offer to the ITA field. As some researchers observe, “ … it seems inevitable that ITA researchers will need to engage with LLMs to enhance their work” [ 11 ] . Some earlier concerns regarding the use of LLMs to support ITA have already been partially or completely addressed in the recent, more advanced LLMs. These include concerns related to the limited length of textual data that can be inputted to the model, the memory-less operation of the model, the need to upload qualitative data to cloud-computing resources that threatens data privacy and security, and the costly operations that accompany the use of these models [ 11 , 12 , 13 ] . But the more recent LLMs can take 1 million tokens 1 1 1 Tokens are the fundamental units of language that represent words, phrases, or parts of words and therefore remember longer conversations [ 14 ] . These models can be used on local devices and are becoming less costly to operate [ 2 ] . Yet other concerns such as those related to the training data limitations, the ethics of using LLMs, and the model hallucinations as previously mentioned require careful consideration, and deliberations elsewhere should be fully supported. In addition, another issue that has been repeatedly discussed by multiple ITA researchers is the crucial importance of prompts that are used to generate LLM responses. Since LLMs function by probabilistically predicting the next word in a sequence based on the input they receive, a slight variation in the language used in prompts can significantly influence the quality of the model’s responses. A recent study reported a significant shift in the attitudes of seventeen junior qualitative researchers from skepticism to confidence in using LLMs as they became more knowledgeable about effective mechanisms to interact with these models [ 7 ] . Thus, a carefully designed and systematically refined prompt can lead to more accurate, relevant,