Assessing ChatGPT’s Reliability as an Educational Resource for Information on Breast Reduction Surgery Authors: Emanuella M Brito, Isabel C Bernal, Andres A Mascaro-Pankova, James W Fletcher, Eric S Weiss, Christopher J Low, Michael C Cheung, Martin I Newman Published: 2024 ABSTRACT Introduction: With advances in AI and machine learning, platforms like OpenAI’s ChatGPT are emerging as educational resources. While these platforms offer easy access and user-friendliness due to their personalized conversational responses, concerns about the accuracy and reliability of their information persist. As one of the most common surgical procedures performed by plastic surgeons worldwide, breast reduction surgery (BRS) offers relief for the physical and emotional burdens of large breasts. However, like any surgical procedure, it can raise a multitude of questions and anxiety.Methods: To evaluate the quality of medical information provided by ChatGPT in response to common patient inquiries about breast reduction surgery, we developed a 15-question questionnaire with typical patient questions about BRS. These questions were presented to ChatGPT, and the answers were compiled and presented to five board-certified plastic surgeons. Each specialist categorized the response as (1) Appropriate, the response accurately reflects current medical knowledge and best practices for BRS; (2) No, not thorough, the response lacks sufficient detail to be a helpful educational resource; (3) No, inaccurate, the response contains misleading or incorrect information.Results: A total of 75 survey responses were obtained, with five experts each analyzing 15 answers from ChatGPT. Of these, 69 (92%) responses were determined to be accurate. However, six (8%) responses were concerning to our experts: four (5.3%) lacked detail, and two (2.7%) were found to be inaccurate. Chi-square analysis revealed no statistical significance in the distribution of responses categorized as “accurate” versus “not thorough/inaccurate,” and “not thorough” versus “inaccurate” (p=0.778 and p=0.306, respectively).Conclusion: While ChatGPT can provide patients with basic background knowledge on BRS and empower patients to ask more informed questions during consultations, it should not replace the consultation and expert guidance of a board-certified plastic surgeon. Introduction Breast reduction surgery (BRS) offers significant relief for patients experiencing physical and emotional burdens due to large breasts and ranks as the 10th most common procedure performed by plastic surgeons globally [1]. However, the decision to undergo any surgical procedure often comes with a multitude of questions and anxieties. Potential complications, recovery timelines, and overall expectations are just a few of the patient’s more common concerns. To empower patients to make informed choices and optimize surgical outcomes, thorough patient education is paramount [2]. The internet has become a dominant source of health information, with patients increasingly turning to online resources to answer their medical queries before consulting a healthcare professional [3]. With the expansion of artificial intelligence (AI) and machine learning (ML), platforms like Open AI’s ChatGPT are threatening to replace “Dr Google'' as an educational resource. ChatGPT's ability to provide personalized answers in a conversational format is a significant draw for patients seeking information [4]. While these platforms offer undeniable advantages in terms of accessibility and user-friendliness, concerns linger regarding the accuracy and reliability of the information they provide [5]. This paper aims to evaluate the quality of medical information provided by ChatGPT in response to common patient inquiries about breast reduction surgery and further explore the potential of such platforms as a patient education tool for those considering this procedure. Materials and methods Study design We developed a 15-item questionnaire to encompass a wide range of topics commonly inquired about by patients considering breast reduction surgery (BRS). Questions are detailed in Table 1 and ChatGPT's responses are given in the Appendix. The questions were formulated based on their ability to address areas such as candidacy, surgical procedures, recovery, risks, and expected outcomes. The list of questions was also developed using resources from the American Society of Plastic Surgeons (ASPS) website, drawing from their compilation of frequently asked questions about BRS [6]. These questions were then used as prompts for OpenAI's ChatGPT version 3.5.0 (OpenAI, San Francisco, CA). Each question was presented to the language model independently, and its corresponding response was accessed on May 9, 2024. The full list of questions and their corresponding answers from ChatGPT is provided in the supplementary table. To assess the accuracy and suitability of the ChatGPT-generated responses for patient education, five board-certified plastic surgeons who specialize in BRS were recruited. A web-based survey platform, SurveyMonkey (SurveyMonkey Inc., San Mateo, CA), was utilized to present each ChatGPT response. The plastic surgeons were aware that the information they were evaluating AI-generated responses. Each evaluator anonymously and independently reviewed all responses and categorized them according to the following criteria: (1) Appropriate; the response accurately reflects current medical knowledge and best practices for BRS; (2) No, not thorough; the response lacks sufficient detail to be a helpful educational resource; (3) No, inaccurate; the response contains misleading or incorrect information. This categorization process aimed to systematically evaluate the quality of information provided by ChatGPT as a patient education tool for breast reduction surgery. To analyze the distribution of these expert ratings, we performed two independent chi-square tests. Table 1Common questions on BRS answered by ChatGPTCommon breast reduction questions that were answered by ChatGPT. Discussing treatment, lifestyle, and recovery.Question NumberCommon Patient Questions About BRSQ1Who is a good candidate for breast reduction surgery (BRS)?Q2Who is not a good candidate for BRS?Q3What kind of surgeon performs BRS?Q4Does my surgeon need to be Board Certified in Plastic Surgery to perform a breast reduction surgery?Q5Does my surgeon need to have hospital privileges to perform a BRS?Q6Can I pick my “cup size” for a BRS?Q7Does breast reduction surgery leave scars?Q8Can I breastfeed following BRS?Q9Is breast reduction surgery painful?Q10Does breast reduction surgery include a breast lift?Q11How long is the recovery following breast reduction surgery?Q12Will breast reduction surgery alter my breast sensation?Q13Should I lose weight before getting breast reduction surgery?Q14Should I get breast reduction surgery before or after having kids?Q15What are the risks of breast reduction surgery? Common questions on BRS answered by ChatGPT Common breast reduction questions that were answered by ChatGPT. Discussing treatment, lifestyle, and recovery. Statistical analysis To assess the adequacy of ChatGPT’s responses, we employed chi-square tests using IBM SPSS Statistics 28.0 (IBM Corp., Armonk, NY). We conducted two independent chi-square analyses. The first analysis examined the relationship between individual survey responses and a binary categorization of answer appropriateness to each question as “sufficient” (yes) or “not thorough or inaccurate” (no). The second analysis investigated the statistical significance between the responses categorized as “no, not thorough” and those categorized as “no, inaccurate.” Statistical significance was assessed using a p-value threshold of 0.05. Results Our survey of five board-certified plastic surgeons who routinely perform BRS noted some issues with the ChatGPT responses. Of the five plastic surgeons, three had > 10 years of experience and all five noted to be board-certified in plastic surgery. Our response rate was 100%. Our five experts each analyzed 15 AI-generated responses; a total of 75 responses were then statistically analyzed by independent chi-square tests. Of these 75 ratings by our experts, 69 (92.0%) were in agreement as appropriate responses. However, six (8.0%) separate and distinct AI-generated responses gave concern to our experts as follows. Four (5.3%) ChatGPT responses were noted to lack detail, and two (2.7%) were found to be inaccurate (Figure 1). There was no overlap in this respect. In no case did more than one expert find an issue with any particular response. While these findings suggest ChatGPT's potential as a tool for general information on breast reduction surgery, it may not be sufficient for addressing in-depth or nuanced questions. Figure 1 explains the response breakdown. Survey Results: ChatGPT Answer Analysis Pie chart of survey responses for ChatGPT answer analysis by five board-certified plastic surgeons. Total of 75 survey responses in which 92% (blue) were classified as accurate; 5% (green) were classified as not thorough; and 3% (purple) as inaccurate. The first chi-square test results are presented in Table 2. A chi-square test of independence was performed to evaluate the relationship between the responses categorized as “sufficient” (yes) versus “not thorough/inaccurate” (no). The relationship between these variables was not statistically significant (p<0.05), X2, (14, N=75) = 9.783, p = .778. However, this does not necessarily mean all ChatGPT responses were acceptable. Table 2Chi-Square Test of ChatGPT for Answer Appropriateness Chi-square analysis evaluating the relationship between individual survey responses “sufficient” vs “Not thorough/Inaccurate." a30 cells (100.0%) have expected count less than 5. The minimum expected count is .40. Value of X2 df p-value (two-sided) Pearson chi-square 9.783a 14 0.778 Likelihood ratio 11.791 14 0.623 No. of survey responses 75 Chi-Square Test of ChatGPT for Answer Appropriateness Chi-square analysis evaluating the relationship between individual survey responses “sufficient” vs “Not thorough/Inaccurate." a30 cells (100.0%) have expected count less than 5. The minimum expected count is .40. A chi-square test was performed to evaluate the relationship between the responses categorized as “not thorough” versus “inaccurate”, shown in Table 3. The relationship between these variables was not statistically significant (p<0.05), X2, (5, N=6) = 6, p = .306. Table 3Chi-Square Test of ChatGPT for Not Appropriate AnswersChi-square analysis evaluating the relationship between individual survey responses “Not thorough" vs "Inaccurate."a12 cells (100.0%) have expected count less than 5. The minimum expected count is .33. Value of X2 dfp-value (two-sided)Pearson chi-square6.000a 50.306Likelihood ratio7.63850.177No. of survey responses6 Chi-Square Test of ChatGPT for Not Appropriate Answers Chi-square analysis evaluating the relationship between individual survey responses “Not thorough" vs "Inaccurate." a12 cells (100.0%) have expected count less than 5. The minimum expected count is .33. Although these discrepancies did not reach statistical significance, it is still notable that six out of 75 ratings (8%) were classified as inaccurate, as any form of misinformation can have clinical implications. Therefore, we will examine and discuss the answers to the questions where at least one respondent categorized a ChatGPT answer as insufficient or inaccurate. This may provide insight into the type of information ChatGPT’s responses might lack. The four questions that at least one expert deemed lacking sufficient detail were questions 1, 2, 6, and 10, while questions 7 and 8 were deemed inaccurate by at least one expert (Table 1). Discussion The increasing tendency of patients to turn to large language models like ChatGPT along with the increased availability of this common internet tool raises concerns about the accuracy of medical information patients receive. This study aimed to evaluate the accuracy of information ChatGPT provides to patients considering a commonly performed plastic surgical procedure, in this case, breast reduction surgery. Our analysis as outlined above revealed that six (40%) of the 15 AI-generated responses were in some manner in conflict with the opinions of professional board-certified plastic surgeons. While these discrepancies did not achieve statistical significance, six (8%) out of 75 ratings classified as inaccurate information is still noteworthy, as any kind of misinformation can be clinically significant. Sensitive areas such as surgery must heavily rely on evidence-based medicine, as misleading data in such fields can have real-world implications capable of affecting patients' lives. Especially in plastic surgery, where most cases of breast reduction procedures are elective, risks should be minimized in all aspects, starting with a patient’s understanding of the procedure. A misinformed patient can lead to unrealistic or unreasonable expectations and, ultimately, a decrease in patient satisfaction and patient-reported outcomes. As a growing field, discussing the accuracy of information provided by artificial intelligence is crucial for ensuring clarity for individuals using these platforms. AI developers must be aware of the implications that misinformation can have on patients and use this awareness as a motive to ensure the continuous improvement of information provided by AI. Patients may not always be able to distinguish between accurate and misleading information. Therefore, understanding that machine learning programs are subject to errors, any medical advice provided by such platforms should be accompanied by a disclaimer that it does not substitute for professional medical advice. Transparency is key to implementing ethical parameters and guidelines in this ever-changing field of technology and artificial intelligence. It is essential to make it clear to users that any information provided should be confirmed by an accredited professional. In this paper, ChatGPT largely emphasized the importance of discussing questions with a board-certified plastic surgeon. However, it did not include statements about the possibility of inaccurate or incomplete information. Four of the six questions that conflicted with the experts’ opinions were classified as lacking sufficient detail. These were questions 1, 2, 6, and 10 (Table 1). In response to question 1, which inquired, 'Who is a good candidate for breast reduction surgery?' ChatGPT elaborates on the characteristics that patients must have to be considered good candidates. These include physical discomfort, skin issues, postural problems, emotional distress, a clear understanding of the procedure’s risks and outcomes, overall health, non-smoking status, and psychological readiness. All of these factors are also classified by the ASPS as important criteria to determine a patient’s eligibility for BRS. However, ChatGPT fails to include two factors highlighted by ASPS as good indicators for undergoing a BRS: shoulder indentations caused by bra straps and non-vaping status within the broader category of non-smokers [7]. ChatGPT answers question 2, 'Who is not a good candidate for BRS?' by listing the characteristics of a not-ideal candidate for breast reduction surgery. These include patients with serious comorbidities, smokers, patients with unrealistic expectations, or those planning to undergo significant weight loss. It also addresses those who are currently pregnant or breastfeeding by stating: “Women who are currently pregnant or breastfeeding are advised to wait until after they have finished nursing. Pregnancy and breastfeeding can alter breast size and shape, which would affect the surgical outcome.” However, it makes no mention of someone who is “actively trying” to become pregnant or someone who may be interested in getting pregnant within the foreseeable future. Because pregnancy increases breast hypertrophy, and because breast surgery has unpredictable effects on the ability to breastfeed, a more inclusive response from the AI might also include those with plans to become pregnant in the immediate future [8,9]. A current study has shown a reduction in the breastfeeding success rate by 41% following breast reduction surgery [10]. Such a high decline in breastfeeding success following breast surgery reinforces the need for patients who seek to become pregnant after the surgery to be aware of this possible complication. Question 6 addressed whether patients can choose their own 'cup size' for breast reduction surgery (BRS). The AI platform explains that this decision depends on a discussion between the patient and the surgeon. An underlying theme of this response implies a significant latitude on the part of the operating surgeon to tailor the results of this procedure to a patient’s expectations and desires. Although some latitude is possible when performing breast reduction surgery, the outcomes are more closely tied to the presenting size and shape of the preoperative breast and body habitus. Discussing “the look and feel you are aiming for” is important, but the response could be more complete if it included a further discussion of what “is” and what “is not” possible to achieve safely. In addition, the use of “sizers during the consultation to help you visualize potential outcomes” has more of a role when discussing breast augmentation surgery and not breast reduction surgery [11]. While ChatGPT effectively explains the mechanics of breast lift procedures and their relation to breast reduction surgery, it fails to address the crucial factor of individual patient assessment in its response to question 10, which asked if breast reduction surgery includes a breast lift. Unlike a one-size-fits-all approach, each patient is carefully evaluated for their suitability for either a breast lift or reduction. ChatGPT's description of the breast lift's purpose aligns with the ASPS website, stating the primary goal as lifting and reshaping sagging breasts. This is achieved by removing excess skin and tightening surrounding tissues to create a new, supported breast contour [6]. However, the AI platform does not explain the technical differences between both procedures. Although it is possible and common for them to be done together, not all good candidates for breast reduction are also good candidates for mastopexy or breast lift. Mastopexy will have a greater impact on the nipple and areola positioning, and patients usually seeking this procedure will have greater aesthetic concerns [12]. Therefore, a personalized and tailored approach should be the aim of surgeons assessing patients seeking BRS to determine the indication of a combined procedure with a breast lift. Of the six questions classified as inaccurate, questions 7 and 8 were deemed incorrect by at least one expert (Table 1). In response to question 7, which asked whether BRS leaves scars, ChatGPT confirms that BRS does indeed leave scars. It explains that the pattern of scars depends on the type of incision made by the surgeon, which can be one of three types: anchor or inverted-T incision, vertical or “lollipop” incision, and periareolar or “donut” incision. The OpenAI platform emphasizes the need for patients to discuss with their plastic surgeons the location of their scars and what to expect during the healing process. This allows patients to have realistic expectations about their surgical outcomes. However, the platform attributes the main cause of scar formation to the surgical technique and to the degree of breast reduction. Although different techniques and the extent of reduction do affect scar formation, they are not the only factors. Skin loss and delayed wound healing have been reported as the number one complication encountered in reduction mammoplasty [13]. ChatGPT fails to mention the probability of such complications occurring. Scarring is a complex process, not solely dependent on a surgeon’s technique. Wound healing is affected by individual characteristics, including genetics, ethnicity, age, and history of hypertrophic scars. These are also known as non-modifiable circumstances that must be explained to the patient by their surgeon so both can work together to optimize the healing process [14]. Lastly, regarding question 8 about breastfeeding after breast reduction surgery, ChatGPT notes that while breastfeeding is possible, success is not guaranteed and depends on factors such as the surgical technique and the extent of tissue removed. The platform briefly explains different factors affecting breastfeeding after BRS, such as surgical technique, amount of tissue removed, preservation of nipple sensation, and individual variability. When discussing surgical techniques that allow for a greater chance of breastfeeding, ChatGPT references the “pedicle method,” stating that it has the best outcome for preserving breastfeeding capability. However, this is an umbrella term for different techniques involving multiple pedicles in the breast, all aiming to maintain nerve and blood supply to the nipple [15]. While the AI response offers broad information relevant to a plastic surgeon consultation, it overlooks a crucial point. It's important to understand that difficulty breastfeeding is common, affecting roughly 70% of women regardless of breast reduction surgery [16]. Studies show a 62% success rate for breastfeeding after reduction mammaplasty [17]. This can be compared to the national average of approximately 41.8% of women who ultimately discontinue breastfeeding altogether after six months post-partum [18]. This broader perspective, which ChatGPT missed, is vital for patients considering reduction surgery. While the AI acknowledges potential breastfeeding challenges, emphasizing the overall picture is essential for informed decision-making. It is important to acknowledge the potential benefits of patients using AI for introductory surgical queries. Information provided by AI sources like ChatGPT can empower patients to ask more informed questions during consultations with their board-certified plastic surgeons. This can facilitate a deeper understanding of potential risks, benefits, and alternatives associated with BRS, ultimately leading to a more informed consent process. Limitations Although this study has successfully demonstrated the potential use of ChatGPT in providing useful preliminary information for patients seeking breast reduction surgery, it had some limitations. A larger sample size, both in the number of questions and specialists, could change the chi-square results and consequently the significance of the answers provided by ChatGPT. Additionally, AI is a trainable system that is capable of learning and drawing new conclusions as it comes across more data [19]. Therefore, the quality of information provided by ChatGPT for this study could improve in a very short period of time. Consequently, the data obtained by this study should be viewed as a current snapshot of ChatGPT’s information on BRS, which may change in the future. Another important limitation is ChatGPT's limited ability to address individual cases. Due to the absence of knowing patient-specific medical history and surgeon expertise, ChatGPT is limited in its ability to provide personalized answers to medical questions. Conclusions AI platforms such as ChatGPT are emerging as potential educational tools in healthcare offering patients easy access to information in a conversational format. This study investigated the quality of information ChatGPT provides for BRS patients, evaluating its suitability as a reliable information source, and found a notable rate of inaccuracies and inconsistencies generated by this AI tool. Surgeons and patients should be aware of these inaccuracies and inconsistencies to avoid any potential complications caused by a misinformed patient. AI generators should use the real-world impact of healthcare misinformation on patients’ lives as a motive to seek continuous improvement based on current evidence-based medicine. Additionally, transparency should be a priority, with disclaimers added to every machine learning platform when providing healthcare information. Any information from a machine learning platform, while often instructive, should always be confirmed by an accredited physician. Therefore, while ChatGPT can provide patients with basic background knowledge on BRS and empower patients to ask more informed questions during consultations, it should not replace the consultation and expert guidance of a board-certified plastic surgeons.