Is Artificial Intelligence (AI) currently able to provide evidence-based scientific responses on methods that can improve the outcomes of embryo transfers? No Authors: Argyrios Kolokythas, Michael H. Dahan Published: 2024 ABSTRACT ObjectiveThe rapid development of Artificial Intelligence (AI) has raised questions about its potential uses in different sectors of everyday life. Specifically in medicine, the question arose whether chatbots could be used as tools for clinical decision-making or patients’ and physicians’ education. To answer this question in the context of fertility, we conducted a test to determine whether current AI platforms can provide evidence-based responses regarding methods that can improve the outcomes of embryo transfers.MethodsWe asked nine popular chatbots to write a 300-word scientific essay, outlining scientific methods that improve embryo transfer outcomes. We then gathered the responses and extracted the methods suggested by each chatbot.ResultsOut of a total of 43 recommendations, which could be grouped into 19 similar categories, only 3/19 (15.8%) were evidence-based practices, those being “ultrasound-guided embryo transfer” in 7/9 (77.8%) chatbots, “single embryo transfer” in 4/9 (44.4%) and “use of a soft catheter” in 2/9 (22.2%), whereas some controversial responses like “preimplantation genetic testing” appeared frequently (6/9 chatbots; 66.7%), along with other debatable recommendations like “endometrial receptivity assay”, “assisted hatching” and “time-lapse incubator”.ConclusionsOur results suggest that AI is not yet in a position to give evidence-based recommendations in the field of fertility, particularly concerning embryo transfer, since the vast majority of responses consisted of scientifically unsupported recommendations. As such, both patients and physicians should be wary of guiding care based on chatbot recommendations in infertility. Chatbot results might improve with time especially if trained from validated medical databases; however, this will have to be scientifically checked. INTRODUCTION Since its introduction to the public in November 2022, ChatGPT has experienced a notable and steep increase in usage, reaching one million users within only five days, and 100 million users and one billion visits within three months of its public launch (Ting et al., 2024). Individuals from diverse fields have since tried to incorporate its usage not only in their personal but also their professional lives, with varying levels of success. It is clear that ChatGPT and other chatbots have the potential to revolutionize the search for information on the internet. Up until this point search engines provided links to other websites which could provide information with varying levels of validity to answer questions posed by the user. While the chatbots themselves collect the information on the internet and answer the questions posed by the user bypassing the need to link to other websites. In the medical field, ChatGPT has also been studied intensely with more than 400 related articles, indexed in PubMed until May 2023 (Ting et al., 2024). ChatGPT has reportedly also succeeded in several standardized medical tests including the United States Medical Licensing Examination® (USMLE®) (Gilson et al., 2023; Kung et al., 2023) and passed or nearly passed several Medical Board examinations (Ali et al., 2023; Bhayana et al., 2023). Other evidence suggests that ChatGPT has also failed medical standardized tests (Weng et al., 2023), while other studies showed inconsistency in performance with ChatGPT approximately 50% of the times succeeding medical exams and 50% failing the same exam type (Strong et al., 2023). Interestingly, it was also shown that patients sometimes preferred the ChatGPT interaction and answers rather than the physicians’ consultation on the same subject (Ayers et al., 2023). Surprisingly, these patients found ChatGPT to be more empathetic, a finding that should raise a different question and study. Concurrently, there has been increasing use of ChatGPT by students in both lower levels of schooling and university to write essays and complete different tasks (Sullivan et al., 2023). The results are sometimes so well-written and factual that it is very difficult to impossible to differentiate them from human-written essays (Waltzer et al., 2023). In the medical field, there are several studies warning about the potential implications related to the quality of evidence of scientific texts created by Artificial Intelligence (AI) software, as these are also difficult to distinguish from human-written texts and can provide questionable data (Gao et al., 2023; Homolak, 2023; Macdonald et al., 2023). Although several techniques have been suggested as a screening for AI-produced texts, none so far seems to be able to identify them with certainty (Levin et al., 2023). The purpose of this study is to try to investigate to what extent AI can produce evidence-based scientific medical recommendations in infertility. This has implications for both patient understanding, since they are likely to use these chatbots for guidance with care, and for scientists who attempt to use these AI platforms to generate scientific articles. To answer these questions we selected the most used AI chatbots and asked them to produce scientific texts about recommendations for improvement of embryo transfer outcomes. MATERIALS AND METHODS We used nine of the most popular free AI chatbots available (ChatGPT, Bard, Writesonic, You, Perplexity, Learnt, Bing, Magickpen, and Rytr) and entered the following command: “Write me a 300-word scientific essay about evidence-based methods that can improve the outcomes of embryo transfer” in May 2023. We then collected the responses and extracted the methods each chatbot suggested. When sufficient similarity among answers was present, we categorized the answers under one category to facilitate the study. We then calculated descriptive statistics and the prevalence of each response. We used as a comparator for widely acceptable practices that are proven to improve embryo transfer outcomes, the 2017 ASRM guideline on performing the embryo transfer (Practice Committee of the American Society for Reproductive Medicine and Practice Committee of the American Society for Reproductive Medicine, 2017) but we also took more current literature into consideration. Data was compared using chi-squared tests and a level of p<0.05 was used as the threshold of statistical significance. RESULTS There were 43 recommendations generated by the nine aforementioned AI chatbots. When categorized, they ended up forming 19 different recommendations, some of which were common among the chatbots and some unique to one or few of them (Table 1). The full-text responses of each AI chatbot can be found in the Supplement. Table 1Summary of recommendations per Chatbot.Chat GTPBardWrite-sonicYouPerplexityLearntBingMagickpenRytrNumber of chatbots yielding each recommendationUltrasound-guided embryo transfer (UGET)* X X (-)(-) X X X X X 7Preimplantation genetic testing (PGT) X (-)(-) X X X (-) X X 6Optimal endometrium preparation(-) X (-) X X X (-)(-) X 5Single embryo transfer*(-) X X X X (-)(-)(-)(-)4Endometrial receptivity analysis (ERA)(-)(-)(-)(-) X X (-) X (-)3Use of a soft catheter*(-) X (-)(-)(-)(-) X (-)(-)2Expertise of the medical team(-)(-)(-) X (-) X (-)(-)(-)2Assisted hatching(-)(-)(-)(-)(-) X (-) X (-)2Time-lapse incubator(-)(-)(-)(-)(-)(-) X (-) X 2Endometrial scratching (ES) X (-)(-)(-)(-)(-)(-)(-)(-)1Natural cycle embryo transfer (NCET) X (-)(-)(-)(-)(-)(-)(-)(-)1Use of GnRH antagonist protocol(-) X (-)(-)(-)(-)(-)(-)(-)1Blastocyst transfer(-)(-)(-)(-)(-) X (-)(-)(-)1Use of specialized catheter(-)(-)(-)(-) X (-)(-)(-)(-)1Preimplantation genetic screening (PGS)(-)(-)(-)(-)(-) X (-)(-)(-)1Mock embryo transfer(-)(-)(-)(-)(-) X (-)(-)(-)1Uninterrupted embryo culture(-)(-)(-)(-)(-)(-) X (-)(-)1Minimizing transfer time(-)(-)(-)(-)(-)(-) X (-)(-)1Maintaining temperature and pH of the culture media(-)(-)(-)(-)(-)(-) X (-)(-)1Number of recommendations for each Chatbot451469644* evidence-based recommendationX: indicates the presence of the recommendation in the chatbot-generated text(-): indicates the absence of the recommendation from the chatbot-generated textThe data has been represented as N. The numbers in the last row indicate the number of recommendations for each Chatbot and the numbers in the last column the number of Chatbots yielding each recommendation. Summary of recommendations per Chatbot. * evidence-based recommendation X: indicates the presence of the recommendation in the chatbot-generated text (-): indicates the absence of the recommendation from the chatbot-generated text The data has been represented as N. The numbers in the last row indicate the number of recommendations for each Chatbot and the numbers in the last column the number of Chatbots yielding each recommendation. The range of the recommendations was from one to nine per chatbot (median = 4, IQR=2) and an average of 4.78 suggestions per chatbot. Writesonic was the only chatbot to only suggest one recommendation, while ChatGPT, You, Magickpen, and Rytr suggested four each. Bard gave five recommendations and Perplexity and Bing six each. Learnt with nine, was the chatbot to give the most recommendations, however here it should be mentioned that Learnt’s preset is to generate three different texts for any given command so that could explain the discrepancy. The number of total recommendations, the number of evidence-based recommendations, and the percentage of evidence-based recommendations per Chatbot are illustrated in Figure 1. Figure 1Graphical representation of the total number of recommendations, evidence-based recommendations, and percentage of evidence-based recommendations per Chatbot. Graphical representation of the total number of recommendations, evidence-based recommendations, and percentage of evidence-based recommendations per Chatbot. From the generated answers the most common one was “ultrasound-guided embryo transfer (UGET)”, recommended by 7/9 chatbots (77.8%), an indeed evidence-based recommendation. In second place with 6/9 chatbots suggesting it (66.7%) was “preimplantation genetic testing (PGT)” while in third place came the vague “optimal en-dometrium preparation” with 5/9 chatbots (55.6%), both non-evidence-based practices to improve embryo transfer. Following with 4/9 chatbots (44.4%) was the second evidence-based recommendation “single embryo transfer” and “endometrial receptivity analysis (ERA)” followed, with 3/9 chatbots (33.3%) recommending it. The following four recommendations appeared in two AI responses (2/9; 22.2%) with only the first one being evidence-based: “use of a soft catheter”, “expertise of the medical team”, “assisted hatching” and use of the “time-lapse incubator”. Finally, the majority of the recommendations were unique, with 10 answers appearing only once (1/9; 11.1%), those being “endometrial scratching (ES)”, “natural cycle embryo transfer (NCET)”, “use of GnRH antagonist protocol”, “blastocyst transfer”, “use of specialized catheter”, “preimplantation genetic screening (PGS)”, “mock embryo transfer”, “uninterrupted embryo culture”, “minimizing transfer time”, and “maintaining temperature and pH of the culture media”. None of the chatbots were more likely to provide false recommendations (p=0.54), or correct ones. The total number of recommendations that were evidence-based was 13 (13/43; 30.2%); and not evidence-based: 30 (30/43; 69.8%). DISCUSSION Our results demonstrate that of the total of 19 different recommendations, only three were evidence-based, widely accepted practices, proven to improve the outcomes of embryo transfer, the “ultrasound-guided embryo transfer”, the “use of a soft catheter”, and the “single embryo transfer”. Although “single embryo transfer”, recommended by 4/9 chatbots (44.4%), does not necessarily improve the likelihood of pregnancy, it does decrease the risks of multiple pregnancy and resultant pregnancy complications for the mother and offspring. As such, “single embryo transfer” could be argued to improve the outcomes of embryo transfer. A “mock embryo transfer” may also be argued to improve outcomes. From the two first evidence-based recommendations “ultrasound-guided embryo transfer” was mentioned by the majority of the AI chatbots (7/9; 77.8%), yet the “use of a soft catheter” was presented as a suggestion by only two out of the nine chatbots (2/9; 22.2%), with an additional response of “use of specialized catheter” having been coded separately since it was not offering a precise answer, and while other more vague or questionable recommendations were appearing more often. As an example, the imprecise recommendation of “optimal endometrium preparation” was ranked third, mentioned by five chatbots, a recommendation that cannot add much to a scientific discussion and cannot be accepted as an evidence-based recommendation due to lack of a precise solution. Additionally, some controversial responses were frequently listed with “preimplantation genetic testing (PGT)” and “endometrial receptivity assay (ERA)” ranking second and fifth respectively, and “assisted hatching” and “time-lapse incubator” being suggested by two chatbots. These recommendations are still controversial and current evidence does not support a clear improvement in live birth rates (Insua et al., 2017; Armstrong et al., 2018; Lacey et al., 2021; Yan et al., 2021; Zhang et al., 2022; Arian et al., 2023; Zolfaroli et al., 2023). The recommendation of “maintaining temperature and pH of the culture media” could be argued to improve outcomes if done properly; however, this answer was also lacking hard facts, not specifying what these conditions should be, and as such adds little to the knowledge of a practitioner; and therefore was felt to be nonspecific. Some further questionable methods offered included “endometrial scratching”, which has fallen out of favor, and “natural cycle embryo transfer”, which remains controversial based on the most recent meta-analysis (Busnelli et al., 2022), each appearing once. It should also be mentioned that some of the given answers were either completely irrelevant to the question (e.g. the “use of GnRH antagonist protocol”) or they did not address the real purpose (e.g. “blastocyst transfer”, which would be determined by laboratory protocols and the number and quality of embryos available). In brief, of 43 recommendations; 30 (69.8%) were wrong or provided inadequate information for decision-making which is certainly a discouraging outcome. However, it’s important to note that this does not necessarily condemn the use of AI in reproductive medicine, nor is that the purpose of this paper. In fact, recent evidence within the fertility field suggests that AI could potentially revolutionize image analysis of surgical sperm samples from azoospermic patients. By significantly reducing the time required for sperm identification, AI has the potential to greatly enhance efficiency and improve success rates in these procedures (Goss et al., 2024). Our study is not without its limitations. First, we did not use all available AI chatbots but the free versions of some of the most popular ones. Although we do not believe that the paid versions would have yielded different outcomes, as they usually come with other features to charge for, we cannot rule out the possibility that the paid versions might have been more accurate. Additionally, we know that chatbots tend to give slightly different answers when repeatedly asked the same questions. That means that some of the used chatbots might have given different responses, which might have been more evidence-based upon subsequent attempts. However, we believe that our study effectively captures the prevailing practices of AI chatbot users, as users typically pose a question only once and not repetitively. CONCLUSION The chatbots’ answers on methods to improve embryo transfer were rarely evidence-based; and those that were, were buried within a plethora of questionable, evidence-lacking responses. At present, none of these nine AI platforms can be used either for medical decision-making or for patients’ or physicians’ education, as it is challenging to distinguish the correct scientific evidence within the over-abundance of non-evidence-based given responses. Although some chatbots seem to perform well on standardized medical tests, the same is not true when it comes to clinical decision-making. With time we expect AI medical information to improve, especially if validated medical databases are used to train them. Yet at present, chatbots’ medical background is insufficient and AI does not provide quality evidence on techniques to improve embryo transfer.