Investigating Older Adults’ Perceptions of AI Tools for Medication Decisions: Vignette-Based Experimental Survey Authors: Sarah E Vordenberg, Julianna Nichols, Vincent D Marshall, Kristie Rebecca Weir, Michael P Dorsch, Naomi Cahill, Urjoshi Sinha, Hawa O Abu, Emeka Stanley Obi, Emmanuel Oluokun Published: 2024 ABSTRACT AbstractBackgroundGiven the public release of large language models, research is needed to explore whether older adults would be receptive to personalized medication advice given by artificial intelligence (AI) tools.ObjectiveThis study aims to identify predictors of the likelihood of older adults stopping a medication and the influence of the source of the information.MethodsWe conducted a web-based experimental survey in which US participants aged ≥65 years were asked to report their likelihood of stopping a medication based on the source of information using a 6-point Likert scale (scale anchors: 1=not at all likely; 6=extremely likely). In total, 3 medications were presented in a randomized order: aspirin (risk of bleeding), ranitidine (cancer-causing chemical), or simvastatin (lack of benefit with age). In total, 5 sources of information were presented: primary care provider (PCP), pharmacist, AI that connects with the electronic health record (EHR) and provides advice to the PCP (“EHR-PCP”), AI with EHR access that directly provides advice (“EHR-Direct”), and AI that asks questions to provide advice (“Questions-Direct”) directly. We calculated descriptive statistics to identify participants who were extremely likely (score 6) to stop the medication and used logistic regression to identify demographic predictors of being likely (scores 4-6) as opposed to unlikely (scores 1-3) to stop a medication.ResultsOlder adults (n=1245) reported being extremely likely to stop a medication based on a PCP’s recommendation (n=748, 60.1% [aspirin] to n=858, 68.9% [ranitidine]) compared to a pharmacist (n=227, 18.2% [simvastatin] to n=361, 29% [ranitidine]). They were infrequently extremely likely to stop a medication when recommended by AI (EHR-PCP: n=182, 14.6% [aspirin] to n=289, 23.2% [ranitidine]; EHR-Direct: n=118, 9.5% [simvastatin] to n=212, 17% [ranitidine]; Questions-Direct: n=121, 9.7% [aspirin] to n=204, 16.4% [ranitidine]). In adjusted analyses, characteristics that increased the likelihood of following an AI recommendation included being Black or African American as compared to White (Questions-Direct: odds ratio [OR] 1.28, 95% CI 1.06-1.54 to EHR-PCP: OR 1.42, 95% CI 1.17-1.73), having higher self-reported health (EHR-PCP: OR 1.09, 95% CI 1.01-1.18 to EHR-Direct: OR 1.13 95%, CI 1.05-1.23), having higher confidence in using an EHR (Questions-Direct: OR 1.36, 95% CI 1.16-1.58 to EHR-PCP: OR 1.55, 95% CI 1.33-1.80), and having higher confidence using apps (EHR-Direct: OR 1.38, 95% CI 1.18-1.62 to EHR-PCP: OR 1.49, 95% CI 1.27-1.74). Older adults with higher health literacy were less likely to stop a medication when recommended by AI (EHR-PCP: OR 0.81, 95% CI 0.75-0.88 to EHR-Direct: OR 0.85, 95% CI 0.78-0.92).ConclusionsOlder adults have reservations about following an AI recommendation to stop a medication. However, individuals who are Black or African American, have higher self-reported health, or have higher confidence in using an EHR or apps may be receptive to AI-based medication recommendations.Keywords: older adults, survey, decisions, artificial intelligence, vignette, drug, pharmacology, pharmaceutic, medication, decision-making, geriatric, aging, surveys, attitude, perception, perspective, recommendation, electronic heath record Introduction Approximately 90% of adults 65 years and older (“older adults”) in the United States take at least 1 prescription medication, two-thirds take at least 1 dietary supplement, and one-third take at least 1 over-the-counter (OTC) medication regularly [1,2]. Medications are an important tool to prevent and treat diseases. However, up to one-half of older adults take at least 1 inappropriate medication, which can lead to drug-drug interactions, adverse events, additional medical appointments, emergency department visits, functional decline, reduced quality of life, and increased health care costs [3]. Older adults frequently report that they prefer to participate in shared decision-making [4]. Therefore, they may discuss prescription or OTC medications with physicians or pharmacists as they trust the information provided [5,6]. While these conversations allow patients to receive personalized recommendations, many health care professionals are experiencing high workload and burnout, which interferes with their ability to engage in patient-centered conversations about medications [7,8]. Older adults are increasingly using technology such as smartphone apps and telehealth [9]. Therefore, technology may be an alternative or complementary strategy for older adults to receive personalized medication advice, particularly given the limited time that health care professionals have available with each patient [10]. SEV and MPD sought to address the gap in technological tools for patients related to OTC medications by creating an electronic decision algorithm prototype to help adults identify safe and effective treatments for common ailments [11]. The tool required that participants answer questions about themselves and their symptoms and were given an OTC product recommendation or encouraged to seek advice from a health care professional. Given the complexity and time-intensiveness of manually building and maintaining a comprehensive database, it did not contain detailed information about individual medications (eg, dosage forms, directions for use, and side effects). When the prototype was shared with adults during semistructured interviews, participants reported that they would likely use the technology. However, they simultaneously raised concerns about the accuracy and comprehensiveness of the information, data storage, and accessibility while aligning with broader social and ethical concerns about using technology [12]. While adults reported being interested in using technology to support medication decisions, SEV and MPD quickly identified that creating and maintaining a knowledge base for more than 300,000 OTC medications in the United States would not be feasible [13]. However, the public release of a large language model (LLM) in 2022 revealed how artificial intelligence (AI) can revolutionize how diseases are diagnosed, develop patient-centered treatment plans, and increase patient engagement [14,15]. For example, it is able to provide high-quality, empathetic responses to patient questions [16]. It can also readily provide clarifying information and adjust the health literacy level of the information. Therefore, we shifted our work to identify predictors of the likelihood of older adults starting or stopping a medication and the influence of the source of the information. Methods Study Design and Sample We developed a 15-minute web-based experimental survey informed by our previous research, which focused on factors important to older adults when making medication decisions [17-19]. We used Qualtrics panels, which include more than 3 million people, to distribute the survey. We initially pilot-tested the survey by distributing it to 50 older adults. We did not make any changes to the survey. Overall, we aimed to recruit 1200 US adults aged 65 years and older to participate between December 2023 and January 2024. We used quotas for gender (50% female), age (50% aged 65-69 years and 50% aged 70 years and older), race (15% Black, 5% Asian, and 80% White or another race), ethnicity (18% Hispanic), and education (70% less than a bachelor’s degree). We aimed to align our quotas with the prevalence of these groups in the US population. The sampling algorithm continued to invite eligible participants to complete the survey until all quotas were achieved. Survey invitations did not include the study topic to decrease self-selection bias, and strategies such as checking IP addresses were used to prevent individuals from submitting multiple responses. Intervention First, we defined 5 potential sources of information for learning about OTC and prescription medications (Table 1). These included 2 health care professionals (ie, primary care provider [PCP] or pharmacist at a drugstore) and 3 versions of an AI app (ie, connect with electronic health record [EHR] and sends medication recommendations to PCP for approval before sending it to the patient [“EHR-PCP”], connects with EHR and sends a medication recommendation directly to the patient [“EHR-Direct”], or collects information by asking the patient a question and then provides a medication recommendation directly to the patient [“Questions-Direct”]). We developed the descriptions of the AI apps based on semistructured interviews that we had previously conducted with adults to explore their interest in using technology when making OTC medication decisions [11]. Table 1Study designa.VariableText descriptionSource informationThere are many ways that you can learn about over-the-counter and prescription medications. You may receive a recommendation from a health care professional:You might have an appointment with or message a primary care provider (PCP) who has access to your electronic health record.You might ask a pharmacist at a drugstore who does not have access to your electronic health record. [Page break] Alternatively, you may receive a recommendation from an application (‘app’) that uses artificial intelligence (AI) and all of the available research evidence to make personalized recommendations for you.One type of app may gather information from your electronic health record such as your health conditions and medications to create a personalized recommendation. The recommendation would be shared with your doctor for approval before your receive the information. [EHR-PCP]A second type of app may gather information from your electronic health record such as your health conditions and medications to create a personalized recommendation. You would receive the recommendation immediately and a copy of the recommendation would be stored in your electronic health record. [EHR-Direct]A third type of app may gather information from you as you answer a series of questions about your health conditions and medications to create a personalized recommendation. You would receive the recommendation immediately. [Questions-Direct]MedicationImagine that you are interested in simplifying your medication regimen to avoid unnecessary medications and decrease the risk of side effects. Consider the following three situations and identify how likely you would be to stop the medication if recommended by each of the following sources of information. [randomized order with page breaks] Imagine that you have taken low-dose aspirin once daily for several years to prevent you from having a heart attack or stroke. You have never had a heart attack or stroke. However, you recently learned that aspirin can increase your risk of bleeding.Imagine that you have taken ranitidine once daily for several years to treat your heartburn symptoms. Your symptoms are well controlled. However, you recently learned that ranitidine contains a cancer-causing chemical.Imagine that you have taken prescription simvastatin once daily for several years to decrease your risk of a heart attack or stroke. Your cholesterol lab values are within the normal range. However, you recently learned that it may not provide as much benefit as you age. You are looking for advice as to whether this information means you should stop taking [medication] How likely would you be to stop the [medication] if recommended by [source of information]? 1 = Not at all likely23456 = Extreme likelyaParticipants received 3 vignettes (ie, aspirin, ranitidine, and simvastatin) in a randomized order. For each vignette, participants indicated how likely they would be to stop the medication if recommended by 5 different sources of information (ie, primary care provider, a pharmacist at a drugstore, “EHR-PCP,” “EHR-Direct,” and “Questions-Direct”). Study designa. You might have an appointment with or message a primary care provider (PCP) who has access to your electronic health record. You might have an appointment with or message a primary care provider (PCP) who has access to your electronic health record. You might ask a pharmacist at a drugstore who does not have access to your electronic health record. You might ask a pharmacist at a drugstore who does not have access to your electronic health record. One type of app may gather information from your electronic health record such as your health conditions and medications to create a personalized recommendation. The recommendation would be shared with your doctor for approval before your receive the information. [EHR-PCP] One type of app may gather information from your electronic health record such as your health conditions and medications to create a personalized recommendation. The recommendation would be shared with your doctor for approval before your receive the information. [EHR-PCP] A second type of app may gather information from your electronic health record such as your health conditions and medications to create a personalized recommendation. You would receive the recommendation immediately and a copy of the recommendation would be stored in your electronic health record. [EHR-Direct] A second type of app may gather information from your electronic health record such as your health conditions and medications to create a personalized recommendation. You would receive the recommendation immediately and a copy of the recommendation would be stored in your electronic health record. [EHR-Direct] A third type of app may gather information from you as you answer a series of questions about your health conditions and medications to create a personalized recommendation. You would receive the recommendation immediately. [Questions-Direct] A third type of app may gather information from you as you answer a series of questions about your health conditions and medications to create a personalized recommendation. You would receive the recommendation immediately. [Questions-Direct] Imagine that you have taken low-dose aspirin once daily for several years to prevent you from having a heart attack or stroke. You have never had a heart attack or stroke. However, you recently learned that aspirin can increase your risk of bleeding. Imagine that you have taken low-dose aspirin once daily for several years to prevent you from having a heart attack or stroke. You have never had a heart attack or stroke. However, you recently learned that aspirin can increase your risk of bleeding. Imagine that you have taken ranitidine once daily for several years to treat your heartburn symptoms. Your symptoms are well controlled. However, you recently learned that ranitidine contains a cancer-causing chemical. Imagine that you have taken ranitidine once daily for several years to treat your heartburn symptoms. Your symptoms are well controlled. However, you recently learned that ranitidine contains a cancer-causing chemical. Imagine that you have taken prescription simvastatin once daily for several years to decrease your risk of a heart attack or stroke. Your cholesterol lab values are within the normal range. However, you recently learned that it may not provide as much benefit as you age. Imagine that you have taken prescription simvastatin once daily for several years to decrease your risk of a heart attack or stroke. Your cholesterol lab values are within the normal range. However, you recently learned that it may not provide as much benefit as you age. aParticipants received 3 vignettes (ie, aspirin, ranitidine, and simvastatin) in a randomized order. For each vignette, participants indicated how likely they would be to stop the medication if recommended by 5 different sources of information (ie, primary care provider, a pharmacist at a drugstore, “EHR-PCP,” “EHR-Direct,” and “Questions-Direct”). We presented 3 medication vignettes (ie, aspirin, ranitidine, and simvastatin) to each participant in a randomized order. Aspirin was selected given it is widely accessible as an OTC medication, taken by approximately one-third of older adults without cardiovascular disease, yet not recommended by clinical guidelines given the risks of bleeding [20,21]. Ranitidine was an OTC histamine-2 receptor antagonist, which was withdrawn from the US market in 2020 after being a best-selling medication in the United States, given that it contained a probable carcinogen [22,23]. Simvastatin is a prescription hydroxymethylglutaryl-CoA reductase inhibitor recommended for the primary prevention of cardiovascular events [24,25]. Participants were not able to go backward in the study due to the randomization of the vignettes. Participants were asked to rate the likelihood that they would stop each medication based on the source of information (6-point Likert scale with “Not at all likely (1)” and “Extremely likely (6)” as scale anchors). We subsequently asked participants to think about the AI app that connects with the EHR and provides immediate, personalized advice about OTC medications with a recommendation stored in the EHR. Participants were asked to rate their level of agreement with statements related to access, trust, nervousness, confusion and discomfort, and preferring to use the app before reaching out to the PCP (5-point Likert scale with “1=Strongly disagree” and “2=Strongly agree (5)” as scale anchors). Demographic Characteristics We collected information about participants’ age, gender, race, ethnicity, and highest level of education. Participants reported their health status using a 5-point Likert scale with response options of “Poor (1),” “Fair (2),” “Good (3),” “Very good (4),” and “Excellent (5),” and their health literacy level was measured by their level of confidence filling out medical forms with response options of “Not at all (1),” “A little bit (2),” “Somewhat (3),” “Quite a bit (4),” and “Excellent (5)” [26-28]. We also asked about their confidence using various forms of technology such as the internet, EHR, and apps (5-point Likert scale with “Not at all confident (1)” and “Very confident (5)” as scale anchors). We included 5 items modified from the technology acceptance model (ie, whether the AI app increases access, trustworthiness, nervousness, confusion and discomfort, and is preferred to contacting a PCP) [29,30]. Ethical Considerations The University of Michigan Institutional Review Board deemed this study exempt (HUM00246349). The informed consent information was included on the first page of the survey. All questions were required; however, participants could exit the survey anytime, and all data were collected anonymously. Prospective participants were told the anticipated duration of the survey, where and how long data would be stored, the investigator, and the purpose of the survey. Participants were compensated based on the terms of their panel agreement. Statistical Analysis We calculated descriptive statistics and reported the percentage of participants who reported being extremely likely to stop a medication (score of 6) based on the source of information. High Cronbach α scores indicated that participants responded consistently across medications by the source of information. Therefore, we created an average medication score by the source of information per participant. We used logistic regression to explore demographic characteristics that predict a high likelihood of stopping medications (scores 4-6) compared to those with low scores (scores 1-3). We used linear regression with continuous integer values between “Not at all confident (1)” and “Very confident (5)” to explore demographic characteristics that predict attitudes toward using the Questions-Direct-OTC app. All regression models included the demographic characteristics of age; gender; race; ethnicity; education; health status; health literacy; and confidence using the internet, EHR, and apps. We used a statistical significance level of P<.05. All analyses were conducted with R statistical software (version 4.2.2). We reported our study using the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) checklist and Checklist for Reporting Results of Internet E-Surveys (CHERIES) [31,32]. Results The survey was opened by 1318 individuals, of whom 73 participants were excluded due to not meeting the inclusion criteria (eg, aged younger than 65 years) or demographic quotas being filled. The final sample for analysis included 1245 respondents (94% completion rate). Demographic Characteristics Half (n=625, 50%) of the 1245 participants were female, and the median age was 70 (IQR 67-74 years; Table 2). Participants most often reported earning a high school diploma or less (n=391, 32%) or attending trade school, some college, or earning an associate’s degree (n=484, 39%). Participants most frequently reported having good health (n=595, 48%) and being extremely comfortable filling out medical forms. Most participants were confident making phone calls (n=1070, 86%), sending and receiving emails (n=1025, 82%) and SMS text messages (n=992, 80%), and using the internet (n=906, 73%). Fewer participants were confident using an EHR (n=790, 63%), apps (n=769, 62%), social media (n=607, 49%), or wearable devices (n=536, 43%). Table 2Demographic characteristics of older adult participants (n=1245).Demographic characteristicsValue Age (years), m edian (IQR) 70 (67-74) Gender, n (%) Female625 (50) Male619 (50) Transgender1 (0) Race, n (%) White894 (72) Black or African American195 (16) Asian or Asian American61 (5) Another race95 (8) Ethnicity, n (%) Hispanic229 (18) Education, n (%) High school diploma or less391 (32) Trade school, some college, or associate’s degree484 (39) Bachelor’s degree240 (19) Master’s degree or higher130 (10) Health status, n (%) Excellent55 (4) Very good282 (23) Good595 (48) Fair267 (21) Poor46 (4) Confidence filling out medical forms, n (%) Extremely662 (53) Quite a bit376 (30) Somewhat142 (11) A little bit45 (4) Not at all20 (2) Confidence using technologya, n (%) Phone calls1070 (86) Emails1025 (82) SMS text messages992 (80) Internet906 (73) Electronic health record790 (63) Apps769 (62) Social media607 (49) Wearable devices536 (43)aParticipants who selected scores of 4 or 5 on a 5-point Likert scale with scale anchors 1=not at all confident and 5=very confident. Demographic characteristics of older adult participants (n=1245). aParticipants who selected scores of 4 or 5 on a 5-point Likert scale with scale anchors 1=not at all confident and 5=very confident. Likelihood of Stopping Medications Older adults (n=1245) more frequently reported being extremely likely to stop a medication when the recommendation came from a PCP (range from n=748, 60.1% for aspirin due to a risk of bleeding to n=858, 68.9% for ranitidine due to a cancer-causing chemical; Table 3). In contrast, older adults were less likely to say that they would follow the recommendation from a pharmacist at a drugstore (extremely likely ranging from n=227, 18.2% for simvastatin due to a lack of benefit with age to n=361, 29% for ranitidine). Table 3Older adult’s likelihood of stopping a medication based on the source of the recommendation and score (n=1245)a.Source of informationLow-dose aspirin (risk of bleeding), n (%)Ranitidine (cancer-causing chemical), n (%)Simvastatin (lack of benefit with age), n (%) PCPb 1=Not at all likely42 (3.4)21 (1.7)20 (1.6) 222 (1.8)8 (0.6)15 (1.2) 351 (4.1)35 (2.8)43 (3.5) 4104 (8.4)78 (6.3)100 (8) 5278 (22.3)245 (19.7)279 (22.4) 6=Extremely likely748 (60.1)858 (68.9)788 (63.3) Pharmacist at a drugstore 1=Not at all likely137 (11)104 (8.4)122 (9.8) 2105 (8.4)68 (5.5)105 (8.4) 3190 (15.3)156 (12.5)218 (17.5) 4294 (23.6)261 (21)292 (23.5) 5286 (23)295 (23.7)281 (22.6) 6=Extremely likely233 (18.7)361 (29)227 (18.2) AIc app that connects with EHRd and provides personalized advice to PCP for approval prior to sharing with you 1=Not at all likely270 (21.7)230 (18.5)266 (21.4) 2122 (9.8)95 (7.6)117 (9.4) 3200 (16.1)178 (14.3)193 (15.5) 4237 (19)214 (17.2)252 (20.2) 5234 (18.8)239 (19.2)217 (17.4) 6=Extremely likely182 (14.6)289 (23.2)200 (16.1) AI app with EHR access that provides personalized advice directly to you 1=Not at all likely323 (25.9)270 (21.7)310 (24.9) 2151 (12.1)135 (10.8)175 (14.1) 3261 (21)218 (17.5)242 (19.4) 4231 (18.6)237 (19)243 (19.5) 5149 (12)173 (13.9)157 (12.6) 6=Extremely likely130 (10.4)212 (17)118 (9.5) AI app that collects information by asking you questions and provides personalized advice directly to you 1=Not at all likely337 (27.1)281 (22.6)318 (25.5) 2169 (13.6)151 (21.1)165 (13.3) 3248 (19.9)222 (17.8)271 (21.8) 4228 (18.3)208 (16.7)240 (19.3) 5142 (11.4)179 (14.4)119 (9.6) 6=Extremely likely121 (9.7)204 (16.4)132 (10.6)aThis table shows the percentage of participants who selected each score on a 6-point Likert scale with 1=not at all likely and 6=extremely likely to stop the medication as scale anchors.bPCP: primary care provider.cAI: artificial intelligence.dEHR: electronic health record. Older adult’s likelihood of stopping a medication based on the source of the recommendation and score (n=1245)a. aThis table shows the percentage of participants who selected each score on a 6-point Likert scale with 1=not at all likely and 6=extremely likely to stop the medication as scale anchors. bPCP: primary care provider. cAI: artificial intelligence. dEHR: electronic health record. Older adults were least likely to agree with stopping medication when the recommendation came from an AI app. Between 14.6% (n=182; aspirin) and 23.2% (n=289; ranitidine) of older adults were extremely likely to stop the medication when recommended by the EHR-PCP app. Regardless of integration within the EHR, when the AI app did not include approval by a PCP, only ~10% (n=130, 10.4% for aspirin and n=132, 10.6% simvastatin) and 17% (n=212; ranitidine) of older adults were extremely likely to stop the medication. In adjusted analyses, older adults who were female as opposed to male were more likely to stop a medication when the recommendation came from a PCP (odds ratio [OR] 1.55, 95% CI 1.18-2.03) or pharmacist (OR 1.27, 95% CI 1.10-1.47; Table 4). Table 4Demographic predictors of being likely to stop a medication based on the source of the recommendation using logistic regression (n=1245)a. PCPbPharmacist at a drugstoreAIc app with EHRd access and PCP approvaleAI app with EHR access and direct recommendationfAI app without EHR access and direct recommendationg Age, ORh (95% CI) 1.03 (1.00-1.06)0.99 (0.98-1.00)1.00 (0.99-1.01)1.01 (1.00-1.03)1.01 (1.00-1.02) Gender, OR (95% CI) MaleREFiREFREFREFREF Female1.55 (1.18-2.03)1.27 (1.10-1.47)1.14 (0.99-1.31)1.01 (0.88-1.16)1.04 (0.91-1.20) Race, OR (95% CI) WhiteREFREFREFREFREF Black of African American0.51 (0.37-0.71)0.90 (0.74-1.10)1.42 (1.17-1.73)1.38 (1.14-1.66)1.28 (1.06-1.54) Another race0.93 (0.60-1.41)0.71 (0.57-0.88)1.21 (0.98-1.49)1.23 (1.00-1.51)1.17 (0.95-1.44) Ethnicity, OR (95% CI) Non-HispanicREFREFREFREFREF Hispanic0.88 (0.61-1.27)1.15 (0.95-1.40)0.98 (0.82-1.17)0.96 (0.81-1.15)1.04 (0.87-1.24) Education, OR (95% CI)