Application of Innovation Tournament methodology to inform de-implementation strategies: lessons learned in addressing mammography overscreening in older women Authors: Rachel C. Shelton, Parisa Tehranifar, Savannah P. Alexander, Emmy Burroughs, Anita G. Karr, Nathalie Moise Published: 2026 ABSTRACT BackgroundDe-implementation is a critical, understudied area within implementation science, particularly de-implementation strategy development. Aligned with most national guidelines, de-implementation of routine mammography screening among older women (without considering patient preferences, health, life expectancy), is a compelling area for advancing de-implementation science. While previously applied for implementation, this research seeks to advance understanding of Innovation Tournaments (InTs) as a participatory methodology to inform de-implementation strategy development.MethodsBuilding on formative qualitative research and informed by User-Centered Design (UCD), we conducted an InT to generate, confirm, and refine de-implementation strategies among purposively selected providers and administrators across two healthcare systems in New York City. InT development began 07/2021 and launched 11/2022–05/2023. A key challenge in co-creating, launching, and maintaining the InT was difficulty communicating about de-implementation, especially given differences in opinion about guidelines across provider specialties. To address this complexity, we collaborated with experts in InTs, shared decision-making, healthcare informatics, communication, creative content development, and UCD. Based on input during the UCD process and to facilitate engagement across participants with varying opinions, we expanded the InT to assess ideas and opinions about mammography guidelines for older women and appropriateness of de-implementation, in addition to assessing de-implementation strategy ideas. We conducted descriptive analyses of InT forum engagement data and content analysis of forum posts/comments.ResultsOf 47 participants who entered the InT website, 36 engaged in the forum (voted, commented, or posted). Participants voted 256 times on posted ideas/opinions/strategies and comments, submitted 71 comments, and posted 13 new ideas/opinions/strategies. Most participants engaged in the forum during their initial visit to the InT, with limited engagement over time. Content analysis of posts/comments identified de-implementation strategy categories largely aligned with the ERIC taxonomy (e.g., patient education, provider training/reminders), though system-level strategies specific to de-implementation emerged.ConclusionsFindings advance de-implementation methods by indicating importance of formative qualitative research and UCD to inform InT development and engagement (particularly if stakeholders disagree on de-implementation appropriateness). Unique considerations include: how best to seed InT content; challenges in generating new strategies that meaningfully expand upon formative qualitative research findings; and time and resources needed to develop, elicit and sustain stakeholder engagement over time.Supplementary InformationThe online version contains supplementary material available at 10.1186/s43058-026-00950-x. Contributions to the Literature• This is one of the first evaluations of Innovation Tournament (InT) use for de-implementation strategy development.• Communicating about de-implementation is complex. InTs may require refinements to facilitate engagement when consensus around de-implementation and related guidelines is lacking.• Formative qualitative research and transdisciplinary User-Centered Design are key to developing and facilitating engagement in InTs focused on de-implementation.• There are trade-offs of seeding InTs with content to spur participation and challenges in sustaining engagement.• For de-implementation, InTs help engage clinical stakeholders, elicit their perceptions, and confirm strategies, but may be less suitable for generating a comprehensive list of strategies or prioritizing them. Contributions to the Literature• This is one of the first evaluations of Innovation Tournament (InT) use for de-implementation strategy development.• Communicating about de-implementation is complex. InTs may require refinements to facilitate engagement when consensus around de-implementation and related guidelines is lacking.• Formative qualitative research and transdisciplinary User-Centered Design are key to developing and facilitating engagement in InTs focused on de-implementation.• There are trade-offs of seeding InTs with content to spur participation and challenges in sustaining engagement.• For de-implementation, InTs help engage clinical stakeholders, elicit their perceptions, and confirm strategies, but may be less suitable for generating a comprehensive list of strategies or prioritizing them. Background De-implementation is a critical, understudied area within implementation science (IS), defined as “reducing (frequency and/or intensity) or stopping the use or delivery of health services or practices that are ineffective, unproven, harmful, overused, inappropriate, and/or low-value by practitioners and delivery systems to patients” [1]. Despite implementation science advancements [2], less research has focused on de-implementation. There are gaps in determining optimal methods for de-implementation strategy identification and selection, and recognition that de-implementation determinants and strategies may differ from those for implementation [3]. This research seeks to advance participatory methods for de-implementation strategy identification and development. Innovation Tournaments (InT) are a promising method for engaging stakeholders and eliciting ideas [4, 5], and have been used to engage healthcare system stakeholders and facilitate implementation strategy crowdsourcing [6, 7]. InTs have not been used for de-implementation strategy development, and it is unclear if these same methods can be applied. We conducted the ‘Rethink Mammography’ InT to engage providers and healthcare administrators to exchange ideas about reducing mammography overuse among older women, with the goal of identifying de-implementation strategies at patient, provider, and system levels. De-implementation of routine mammography screening among older women (without considering patient preferences, health, life expectancy) is a compelling case study for advancing the science of de-implementation. Though widely adopted, routine mammography screening may have lower benefits among older women [8–13], and does not always lead to significant breast cancer mortality reduction in those with shorter life expectancies, greater competing health risks, and lower risk for aggressive tumors [14]. Continued screening poses harms, including anxiety, complications from diagnostic tests, false-positive results, unnecessary treatment, and over-diagnosis [10, 11, 15, 16]. A recent study estimated 47% of breast cancer was over-diagnosed among women aged 75–84 years and 54% was over-diagnosed among women 85 and older, with no significant reduction in breast cancer death from screening [11]. While there is variation, many U.S.-based national guidelines do not support routine mammography screening among women 75 and older [17–23]. Yet, ~56% of community-dwelling women of that age report receiving mammography screening in the prior 2 years, including 50% of women with <10-year life expectancy [24, 25], indicating the need to identify strategies to reduce mammography overuse among older women. Using mammography overuse among older women as a case study, this research describes methods, considerations, and lessons learned in exploring usefulness of an InT to engage stakeholders and generate, confirm, and refine de-implementation strategies. Methods Rethink Mammography InT development Our formative mixed-methods research [26–28] highlights intersecting patient, provider, and organizational-level barriers that reinforce routine mammography screening among older women, including: lack of patient and provider education around mammography discontinuation, ease and habitual nature of screening reinforced by institutionalized cues (e.g., reminder letters, electronic health record [EHR] notifications), varying guidelines across professions, and guideline misalignment between departments (e.g., annual mammography reminder letters from radiology irrespective of primary care provider [PCP] referrals). To identify de-implementation strategies that could map onto these barriers, we conducted an online InT to engage providers and administrators in exchanging ideas around mammography overuse in older women. We collaborated with experts in InTs, shared decision-making, informatics, communication, creative content development, and User-Centered Design (UCD) to co-create, pilot, and launch the InT. During development of branding (e.g., ‘Rethink Mammography InT’) and communications (e.g., email invitations), we used a journey mapping process with a creative designer and web developer to inform the participant experience (e.g., four 1–2-hour journey mapping sessions [3 months], followed by iterative development of the initial website [13 months] in collaboration with 6 co-investigators). Co-creation began 07/2021, and the InT was live 11/2022–05/2023. A clinician champion on our team was critical in engaging initial participants and providing insights into the clinician experience. A key challenge that lengthened this process was determining how best to approach de-implementation, given that our formative work ([26–28]; and unpublished data), indicated participants had differing views on guideline appropriateness. We worked closely with the creative designer in crafting, branding, and communicating about the InT to encourage engagement, regardless of participants’ views. We emphasized the importance of provider expertise given lack of consensus on mammography screening among older women and controversy about benefits/harms. Thus, we broadened the InT’s scope beyond generating de-implementation strategies to also eliciting ideas and opinions about de-implementation and related guidelines (Additional Files 1,2). The email invitation contained a link to the InT website. The landing page provided a brief rationale for the InT and hyperlinks to national guidelines, followed by a brief demographic survey. To encourage participants to proceed to the forum, we acknowledged differing views by asking, “National guidelines (e.g., American College of Physicians, USPSTF, ACS) do not recommend routine screening mammography in women 75 years and older without consideration of patients’ life expectancy, health, and informed preferences. Do you endorse these recommendations?” Participants were given the option to indicate level of endorsement from 1 to 5, with 1 labeled as ‘Yes’, 3 labeled as ‘Unsure’, and 5 labeled as ‘No’ (2 and 4 were unlabeled). Participants viewed a brief video illustrating how to submit posts, comments, and votes before entering the forum, modeled after Reddit (Fig. 1; Additional Files 3,4). We seeded the forum with 4 posts based on our formative qualitative research ([26–28]; and unpublished data), including strategies related to patient education, system cues, and a story about the complexity of navigating patient expectations when deciding whether to discontinue screening, recounted by a PCP on our team. Our decisions to include the video, model the forum after a widely-used social media platform, and seed the forum with posts were in response to InT prototype feedback from co-investigators, who flagged that it was unclear how to engage in the forum. The final InT webpage stated, “We invite you to return to the Rethink Mammography Innovation Tournament to see what your colleagues are saying about your ideas and add new ideas and comments of your own!” To incentivize engagement/discussion and gain insight into how to prioritize ideas shared, participants were told that Rethinkers who contributed the 2 ideas that participants interacted with most would receive a $250 prize.Fig. 1Rethink Mammography InT forum. Legend: this image of the Rethink Mammography InT forum shows the forum as participants saw it, except for the buttons enabling deletion of posts and comments, which only the research study team had access to for security purposes and ultimately never utilized. Alt text: screenshot capturing a portion of the Rethink Mammography InT forum, with posts (i.e., ideas/opinions/strategies) and their professional tags (e.g., primary care provider, Resident/fellow) aligned left. Comments on the posts and their respective professional tags are indented underneath the posts. Up/downvotes displayed underneath posts and next to comments (denoted by blue and orange arrows, respectively). Narrow righthand column with menu to sort posts by total number of votes (i.e., sum of upvotes and downvotes), date submitted, and profession (e.g., specialty); or to submit a new post Rethink Mammography InT forum. Legend: this image of the Rethink Mammography InT forum shows the forum as participants saw it, except for the buttons enabling deletion of posts and comments, which only the research study team had access to for security purposes and ultimately never utilized. Alt text: screenshot capturing a portion of the Rethink Mammography InT forum, with posts (i.e., ideas/opinions/strategies) and their professional tags (e.g., primary care provider, Resident/fellow) aligned left. Comments on the posts and their respective professional tags are indented underneath the posts. Up/downvotes displayed underneath posts and next to comments (denoted by blue and orange arrows, respectively). Narrow righthand column with menu to sort posts by total number of votes (i.e., sum of upvotes and downvotes), date submitted, and profession (e.g., specialty); or to submit a new post Recruitment and data collection We recruited participants from healthcare systems in New York City (NYC): New York Presbyterian Hospital [NYPH]/Columbia Doctors (our future de-implementation trial sites) and Mount Sinai Health System to enhance generalizability of findings. We purposively recruited providers and administrators who refer women for mammography (e.g., PCPs), have expertise in mammography and breast cancer (e.g., radiologists, oncologists), and have decision-making authority around mammography (e.g., administrators). We made the methodological choice to focus the InT on clinicians, as our formative work [26–28] underscored challenges of communicating about mammography de-implementation with older patients in an InT. We used email-based snowball sampling beginning with our contacts across the healthcare systems and co-created branded email invitations and updates on InT progress (e.g., number of submitted votes, comments, posts), including a hyperlink to the InT and encouragement to share with colleagues (Additional Files 1,2,5,6). Data analysis We conducted descriptive analyses of the quantitative InT data (Tables 1, 2). Two researchers (RCS,SPA) trained in qualitative methods independently double-coded and performed content analysis of the posts/comments [29] to identify overarching categories and perceptions of de-implementation strategies. The coded data was mapped onto previously identified de-implementation barriers (unpublished data [26–28]; and unpublished data), to inform de-implementation strategy development, using these steps: 1) identifying and grouping barriers at patient, provider, organizational levels; and 2) linking barriers and potential change domains to match de-implementation strategies. De-implementation strategies were broadly organized according to the Expert Recommendations for Implementing Change (ERIC) taxonomy, with consideration of refinement/additions for de-implementation [30]. We adhered to Standards for Reporting Qualitative Research (Additional File 7).Table 1Characteristics of participants in the InTa (n = 47)Specialty% (n)Primary Care53.2% (25)ObGyn6.4% (3)Geriatrics2.1% (1)Radiology19.1% (9)Breast Surgery6.4% (3)Breast Oncology8.5% (4)Other6.4% (3)Role% (n)Attending Physician80.9% (38)Researcher12.8% (6)Resident/Fellow10.6% (5)Hospital/Clinical Administration6.4% (3)Nurse2.1% (1)Community Health Worker/Health Educator2.1% (1)Practice% (n)New York Presbyterian Hospital51.1% (24)Columbia Doctors29.8% (14)Mount Sinai34.0% (16)Other2.1% (1)Age% (n)Under 250.0% (0)25–3421.3% (10)35–4429.8% (14)45–5423.4% (11)55–6417.0% (8)65 and Above8.5% (4)Gender% (n)Female80.9% (38)Male19.1% (9)Other0.0% (0)aParticipants could select more than one specialty, role, and practice, respectively (e.g., why counts sum to more than 47 and percentages sum to more than 100%)Table 2Types of engagement in the InT from November 2022 to May 2023Collective engagement the 1sttime participants entered the forumCollective engagement the 2ndtime participants entered the forumCollective engagement the 3rdtime participants entered the forumCollective engagement over timeEngagement per participant over timeAverage (Range)Participants347 (2 new & 5 returning participants)a1 (returning)b36N/AVotes2312502565.4 (0–26) votesComments*6542711.5 (0–15) commentsPosts*1120130.3 (0–2) posts*Excludes 9 posts and 1 comment entered by the research study team to facilitate participant engagement; a9 out of 47 participants logged in a second time, but only 7 of the 9 contributed any votes, comments, or posts; b2 out of 47 participants logged in a third time, but only 1 of the 2 contributed any votes, comments, or posts Table 1Characteristics of participants in the InTa (n = 47)Specialty% (n)Primary Care53.2% (25)ObGyn6.4% (3)Geriatrics2.1% (1)Radiology19.1% (9)Breast Surgery6.4% (3)Breast Oncology8.5% (4)Other6.4% (3)Role% (n)Attending Physician80.9% (38)Researcher12.8% (6)Resident/Fellow10.6% (5)Hospital/Clinical Administration6.4% (3)Nurse2.1% (1)Community Health Worker/Health Educator2.1% (1)Practice% (n)New York Presbyterian Hospital51.1% (24)Columbia Doctors29.8% (14)Mount Sinai34.0% (16)Other2.1% (1)Age% (n)Under 250.0% (0)25–3421.3% (10)35–4429.8% (14)45–5423.4% (11)55–6417.0% (8)65 and Above8.5% (4)Gender% (n)Female80.9% (38)Male19.1% (9)Other0.0% (0)aParticipants could select more than one specialty, role, and practice, respectively (e.g., why counts sum to more than 47 and percentages sum to more than 100%) Characteristics of participants in the InTa (n = 47) aParticipants could select more than one specialty, role, and practice, respectively (e.g., why counts sum to more than 47 and percentages sum to more than 100%) Table 2Types of engagement in the InT from November 2022 to May 2023Collective engagement the 1sttime participants entered the forumCollective engagement the 2ndtime participants entered the forumCollective engagement the 3rdtime participants entered the forumCollective engagement over timeEngagement per participant over timeAverage (Range)Participants347 (2 new & 5 returning participants)a1 (returning)b36N/AVotes2312502565.4 (0–26) votesComments*6542711.5 (0–15) commentsPosts*1120130.3 (0–2) posts*Excludes 9 posts and 1 comment entered by the research study team to facilitate participant engagement; a9 out of 47 participants logged in a second time, but only 7 of the 9 contributed any votes, comments, or posts; b2 out of 47 participants logged in a third time, but only 1 of the 2 contributed any votes, comments, or posts Types of engagement in the InT from November 2022 to May 2023 *Excludes 9 posts and 1 comment entered by the research study team to facilitate participant engagement; a9 out of 47 participants logged in a second time, but only 7 of the 9 contributed any votes, comments, or posts; b2 out of 47 participants logged in a third time, but only 1 of the 2 contributed any votes, comments, or posts Results InT participant recruitment At InT launch in 11/2022, we reached out to 58 of our contacts and followed up with them and 29 others recruited via snowball sampling. While infeasible to track all sampling, we were included on email invitations to 15 individual providers/administrators; 6 family/internal medicine, geriatric, and breast radiology provider groups; and 3 breast surgeon/pathology and family/internal medicine listservs. Due to low engagement (e.g., possibly related to launching during winter holidays), we seeded the forum with 5 additional posts and 1 comment in 12/2022, and in 01/2023, shortened the invitation (Additional Files 1,2). Subsequently, participant engagement increased (Fig. 2). We emailed updates in late February/early March, mid-April, and late April 2023, encouraging new and returning participants to see what was “trending”, highlighting the number of submitted ideas, opinions, and votes (Additional Files 5,6). After the InT closed, participants received a newsletter reporting the results and identifying the 2 new (not seeded) ideas participants engaged with the most (Additional File 8).Fig. 2Outreach and engagement. Legend: this multi-line graph shows the level of engagement in terms of how many votes, comments, and posts participants had collectively submitted at key time points, i.e., just prior to our research team’s decision to implement additional outreach. The graph shows the impact of that outreach on the level of engagement over time. As a key example, as of 1/12/2023, nearly 2 months after launching the InT, 7 participants had collectively submitted 17 votes, 3 comments, and 2 posts. We then shortened the email invitation and added URLs (i.e., rather than just hyperlinks) to improve readability and sent 17 follow-up emails and 3 new email invitations. We also have record of our email invitation being forwarded to 14 individuals, 3 provider listservs, and 3 provider groups soon after 1/12/2023. By 2/2/2023, 34 participants had collectively submitted 189 votes, 55 comments, and 12 posts. Alt text: none. Legend is sufficient Outreach and engagement. Legend: this multi-line graph shows the level of engagement in terms of how many votes, comments, and posts participants had collectively submitted at key time points, i.e., just prior to our research team’s decision to implement additional outreach. The graph shows the impact of that outreach on the level of engagement over time. As a key example, as of 1/12/2023, nearly 2 months after launching the InT, 7 participants had collectively submitted 17 votes, 3 comments, and 2 posts. We then shortened the email invitation and added URLs (i.e., rather than just hyperlinks) to improve readability and sent 17 follow-up emails and 3 new email invitations. We also have record of our email invitation being forwarded to 14 individuals, 3 provider listservs, and 3 provider groups soon after 1/12/2023. By 2/2/2023, 34 participants had collectively submitted 189 votes, 55 comments, and 12 posts. Alt text: none. Legend is sufficient Participant characteristics Forty-seven participants completed the demographic survey, representing three NYC settings: NYPH (51%), Columbia Doctors (30%), Mount Sinai (34%). Over half identified their specialty as primary care (Table 1). Most (58%) strongly endorsed national guidelines that do not recommend routine screening mammography in women 75 and older without consideration of patients’ life expectancy, health, and informed preferences. Endorsement varied across specialty: 38% of radiologists strongly endorsed versus 67% of PCPs, geriatricians, ObGyns (Fig. 3).Fig. 3Endorsement of mammography screening guidelines among InT participants (n = 45). Legend: none. Alt text: stacked bar chart displays the proportion of individuals who responded on a scale of 1 (yes), 2, 3 (unsure), 4, and 5 (no) to the following question that was asked just before entering the Rethink Mammography InT forum: “National guidelines (e.g., American College of Physicians, USPSTF, ACS) do not recommend routine screening mammography in women 75 years and older without consideration of patients’ life expectancy, health, and informed preferences. Do you endorse these recommendations?” The percentages of those who responded 1 (yes) are 58% (all participants across specialties, n = 45), 67% (Primary Care, Geriatrics, ObGyn, n = 27), 50% (Oncology, Surgery, or Other specialty, n = 10), and 38% (Radiology, n = 8) Endorsement of mammography screening guidelines among InT participants (n = 45). Legend: none. Alt text: stacked bar chart displays the proportion of individuals who responded on a scale of 1 (yes), 2, 3 (unsure), 4, and 5 (no) to the following question that was asked just before entering the Rethink Mammography InT forum: “National guidelines (e.g., American College of Physicians, USPSTF, ACS) do not recommend routine screening mammography in women 75 years and older without consideration of patients’ life expectancy, health, and informed preferences. Do you endorse these recommendations?” The percentages of those who responded 1 (yes) are 58% (all participants across specialties, n = 45), 67% (Primary Care, Geriatrics, ObGyn, n = 27), 50% (Oncology, Surgery, or Other specialty, n = 10), and 38% (Radiology, n = 8) InT engagement Figure 2 illustrates participant engagement throughout the InT. Of the 47 participants who entered the InT website, 36 actively engaged in the forum (voted, commented, or posted). Nearly all active participants voted on at least one post or comment (35/36), whereas less than one-third posted at least one new idea/opinion/strategy, and half commented on at least one post. Participants cast 256 votes on ideas/opinions/strategies (28 downvotes, 228 upvotes) across all posts and comments, submitted 71 comments, and posted 13 new ideas/opinions/strategies, excluding seeded content. Most votes, comments, and posts were contributed during participants’ first time entering the InT (Table 2). We organized ideas, opinions, strategies into de-implementation strategies using these overarching categories: 1) Patient Education; 2) Provider Training; 3) Shared Decision-Making; 4) Integrated EHR Feature; 5) Health System Alignment/Reminder Letters (Table 3). While strategies largely aligned with the ERIC taxonomy (e.g., patient education, provider training/reminders), strategies specific to de-implementation were proposed at the system level (Table 3).Table 3Summary of key de-implementation strategies identified in the Innovation Tournament which were mapped onto barriers addressed (by the research team)De-implementation Strategy with related Quotes and Alignment with ERIC TaxonomyUp-votesDown-votesBarriers Addressed1) Patient Education: (Aligned with ‘Use Mass Media’ and ‘Conduct Educational Meetings’ in ERIC taxonomy) • Broader education and communication campaigns for older women and their families providing information about the age transition for mammography recommendations (seeded)- “The evidence for cancer screening harms for older sicker patients needs to [be] told outside of the clinical encounter through media. The lonesome PCP counseling can feel ineffective when the rest of the world is awashed in industry supported pink ribbons.”- “PCPs are ill equipped to discuss screening with patients in many cases – too much to expect the PCP to be up to date on mammography, hypertension, diabetes, etc. This should be a public health campaign that included breast cancer experts.”72Patient level: Lack of knowledge, societal norms, sense of agency over health, anxieties around stoppingProvider level: Limited clinical time, challenging nature of discussions2) Provider Education and Training: (Aligned with ‘Conduct Ongoing Training’ in ERIC taxonomy)•Training for medical students/residents on evaluating patient health holistically; emphasis on prognosis, patient’s values/goals, risks/benefits of screening (emergent)140• Education/training for primary care providers (e.g., Grand Rounds) about evidence of benefits and harms of screening mammography for older women (seeded)100- “Messaging at Grand Rounds more traditionally have greatly emphasized underdiagnosis and need for more screening, a more evidence-based discussion covering overdiagnosis, improved cancer treatment lowering yield of screening have rarely made headlines, agree it is time”- “Medical students and residents need more education on evaluating their patients with a holistic lens, instead of just checking boxes. An emphasis on prognosis and function, and understanding the risks/benefits of a screening test can help with this.”Provider level: Lack of provider awareness and disagreement around guidelines; provider communication3) Shared Decision-Making: (Aligned with ‘Prepare patients/consumers to be active participants’ and ‘Remind Clinicians’ in ERIC taxonomy)• Patient-facing decision aid to encourage education and reflection around risks and benefits of mammography screening (seeded)161Patient level: Lack of knowledge, sense of agency, anxieties around stoppingProvider level: Provider communication with patients; challenging discussion about life expectancy; limited clinical timeSystem level: Automatic cues to screen patients• Alert in Epic about order gaps and patient prognosis to prompt shared decision-making during visit. This could include email or print versions of tailored patient information in post-visit follow-up communication. (seeded)72- “e-Prognosis has excellent decision-aids that could be shared with patients”- “I agree the automated screening alert should explain the importance of shared decision-making (SDM) for those > 75 years with linkage to a SDM tool or at least information that can be conveyed to patients so they have accurate information with which to decide”Disagreement about appropriateness of age-based guidelines & SDM with patients, particularly among radiologists:- “While I think it is great medicine to discuss the science of why we do anything (including mammography) with our patients, I don’t think any discussion with a healthy woman of any age around stopping mammography based on an age number is appropriate. We have limited time with patients and that time should be spent focused on the most high-yield issues. A discussion around paring back on screening (of any kind) is appropriate in patients with limited life expectancy and not before.”- “Age alone should not be the basis to continue or discontinue screening. Beyond age 75 years, the decision to discontinue screening mammography should be based on a shared decision-making process informed by the woman’s health status and longevity.”4) Integrated EHR Features: (Aligned with ‘Remind Clinicians’ in ERIC taxonomy) • Include reminder flags and talking points in EHR to help remind clinicians and patients to talk through the implications of screening (emergent)71• Adjusting automated Epic screening guidelines/alerts under Care Gaps to reflect guidelines for older women (emergent)92 - “I would love to have that when I put the mammography order in, [the EHR] says, “Hey, did you know this is a 76-year-old? Have you had a conversation about shared decision-making?” … It would make my life a little easier because generally, it’s a pretty small part of my focus.” - “The automated Epic screening guidelines/alerts under Care Gaps is the ACS recommendation (Annual screening in females: 40 to 85 years) which I think is not appropriate/applicable to the majority of patients in my panel. I think that replacement with USPSTF guidelines for up to age 75 followed by a separate guideline starting at age 75 for a risk-benefit discussion regarding continuing mammogram would be more helpful and effective.”Provider level: Lack of provider awareness and disagreement about guidelines; automatic cue to screenSystem level: Automatic cue to screen patients, misalignment between departments within healthcare system5) Health System Alignment and Reminder Letters: (Aligned with ‘Conduct local consensus discussions’ in ERIC taxonomy; Communications/Revising Patient Reminder Letters to Address De-implementation may be specific new strategy related to De-implementation)• A lot of PCPs think patients over 75 get too many reminder letters, making them think that mammography is more urgent than it is. We could convene a hospital taskforce of PCPs, radiologists and administrators to reach consensus on how to best approach the language about screening guidelines in the mammography results letter mailed to older patients. (seeded)244Patient level: Habitual; inconsistent messaging and patient confusionProvider level: Disagreement about guidelinesSystem level: Automatic cue to screen, misalignment between departments within healthcare system - “Agree fully these letters has to be re-examined, the language can be more evidence based/nuanced for the over 75 yr old …. Generally speaking for all ages, I have often worried that these letter trigger significant anxiety from a small group of pts who are vigilant and already screening very closely; while others who are avoidant are unmoved. I am sorry to say the letters can trigger negative emotions in myself as a busy PCP, because when the pt calls about it or wave it at me mid-visit, I often have to stop my train of thoughts and work flow to address it (mostly to reassure) and then perform the numerous clicks in another tab in Epic to order the screening mammo (hunting the same ICD code over and over).”- “Rewording of reminder letters for older patients is not a bad idea, emphasizing the optional nature of screening at older ages. It is really about patient preference for medical intervention if a disease process (breast cancer) is identified.” Table 3Summary of key de-implementation strategies identified in the Innovation Tournament which were mapped onto barriers addressed (by the research team)De-implementation Strategy with related Quotes and Alignment with ERIC TaxonomyUp-votesDown-votesBarriers Addressed1) Patient Education: (Aligned with ‘Use Mass Media’ and ‘Conduct Educational Meetings’ in ERIC taxonomy) • Broader education and communication campaigns for older women and their families providing information about the age transition for mammography recommendations (seeded)- “The evidence for cancer screening harms for older sicker patients needs to [be] told outside of the clinical encounter through media. The lonesome PCP counseling can feel ineffective when the rest of the world is awashed in industry supported pink ribbons.”- “PCPs are ill equipped to discuss screening with patients in many cases – too much to expect the PCP to be up to date on mammography, hypertension, diabetes, etc. This should be a public health campaign that included breast cancer experts.”72Patient level: Lack of knowledge, societal norms, sense of agency over health, anxieties around stoppingProvider level: Limited clinical time, challenging nature of discussions2) Provider Education and Training: (Aligned with ‘Conduct Ongoing Training’ in ERIC taxonomy)•Training for medical students/residents on evaluating patient health holistically; emphasis on prognosis, patient’s values/goals, risks/benefits of screening (emergent)140• Education/training for primary care providers (e.g., Grand Rounds) about evidence of benefits and harms of screening mammography for older women (seeded)100- “Messaging at Grand Rounds more traditionally have greatly emphasized underdiagnosis and need for more screening, a more evidence-based discussion covering overdiagnosis, improved cancer treatment lowering yield of screening have rarely made headlines, agree it is time”- “Medical students and residents need more education on evaluating their patients with a holistic lens, instead of just checking boxes. An emphasis on prognosis and function, and understanding the risks/benefits of a screening test can help with this.”Provider level: Lack of provider awareness and disagreement around guidelines; provider communication3) Shared Decision-Making: (Aligned with ‘Prepare patients/consumers to be active participants’ and ‘Remind Clinicians’ in ERIC taxonomy)• Patient-facing decision aid to encourage education and reflection around risks and benefits of mammography screening (seeded)161Patient level: Lack of knowledge, sense of agency, anxieties around stoppingProvider level: Provider communication with patients; challenging discussion about life expectancy; limited clinical timeSystem level: Automatic cues to screen patients• Alert in Epic about order gaps and patient prognosis to prompt shared decision-making during visit. This could include email or print versions of tailored patient information in post-visit follow-up communication. (seeded)72- “e-Prognosis has excellent decision-aids that could be shared with patients”- “I agree the automated screening alert should explain the importance of shared decision-making (SDM) for those > 75 years with linkage to a SDM tool or at least information that can be conveyed to patients so they have accurate information with which to decide”Disagreement about appropriateness of age-based guidelines & SDM with patients, particularly among radiologists:- “While I think it is great medicine to discuss the science of why we do anything (including mammography) with our patients, I don’t think any discussion with a healthy woman of any age around stopping mammography based on an age number is appropriate. We have limited time with patients and that time should be spent focused on the most high-yield issues. A discussion around paring back on screening (of any kind) is appropriate in patients with limited life expectancy and not before.”- “Age alone should not be the basis to continue or discontinue screening. Beyond age 75 years, the decision to discontinue screening mammography should be based on a shared decision-making process informed by the woman’s health status and longevity.”4) Integrated EHR Features: (Aligned with ‘Remind Clinicians’ in ERIC taxonomy) • Include reminder flags and talking points in EHR to help remind clinicians and patients to talk through the implications of screening (emergent)71• Adjusting automated Epic screening guidelines/alerts under Care Gaps to reflect guidelines for older women (emergent)92 - “I would love to have that when I put the mammography order in, [the EHR] says, “Hey, did you know this is a 76-year-old? Have you had a conversation about shared decision-making?” … It would make my life a little easier because generally, it’s a pretty small part of my focus.” - “The automated Epic screening guidelines/alerts under Care Gaps is the ACS recommendation (Annual screening in females: 40 to 85 years) which I think is not appropriate/applicable to the majority of patients in my panel. I think that replacement with USPSTF guidelines for up to age 75 followed by a separate guideline starting at age 75 for a risk-benefit discussion regarding continuing mammogram would be more helpful and effective.”Provider level: Lack of provider awareness and disagreement about guidelines; automatic cue to screenSystem level: Automatic cue to screen patients, misalignment between departments within healthcare system5) Health System Alignment and Reminder Letters: (Aligned with ‘Conduct local consensus discussions’ in ERIC taxonomy; Communications/Revising Patient Reminder Letters to Address De-implementation may be specific new strategy related to De-implementation)• A lot of PCPs think patients over 75 get too many reminder letters, making them think that mammography is more urgent than it is. We could convene a hospital taskforce of PCPs, radiologists and administrators to reach consensus on how to best approach the language about screening guidelines in the mammography results letter mailed to older patients. (seeded)244Patient level: Habitual; inconsistent messaging and patient confusionProvider level: Disagreement about guidelinesSystem level: Automatic cue to screen, misalignment between departments within healthcare system - “Agree fully these letters has to be re-examined, the language can be more evidence based/nuanced for the over 75 yr old …. Generally speaking for all ages, I have often worried that these letter trigger significant anxiety from a small group of pts who are vigilant and already screening very closely; while others who are avoidant are unmoved. I am sorry to say the letters can trigger negative emotions in myself as a busy PCP, because when the pt calls about it or wave it at me mid-visit, I often have to stop my train of thoughts and work flow to address it (mostly to reassure) and then perform the numerous clicks in another tab in Epic to order the screening mammo (hunting the same ICD code over and over).”- “Rewording of reminder letters for older patients is not a bad idea, emphasizing the optional nature of screening at older ages. It is really about patient preference for medical intervention if a disease process (breast cancer) is identified.” Summary of key de-implementation strategies identified in the Innovation Tournament which were mapped onto barriers addressed (by the research team) • Broader education and communication campaigns for older women and their families providing information about the age transition for mammography recommendations (seeded) - “The evidence for cancer screening harms for older sicker patients needs to [be] told outside of the clinical encounter through media. The lonesome PCP counseling can feel ineffective when the rest of the world is awashed in industry supported pink ribbons.” - “PCPs are ill equipped to discuss screening with patients in many cases – too much to expect the PCP to be up to date on mammography, hypertension, diabetes, etc. This should be a public health campaign that included breast cancer experts.” Patient level: Lack of knowledge, societal norms, sense of agency over health, anxieties around stopping Provider level: Limited clinical time, challenging nature of discussions - “Messaging at Grand Rounds more traditionally have greatly emphasized underdiagnosis and need for more screening, a more evidence-based discussion covering overdiagnosis, improved cancer treatment lowering yield of screening have rarely made headlines, agree it is time” - “Medical students and residents need more education on evaluating their patients with a holistic lens, instead of just checking boxes. An emphasis on prognosis and function, and understanding the risks/benefits of a screening test can help with this.” Patient level: Lack of knowledge, sense of agency, anxieties around stopping Provider level: Provider communication with patients; challenging discussion about life expectancy; limited clinical time System level: Automatic cues to screen patients - “e-Prognosis has excellent decision-aids that could be shared with patients” - “I agree the automated screening alert should explain the importance of shared decision-making (SDM) for those > 75 years with linkage to a SDM tool or at least information that can be conveyed to patients so they have accurate information with which to decide” Disagreement about appropriateness of age-based guidelines & SDM with patients, particularly among radiologists: - “While I think it is great medicine to discuss the science of why we do anything (including mammography) with our patients, I don’t think any discussion with a healthy woman of any age around stopping mammography based on an age number is appropriate. We have limited time with patients and that time should be spent focused on the most high-yield issues. A discussion around paring back on screening (of any kind) is appropriate in patients with limited life expectancy and not before.” - “Age alone should not be the basis to continue or discontinue screening. Beyond age 75 years, the decision to discontinue screening mammography should be based on a shared decision-making process informed by the woman’s health status and longevity.” - “I would love to have that when I put the mammography order in, [the EHR] says, “Hey, did you know this is a 76-year-old? Have you had a conversation about shared decision-making?” … It would make my life a little easier because generally, it’s a pretty small part of my focus.” - “The automated Epic screening guidelines/alerts under Care Gaps is the ACS recommendation (Annual screening in females: 40 to 85 years) which I think is not appropriate/applicable to the majority of patients in my panel. I think that replacement with USPSTF guidelines for up to age 75 followed by a separate guideline starting at age 75 for a risk-benefit discussion regarding continuing mammogram would be more helpful and effective.” Provider level: Lack of provider awareness and disagreement about guidelines; automatic cue to screen System level: Automatic cue to screen patients, misalignment between departments within healthcare system Patient level: Habitual; inconsistent messaging and patient confusion Provider level: Disagreement about guidelines System level: Automatic cue to screen, misalignment between departments within healthcare system - “Agree fully these letters has to be re-examined, the language can be more evidence based/nuanced for the over 75 yr old …. Generally speaking for all ages, I have often worried that these letter trigger significant anxiety from a small group of pts who are vigilant and already screening very closely; while others who are avoidant are unmoved. I am sorry to say the letters can trigger negative emotions in myself as a busy PCP, because when the pt calls about it or wave it at me mid-visit, I often have to stop my train of thoughts and work flow to address it (mostly to reassure) and then perform the numerous clicks in another tab in Epic to order the screening mammo (hunting the same ICD code over and over).” - “Rewording of reminder letters for older patients is not a bad idea, emphasizing the optional nature of screening at older ages. It is really about patient preference for medical intervention if a disease process (breast cancer) is identified.” Discussion InTs are one of many possible crowdsourcing methods to engage stakeholders [31, 32] and have been successfully used to generate implementation strategies. For example, Stewart and colleagues’ InT generated 65 implementation strategies over 5 weeks [7]. Van Pelt and colleagues’ InT generated 73 implementation strategies over 9 weeks [33]. While we had strong engagement (e.g., 256 votes, 71 comments submitted), our InT generated 13 new ideas over 6 months. This narrower range suggests unique challenges when leveraging InTs for de-implementation, given its complexity and disagreements around guideline appropriateness. Due to this lack of consensus, our interdisciplinary research team applied expertise as clinicians, health system researchers and leaders, and creative content developers to a 16-month UCD process to develop the Rethink Mammography InT website and communications, a lengthier process than anticipated. We sought to create an intuitive, low-burden participant experience, while effectively communicating the complex idea of mammography de-implementation. We modeled the InT forum after the familiar Reddit platform, invited participants to lend their expertise, and encouraged them to “see what [was] trending” and “what others [were] saying about [their] ideas and opinions” to facilitate discussion and welcome a range of opinions. Despite our efforts, disagreements may have stymied strategy elicitation, especially since the forum did not facilitate real-time discussion and most participants did not return after their initial visit. Additionally, our seeded ideas (based on our formative qualitative work) were intended to and did encourage engagement, but may have constrained idea generation. Finally, in contrast to InTs focused on implementation strategy generation [7, 33], we solicited broader ideas and opinions about mammography overuse and de-implementation in older women to encourage participation among those with differing views, potentially limiting strategy elicitation. This lack of consensus might be a particularly salient issue for de-implementation, distinct from studies focused on implementation. Returning to the above examples of implementation-focused InTs, Stewart and colleagues invited > 700 behavioral health system leaders, clinicians, and community stakeholders to submit ideas about how their organization could help them use evidence-based practices (EBPs) [7] – in this context, there was likely consensus on the importance of improving EBP use [34]. Additionally, Van Pelt and colleagues implemented InT recruitment throughout a large urban area, inviting sexual minority men to share ideas for delivering a new form of PrEP to their community – as such, this study was also conducted among a population likely to support its implementation [33]. Our study suggests adaptations may be needed when leveraging InTs for de-implementation. We employed a multi-step approach, in which formative research was critical in understanding multilevel factors that reinforce routine mammography among older women, and providing initial de-implementation strategy ideas to seed the InT. However, many elicited strategies overlapped with our formative research findings. InTs may be more useful for confirming de-implementation strategies (e.g., through up/downvotes) and generating potential delivery modalities (e.g., video patient education), rather than generating or prioritizing a comprehensive range of strategies. This is consistent with research from Becker-Haimes and colleagues [35], who compared use of field observations/qualitative interviews and an InT in generating ideas to increase EBPs among staff working with children with autism. Observations/interviews generated nearly double the strategies than the InT (26 versus 14), with overlap in identified strategies. Strategies from observations/interviews were more specific, though the InT yielded unique strategy types. Given time and resources needed to develop and elicit engagement in our InT, research is needed to understand InT value and cost-effectiveness for developing de-implementation strategies, as well as trade-offs of seeding InTs to spur participation. While many de-implementation strategies identified in our InT align with implementation strategies (e.g., training/reminding clinicians) [36, 37], other strategies (e.g., health system alignment, revising patient letters) may be specific to de-implementation and not reflected in the ERIC taxonomy [30]. Further, while our InT provided insights into perceived strategy feasibility, it was less useful in determining which strategies to prioritize for development. Additionally, few strategies addressed the ‘unlearning’ needed for de-implementation when practices are institutionalized or normative within systems [26, 38]. Future research should explore methods that are well-suited for de-implementation and reflect strategies addressing learning and un-learning processes. Conclusions Despite limitations (e.g., limited generalizability beyond urban health systems), this is one of the first studies to evaluate use of InT methodology for de-implementation strategy development. It is possible our approach to facilitate engagement influenced the number and nature of ideas posted; for example, we seeded ideas to encourage dialogue and incentivized participants by offering a prize to those who submitted ideas with the most engagement/interaction. However, all seeded ideas were informed by our formative research and were intended to facilitate idea refinement; further, engagement/interaction metrics were not used to measure ‘best’ idea rating. Key takeaways include importance of formative qualitative research and UCD to inform InT engagement and strategy elicitation, complexity of navigating differing views on de-implementation in an InT, and need for efforts to sustain InT involvement. While the InT was more time-intensive than anticipated and did not provide a large number of strategies, it was useful for engaging and understanding clinician perceptions of de-implementation, and elicited or confirmed a foundation of strategies to inform subsequent strategy prioritization. Future research should explore methods that are useful in selecting and tailoring de-implementation strategies beyond this initial phase. Electronic supplementary material Below is the link to the electronic supplementary material. Supplementary material 1: Rethink InT email invitation 1: Description of data: This is the initial email invitation to the Rethink Mammography InT that we disseminated. Supplementary material 2: Rethink InT email invitation 2: Description of data: This is the revised email invitation that we began disseminating about 2 months after the launch of the Rethink Mammography InT, in which we shortened the language and added URLs (i.e., rather than just hyperlinks) to improve readability and engagement. These revisions were part of the ‘new invites and follow-up’ shown immediately after 1/12/23 in Figure 2. Supplementary material 3: Rethink InT instructional video (desktop): Description of data: After completing the brief demographic survey and answering the question about guideline endorsement (see Methods section in main text), participants viewed this brief instructional video illustrating how to submit posts, comments, and votes before proceeding to the InT forum. This is the desktop version of that video. Supplementary material 4: Rethink InT instructional video (mobile): Description of data: After completing the brief demographic survey and answering the question about guideline endorsement (see Methods section in main text), participants viewed this brief instructional video illustrating how to submit posts, comments, and votes before proceeding to the InT forum. This is the mobile version of that video. Supplementary material 5: Rethink InT email update: Description of data: This email update included the then-current number of posts and comments (collectively referred to as ‘ideas and opinions’) and votes as well as language (e.g., ‘trending’) aimed at boosting engagement (i.e., returning to or engaging in the Rethink Mammography InT forum for the first time). This email update is the update referenced immediately after 2/27/23 in Figure 2 and was sent to a) those who had provided informed consent to participate and completed the sociodemographic survey (regardless of whether they had yet contributed a vote, comment, or post in the forum), b) those whom we had originally invited (regardless of whether they had yet provided informed consent to participate), c) those to whom our email invitation had been forwarded, and d) those whom participants had recommended we reach out to on the last page of the InT website. Supplementary material 6: Rethink InT email last call: Description of data: Similar to the ‘Rethink InT email update,’ the ‘Rethink InT email last call’ included the then-current number of posts and comments (collectively referred to as ‘ideas and opinions’) and votes as well as language (e.g., ‘trending’) aimed at boosting engagement (i.e., returning to or engaging in the Rethink Mammography InT forum for the first time), highlighting that the InT was about to close (i.e., ‘LAST CHANCE’). This email last call corresponds with the last calls referenced immediately after 4/18/23 and 4/25/23 in Figure 2. As with the ‘Rethink InT email update’, this email was sent to a) those who had provided informed consent to participate and completed the sociodemographic survey (regardless of whether they had yet contributed a vote, comment, or post in the forum), b) those whom we had originally invited (regardless of whether they had yet provided informed consent to participate), c) those to whom our email invitation had been forwarded, and d) those whom participants had recommended we reach out to on the last page of the InT website. Supplementary material 7: Standards for Reporting Qualitative Research checklist: Description of data: This is the completed Standards for Reporting Qualitative Research checklist. Supplementary material 8: Rethink InT newsletter: Description of data: After the conclusion of the Rethink Mammography InT, we sent all participants who had provided informed consent to participate and completed the sociodemographic survey (regardless of whether they had contributed a vote, comment, or post in the forum) a newsletter that reported the results of the InT, including the 2 new (i.e., not seeded) ideas that participants engaged with the most and the 5 overarching categories of strategies elicited from the InT. Supplementary material 1: Rethink InT email invitation 1: Description of data: This is the initial email invitation to the Rethink Mammography InT that we disseminated. Supplementary material 2: Rethink InT email invitation 2: Description of data: This is the revised email invitation that we began disseminating about 2 months after the launch of the Rethink Mammography InT, in which we shortened the language and added URLs (i.e., rather than just hyperlinks) to improve readability and engagement. These revisions were part of the ‘new invites and follow-up’ shown immediately after 1/12/23 in Figure 2. Supplementary material 3: Rethink InT instructional video (desktop): Description of data: After completing the brief demographic survey and answering the question about guideline endorsement (see Methods section in main text), participants viewed this brief instructional video illustrating how to submit posts, comments, and votes before proceeding to the InT forum. This is the desktop version of that video. Supplementary material 4: Rethink InT instructional video (mobile): Description of data: After completing the brief demographic survey and answering the question about guideline endorsement (see Methods section in main text), participants viewed this brief instructional video illustrating how to submit posts, comments, and votes before proceeding to the InT forum. This is the mobile version of that video. Supplementary material 5: Rethink InT email update: Description of data: This email update included the then-current number of posts and comments (collectively referred to as ‘ideas and opinions’) and votes as well as language (e.g., ‘trending’) aimed at boosting engagement (i.e., returning to or engaging in the Rethink Mammography InT forum for the first time). This email update is the update referenced immediately after 2/27/23 in Figure 2 and was sent to a) those who had provided informed consent to participate and completed the sociodemographic survey (regardless of whether they had yet contributed a vote, comment, or post in the forum), b) those whom we had originally invited (regardless of whether they had yet provided informed consent to participate), c) those to whom our email invitation had been forwarded, and d) those whom participants had recommended we reach out to on the last page of the InT website. Supplementary material 6: Rethink InT email last call: Description of data: Similar to the ‘Rethink InT email update,’ the ‘Rethink InT email last call’ included the then-current number of posts and comments (collectively referred to as ‘ideas and opinions’) and votes as well as language (e.g., ‘trending’) aimed at boosting engagement (i.e., returning to or engaging in the Rethink Mammography InT forum for the first time), highlighting that the InT was about to close (i.e., ‘LAST CHANCE’). This email last call corresponds with the last calls referenced immediately after 4/18/23 and 4/25/23 in Figure 2. As with the ‘Rethink InT email update’, this email was sent to a) those who had provided informed consent to participate and completed the sociodemographic survey (regardless of whether they had yet contributed a vote, comment, or post in the forum), b) those whom we had originally invited (regardless of whether they had yet provided informed consent to participate), c) those to whom our email invitation had been forwarded, and d) those whom participants had recommended we reach out to on the last page of the InT website. Supplementary material 7: Standards for Reporting Qualitative Research checklist: Description of data: This is the completed Standards for Reporting Qualitative Research checklist. Supplementary material 8: Rethink InT newsletter: Description of data: After the conclusion of the Rethink Mammography InT, we sent all participants who had provided informed consent to participate and completed the sociodemographic survey (regardless of whether they had contributed a vote, comment, or post in the forum) a newsletter that reported the results of the InT, including the 2 new (i.e., not seeded) ideas that participants engaged with the most and the 5 overarching categories of strategies elicited from the InT.