Assessing healthcare organizations’ readiness to implement a learning health system: questionnaire validation using a Delphi method Authors: Catherine M Giroux, Paula L Bush, Mohammed Alkhaldi, Pascaline Kengne Talla, Yves Couturier, Aliki Thomas, André Bussières, Sabrina T Wong, Marie-Eve Poitras, Sara Ahmed Published: 2025 ABSTRACT AbstractIntroductionAdopting a learning health system (LHS) approach holds promise for bridging knowledge between policymakers, health professionals, managers, researchers, and patients and their families to collaboratively improve health care. Organizational readiness assessments exist in the quality improvement literature, but may not consider LHS components. This study aimed to establish the content validity of a new LHS readiness questionnaire.MethodsA three-round Delphi study was conducted to establish consensus on the importance, relevance, clarity, and comprehensiveness of the domains and items included in this questionnaire. Purposive sampling was used to recruit participants with expertise in LHS who are involved in healthcare organizations across Canada and internationally (n = 41). A minimum of 70% agreement represented consensus. A steering committee reviewed findings and refined items for clarity. Modified items were re-tested in subsequent rounds.ResultsIn Round 1, 85 items were tested, of which 41 achieved consensus, 7 were removed, 21 underwent major modification, 16 were clarified and retested, and 11 new items were proposed. Round 2 tested 36 items (25 revised, 11 new). 18 items achieved consensus, 8 were removed, and 10 were modified. In Round 3, 10 items were tested, 5 achieved consensus, 1 was removed, and 4 were clarified and included post-expert panel discussion. Overall, 41 items were retained in their original form, 20 were modified, and 7 new items were added. The final measure includes 68 items reflected by four domains: (1) performance to data (n = 13 items), (2) data to knowledge (n = 13 items), (3) knowledge to performance (n = 22 items), and (4) LHS core values (n = 20 items).ConclusionThe proposed new measure can help establish organizational readiness for change. Future research should seek to test the psychometric properties of this tool and explore potential barriers to its adoption amongst interested parties.Supplementary InformationThe online version contains supplementary material available at 10.1186/s12913-025-13636-2.Keywords: Learning health system, Organizational readiness to change, Delphi method, Content validation, Health services research, Measurement and evaluation Text box 1. Contributions to the literature• Content validation of a novel LHS-specific organizational readiness assessment tool, that has been informed by theory• Comparison between readiness assessments and maturity matrices in the context of LHS implementation and scale up• Rigorous Delphi method, with stable responses across three rounds of surveys Background Translating evidence into practice remains a persistent challenge in the health sciences. Research indicates that it can take up to 17 years for just 14% of empirical findings to be integrated into routine care, ultimately benefiting patients [1–3]. Knowledge, user experience, policy, and the contextual environment are constantly evolving, requiring an adaptation of practices to deliver integrated, high quality, and evidence-informed care [4]. Moreover, adapting or integrating these practices can support the well-being of both patients and health professionals while helping to mitigate costs associated with care provision [5, 6]. The COVID-19 pandemic challenged all healthcare and social services systems in terms of knowledge, resources, and workforce [2, 7, 8], illustrating a pressing need to assess knowledge-practice gaps related to organizations’ capacity to adapt and implement change amongst policymakers, health and social services system managers and professionals, researchers, and patients and their families and caregivers. One potential approach to bridging this gap is by adopting a learning health system (LHS), which can contribute to accelerating the translation of research into practice and the development of interventions that improve patient care and outcomes [9]. The LHS approach allows for cyclical transitions of real-time data from medico-administrative, clinical, and patient-reported data to knowledge, from knowledge to clinician and manager performance, and from performance back to medico-administrative, clinical, and patient reported data [5, 10]. Adopting an LHS approach can help healthcare organizations take a systematic and data-driven approach to generating and applying knowledge, ultimately improving the quality and value of the care they deliver, while stimulating innovation [9]. Thus, an LHS approach may improve care that focuses on patient needs, population health, patient and health professional experience, health equity, and health-related costs [11, 12]. Despite the potential benefits of an LHS approach, there are few specific examples of successful LHS implementation in practice [13]. This gap raises important questions about the factors that influence an organization’s ability to adopt and sustain LHS initiatives. In particular, little is known about healthcare organizations’ readiness to assess, improve, and monitor a given clinical practice or health condition. Organizational readiness for change has been discussed in the health services literature since the late 1970s [14] and may provide a useful framework for organizations to assess their capacity to engage with an LHS approach. According to Weiner (2009), organizational readiness refers to the members of an organization’s shared resolve to implement a change (i.e., change commitment) and their shared belief in their collective capacity to do so (i.e., change efficacy) [15–18]. Typically, organizational readiness assessments focus on the individuals who work within organizations (i.e., health professionals, managers/leaders, support and administrative staff, etc.) and often draw on theories, such as Weiner’s theory of organizational change or Bandura’s self- or collective efficacy [15–17, 19]. Maturity, a concept that is related to organizational readiness, originated in the field of computer sciences and has been applied to the use of healthcare technologies [20–22]. Maturity is defined as the degree to which a process can achieve a specific objective or outcome in a predictable way [23–25]. Within the context of LHS, maturity has been used to develop such assessment tools as network maturity grids [23]. While related, organizational readiness and maturity address different objectives. Firstly, studies discussing organizational readiness generally focused on the people in single organizations [16, 17, 26], while those addressing maturity focused on the technological capacities of organizations that belong to wider networks [20, 23, 24]. The extant literature suggests that organizational readiness is typically assessed before implementating an initial change (e.g., a new intervention) [16–18, 26], whereas maturity is assessed post-implementation but pre-scale up [23, 25, 27]. Within the context of LHS, organizational readiness assessments can help healthcare organizations prepare to adopt an LHS approach, while maturity assessments may support organizations already engaged in an LHS as they seek to expand or scale their efforts across a larger network. In terms of design, organizational readiness assessments tend to use Likert scales [17, 19, 26, 28], whereas maturity assessments use behaviourally anchored scales (e.g., not started, beginning, intermediate, mature, idealized state) [23, 27–30] to measure the concepts of interest. Within the health systems literature, work has been conducted on assessing the maturity of existing LHS [23]. Less is known about how health organizations can assess their readiness to adopt an LHS approach. Several tools are available to assess organizational readiness for change, such as the Organizational Readiness to Change Assessment (ORCA) [26] and the Organizational Readiness for Implementing Change (ORIC) scale [17]. However, few are designed to address a transformation as complex as implementing an LHS. The ORIC, for instance, is grounded in Weiner’s (2009) theory and is primarily intended to assess readiness for more general types of change (e.g., “we need to implement this change,” “we believe this change will improve outcomes”). While such tools capture readiness for change, they do not include the content needed to evaluate readiness to adopt an LHS approach. Readiness tools must capture the relevance and value of an LHS to an organization, reflecting specific resources and strategies that will contribute to the success of a LHS; if those who have the power to implement an LHS approach do not see the value for their specific context, uptake of the intervention may be hindered [31, 32]. A previous scoping review [33] conducted by members of our team found few empirical examples of LHS in the extant literature; moreover, while some articles [23, 29] presented tools to help assess the maturity of an LHS, these tools appeared to often assume that an LHS approach is already in place. We found no articles that catalogued tools designed to help healthcare organizations assess their readiness to implement an LHS approach. To address these concerns, our research team developed a questionnaire (hereafter ‘LHS readiness questionnaire’) based on Weiner’s (2009) theory of organizational change, intended to help healthcare organizations assess their current level of readiness to implement an LHS approach. The purpose of the present study was to enrich the existing knowledge on assessing organizational readiness to implement an LHS approach and to generate evidence in support of the content validity of our LHS readiness questionnaire. Methods Delphi study design Content validity is the degree to which the content of an instrument measures the intended construct; it refers to the relevance, comprehensiveness, and comprehensibility (i.e., clarity) of the construct being measured [34]. To establish consensus on these aspects of our LHS readiness questionnaire, we conducted a three-round Delphi using online surveys, evaluating the importance, relevance, clarity and comprehensiveness of its domains and items. We chose to include importance in our content validation since we were interested in whether participants would identify any domains or items as important but not relevant or vice versa. The Delphi approach is a commonly used consensus method where participants iteratively rank items using anonymous surveys until consensus is reached [35–37]. Consensus methods are useful when empirical evidence is lacking, limited, or contradictory; they are predicated on the idea that an accurate and reliable assessment can best be achieved by consulting a panel of experts and accepting the group consensus [37, 38]. Specifically, we addressed the following research questions: To what extent do participants consider the questionnaire domains and items to be important for assessing LHS implementation readiness? To what extent do participants consider the questionnaire domains and items to be important for assessing LHS implementation readiness? To what extent do participants consider the questionnaire domains and items to be relevant for assessing LHS implementation readiness? To what extent do participants consider the questionnaire domains and items to be relevant for assessing LHS implementation readiness? To what extent do participants consider the questionnaire items clearly articulated? To what extent do participants consider the questionnaire items clearly articulated? To what extent do participants consider the questionnaire domains and items to be comprehensive in their assessment of LHS implementation readiness? To what extent do participants consider the questionnaire domains and items to be comprehensive in their assessment of LHS implementation readiness? All study components have been reported using the ACCORD framework for consensus studies. Our study protocol is available on Open Science Framework [39]: https://osf.io/69ugd. Development of the Delphi surveys The Delphi surveys addressed key content validation questions related to the four domains of the LHS readiness questionnaire. Specifically, participants were instructed to assess each of the LHS readiness questionnaire items on the following three aspects: Importance: is this an important item to include in a tool assessing readiness to implement an LHS? Importance: is this an important item to include in a tool assessing readiness to implement an LHS? Relevance: is this useful information for evaluating readiness for an LHS? and Relevance: is this useful information for evaluating readiness for an LHS? and Clarity: is the item clearly written and does it make sense?) [40]. Clarity: is the item clearly written and does it make sense?) [40]. Participants were asked to rate the importance, relevance, and clarity of each item and domain on a three-point Likert scale (important/relevant/clear, somewhat important/relevant/clear, not important/relevant/clear). Similarly, participants rated the importance, relevance, and clarity of the introduction text included as a preamble to the LHS readiness questionnaire, alongside the text explaining the four domains included in the tool. Finally, they were asked to rate the comprehensiveness of each domain on a three-point scale and to identify any areas not covered by the items or to suggest additional items. We also asked participants to rate the comprehensiveness of the LHS readiness questionnaire as a whole. We determined that consensus would be reached with a minimum threshold of 70% agreement on each question, which aligns with other Delphi studies [35, 36, 41]. All criteria (importance, relevance, and clarity) needed to meet the 70% threshold to be eligible for inclusion. If an item only met the threshold for importance, for example, it was re-tested in a subsequent round. The Delphi surveys were piloted with three members of the Person-Centred Health Informatics lab at McGill University to test the clarity and flow of questions asked. The pilot also provided an estimate of how long each survey would take participants to complete. Participant recruitment Two types of participants were recruited to be involved in this Delphi study. These participants included a Delphi steering committee who would oversee the study and the expert participants who would respond to each Delphi survey. The Delphi steering committee We assembled a steering committee to review and interpret the results of each Delphi round. This committee consisted of academics, clinicians, and educators from occupational therapy, physical therapy, nursing, social work, implementation science, and health professions education, selected based on individuals’ expertise in digital health, measurement, evaluation, health policy research, and learning health systems. The committee was responsible for meeting in between Delphi rounds to discuss the findings and make decisions about which items needed to be reworded for clarity and re-tested in subsequent rounds. Decisions were made using a discussion and consensus approach. Delphi expert participant panel In May 2024, a purposive sampling strategy [42] was used to recruit a panel of participants with expertise or interests in LHS who are involved in healthcare organizations across Canada and internationally. We identified approximately 163 potential participants with LHS expertise or experience based on their recent publications, research activities, and academic or professional affiliations. Specifically, participants were identified through key informant recommendations, environmental/website scans of organizations working with LHS, searching Scopus for recent publications, and using existing provincial and national research and policy networks. We first contacted potential participants by sending personalized emails using Redcap™ (https://projectredcap.org/), with the goal of recruiting approximately 40 participants as recommended for Delphi studies to ensure the stability of responses over the three rounds [43, 44] and included more to mitigate potential attrition over the three rounds. During a two-week preparation phase, each particpant received an information letter, informed consent form, and a copy of our literature review and Delphi protocol. Informed consent was obtained from all participants prior to commencing this Delphi study. Participants completed a brief demographic survey at this time. We invited all prospective participants to share the invitation with members of their organization whom they believed to have LHS-specific knowledge or experience [42]. Participants were included if they identified as working within the field of LHS locally, nationally, or internationally, and if they represented diverse settings (e.g., academia, healthcare services, advocacy, policy) and roles (e.g., researchers, clinicians, managers, decision-makers, and patients and/or caregivers) within healthcare and social services systems. Development of the initial LHS readiness questionnaire The LHS readiness questionnaire was informed by key literature on theories of organizational change and the scoping review, previously conducted by members of our team, that identified potentially relevant domains and items [33]. To inform item development, we extracted information about LHS constructs from our scoping review and classified these constructs under four domains: (1) performance to data, (2) data to knowledge, (3) knowledge to performance, and (4) LHS core values. The initial LHS readiness questionnaire contained 11 multi-part items that served to assess organizational readiness across these four domains. Table 1 depicts a blueprint of the domains, definitions, and associated items in the initial questionnaire. Table 1LHS readiness questionnaire blueprintQuestionnaire domainQuestionnaire sub-domainCorresponding item numbersPerformance to DataThis domain focuses on patient health, health practices and/or heath system management; prioritizing the most important needs; identifying the problem of interest to study; and determining which outcomes or indicators to measure, how to measure them, and which sources and mechanisms should be used for data collection. The problem of interest could be at the patient, health professional, and/or organizational level.Interested Parties’ Needs and PrioritizationThis subdomain assesses organizations’ capacity to regularly identify, prioritize, and evaluate the needs of interested parties related to a specific problem of interest.1Defining Outcome and Indicator MeasuresThis subdomain assesses organizations’ capacity to identify outcomes or indicators to address specific objectives and measure them for a given problem of interest.NOTE: Outcomes and indicators could be qualitative or quantitative, including but not limited to, patient-reported experiences, patient-reported outcomes, patient clinical outcomes, health professional-reported experiences, health professional-reported outcomes, health manager-reported experiences, health manager-reported outcomes, and medico-administrative data.1, 3Data CollectionThis subdomain assesses organizational access to the mechanisms and resources required to collect data from all interested parties in real time for a given problem of interest.NOTE: The data sources could include medico-administrative data, self-report questionnaires, interviews, focus groups, clinical data, laboratory data, medical imaging data, biobank data, and direct observation data.1, 2Data to KnowledgeThis domain consists of assembling and analysing data, interpreting the results, and performing the knowledge translation activities.Data Assembly and AnalysisThis subdomain assesses what resources and infrastructure organizations can access that would enable the process of collecting and analysing data.NOTE: Data are collected in real time using digital platforms and various sources, like medico-administrative data sources, self-report questionnaires, interview sources, focus group sources, clinical data sources, laboratory data sources, medical imaging data sources, and direct observation data sources. Data are inspected, cleaned, transformed, visualized, and modelled with the goal of discovering useful information, informing conclusions, and supporting decision-making on a regular and timely basis.2, 4, 5Result InterpretationThis subdomain assesses organizations’ capacity to interpret and exchange results with relevant interested parties and determine which results to prioritize for continuous quality improvement.4, 5Knowledge to PerformanceThis domain consists of implementing and evaluating a practice that aims to resolve a problem of interest. Data can be collected during and after the implementation of practice change.Knowledge Synthesis and MobilizationThis subdomain assesses organizations’ capacity to synthesize the evidence (e.g., literature reviews, environmental scanning) to assess the current state of a given issue and identify gaps. It also consists of organizations’ readiness to transform the results obtained from data analyses into knowledge mobilization tools and activities on a regular and timely basis.6, 7, 8Practice Change Implementation and EvaluationThis subdomain assesses organizational readiness to design, implement, and evaluate the practice change.NOTE: Evaluation will focus on implementation outcomes, service outcomes, patient and health professional team outcomes, and performance indicators.3, 7, 8LHS Core ValuesThis domain is defined by policies, governance, regulations, and other core values that facilitate research, collaboration, and continuous learning.Governance Structures and PartnershipsThis subdomain assesses organizational readiness to implement iterative, adaptive, and collaborative governance structures and partnerships that are necessary to support an LHS’ sustainable operation, set required standards, maintain relationships, and meaningfully empower interested parties to participate in LHS decisions and activities.7, 9,Equity and AccessibilityThis subdomain assesses organizational readiness to implement an LHS that can:- Protect and improve the health of individuals, families, communities, and populations;- Examine problems and solutions through an equity lens, while ensuring impacts are distributed to reduce health disparities;- Be available and deliver value to all; and- Include and encourage diverse populations to participate actively.9Integrity and PrivacyThis subdomain assesses organizational readiness to implement an LHS that consists of appropriate ethics, legal, and privacy expertise. The organization and LHS will adhere to ethical guidelines, privacy legislation, data security and research integrity best practices, and full transparency to ensure the trust of all interested parties.9, 10Overall Perceived Readiness to ChangeThis subdomain assesses organizational members’ perceived readiness to change, specifically whether they agree that the change is important and feasible to undertake.11 LHS readiness questionnaire blueprint Performance to Data This domain focuses on patient health, health practices and/or heath system management; prioritizing the most important needs; identifying the problem of interest to study; and determining which outcomes or indicators to measure, how to measure them, and which sources and mechanisms should be used for data collection. The problem of interest could be at the patient, health professional, and/or organizational level. Interested Parties’ Needs and Prioritization This subdomain assesses organizations’ capacity to regularly identify, prioritize, and evaluate the needs of interested parties related to a specific problem of interest. Defining Outcome and Indicator Measures This subdomain assesses organizations’ capacity to identify outcomes or indicators to address specific objectives and measure them for a given problem of interest. NOTE: Outcomes and indicators could be qualitative or quantitative, including but not limited to, patient-reported experiences, patient-reported outcomes, patient clinical outcomes, health professional-reported experiences, health professional-reported outcomes, health manager-reported experiences, health manager-reported outcomes, and medico-administrative data. Data Collection This subdomain assesses organizational access to the mechanisms and resources required to collect data from all interested parties in real time for a given problem of interest. NOTE: The data sources could include medico-administrative data, self-report questionnaires, interviews, focus groups, clinical data, laboratory data, medical imaging data, biobank data, and direct observation data. Data to Knowledge This domain consists of assembling and analysing data, interpreting the results, and performing the knowledge translation activities. Data Assembly and Analysis This subdomain assesses what resources and infrastructure organizations can access that would enable the process of collecting and analysing data. NOTE: Data are collected in real time using digital platforms and various sources, like medico-administrative data sources, self-report questionnaires, interview sources, focus group sources, clinical data sources, laboratory data sources, medical imaging data sources, and direct observation data sources. Data are inspected, cleaned, transformed, visualized, and modelled with the goal of discovering useful information, informing conclusions, and supporting decision-making on a regular and timely basis. Result Interpretation This subdomain assesses organizations’ capacity to interpret and exchange results with relevant interested parties and determine which results to prioritize for continuous quality improvement. Knowledge to Performance This domain consists of implementing and evaluating a practice that aims to resolve a problem of interest. Data can be collected during and after the implementation of practice change. Knowledge Synthesis and Mobilization This subdomain assesses organizations’ capacity to synthesize the evidence (e.g., literature reviews, environmental scanning) to assess the current state of a given issue and identify gaps. It also consists of organizations’ readiness to transform the results obtained from data analyses into knowledge mobilization tools and activities on a regular and timely basis. Practice Change Implementation and Evaluation This subdomain assesses organizational readiness to design, implement, and evaluate the practice change. NOTE: Evaluation will focus on implementation outcomes, service outcomes, patient and health professional team outcomes, and performance indicators. LHS Core Values This domain is defined by policies, governance, regulations, and other core values that facilitate research, collaboration, and continuous learning. Governance Structures and Partnerships This subdomain assesses organizational readiness to implement iterative, adaptive, and collaborative governance structures and partnerships that are necessary to support an LHS’ sustainable operation, set required standards, maintain relationships, and meaningfully empower interested parties to participate in LHS decisions and activities. Equity and Accessibility This subdomain assesses organizational readiness to implement an LHS that can: - Protect and improve the health of individuals, families, communities, and populations; - Examine problems and solutions through an equity lens, while ensuring impacts are distributed to reduce health disparities; - Be available and deliver value to all; and - Include and encourage diverse populations to participate actively. Integrity and Privacy This subdomain assesses organizational readiness to implement an LHS that consists of appropriate ethics, legal, and privacy expertise. The organization and LHS will adhere to ethical guidelines, privacy legislation, data security and research integrity best practices, and full transparency to ensure the trust of all interested parties. Overall Perceived Readiness to Change This subdomain assesses organizational members’ perceived readiness to change, specifically whether they agree that the change is important and feasible to undertake. Subsequently, we mapped our items onto Weiner’s (2009) theory of organizational change to ensure our items captured the organizational readiness concepts. Table 2 depicts the alignment between each item and Weiner’s organizational readiness domains. Table 2Alignment between our LHS readiness questionnaire items and organizational readiness domainsDomainSubdomainDefinitionItemsChange CommitmentChange ValenceThe more organizational members value the change, the more they want to implement the change11a, 11bChange EfficacyTask DemandsKnowing what courses of action are necessary, what kinds of resources are needed, how much time is needed, and how activities should be sequenced1d, 1 h, 4b, 4c, 4d, 4e, 4f, 4 g, 6a, 6b, 6c, 6d, 6e, 6f, 6 g, 6 h, 6i, 6j, 9dResource AvailabilityWhether the organization has the human, financial, material, and informational resources necessary to implement the change well1a, 1b, 1c, 1e, 1f, 1 g, 2, 4a, 5, 7, 8, 9c, 10Situational FactorsWhether sufficient time exists to implement the change well or whether the internal political environment supports implementation3, 9a, 9b, 9d, 9e, 9f, 9 g, 9 h, 9i, 9j, 9k Alignment between our LHS readiness questionnaire items and organizational readiness domains We refined our initial questionnaire through an iterative process of feedback collection and integration. We consulted six researchers, who were not members of the Delphi Steering Committee, with expertise in LHS, knowledge mobilization, implementation science, health measurement, and primary care; four primary care professionals (family physicians and nurses); two health and social service managers; and three patient partners. Participants reviewed the questionnaire for clarity, suggested rewording when needed, and identified irrelevant or missing items. Finally, we translated the questionnaire from English to French. The questionnaire was translated by a bilingual language expert (English-French) and then reviewed by a research assistant and the bilingual research team for accuracy and consistency between the French and English versions. Administration of the Delphi surveys The Delphi surveys were administered via Redcap™ which ensured that data was entered into a secure web database that complied with Provincial and Federal regulatory requirements through any browser. Redcap™ also offered several affordances, including dashboards, analysis, reporting, and data extraction functionalities. Data collection for the Delphi occurred during the Summer of 2024. We followed a modified version of Dillman’s Tailored Design Method, which included multiple follow-up emails to the Delphi participants [45, 46]. Specifically, we followed up with participants on the 7th and 14th day of the data collection period for each Delphi round [45, 46]. This approach is known to achieve high response rates [45]. The survey took participants approximately 60 min to complete in Round 1 and 30 min in Rounds 2 and 3; all questions were optional. All data collection tools, documents, and communications were in both English and French. In each round, a link to the survey was sent to all participants. The survey was open for 14 days. The research team aggregated all findings and made decisions about what items met the criteria regarding clarity, importance, and relevance and could therefore be included in the final LHS readiness questionnaire and removed from future Delphi rounds. Items that required further review–and new items added in each round–were included in the surveys of the 2nd and 3rd rounds. Items that did not meet the consensus thresholds on importance and relevance were removed. Anonymized versions of the findings of each round were sent to participants for review, using Redcap, prior to the start of the next round. Data analysis The questionnaires for each round of the Delphi consisted of both closed- and open-ended responses. To analyse the closed-ended responses, we used Redcap and Microsoft Excel to calculate percent agreement for each item in each of the three Delphi rounds. To analyse the open-ended responses, we conducted a conventional content analysis [47]. In a conventional content analysis, researchers avoid using preconceived categories, coding inductively from the data instead [47]. All closed- and open-ended data were used by the steering committee to make decisions about which items to include, exclude, or retest in future rounds. The steering committee also used the open-ended data to clarify the language of any unclear items being retested in subsequent rounds. Ethics This study received ethical approval from the McGill University Faculty of Medicine and Health Sciences Review Board (study number A03-E23-24B) in March of 2024. Results Of the approximately 160 people invited to participate, 41 accepted and took part in this Delphi. Overall, respondents were predominantly female (n = 32; 73%), PhD-trained (n = 31; 71%), and researchers (n = 33; 75%) in academic (n = 28; 64%) or healthcare institutions (n = 9; 21%) between the ages of 30 and 45 (n = 27; 61%). Thirty-one (76%) participants were from Canada; ten (24%) were international. While we invited participants from every continent, only two additional country contexts were represented by international participants (i.e., the United States and the Netherlands). Most participants identified as having engaged with the topic of LHS for three to five years (n = 18; 41%). Twenty-four participants completed every round; since the Delphi rounds coincided with summer vacations, the remaining 17 participants completed the preparation phase and two (out of three) rounds. Table 3 provides an overview of the evolution of the LHS readiness questionnaire across rounds. Overall, 41 items were retained in their original form, 20 were modified, and 7 new items were added. Appendix A (supplemental file) provides an overview of all items; their importance, relevance, and clarity scores; and the decisions made after each round. A copy of the final LHS readiness questionnaire is available for download at https://rehab-projects.ca/clinician-resources/lhs-readiness/lhs-readiness-access-a-copy-of-the-questionnaire/. Table 3Evolution of LHS readiness questionnaire across Delphi roundsRoundParticipantsItems testedConsensus achievedItems removedItems modifiedNew items proposedComprehensivenessRound 1368541 (3 from D1, 11 from D2, 12 from D3, 15 from D4)7 (3 from D1, 3 from D3, 1 from D4)37 (21 major modifications and 16 clarified and retested)11No domains met thresholdRound 23336(25 revised and condensed, 11 new)18 (6 from D1, 2 from D2, 7 from D3, 3 from D4)8 (1 from D1, 1 from D2, 4 from D3, 2 from D4)10 (for retesting)—All domains and questionnaire met thresholdRound 333105 (2 from D1, 2 from D3, 1 from D4)1 (from D1)4 (clarified post-expert panel discussion; 2 from D1, 1 from D3, 1 from D4)—Final questionnaire completedNote. D1: Domain 1 – Performance to Data; D2: Domain 2 – Data to Knowledge; D3: Domain 3 – Knowledge to Performance; D4: Domain 4 – LHS Core Values Evolution of LHS readiness questionnaire across Delphi rounds 36 (25 revised and condensed, 11 new) Note. D1: Domain 1 – Performance to Data; D2: Domain 2 – Data to Knowledge; D3: Domain 3 – Knowledge to Performance; D4: Domain 4 – LHS Core Values Discussion The purpose of this study was to generate evidence in support of the content validity of our LHS readiness questionnaire. Creating validated tools for LHS and building consensus around the content validity of these tools is important for promoting a common understanding of what an LHS approach is amongst healthcare organizations [23, 48]. Following this content validation, our questionnaire is designed to assess four domains: (1) performance to data (n = 13 items), (2) data to knowledge (n = 13 items), (3) knowledge to performance (n = 22 items), and (4) LHS core values (n = 20 items). The initial LHS readiness questionnaire focused on assessing the knowledge, skills, and attitudes of people working within healthcare organizations, alongside their current practices and ability to access the resources required to collect, analyze, and use data. Preliminary items related to recruitment of specific expertise (e.g., legal expertise, knowledge translation expertise), current or future use of specific technologies or systems (e.g., data visualization software, data warehouses), and cataloguing current practices (e.g., measuring specific indicators, engaging with interested parties). Over the three Delphi rounds, participants indicated that several questions were redundant and could thus be removed or consolidated with other questions. These items included some of the new additions proposed, which related to specific types of expertise (e.g., plain language, content area, research, quality improvement). Participants noted that while dedicated time for continuing education was important, time for research or QI activities was less important to an LHS approach. Participants also emphasized that understanding interested parties’ needs was less important than knowing how to engage and leverage community expertise. Although considered important, there are few specific resources for how to effectively engage with LHS partners [33]. Going forward, early involvement of patients and the public in assessing organizational readiness to implement an LHS approach may provide important insights that would otherwise be overlooked. Establishing clear protocols of how to effectively involve patients and the public in an LHS approach may provide a roadmap for realizing this objective. Our LHS readiness questionnaire complements existing readiness and maturity tools. We intentionally mapped our items to Weiner’s (2009) theory of organizational readiness to change, which is similar to existing tools like the ORIC and ORCA [15, 17, 26]. While we did not identify any specific LHS readiness assessment, our previously conducted scoping review identified several LHS-specific maturity tools [33]. These tools shared similar domains and items to those presented in our current LHS readiness questionnaire. Specifically, leadership, governance, and strategy were consistently highlighted, underscoring the importance of supportive leadership, clear governance structures, and strategic commitment to improvement [23, 27, 49]. Culture, environment, and teamwork also recurred, emphasizing collaboration and strong organizational climates that enable change [29, 50]. Frameworks included domains related to learning, research, and innovation, pointing to the need for structured processes that embed evaluation, knowledge generation, and continuous improvement into practice [29, 50]. Patient and community engagement was another shared element, stressing the value of tailoring interventions to patient needs and involving external partners in implementation efforts [51, 52]. Data, digital infrastructure, and information systems further appeared as critical enablers of readiness, ensuring interoperability, analytics capacity, and digital literacy [24, 53]. Finally, existing tools accounted for resources and implementation complexity, recognizing that sustainability depends on the availability of time, staff, funding, and scalable infrastructure [23, 54]. Taken together, these domains represent important factors to consider when assessing and strengthening organizational readiness to implement and sustain learning health systems. Participants expressed that the preliminary questionnaire’s focus was largely on technical resources and expertise, with less attention given to the human learning components. Accordingly, they recommended adding new items related to, for example, organizational members’ ability to: (1) use the data collected, (2) reflect on their learning with the LHS approach, and (3) speak up without risk of consequences. Overall, the initial questionnaire prioritized data and informatics, whereas the final questionnaire also includes content on the humans within an organization and providing the time, space, and resources to enable learning within an organization (i.e., both machine learning and human learning). Participants’ responses highlighted the importance of emphasizing the human factors included in an LHS approach; having the resources to collect data is important but having confidence that the people within an organization can interpret, use, and learn from the data is an essential component of readiness [55]. These findings may contrast with existing literature, where LHS maturity models often emphasize the technological infrastructure needed to support an LHS across healthcare networks [23, 29]. It also shows the need for an organizational readiness assessment focused specifically on LHS, since existing organizational readiness tools and theories are predicated on these human factors, specifically organizational members’ readiness and willingness to embrace a change [15, 17, 26]. Despite our early questionnaire focusing heavily on data and informatics [33], the expert participants in our Delphi emphasized the importance of social science and educational knowledge, theories, and frameworks. Going forward, LHS researchers and policymakers should consider the interdisciplinary nature of the health system and who is being expected to learn within it. It is important to support knowledge, skills, and attitude development by the humans within an organization–alongside their technical capacity to collect and use data in practice [56, 57]. While most items clearly met the inclusion thresholds, some required further discussion due to perceived clarity. For example, the third question on organizational indicators underwent major modification in Round 2 based on participant feedback but was reverted in Round 3 to a slightly refined version of its original wording. The steering committee played a key role in weighing importance, relevance, and clarity scores while also considering participants’ open-ended feedback. Additionally, because the LHS readiness questionnaire was tested in both English and French, some feedback applied only to one version. Ambiguous items were deliberated from multiple perspectives–drawing on committee members’ diverse backgrounds, expertise, and linguistic preferences–before finalizing decisions. Notably, some items the committee initially expected to score highly in importance or relevance, such as legal expertise as a required resource, were ultimately removed for not meeting thresholds. This underscores the value of content validation through a consensus approach, as expert participants’ perspectives may differ from those of the steering committee or research team [36, 37]. We developed core items that are useful for helping diverse healthcare settings to assess their readiness to implement an LHS approach. Based on our findings, it appears that participants find the items included within these four domains to be important, relevant, and clear. Results suggest our LHS readiness questionnaire is a comprehensive organizational readiness assessment that health organizations can use to understand their degree of readiness to implement an LHS approach. Future research should implement and validate this LHS readiness questionnaire in diverse settings (e.g., primary care, hospitals, rehabilitation, additional country contexts), including cognitive interviewing and/or focus groups alongside further psychometric testing to validate scoring. Additionally, we aim to adapt this questionnaire to help patients and caregivers meaningfully assess whether their care setting is ready for, or already using, an LHS approach. This may involve developing new items (e.g., whether patients have access to their data or if their data is used in health professional education). To achieve this, we will engage partners to review and refine the questionnaire for relevance to patients, caregivers, and/or other equity-seeking groups. We will also test and validate any revised versions of these questionnaires using a similar approach to the one presented here. Currently, this LHS readiness questionnaire serves as a formative assessment tool, enabling organizations interested in adopting an LHS approach to evaluate whether they have the resources, capacity, and buy-in required for successful implementation. It can be used by single healthcare organizations or larger health systems, but is intended for use early in the LHS contemplation phase–that is, before an LHS has been implemented. Higher scores in each domain may indicate an increased level of readiness to implement an LHS approach, whereas lower scores may highlight areas needing improvement or attention prior to adopting an LHS approach. Specifically, the use of this LHS readiness questionnaire may help organizations identify what specific resources need to be in place or what tasks need to be done to increase, maintain, or optimize their level of readiness. Once an LHS approach is adopted at an organizational level, the use of LHS maturity tools may be appropriate to determine whether and how to scale the LHS approach to a network level [23]. Future work that explores the relationship between organizational readiness and maturity within the context of LHS is recommended. Study strengths and limitations The findings of this study directly contribute to health services and policy research by generating evidence in support of the content validity of a questionnaire that can help healthcare organizations determine their level of readiness to implement a learning health system. Moreover, study findings have implications for the broader health systems, since the main goal of an LHS is to improve organizational learning across a health centre’s clinical, operational, and research activities to achieve high value and quality healthcare [31, 32]. Methodologically, this study was rigorous, using a Delphi approach to establish consensus on the items included in our LHS readiness questionnaire. We also employed strategies known to achieve high response rates, such as a Dillman technique, throughout. We used triangulation of closed- and open-ended findings, peer debriefing through our steering committee meetings, and the creation of an audit trail as strategies to ensure the reliability and trustworthiness of our findings [58–60]. Our questionnaire was intended to align with Weiner’s theory of organizational change, which is a key theory underpinning other organizational readiness tools found in the QI literature [15, 17, 61]. Nonetheless, this study is not without limitations. We aimed to recruit participants representing diverse interested parties, including clinicians, policymakers, patients and caregivers, and healthcare managers or leaders. Despite our sampling strategy, participants were predominantly researchers from high-income countries. Thus, our results may be influenced by their perspectives and experiences. Moreover, we had planned to explore the extent to which responses varied by interested party group but were unable to complete this analysis due to the homogeneity of our respondents. Since most of our respondents were researchers, our LHS readiness questionnaire should be tested with other interested party groups to ensure its appropriateness. Additionally, while this study included representation from international experts, international expert participation was not as comprehensive as originally planned. Expansion of participation from all health system levels (e.g., policy, managerial, and clinical), more locations and regions (e.g., LMICs), more disciplines, and more interested party groups (e.g., patients, caregivers, equity-seeking groups) is recommended in similar and future research. Additional research that focuses on the acceptability and feasibility of using this LHS readiness questionnaire in low- and middle-income country contexts is recommended. Finally, this study did not aim to implement this LHS readiness questionnaire in practice nor did it aim to perform psychometric analysis on the items included within it. Future content validation work in diverse settings (e.g., primary care, tertiary care) is recommended to help inform scoring. Conclusion This study validated the content of a newly developed LHS readiness questionnaire. While existing tools assess organizations’ readiness for change, adopting an LHS approach requires a dedicated assessment due to its complexity, resource demands, and need for sustained commitment. We used a Delphi approach to confirm the importance, relevance, and clarity of the items included under each domain of our LHS readiness questionnaire, alongside the comprehensiveness of the questionnaire’s items. Future research should seek to test the psychometric properties of this tool and explore potential barriers to adoption amongst interested parties. This tool is intended to be used by members of diverse healthcare organization types (e.g., family medicine groups, individual hospitals, health networks) and can be completed by individual members or in collaboration with health leaders, healthcare professionals, and patients or caregivers. By enabling organizations to assess their level of readiness, this tool can help position them to achieve the quintuple aim and improve patient care, population health, patient and care provider experience, health equity, and health-related costs. Supplementary Information Below is the link to the electronic supplementary material. Supplementary Material 1 Acknowledgements The authors would like to acknowledge Herve Tchala Vignon Zomahoun and Sophie Boies for their early contributions to this work. Author contributions All authors contributed to the conceptualization and design of this study. CG, PB, MA, and SA contributed to the analysis in between Delphi rounds. CG drafted the manuscript. All authors provided feedback and approved the final version. Funding Unité de soutien SSA du Québec, INSPIRE program Lindsay Foundation, BRILLIANT program Habilitas Foundation. Data availability Data supporting the results reported in this article can be found in Appendix A of the Supplementary Materials files. Declarations Ethical approval This study received Research Ethics Board approval from McGill University’s Health Sciences Institutional Review Board (AO3-E23-24B) in March 2024. This study adhered to the principles set forth in the Declaration of Helsinki – ethical principles for medical research involving human participants. Informed consent was obtained from all participants prior to their participation in this Delphi study. Consent for publication Not applicable. Competing interests The authors declare no competing interests. Footnotes Correspondence concerning this article should be addressed to Sara Ahmed, PT, PhD, Professor, McGill University Faculty of Medicine and Health Sciences, School of Physical and Occupational Therapy, Research Scientist, Centre for Interdisciplinary Research in Rehabilitation (CRIR); Research Institute of the McGill University Health Centre; Site Research Manager, CIUSSS West-Central Montreal Lethbridge-Layton-Mackay Rehabilitation Centre; Scientific Director, Biomedical Research and Informatics Living Laboratory for Innovative Advances of New Technologies (BRILLIANT). Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. References Associated Data Supplementary Materials Supplementary Material 1 Data Availability Statement Data supporting the results reported in this article can be found in Appendix A of the Supplementary Materials files.