Evaluation of an evidence-based practice education workshop focused on critical appraisal for advanced practice nurses: A before-after intervention study Authors: Ai Tomotaki, Ikuko Sakai, Hiroki Fukahori, Yasunobu Tsuda Published: 2024 ABSTRACT BackgroundRegarding the evidence-based practice (EBP) process, nurses tend to perceive that critical appraisal of research articles is challenging. Though critical appraisal in EBP has been a frequent topic in medical education, implementing and disseminating existing EBP education for nurses are challenging in countries with limited EBP educational resources. This study aimed to evaluate changes in practice, attitude, knowledge and skills of EBP before and after an EBP workshop focused on critical appraisal among advanced practice nurses.MethodsThis study was a before and after educational intervention study. The workshops were provided in small groups in 2017 and 2018. The targeted nurses were certified nurse specialists (CNSs) who provide patient care and are certified as advanced practice nurses. The workshop was designed to include at least one CNS in each group. The Japanese version of the Evidence-based Practice Questionnaire, with four subscales—practice, attitude, and knowledge and skills of research and practice—was evaluated four times: baseline, immediately after the workshop, and 3 and 6 months after the workshop. At the end of the workshop, the participants wrote their learning objectives and plans for improving their critical appraisal of EBP. A self-reflection questionnaire was distributed 3 months after the workshop, and a qualitative descriptive analysis was used.ResultsEleven CNSs participated in this study. Only the score of the knowledge and skills of research showed a statistically significant change. Many participants did not perform critical appraisal of research articles after the workshop. The facilitators were to have a role in integrating research evidence into practice and the barriers to being too busy, personnel transfers, and insufficient conditions.ConclusionsSelf-efficacy for critical appraisal in EBP might increase through the EBP education workshop. Even nurses interested in EBP require ongoing learning opportunities and organizational support for EBP activities.Trial RegistrationThis study was retrospectively registered with University hospital Medical Information Network Clinical Trial Registry (UMINCTR) on 09/14/2018 (Registration Number. UMIN000034146).Supplementary InformationThe online version contains supplementary material available at 10.1186/s12909-024-06315-z. Introduction Evidence-based practice (EBP) is the integration of the best available evidence with clinical expertise and patient values [1]. Clinical decision-making should be supported by current clinical information that reflects the best available research evidence [2]. Among the five steps of EBP (Ask, Acquire, Appraise, Apply, and Assess) [1], “Appraise” involves the critical evaluation of the integrity, reliability, and applicability of health-related research [3] to identify and select the most relevant evidence. Nurses are expected to contribute to and enhance EBP [4], and they generally have positive beliefs and attitudes toward it [5]. However, they practice critical appraisal infrequently and often lack the knowledge or skills required, with “Appraise” being the least performed step [6–10]. Advanced practice nurses (APNs) with high EBP competency levels [11, 12] are expected to play a leading role in EBP [2, 13]. EBP education is a fundamental approach to the development of EBP competencies. Numerous educational intervention studies have already been conducted [14–18]. “Appraisal” is the most frequently addressed topic in EBP medical education [19]. However, many nurses have perceived infrequent action of critical appraisal and inadequate knowledge and skills [5–10]. In Japan, many nurses also perceive low frequency of participation and low ability for critical appraisal [6, 13]. Furthermore, EBP education studies from Japan are limited, suggesting challenges in implementing and disseminating current EBP education for Japanese nurses and nursing students. For instance, nursing baccalaureate and master’s programs for certified nurse specialists (CNSs) as APNs are not regulated, although some official statements refer to the need for EBP education [20]. Although there are nursing research courses, there is often a lack of distinction between creating new evidence and integrating evidence into clinical practice (i.e., EBP). Additionally, there is a limited learning environment and practice of critical appraisal in clinical settings after graduating from a university or graduate school [13]. Even CNSs have few opportunities to participate in journal clubs and human resources are insufficient to teach critical appraisal in the context of EBP in their organizations [13]. Considering Japan’s unique cultural and societal context, directly implementing educational interventions that have been successful elsewhere can be challenging. EBP educational interventions, often based on problem-based learning, are typically conducted at single institutions or specific multisite settings, such as residency or organizational programs [13, 21, 22]. In these contexts, participants are often colleagues who collaborate on EBP within the same facility. However, in countries like Japan, implementing such programs can be difficult due to factors such as shortage of EBP mentors and educators, limited organizational readiness for EBP, and challenges in recruiting participants from a single institution. In such cases, participants may come from various institutions, potentially creating a disconnect between the learning environment and the actual clinical setting. Therefore, developing innovative educational approaches is essential in resource-constrained settings. A new educational program must be designed with continuity between the place of learning in educational interventions and the place of daily practice, considering inadequate EBP resources, as well as evidence-based educational strategies, adult learning theory, and self-directed learning for participants [23–30]. Additionally, to evaluate its impact, tracking learners’ subsequent behaviors and outcomes is essential. Although the duration of educational interventions has been reported to vary among studies [15, 31], the long-term impact of the program may depend on the environment in which the participant is placed. Therefore, this study aimed to evaluate the impact of a new EBP workshop that focused on critical appraisal in a setting with limited EBP resources for nurses’ practice, attitude, and knowledge and skills of research and practice at baseline, immediately, 3 months, and 6 months after the workshop and the learning process after the workshop. Methods Study design This educational interventional study with a before-after single-arm trial was conducted from October 2017 to March 2018 in Japan. The education program was developed based on the Critical Appraisal Skills Program (CASP) [32]. The CASP has been adopted for EBP education of clinical nurses [33, 34]. This study was designed based on the learning package of evidence-based medicine developed by CASP Japan. The details of our educational program are shown in Supplemental file 1, which refers to the guideline for reporting EBP educational interventions and teaching [23]. Participants and settings The target population was CNSs with a master’s degree who were certified by the Japan Nursing Association as advanced practice nurses and working as a nurse in patient care. The participants were recruited through convenience sampling and a mailing list distributed to CNSs. Participants applied to join the study as groups, forming their own groups that met the following criteria: (a) a group consisted of no more than four members, and (b) at least one member of the group was a CNS. If these criteria were met, other nurses without a CNS license or those working as faculty with a CNS license, were allowed to participate. This setting was used because this study aimed to provide a setting as similar as possible to an actual and daily clinical setting in Japan. Educational intervention Learning objectives The overall objectives of this educational program were that learners would be able to formulate clinical questions using the PICO (patients/population, intervention, comparison, and outcome) format, appraise research articles while critically acquiring knowledge and skills, identify their challenges in formulating clinical questions and appraising research articles critically, and recognize their opinions on EBP and its process. Theoretical frameworks The educational program adopted the following theoretical frameworks: the experiential learning model for designing this program was based on an experimental learning cycle, instructional design of this program to enhance learning, Knowles’ adult learning theory for clinical nurses as adult learners, and problem-based learning for creating a setting that is closely associated with the EBP process in clinical practice. The three modules of the workshop and the follow-up period after the workshop were structured such that the participants experienced the learning cycle of “concrete experience,” “reflective observation,” “abstract conceptualization,” and “active experimentation.” Planned program The educational program was designed based on the EBM learning package developed by CASP Japan. It was based on a clinical scenario based on the participants’ clinical questions in each group and comprised three modules: Module 1 focused on critical appraisal in EBP, Module 2 was a self-learning period, and Module 3 included reformulating the clinical question and searching the literature. This learning strategy, which combines lectures and small group discussions, has been used in several EBP studies [33, 34]. This workshop was held per group, which comprised 2–4 participants, including at least one CNS and one facilitator. Educational materials The materials included the handouts provided for lectures, worksheets, a scenario, and research articles. The EBM learning package was modified to suit this study, and CASP Japan approved the revisions. Scenarios were prepared for each group tailored to the interests of the participants, and the facilitator participating in the workshop selected a research article that matched the scenario. Data collection Data were collected from October 2017 to August 2018, four times per participant: pre (baseline and time 1) and post (time 2 and time 3) during the educational program and 3 and 6 months after the educational program as follow-ups. The participants’ demographics were collected at baseline, and a self-reported questionnaire regarding evidence-based practice was distributed at all four times. At time 3, a reflection sheet that describes the participants’ learning process was distributed by mail(Supplemental file 2). The timeline of the educational program and data collection are shown in Table 1. Table 1Timeline of educational program and data collectionEducational programData collectionModuleHow to provide educational programsTimeCollecting dataBaseline data before Module1(Time 1)QuestionnaireModule 1Workshop (face-to-face, 5 h in one day)Module 2Self-learning (1 month after Module1)Module 3Workshop(face-to-face, 3 h in one day)After Module 3(Time 2)Questionnaire3 month after Module 3 (Time 3)Questionnaire and focus-group interviews6 month after Module 3 (Time 4)Questionnaire Table 1Timeline of educational program and data collectionEducational programData collectionModuleHow to provide educational programsTimeCollecting dataBaseline data before Module1(Time 1)QuestionnaireModule 1Workshop (face-to-face, 5 h in one day)Module 2Self-learning (1 month after Module1)Module 3Workshop(face-to-face, 3 h in one day)After Module 3(Time 2)Questionnaire3 month after Module 3 (Time 3)Questionnaire and focus-group interviews6 month after Module 3 (Time 4)Questionnaire Timeline of educational program and data collection Baseline data before Module1 (Time 1) Workshop (face-to-face, 5 h in one day) Self-learning (1 month after Module1) Workshop (face-to-face, 3 h in one day) After Module 3 (Time 2) Measurements The primary outcome was the total score of the evidence-based practice questionnaire of the Japanese version (EBPQ-J) [35]. This is the translated version in Japanese from the original EBPQ in English [36]. The EBPQ-J comprises four subscales: practice, attitude, knowledge and skills of research, and practice. This self-report questionnaire comprises 18 items ranked on a 7-point Likert scale (1 = never and 7 = frequent), and the range of the total score is 7–126. The higher the score, the better the participant’s outcomes for EBP. The reliability and validity of the EBPQ-J have already been tested [35]. The secondary outcomes were the four subscales of the EBPQ-J. In addition, quantitative changes were evaluated using reflection sheets. Data analysis The participants’ demographics were descriptively analyzed. The EBPQ-J scores from the four subscales were analyzed to determine the difference between the four subscales using repeated two-way analysis of variance (significance level of α = 0.05). The learning process was qualitatively analyzed using text data from the self-report questionnaires, which focused on whether the participants had achieved their behavior goals and reasons why they had or had not achieved them. The target population was the CNSs eligible in this study; therefore, the data for other nurses are described in Supplemental files 3 and 4. Statistical analysis was performed using SAS 9.4 (SAS Institute Inc., Cary, NC, USA.) and Microsoft Excel 2010. Sample size The sample size was 20 CNSs, calculated to provide the mean difference (standard deviation) of the total scores of the EBPQ-J as 20 (20) before and after the educational program, with α error 0.05 and power 0.80. Ethical considerations This study was approved by the Institutional Review Board. After the approval, the researchers informed all the participants about the study in writing and orally, and informed consent was obtained from all participants in this study. As an incentive for participating in this study, each participant received a gift card of 5,000 yen. The study was retrospectively registered with University hospital Medical Information Network Clinical Trial Registry (UMIN-CTR) on 09/14/2018 (Registration Number. UMIN000034146). Results Participants Nine groups applied to participate in this study, and eleven CNSs were enrolled. Along with these participants, fifteen nurses without a CNS license or with a CNS license working as faculty members participated in this study with the eleven CNSs. Finally, seven CNSs and eleven other nurses completed the program and all questionnaires (Fig. 1). Each group contained two to four nurses and included 1–2 CNSs. The most common unit of application was “community colleagues” (5 groups), followed by study groups with hospital staff (including different hospital affiliations in the same family) (2 groups) and study groups with staff in hospital wards (2 groups). The characteristics of the CNS participants are detailed in Table 2. The mean age of the CNS participants was 38.4 years, and the mean clinical experience was 15.5 years. They had held their CNS certification for an average of 3.5 years. Most CNSs were employed in hospitals (81.8%) and worked as staff nurses (72.7%). Additionally, 72.7% of the CNSs received education on EBP and had experience with EBP in their practice, although few participated in journal clubs in the past year. Other nurses had similar characteristics in terms of age and years of clinical experience; however, they had less experience learning and implementing EBP compared to the CNSs (Supplemental File 3). Fig. 1Participants’ flow Participants’ flow Table 2Participants’ characteristicsCNS eligible in this study(n = 11)AgeMean (standard deviation)38.4(3.3)Years of clinical experienceMean (standard deviation)15.5 (3.5)Academic backgroundMaster’s degree11 (100%)Certification of certified nurse specialists*1)Yes11 (100%)Year after being certified as a certified nurse specialistMean (SD)3.5 (2.9)OrganizationHospitals9 (81.8%)Visiting nursing station0 ( 0.0%)University0 ( 0.0%)Research Institute0 ( 0.0%)The others2 (18.2%)Job position*2Staff nurses8 (72.7%)Chief nurses3 (27.2%)Specialized nurses2 (18.2%)Others0 (0.0%)Learning experience of EBPYes8 (72.7%)Experience doing EBPYes8 (72.7%)Participation in journal club per yearNone or missing10 (91.1%)One time0 (0.0%)30 times1 (8.9%)More than 60 times0 (0.0%)EBP: evidence-based practice, CNS: certified nurse specialist, SD: standard deviation*1) The CNSs’ specialties were cancer nursing, pediatric nursing, maternal nursing, chronic disease nursing, acute and critical care nursing, and infectious disease nursing*2) Job positions were selected via multiple choice Table 2Participants’ characteristicsCNS eligible in this study(n = 11)AgeMean (standard deviation)38.4(3.3)Years of clinical experienceMean (standard deviation)15.5 (3.5)Academic backgroundMaster’s degree11 (100%)Certification of certified nurse specialists*1)Yes11 (100%)Year after being certified as a certified nurse specialistMean (SD)3.5 (2.9)OrganizationHospitals9 (81.8%)Visiting nursing station0 ( 0.0%)University0 ( 0.0%)Research Institute0 ( 0.0%)The others2 (18.2%)Job position*2Staff nurses8 (72.7%)Chief nurses3 (27.2%)Specialized nurses2 (18.2%)Others0 (0.0%)Learning experience of EBPYes8 (72.7%)Experience doing EBPYes8 (72.7%)Participation in journal club per yearNone or missing10 (91.1%)One time0 (0.0%)30 times1 (8.9%)More than 60 times0 (0.0%)EBP: evidence-based practice, CNS: certified nurse specialist, SD: standard deviation*1) The CNSs’ specialties were cancer nursing, pediatric nursing, maternal nursing, chronic disease nursing, acute and critical care nursing, and infectious disease nursing*2) Job positions were selected via multiple choice Participants’ characteristics EBP: evidence-based practice, CNS: certified nurse specialist, SD: standard deviation *1) The CNSs’ specialties were cancer nursing, pediatric nursing, maternal nursing, chronic disease nursing, acute and critical care nursing, and infectious disease nursing *2) Job positions were selected via multiple choice Changes in EBPQ-J scores before and after the education program At baseline of the CNSs shown in Table 3, the EBPQ-J’s total score and the subscales’ score, including the practice and knowledge and skills of research and practice, were moderate, with means and standard deviations of 70.4 (19.3), 20.0 (9.3), 21.7 (8.3), and 9.3 (2.9), respectively. The attitude score was particularly high, with a mean of 19.4 (1.4). Changes in scores over the four time points through 6 months (Table 3) showed a slight upward trend, but no statistically significant changes were observed in the total score or the three subscale scores—practice, attitude, and knowledge and skills of practice. Only the score for knowledge and skills of research increased by an average of approximately 10 points, which was statistically significant. The other nurses’ scores trended to be lower than those of the CNSs, with some significant changes noted in the total scores and the subscales of practice and knowledge/skills of research (Supplemental File 4). Table 3Changes in the evidence-based practice questionnaire of the Japanese version (EBPQ-J) scores for six monthsEBPQ-JCNSs eligible for this study (n = 7)p valueTime 1Time 2Time 3Time 4Mean (SD)Mean (SD)Mean (SD)Mean (SD)Total scores(range: 18–126)70.4 (19.3)81.9 (12.0)78.0 (16.5)85.6 (16.1)0.21Practice(range: 6–42)20.0 ( 9.3)23.0 ( 8.3)21.3 ( 8.2)24.3 ( 7.7)0.61Attitude(range: 3–21)19.4 ( 1.4)17.3 ( 6.0)18.9 ( 2.0)17.0 ( 5.4)0.64Knowledge/skills of research(range: 7–49)21.7 ( 8.3)30.4 ( 4.6)27.3 ( 6.7)31.1 ( 6.0)< 0.05Knowledge/skills of practice(range: 2–14)9.3 ( 2.9)11.1 ( 1.5)10.6 ( 2.3)11.1 ( 1.7)0.23SD: standard deviation*1) Time 1: baseline before the workshop, time 2: immediately after the workshop, and times 3 and 4: three and six months after the workshop Table 3Changes in the evidence-based practice questionnaire of the Japanese version (EBPQ-J) scores for six monthsEBPQ-JCNSs eligible for this study (n = 7)p valueTime 1Time 2Time 3Time 4Mean (SD)Mean (SD)Mean (SD)Mean (SD)Total scores(range: 18–126)70.4 (19.3)81.9 (12.0)78.0 (16.5)85.6 (16.1)0.21Practice(range: 6–42)20.0 ( 9.3)23.0 ( 8.3)21.3 ( 8.2)24.3 ( 7.7)0.61Attitude(range: 3–21)19.4 ( 1.4)17.3 ( 6.0)18.9 ( 2.0)17.0 ( 5.4)0.64Knowledge/skills of research(range: 7–49)21.7 ( 8.3)30.4 ( 4.6)27.3 ( 6.7)31.1 ( 6.0)< 0.05Knowledge/skills of practice(range: 2–14)9.3 ( 2.9)11.1 ( 1.5)10.6 ( 2.3)11.1 ( 1.7)0.23SD: standard deviation*1) Time 1: baseline before the workshop, time 2: immediately after the workshop, and times 3 and 4: three and six months after the workshop Changes in the evidence-based practice questionnaire of the Japanese version (EBPQ-J) scores for six months Total scores (range: 18–126) Practice (range: 6–42) Attitude (range: 3–21) Knowledge/skills of research (range: 7–49) Knowledge/skills of practice (range: 2–14) SD: standard deviation *1) Time 1: baseline before the workshop, time 2: immediately after the workshop, and times 3 and 4: three and six months after the workshop Learning processes Almost all the participants were satisfied with the workshop. However, many participants did not follow the plan of performing critical appraisal of research articles after the workshop. The CNSs who practiced critical appraisal told the facilitators that they found instances in their clinical practice that provoked them to read research articles. The CNSs who did not practice critical appraisal had barriers such as a job transfer, a job change, or changes in their physical condition or workload. Other reasons for not doing critical appraisal were a mismatch of the interests of EBP between the CNSs and the organization and their unspecific action plan, which they devised at the end of Module 3. Discussion The educational program in this study showed an increasing trend in the knowledge and skills subscale regarding research during the 6 months, but the total score and the other subscales did not change significantly. Furthermore, even for the CNSs with a strong interest in EBP and critical appraisal, continuous learning and engaging in critical appraisal of research articles in clinical settings was difficult. Previous studies reported that health care professionals with a master’s degree required ongoing support in learning EBP [37] and that learners in a monthly 6-month educational program using CASP required subsequent learning of EBP [34]. Our findings indicate that this program needs an additional follow-up program for enhancing CNSs’ action of critical appraisal in clinical settings or the CNSs require support to increase opportunities to read research articles for EBP after completing the program. This program was evaluated in terms of three domains: knowledge and skills, attitude, and practices for EBP. First, this study indicated that providing learning opportunities focused on critical appraisal in EBP for CNSs may likely contribute to positive self-efficacy for knowledge and skills; this result is similar to previous EBP educational studies [15–17, 34]. Second, the insignificant changes in practice scores on the EBPQ-J suggest that this intervention does not encourage a behavior change in CNSs. It implies that only action plan development and evaluation after completing the program are insufficient. Third, a small change in the attitude score of the EBPQ-J after the educational intervention is probably due to the higher score at baseline. In general, many studies have high scores for attitudes toward EBP [33, 38]. This study included CNSs who are interested in EBP, and their attitude toward EBP was less affected by the educational program. To not only acquire knowledge and skills in critical appraisal, but also to increase its frequency, there is a need to further design an approach to provide a clinically integrated EBP education program [39] as a part of continuous education and the lifelong learning process. The findings that this educational program did not notably improve the total score of the EBPQ-J over the six months are interpreted that the program did not enhance self-directed learning. To increase the total score of the EBPQ-J, participants need to continue reinforcing their EBP efforts in the workplace. Even including action plan development for conducting critical appraisal in EBP for APNs interested in EBP, the program did not enable them to overcome the barriers to engaging in critical appraisal. For example, the absence of the ability to proactively manage their own time for EBP and the authorities to change usual care into alternative care is often considered a barrier to EBP [40, 41]. The lack of such organizational readiness may have hindered the implementation of EBP by the participant APNs [42]. Strengths and limitations In this study, university faculty members facilitated an educational program as an external educational resource for learning critical appraisal for EBP. This study setting could serve as a model for collaboration between academia and clinical practice. Another study reported that clinical nurses’ journal clubs were held with researchers or faculty members who supported selecting articles and managing the journal clubs [43]. Such activities might provide an opportunity to encourage the interpretation of research evidence in clinical practice and its incorporation into practice. This study has some limitations. First, this study did not have a control group; thus, whether the change was affected by only the educational intervention remains unknown. For example, if CNSs conduct or participate in nursing research and projects, the scores related to EBP might change even if the educational program has no effect. Second, there was a strong selection bias for participants in terms of generalizability; thus, the findings would not be applicable to CNSs uninterested in EBP and critical appraisal. Third, the number of the participants for data analysis was smaller than the designed sample size; thus, the power was low. Implications for education Learning opportunities for the critical appraisal of EBP should be disseminated to countries where EBP is deficient. In-house educational human resources for EBP and critical appraisal are limited in such countries. The educational program developed in this study has the potential to enhance the participants’ self-efficacy in research skills and knowledge of EBP, despite some limitations. In Japan, there is a shortage of EBP instructors who can educate critical appraisal in the context of EBP. CNSs are expected to become instructors; however, EBP is not a mandatory role for CNSs [44]. Education for the critical appraisal of research questions is partially provided in graduate school, but there are few opportunities to apply it to solve clinical questions [13]. In nursing education, especially for APNs including CNSs, an education of critical appraisal for EBP should be provided apart from that in traditional research design education. Conclusions This study developed and evaluated an educational program based on CASP, developed by CASP Japan. The EBP workshop, which focused on critical appraisal of EBP, improved the score of knowledge and skills of research on the EBPQ-J six months after the workshop, but the total score and those of the other subscales did not change significantly. Additionally, even if CNSs had a strong interest in EBP and critical appraisal, ongoing learning of critical appraisal of research articles was difficult for them in practice. Continuous learning and practicing EBP and critical appraisal require organizational support and readiness for EBP. Electronic supplementary material Below is the link to the electronic supplementary material. Supplementary Material 1 Supplementary Material 1