A satisfaction-focused CIPP evaluation of Mongolia’s undergraduate occupational therapy program: a cross-sectional study Authors: Bulganchimeg Sanjmyatav, Erdenetsetseg Myagmar, Karen P. Y. Liu, Janet Lok Chun Lee, Munkh-Erdene Bayartai, Batgerel Oidov, Solongo Bandi, Oyungoo Badamdorj Published: 2025 ABSTRACT PurposeThis study evaluated Mongolia’s undergraduate occupational therapy program using the CIPP (Context, Input, Process, Product) model, drawing on feedback from students, alumni, faculty, and program managers to understand its current state and identify areas for improvement.MethodsWe conducted a cross-sectional survey using the CIPP evaluation model. The model-based questionnaire requested responses using a Likert scale from 1 to 5. Responses were collected from 34 students, 19 alumni, 24 faculty members, and five program management team members. They were analyzed quantitatively using descriptive and inferential statistics.ResultsAll participants expressed satisfaction with the context, process, and product components of the occupational therapy program, with scores of 3.4 or higher. For the input component, the faculty members and program managers gave satisfactory ratings, while the students and alumni gave neutral ratings, on average. There was a significant difference between the average input scores of the first-year and third-year students, and between the first-year and fourth-year students (p<0.05). Lecturers teaching basic professional subjects rated the program’s inputs higher than those teaching general subjects. On average, the students rated the program’s “context” higher than alumni, while faculty members rated “input” higher than both students and alumni. Additionally, the program’s managers rated “input” higher than the students (p<0.05).ConclusionThe occupational therapy program in Mongolia has been improving annually. While it is generally rated as “satisfactory,” there is a need for enhanced resources to better support the program. Introduction Comprehensive evaluation of a healthcare education program provides its stakeholders with useful information about its past achievements, its current state, and areas needing adjustment to accommodate future developments [1]. Students, alumni, faculty, and program managers can all participate in evaluations. Evaluation is fundamental to all improvements, whether in the realm of educational quality or the efficient operation of a school. Educational assessment significantly influences the quality of student learning, teacher instructional practices, and the evaluation of curricula, school quality, and system performance. It functions as a mechanism for both improving teaching and learning in a formative manner and evaluating teachers, schools, and administration in a summative context. Due to its substantial impact, exploring its historical development, current status, and future potential is beneficial [2]. In fact, though, a thorough quality evaluation of a school should encompass all facets of the institution and its impact on students, positioning it as the initial step toward achieving quality improvement and development [3]. A bachelor’s degree program in occupational therapy was introduced in Mongolia in 2014 at the Mongolian National University of Medical Sciences (MNUMS), still today the only university in the nation offering such training. In 2018, the first six graduates began working in Mongolian hospitals and a center for children with disability. By 2024, 52 occupational therapists have graduated from the program. Although the program has undergone periodic reviews and minor adjustments, a full evaluation has not yet been conducted. Due to Mongolia’s historical context as a nation previously under Soviet influence, the development of occupational therapy services, policies, and workforce has been constrained, akin to the situation in numerous other post-Soviet states [4-6]. Although the occupational therapy program at MNUMS has made advancements since its establishment in 2014, a comprehensive evaluation is now imperative to critically assess its effectiveness, guide strategic improvements, and ensure alignment with international standards. Various designs and models exist for evaluating education programs, with the Context, Input, Process and Product (CIPP) model being widely used in education for the health professions [7]. That model was proposed by Madaus et al. [8] in 1983. It supports both summative and formative evaluations, offering a comprehensive view by assessing context, input, process as well as output. It is a management-focused framework that evaluates programs during and after their implementation. The context component examines needs, goals, and the conditions for achieving outputs. The input component assesses resources and techniques impacting a program’s educational objectives. The process component evaluates a program’s execution against goals and timelines. The product component measures the program’s effectiveness and efficiency. Such an evaluation provides decision-makers with essential information, including that needed to determine whether to continue or discontinue a program [9]. In contrast to outcome-oriented models such as Kirkpatrick’s four levels or logic models, the CIPP model provides a more expansive and dynamic framework that emphasizes the importance of continuous feedback, comprehensive context analysis, and adaptability. These features are particularly valuable for programs that are either in the developmental phase or have been recently implemented [10]. The model’s flexibility and comprehensive scope have contributed to its increasing application in the evaluation of complex educational environments, particularly within the domain of health professions education [11]. This study assessed the occupational therapy program at the MNUMS, gathering feedback from the students, alumni, faculty, and the program’s management. The comprehensive evaluation focused on determining whether the program is meeting its education objectives and identifying areas for potential enhancement. This study provides insights into health professions education by documenting the first comprehensive evaluation of an occupational therapy program in a post-Soviet, low-resource setting. Insights from Mongolia’s experience offer valuable implications for similar countries in Eastern Europe and Central Asia where occupational therapy is newly emerging and formal evaluation practices remain limited [12-14]. Methods 1. Participants This cross-sectional study was conducted from September 2020 to October 2021. It evaluated the Occupational Therapy bachelor’s program implemented from 2014 to 2020, collecting information from students, faculty, alumni, and staff associated with the program during that period. A total of 82 individuals were recruited using convenience sampling: 34 students, 19 alumni, 24 teaching faculty, and five management team members. That group included students from all of the courses, graduates with over 6 months of work experience, and teachers and program managers who had been involved in the program for at least 2 years. 2. Data collection and analysis To gather quantitative data on stakeholder satisfaction, the research team designed a Mongolian-language questionnaire informed by the four areas of the CIPP model: context, input, process, and product [15]. The questionnaire included 27 to 50 items, as shown in Table 1, using a Likert-type scale from 1 (strongly disagree) to 5 (strongly agree) (See Supplement 1 for the questionnaires.). To test the construct validity of the survey questionnaire, the importance of each question was rated by 20 respondents on a Likert scale with 1 for not at all important to 5 for very important. All items with a mean score of greater than 1.5 were retained as having acceptable validity [16]. The questionnaire’s internal consistency was quantified using Cronbach’s alpha. An alpha between 0.8 and 0.9 was taken as indicating acceptable internal consistency [17]. The means and standard deviations of the Likert scale scoring on each component of the CIPP model were analyzed using the IBM SPSS software suite ver. 25.0 (IBM Corp., Armonk, USA). The responses of each group of participants were compared as independent samples for each subgroup of the questionnaire. The subgroup scores were compared using analysis of variance. The results were checked for normality, and the significance of any differences was assessed using t-tests for normally-distributed data and Kruskal Wallis tests and Mann-Whitney U tests for data that was not normally distributed. 3. Ethics statement The research protocols were approved by the Research Ethics Review Committee of the MNUMS (approval no., 2020/3-05 dated June 12, 2020). The work was conducted in compliance with the Helsinki Declaration of 1975 as revised in 2008. The study guaranteed voluntary participation, anonymity, and confidentiality of the participants’ data. The participants were informed in advance about the study’s purpose, and of their right to withdraw at any time. Each provided written consent for their participation. Results This satisfaction-focused evaluation, guided by the CIPP framework, included 82 participants across four stakeholder groups: 34 students (97.3% response rate), 19 alumni (95%), 24 teaching faculty members (77.4%), and five program managers (83.4%). The participants ranged in age from 19 to 56 years, and 80.5% were female (Table 2). Cronbach’s alpha for the context, input, process, and product of the CIPP was 0.87, 0.94, 0.89, and 0.93, respectively. All of the respondents evaluated themselves as “satisfied” with the occupational therapy program’s context, process, and product components. The faculty and management team respondents all responded “agree” (average ratings: 3.41 and 4.20, respectively) or“ strongly agree” (4.21 and 5.00) to the statements about the program’s inputs. The students’ average score given to this component was“ neutral” (2.91±0.8), as was that of the alumni (3.03±0.78) (Table 3). Faculty members gave the input items higher scores than the students and alumni, on average, and members of the program management team provided higher average scores than the students. The input items with the lowest scores were the adequacy of library services and textbooks (2.32±0.35), the adequacy of professional books in Mongolian (2.2±0.48), student support services like internet speed and printing (2.02±0.78), the availability and pricing of cafeteria services (1.97±0.29), and the adequacy of the skills lab facilities (2.58±0.87). There were statistically significant differences between the CIPP input scores of the first year and third year students (p≤0.01), as well as between the first year and fourth year students (p≤0.05). Additionally, lecturers teaching basic professional subjects provided significantly higher input scores than those teaching general subjects (p≤0.01). Scores given by students for the “context” items were significantly higher than those of the alumni, on average (Table 4). When the occupational therapy graduates were asked about the challenges they faced when first beginning to work, 84.2% indicated that they lacked professional hands-on skills. Surprisingly, only 68.4% agreed that improving the learning environment and study resources would benefit the occupational therapy program. But only 26.3% expressed dissatisfaction with the program’s organizational aspects. Discussion The occupational therapy program is integral to the School of Nursing at the MNUMS, and the study data affirmed that the program’s context component is improving annually. The input component received the lowest ratings, possibly due to limited preparation, weak policy support, and unclear understanding of occupational therapy. The process component responses emphasized the need for an outcome-based curriculum to improve the program’s quality. The context evaluation aims to define, identify and address the needs of the target population and the main problems. It seeks to determine whether the program’s goals effectively respond to the identified needs [18]. All of the survey’s respondents evaluated context of the occupational therapy program as “satisfied” on the Likert scale. Occupational therapy is a new profession with trained therapists in high demand. Indeed, all of rehabilitation medicine is a growing discipline in Mongolia, but there is a lack of definitive official data regarding its specialist workforce [19,20]. In this survey the alumni group gave significantly lower average context ratings than the students, reflecting their different perspectives. The initial graduates of the occupational therapy program may have lacked sufficient understanding of occupational therapy. Consequently, they encountered numerous uncertainties, including questions about the type of service they should be providing to people in need after their graduation. Another factor affecting students’ understanding about occupational therapy could have been the effectiveness of the school’s management team at that time. The teachers too may have had insufficient knowledge of occupational therapy, insufficient preparation, and only limited teaching capacity. Over time, as the program has developed, the need for and value of occupational therapy has come to be better understood among the faculty and the program’s management, as well as among the alumni and eventually the students. The MNUMS initially based its occupational therapy curriculum on the curriculum of Gunma University in Japan recognized by the World Federation of Occupational Therapists (WFOT). The findings here suggest that the occupational therapy program needs to be improved in accordance with Mongolia’s unique geography, disease prevalence, and cultural context [21]. These findings also revealed notable differences in perceptions between alumni and current students, particularly within the “context” and “input” components of the program. Alumni, particularly those from the earlier cohorts, tended to give lower ratings, likely due to their limited exposure to occupational therapy during training, unclear career prospects at the time, and inadequate educational infrastructure. Conversely, recent students reported higher satisfaction, benefitting from expanded institutional support, enhanced learning resources, and clearer professional direction. Such contrasts underscore the evolving nature of the program and the significant role that program maturity, stakeholder awareness, and contextual improvements play in shaping evaluation outcomes. Stakeholder views differ based on how the program has evolved over time. Alumni from earlier years faced a program with fewer resources and uncertain goals, which may explain their critical feedback. In contrast, current students have access to better support and growing recognition of the profession, leading to a more positive perspective. These findings show that structural barriers in Mongolia’s medical education system, including limited clinical training opportunities and too few qualified occupational therapy instructors, contributed to the lack of practical skills among graduates. These challenges were more severe in the program’s early years when occupational therapy was new and institutional support was still growing. Input evaluation aims to identify the available resources and how they could best be used to meet the program’s objectives [22]. In this survey the input component of the CIPP model received the lowest scores, with the students and alumni rating it lower than the managers and faculty. Rechel and McKee [14] has suggested that dissatisfaction with the input component is often a result of an inadequate budget and weak financial resources in general. In this study the third- and fourth-year students gave lower average input scores the first-year students. The issues most commonly raised by occupational therapy students and alumni were insufficient library services, a lack of professional textbooks in Mongolian, limited internet access, unsatisfactory cafeteria services, and inadequate student services in general. A previous investigation examining a doctoral nursing program in Japan also demonstrated the importance of having an adequate number professors as well as equipment and facilities such as a good library and computer system [21]. That is in line with the findings here. Mongolia’s medical education has evolved from a lower-middle-income context [23], with its higher education system primarily funded by private sources like tuition fees, which now account for 60 of the system’s financial resources. A report on the Mongolian education system suggested that 80% of academic staff salaries are covered by student fees, indicating limited resources for essential facilities such as libraries and equipment [24]. And the learning environment and resource materials for the program have indeed improved, probably due to increased management understanding of and support for occupational therapy. This progress aligns with Mongolia’s goal of increasing spending on maintenance and equipment to 20% of total capital expenditures for higher education. They have in fact risen from 16.32% in 2021 to 19.16% in 2023 [24]. The survey’s process component evaluated how the program is being implemented and its impact on the learners. It focused on teaching activities and instructors’ behavior, knowledge, and experience. With previous evaluation results, the data can be used to improve program implementation [25]. The process feedback for the occupational therapy program was satisfactory, though the respondents did note some areas for improvement. The program’s structure includes general subjects in the first year, basic subjects in the second year, and professional core subjects in the third and fourth years. For improvement it is important to ensure that the material taught aligns with the profession’s characteristics and needs. That implies a need to update the curriculum regularly, which in turn implies a need to periodic evaluations. To diversify the teaching methods, incorporate more practical activities, and focus on the number and quality of both teachers and students would seem fundamental. In addition, teachers need time and support for developing Mongolian materials and undertaking local research to underpin the occupational therapy content. The product evaluation examined the program’s impact on its graduates, comparing results to goals and identifying the gap between expectations and actual outcomes [26]. The product evaluation indicated that graduates’ limited practical skills are potentially influenced by three interrelated factors: the qualifications of teaching staff, the availability of clinical training sites, and the relevance of the curriculum. In the program’s early stages, the lack of faculty specialized in occupational therapy and limited clinical placements hindered students’ opportunities to develop applied skills. Additionally, instructors without occupational therapy backgrounds often lacked the expertise to provide practice-focused teaching. Although the curriculum offered solid theoretical foundations, it required adaptation to better reflect Mongolia’s specific healthcare context and resource limitations. Enhancing faculty expertise, expanding access to clinical learning environments, and aligning the curriculum with competency-based educational goals are critical steps to better prepare graduates for professional practice. Lecturing can provide adequate professional and theoretical knowledge essential for occupational therapists, but there is also a need to teach more practical applications crucial for occupational therapy practice. The survey’s respondents found the occupational therapy program to be satisfactory in this regard. Yazdani and Moradi [27] are among those who have emphasized the importance of balancing theoretical and practical subjects. Implementing these actions could improve the quality of the occupational therapy program in Mongolia, potentially encouraging occupational therapy graduates to reach the minimum standards suggested by the WFOT. The 2016 version of the WFOT’s Minimum Standards for the Education of Occupational Therapists serve as a guide for planning new educational programs that aim to achieve and maintain WFOT Approved or Accredited status. They support the development of occupational therapy education within higher education institutions and aim to produce graduates proficient in keeping abreast of current theories, research findings, and evidence, as well as in evaluating their effectiveness and value [28]. 1. Limitations and strengths In interpreting these findings, it is important to bear in mind that all of the new graduates responding had a maximum of 3 years of work experience, and the clinical placements for students were limited in the program’s early years. Further qualitative study involving stakeholders, policy makers, employers, and clients is now needed to evaluate further the results from the quantitative analysis. Additionally, the use of convenience sampling in the present study limits the generalizability and representativeness of the findings. But a notable strength is the high response rate, demonstrating the commitment to occupational therapy of those surveyed and their concern for its future. Although this study utilized the CIPP framework to assess the occupational therapy program, the results are primarily based on participants’ satisfaction. While satisfaction provides useful insight, it alone does not encompass the full scope of the CIPP components. To gain a more complete understanding, future evaluations should incorporate broader indicators, including student academic achievement, graduate employment rates, and in-depth feedback from relevant stakeholders. That said, the responses reflect the program’s past achievements, usefully evaluate its effectiveness, and identify directions for potential future development. Additionally, the study’s findings can serve as a guideline for other countries introducing an occupational therapy program for the first time and experiencing a similar development trajectory. 2. Conclusion Mongolia’s occupational therapy education program continues to progress steadily, despite ongoing challenges related to limited resources. This study identifies critical areas for improvement, including enhanced resources, increased clinical training opportunities, curriculum updates, and strengthened faculty expertise, all of which are essential for successful occupational therapy program development in Mongolia. The results not only reinforce Mongolia’s advancements but also provide valuable insights for other nations developing occupational therapy education programs. Future research combining quantitative and qualitative approaches in future evaluations could provide a deeper and more holistic assessment of the program’s effectiveness.