Development of Evaluation Criteria for Occupational Health Management Systems: Delphi Study Authors: Hansoo Song, Jaehoo Lee, Bohyun Sim, Yu Min Lee, Hyoung-Ryoul Kim Published: 2025 ABSTRACT BackgroundThis study aimed to develop evaluation criteria for assessing a large-scale enterprise’s occupational health management system (OHMS).MethodsA literature review was conducted, and the OHMS evaluation criteria were selected. A two-round Delphi survey was conducted with 20 panelists in the fields of occupational and environmental medicine and industrial hygiene. The evaluation items were formulated based on the results of the first-round Delphi survey, written opinions submitted by experts, and focus group interviews with company health management managers. If the content validity ratio (CVR) of each evaluation item was less than 0.42, the item was rejected.ResultsThe first Delphi survey questionnaire consisted of 36 evaluation criteria in 5 areas according to the Plan-Do-Check-Act cycle based on the literature review. These areas included 5 plan items, 6 do items, 4 check items, 3 action items, and 12 occupational health service items. From the first Delphi survey, out of the 36 evaluation items, 10 exhibited excellent content validity with a score of 0.7 or higher, and 25 demonstrated good content validity with a score lower than 0.7, but higher than 0.474. The evaluation items underwent significant revisions by incorporating written opinions from experts and feedback obtained from focus group interviews with health managers. The second Delphi survey questionnaire presented 31 evaluation criteria across four domains. Three criteria did not meet the CVR standards, and 28 items in four domains were finally selected.ConclusionUsing this evaluation tool, the company’s health management director will be able to continuously monitor and improve the system by evaluating the system that produces performance rather than evaluating performance. INTRODUCTION Traditionally, occupational health problems have been regarded as issues requiring technical solutions. However, it has been recognized that leadership, human factors, and organizational culture must be considered comprehensively.1 Consequently, there is now a consensus on the need for a systems-based approach and cultural transformation in the workplace.23 A system is an organic entity in which components such as inputs, processes, and outputs interact to achieve a specific goal. From this perspective, adverse outcomes such as fatalities, injuries, and losses are viewed as the result of system operations. Continuous improvement is achieved by assessing outputs and providing feedback to inputs, thereby ensuring an ongoing cycle of enhancement.4 A systems-based approach in occupational safety and health has led to the development and implementation of various evaluation frameworks based on the Plan-Do-Check-Act (PDCA) model.5 Representative examples include the International Organization for Standardization (ISO) standards, such as ISO 45001:2018 (occupational health and safety management systems)6 and ISO 45003:2021 (the first international standard providing practical guidance on psychological health in the workplace).7 In South Korea, the Korea Occupational Safety and Health Agency (KOSHA)-MS system, developed by the KOSHA, is widely used.8 These evaluation frameworks offer the advantage of integrating occupational safety and health, enabling a comprehensive assessment of both aspects. However, compared to the safety domain, the health domain is presented in a more simplified manner and does not adequately reflect the distinct context and complex interrelations inherent to occupational health.9 While safety management focuses on primary prevention aimed at controlling the causes of accidents, health management extends beyond this to include secondary prevention—identifying and addressing high-risk individuals—and tertiary prevention—supporting a successful return to work following illness or injury. Moreover, as the workplace represents a critical setting for public health, non-occupational health management should also be incorporated into the overall framework. Specifically, the health effects of hazardous exposure often take a long time to manifest. Additionally, occupational health outcomes are influenced not only by workplace-related factors but also by individual health conditions, requiring integration with medical management. These unique aspects have not yet been incorporated into occupational safety and health management systems. Appropriate performance indicators are required to establish well-defined occupational health management frameworks. It is also necessary to develop comprehensive performance metrics that encompass various occupational health initiatives. At Pohang Iron and Steel Company (POSCO), an increasing number of workers, particularly subcontractor employees, have been diagnosed with hematologic cancers, lung cancer, and respiratory diseases. In response to this growing concern, POSCO requested an assessment and improvement of its occupational health management system (OHMS). During the evaluation of POSCO’s occupational health activities, the research team faced challenges in identifying an appropriate assessment tool. The PDCA model is well-suited for process-oriented, autonomous safety and health management. It aligns with the global shift from regulation-based industrial safety and health practices toward autonomous regulatory approaches. Accordingly, a new evaluation framework was developed based on the PDCA model, which is widely adopted in international certification systems. This approach facilitates consistency with existing legal and policy frameworks and enhances applicability to large-scale enterprises, particularly those employing multiple subcontractors. METHODS In this study, a literature review was conducted on occupational safety and health management systems based on the PDCA model, including International Labour Organization (ILO)-OSH 2001 of the ILO,10 the United States Occupational Safety and Health Administration,11 the European Agency for Safety and Health at Work,12 the Health and Safety Executive,13 the ISO,67 and the KOSHA8 and the ILO Occupational Health Conventions.14 A two-round Delphi method was used to develop evaluation criteria. Twenty experts were recruited with consideration for regional and gender representation through recommendations from the Korean Society of Occupational and Environmental Medicine and the Korean Industrial Health Association. Among them, 80% were occupational medicine specialists and 20% were industrial hygiene specialists, with 5 to 33 years of experience (Table 1). Table 1Characteristics of study subjectsCategoryRound 1Round 2Experts number1920Age44.8 ± 7.544.2 ± 7.4Career16.5 ± 8.514.9 ± 8.8AffiliationKorean Society of Occupational & Environmental Medicine1216Korean Industrial Hygiene Association74SexMen1618Women32Career of specialtyOccupational disease12.4 ± 7.511.8 ± 6.5Occupational hygiene23.6 ± 6.627.3 ± 5.3Values are presented as number or mean ± standard deviation. Characteristics of study subjects Values are presented as number or mean ± standard deviation. The first-round questionnaire, structured according to the PDCA model, included 36 evaluation criteria across five domains, including occupational health services. Items were rated on a 5-point Likert scale, and both quantitative (mean, standard deviation, content validity ratio [CVR]) and qualitative feedback were analyzed. Items were classified as basic or highly advanced (H), the latter focusing on future-oriented or complex aspects such as subcontractor involvement or cost burden. Focus group interviews were also conducted with POSCO’s occupational health managers and subcontractor supervisors, confirming the relevance of the proposed items. Based on the first-round results, a revised questionnaire with 31 items in four domains was developed for the second round. After analyzing CVR values (cut-off: 0.42 for N = 20), 28 final evaluation criteria were selected.15 The CVR was calculated using the following formula: where N represents the total number of panel members and NE represents the number of respondents who rated the item as either 4 (valid) or 5 (highly valid) on the Likert scale. According to Lawshe’s table, with 20 expert panel members, the minimum acceptable CVR value was set at 0.42.16 Ethics statement The study was approved by the Chosun University Hospital Institutional Review Board and the requirement for informed consent was waived (approval No. CHOSUN 2022-10-013). RESULTS In the first-round Delphi survey, out of the total 36 evaluation criteria, 10 items demonstrated strong content validity with a CVR ≥ 0.7, while 25 items were considered acceptable with a CVR between 0.474 and 0.7. One item had a CVR < 0.474, indicating insufficient validity. After reviewing the opinions of experts along with these quantitative evaluation results, the revised results were as follows. First, the conceptual definitions of the PDCA cycle were refined. Plan refers to policies and strategic planning, Do pertains to the organization and implementation of initiatives, Check involves monitoring and evaluation through performance indicators, and Action focuses on deriving improvements based on analysis and assessment. The items were reorganized accordingly and redundancies were addressed. Second, occupational health services were removed. This decision was based on the recognition that the necessary occupational health services may vary by workplace and that these items overlap with the Do domain. Instead of checking whether individual occupational health services were implemented, the system was changed to standardize occupational health programs tailored to the characteristics of the workplace and to evaluate whether they were properly implemented. Third, definitions were provided for unclear concepts to enhance the precision and consistency of the interpretation. Strategic goals and tasks were presented separately, and input and process, leading, and performance indicators were set and reviewed for evaluating occupational health projects. Fourth, in the first round, B and H items were distinguished. However, in the second round, the basis for this distinction was not clear, so the distinction was eliminated (Table 2). Table 2Results of the Delphi study for evaluation criteria of occupational safety and health management system (first round)ItemsMean ± SDPositive answerCVRDomain 1 – PlanB.1.1. There is a mission of “occupational health,” as recognized by its members.4.47 ± 0.84170.789B.1.2. Goals and responsibilities consistent with the mission are presented.4.32 ± 0.89160.684B.1.3. Occupational health organizations have autonomy and authority to achieve their mission and strategic goals.4.47 ± 0.70170.789B.1.4. Performance indicators of occupational health projects are defined according to mission and strategic goals.4.37 ± 0.90160.684B.1.5. The budget for occupational health projects is appropriately allocated.4.37 ± 0.96150.579H.1.1. The company is practicing social responsibility for the health management of its partners.4.53 ± 0.77180.895Domain 2 – DoB.2.1. Occupational health organizations evaluate workplace risks using scientific methods.4.21 ± 0.79150.579B.2.2. Occupational health organizations have the expertise and implementation capacity to conduct risk assessments in the workplace.4.11 ± 0.94140.474B.2.3. Strategies to reduce exposure to harmful risk factors are evidence-based.4.21 ± 1.13150.579B.2.4. The company has a communication system for risk recognition and prevention.4.42 ± 0.84170.789B.2.5. The data and records necessary for occupational health projects are appropriately managed.4.63 ± 0.60180.895B.2.6. The priority of implementing occupational health programs is determined according to the level of disease risk.4.32 ± 0.95170.789H.2.1. A cohort has been established and operated to track exposure to harmful factors and health effects.3.79 ± 1.44120.263H.2.2. New health management issues are identified and reflected in occupational health projects.4.11 ± 1.10150.579Domain 3 – CheckB.3.1. Legally mandated work environment measurements and health surveillance are being conducted appropriately.4.58 ± 0.61180.895B.3.2. Occupational health organizations are conducting reactive monitoring of work environment measurements.4.21 ± 1.03150.579B.3.3. The company is conducting additional assessments (in addition to legally mandated employee health surveillance) to determine the health status.4.26 ± 0.81150.579B.3.4. An investigation is conducted when a suspected case of poisoning or occupational disease occurs.4.16 ± 1.21140.474H.3.1. Every member (contract workers, night workers, part-time employees, employees of partner companies, etc.) has universal access to occupational health services.4.21 ± 0.92150.579Domain 4 – ActionB.4.1. The occupational health organization is monitoring changes in performance indicators before and after the implementation of the health project.4.42 ± 0.69170.789B.4.2. Every year, the results of the occupational health project are reported to members.4.26 ± 0.93170.789B.4.3. The evaluation results are used to improve occupational health activities.4.26 ± 1.05150.579Domain 5 – ServiceB.5.1. Hearing conservation programs are being implemented appropriately.4.32 ± 0.75160.684B.5.2. Respiratory protection (dust gas exposure reduction) programs are being implemented appropriately.4.32 ± 0.82150.579B.5.3. The musculoskeletal disease prevention and management program is being implemented appropriately.4.47 ± 0.77160.684B.5.4. Mental health management programs are being implemented appropriately.4.37 ± 0.90160.684B.5.5. We are implementing programs for smoking cessation, drinking moderation, and obesity management.4.00 ± 1.05140.474B.5.6. A shift worker health management program is being implemented.4.32 ± 0.82150.579B.5.7. We are implementing programs to improve work-life balance.4.21 ± 0.85160.684B.5.8. Medical services (routine and emergency medical care) are provided according to the needs of workers.4.21 ± 0.92150.579B.5.9. A return-to-work program led by an occupational health doctor is being implemented.4.42 ± 0.84170.789B.5.10. Follow-up care consultations are provided for health examinations.4.26 ± 0.87160.684B.5.11. Additional screening programs (e.g., comprehensive employee health screening) were appropriately designed based on the evidence.4.05 ± 1.13150.579B.5.12. Safety and health education are implementation-oriented and reflect on-site cases.4.26 ± 0.99140.474H.5.1. Occupational health organizations develop and operate health promotion programs that take into account workers’ socio-demographic characteristics, exposure to harmful factors, and disease prevalence.4.16 ± 0.76150.579H.5.2. The company is analyzing and improving organizational culture and work practices that have a negative effect on the safety and health culture of the workplace.4.05 ± 0.97150.589Minimum value by Lawshe’s table = 0.474 (19 experts).SD = standard deviation, CVR = content validity ratio. Results of the Delphi study for evaluation criteria of occupational safety and health management system (first round) Minimum value by Lawshe’s table = 0.474 (19 experts). SD = standard deviation, CVR = content validity ratio. The CVR of the 8 criteria in the Plan domain ranged from 0.60 to 0.80, with expert consensus. Among the 9 criteria in the Do domain, all except D6 (primary medical services) had a CVR of 0.60–0.90, with expert consensus. D6 was excluded because many experts believed that Korea has excellent access to primary medical care. Even in the absence of a primary medical institution in the workplace, community medical institutions can provide adequate care. Among the 9 criteria of the Check domain, all except C8 (additional health check-up) and C9 (a disease cohort) had a CVR of 0.60–1.00, with an expert consensus. C8 was excluded because many experts viewed the company-sponsored comprehensive health examination in Korean companies as a form of employee welfare and support, rather than an occupational health measure. C9 was excluded because experts considered it more appropriate for the government to take the lead, with companies participating in cohort studies for diseases requiring long-term follow-up, rather than each company establishing its own cohort. Additionally, not all companies require such a cohort. For the 5 criteria in the Act domain, experts reached a consensus on a CVR range of 0.60–0.80. This confirmed the OHMS evaluation criteria, which consisted of 28 items across 4 domains (Table 3). Table 3Results of the Delphi study for evaluation criteria of occupational safety and health management system (second round)Evaluation criteriaMean ± SDPositive answerCVRFinal adoption1. Plan domainP1. The mission of occupational health is conveyed to members through written regulations and CEO leadership.4.55 ± 0.67180.80oP2. Strategic goals and tasks for occupational health were established according to the results of risk assessment.4.25 ± 0.94170.70oP3. Occupational health organizations have an appropriate operating system to promote projects.4.45 ± 0.80180.80oP4. Roles and responsibilities are appropriately allocated according to strategic goals and tasks.4.40 ± 0.73170.70oP5. Performance indicators of the occupational health project are set to evaluate the implementation of strategic tasks.4.30 ± 0.78160.60oP6. A communication system has been established to recognize occupational health risks and to efficiently pursue projects.4.40 ± 0.97170.70oP7. The mission of occupational health clearly states social responsibility for the health management of partner companies.4.30 ± 0.78160.60oP8. The occupational health organization includes coordinators for health management with partners.4.15 ± 0.73160.60o2. Do domainD1. Employee health surveillance and follow-up are appropriately implemented.4.50 ± 0.74170.70oD2. Risk assessment and work environment measurement are implemented to improve the hazardous work environment.4.60 ± 0.73170.70oD3. There is a standardized occupational health program that is implemented regularly.3.95 ± 0.86160.60oD4. There is a manual for responding to disasters and emergency patients, and regular inspections and training are conducted.4.35 ± 0.79180.80oD5. Occupational health doctors provide consultations and assessments of fitness for work.4.45 ± 0.74170.70oD6. Primary medical services are appropriately provided at the workplace.3.75 ± 0.77130.30xD7. We provide hazardous risk information to our partner companies and support occupational health projects.4.55 ± 0.59190.90oD8. We are also improving our partner companies’ occupational health hazards.4.50 ± 0.67180.80oD9. Every member has universal access to occupational health services.4.30 ± 0.64180.80o3. Check domainC1. Major health indicators are monitored through the occupational health computer system.4.45 ± 0.67180.80oC2. We are monitoring the input and process indicators of the occupational health project.4.25 ± 0.77160.60oC3. Audits are being conducted to manage the quality of occupational health projects.4.25 ± 0.70170.70oC4. We are measuring leading indicators that can predict the occurrence of disease.4.20 ± 0.68170.70oC5. We are measuring the lagging indicators of occupational health.4.35 ± 0.79180.80oC6. We are evaluating the return to work status of sick and injured employees.4.60 ± 0.49201.00oC7. New risk factors are being identified through reactive monitoring exposure assessments.4.35 ± 0.85170.70oC8. In addition to the legally mandated health surveillance, additional health checkups are being conducted.3.35 ± 0.858−0.20xC9. We have established a disease cohort that includes the main contractor and partner companies, and are continuously tracking it.3.95 ± 0.86140.40x4. Act domainA1. Changes in performance indicators were analyzed before and after the implementation of the occupational health project.4.60 ± 0.86170.70oA2. The results of the occupational health project were reported to members, opinions were collected, and reflected in system improvement.4.60 ± 0.73170.70oA3. The results of the analysis of importance and implementation of each occupational health program were later reflected in the overall occupational health plan.4.60 ± 0.66180.80oA4. We inspected and improved organizational culture and work practices that had a negative effect on the safety and health culture of the workplace.4.25 ± 0.89160.60oA5. The results of the occupational health project were shared with partners and an improvement plan was prepared.4.45 ± 0.86170.70oMinimum value by Lawshe’s table = 0.420 (20 experts).SD = standard deviation, CVR = content validity ratio. Results of the Delphi study for evaluation criteria of occupational safety and health management system (second round) Minimum value by Lawshe’s table = 0.420 (20 experts). SD = standard deviation, CVR = content validity ratio. DISCUSSION The Delphi process finalized 28 validated evaluation criteria across four domains of the PDCA-based OHMS framework (Supplementary Table 1). Most criteria exhibited strong expert consensus, with CVRs ranging from 0.60 to 1.00. The purpose of evaluating an OHMS is to assess a system that produces outcomes, rather than evaluating the outcomes themselves. Evaluators should examine whether the PDCA cycle in occupational health operates effectively. They must also identify the reasons for any dysfunction and propose measures for improvement. In Korea, the OHMS in workplaces tends to be driven by legal requirements rather than by voluntary regulations.17 Consequently, while planning and implementation are generally carried out well, the legal framework does not extend to evaluating or improving these processes. This leads to stagnation in the development of OHMS. This hinders the continuous development of the system. Although various standards for evaluating the OHMS have been discussed in the literature, they have not been presented as integrated tools that can be applied in real-world workplaces. This study aimed to synthesize the existing discussions and propose a universally applicable set of evaluation criteria for the OHMS. In the PDCA cycle, the Plan domain involves policy and planning, the Do domain refers to organizing and implementation, the Check domain includes indicators and audits, and the Action domain entails analysis and review. Workplace environment measurements and special health examinations are often classified under the Check domain because they assess exposure levels and health effects. However, in occupational health, they are typically placed within the Do domain because they are followed by corrective actions.13 The Do domain also includes various health promotion and protection programs, such as safety and health education, personal protective equipment use, hearing conservation programs, ergonomic improvements, and organizational management of job stress.18 For example, the Check domain activities such as noise measurement should lead to Do actions such as noise source control. To ensure this link, the Plan stage must define occupational health objectives and performance indicators. This evaluation framework categorizes performance indicators into input, process, leading, and lagging indicators and places them within the Check domain. In occupational health, a wide range of performance indicators can be established, each linked to workplace health objectives. The PDCA cycle cannot function effectively without measurable indicators. Evaluators can review the core of the OHMS by assessing the appropriateness of performance indicators, measurement results, health managers’ interpretations, and corresponding corrective actions.1920 It is designed for large enterprises with multiple subcontractors and dedicated occupational health organizations. Modern businesses do not operate in isolation; they collaborate with numerous partner companies, whose occupational health management is influenced by the policies of large enterprises. Therefore, corporate social responsibility extends to partner companies, a principle reflected in criteria P7, P8, D7, and A5. This framework evaluates a system that produces outcomes, rather than evaluating the occupational health outcomes themselves. If the outcomes are unsatisfactory, this framework can be used to assess and improve the OHMS. It also serves as a tool for assessing annual quality. Evaluators conduct multidimensional qualitative assessments through stakeholder interviews, document reviews, data analyses, surveys of workers and supervisors, and focus group interviews. Final evaluations classify the results as excellent, adequate, or non-compliant, with clear justifications required for excellent and non-compliant ratings. The framework provides judgment criteria to guide assessments by considering both content and context. However, meeting every example within the criteria is not mandatory. Rather, they serve as reference points to ensure alignment with the intent of the framework. “Adequate” is when it is judged that the purpose of the standard is being achieved. “Excellent” refers to good practices that can be recommended to other workplaces. “Non-compliant” is a case where the goal of the standard has not been achieved, and the reason for the judgment and direction for improvement must be presented. This judgment must be made through a consensus within the evaluation team. Application of the OHMS framework to POSCO’s system revealed both strengths and areas in need of improvement. While the Plan and Do domains were well-structured, the absence of appropriate performance indicators impeded effective implementation of the Check phase, thereby suggesting feedback and corrective action in the Action stage. This framework serves as a valuable tool for large enterprises seeking to enhance and sustain their OHMSs. The key considerations for applying this evaluation framework are as follows. First, this framework was designed for large enterprises, which have sufficient human and financial resources and are increasingly motivated to adopt it because of the growing emphasis on environmental social governance.21 In contrast, small- and medium-sized enterprises may face challenges in implementing and maintaining such a system because of cost, workforce, and expertise constraints. Second, although this framework deepens the assessment of occupational health by distinguishing it from broader occupational safety and health systems, both aspects should still be considered together. Therefore, it is best applied when a more detailed evaluation of occupational health management is required for the overall occupational safety and health assessment. Third, the framework was qualitative rather than quantitative. Quantifying the evaluation results has limited significance. The key is to identify deficiencies within the criteria and explore the most effective ways to improve them.22 This study proposed a PDCA-based evaluation framework for OHMS, comprising 28 validated criteria specifically designed for large enterprises operating with subcontractor networks. The framework prioritizes system-level assessment over outcome-based evaluation and offers practical, qualitative guidance for continuous improvement, utilizing measurable indicators and structured expert judgment. While most companies may meet the criteria for Plan and Do, they are likely to be weaker in Check and Action. Furthermore, plans and implementation efforts based on an incomplete Check and Action phase may be ineffective. Thus, this framework is particularly useful for strategic development and encourages organizational members to explore new approaches. Ultimately, it is well-suited for companies committed to fostering a proactive occupational health and safety culture.