Synthesis of existing instruments for the evaluation of the implementation of system of care for children and adolescents: a systematic review protocol Authors: Masaki Kojima, Abir Nagata, Momoka Suda, Sayaka Kawakami, Md Shafiur Rahman, Toshiki Iwabuchi, Atsushi Senju Published: 2025 ABSTRACT Abstract AbstractIntroductionThe global need for developing comprehensive mental healthcare systems for children and adolescents has been increasingly evident. The system-of-care (SOC) model in the USA is among the most studied cases for such a system, which demonstrates how integrated services, encompassing psychiatric care, can effectively support children and adolescents with severe emotional disturbance. Over the past decades, the SOC framework has been widely expanded, with nearly every state in the USA receiving federal funding to implement or expand the SOC initiatives. The number of studies investigating the efficacy of the SOC is increasing, but no systematic review of the instruments for the evaluation of the implementation of the SOC exists today. This protocol paper documents the plan of a systematic review, which aims to identify and synthesise the existing instruments, which are designed and used to evaluate the implementation of the SOC.Methods and analysisThis systematic review will be reported according to the Preferred Reporting Items for Systematic reviews and Meta-Analysis Protocol guidelines. To identify relevant studies, we will perform a comprehensive search in five databases: PubMed, PsycINFO, ERIC, Social Work Abstracts and Web of Science. The title and abstracts of all searched articles will be screened independently by two reviewers according to the eligible criteria. Full-text screening of potentially relevant articles will be performed by at least two independent reviewers. A summary of included articles will describe the availability, progress and cultural adaptability of instruments.Ethics and disseminationThe current systematic review will be solely based on previously reported data and will not involve new data collection. There are no concerns that require ethical vetting. Findings of the review will be disseminated through conference presentations and publication in a peer-reviewed journal.PROSPERO registration number1065693.Keywords: MENTAL HEALTH, Child, Systematic Review, Child & adolescent psychiatry, Health Services Strengths and limitations of this study This review protocol outlines methods based on the Preferred Reporting Items for Systematic reviews and Meta-Analysis Protocol guidelines for synthesising instruments used to evaluate the implementation of system-of-care frameworks. This review protocol outlines methods based on the Preferred Reporting Items for Systematic reviews and Meta-Analysis Protocol guidelines for synthesising instruments used to evaluate the implementation of system-of-care frameworks. Study selection, data extraction and quality assessments will be performed by two independent reviewers for accuracy and consistency. Study selection, data extraction and quality assessments will be performed by two independent reviewers for accuracy and consistency. Variability of instruments may introduce heterogeneity, which has a potential risk of complicating the synthesis of results. Variability of instruments may introduce heterogeneity, which has a potential risk of complicating the synthesis of results. Introduction Mental disorders account for a large proportion of the disease burden affecting young people globally. Poor mental health is strongly related to other mental and social challenges, such as lower educational achievement, substance abuse and violence.1 Efforts have been made to build organised and coordinated care systems in several countries and regions, but such systems differed substantially between countries and regions,2,5 which may not meet the needs of diverse children, especially those living in many low-income and middle-income countries.6 7 Several studies also reported that in the currently available systems, children suffering from pronounced emotional difficulties are facing particular barriers in accessing appropriate mental healthcare.8,12 Given this context, it may not be surprising that the need for developing comprehensive and standardised mental healthcare systems for children, especially those with pronounced emotional difficulties, has been increasingly recognised globally.13,15 Several countries have developed their own integrated mental care systems for children and adolescents. In England, for example, the THRIVE Framework and its implementation programme, i-THRIVE, have been widely used since 2016.16 This approach emphasised collaboration between services and integrated care, aimed to break down traditional barriers between health, social care and voluntary sectors to create a ‘whole system-based’ approach.17 The headspace initiative in Australia, on the other hand, is advocated as a nation-wide youth mental healthcare reform, which mainly focused on clients with depression and anxiety issues.18 Finally, the system of care (SOC) in the USA is among the most longstanding examples, with a relatively substantial body of empirical studies. The SOC model in the USA demonstrates how integrated services, encompassing psychiatric care, can effectively support children and adolescents with severe emotional disturbances (SED). The SOC was originally defined by Stroul and Friedman19 in 1986 for a coordinated network and continuum of care to meet the multiple and changing needs of children and adolescents with SED.20 In the beginning, the SOC concept emphasised a broad array of services, including (a) ‘a comprehensive range of intensive non-residential and residential services and supports’; (b) ‘treatment in the least restrictive, most appropriate setting’; (c) ‘individualised and flexible treatment and services’; (d) ‘interagency collaboration among the various agencies and systems that share responsibility for children and youth with SED’; (e) ‘family involvement in planning and delivery of services’ and (f) ‘culturally competent services’.19 21 22 In the USA, the SOC has grown largely through federal support, primarily via the Comprehensive Mental Health Services for Children and Their Families Programme.22 Over the past decades, the SOC framework has been widely expanded, with nearly every state in the USA receiving federal funding to implement or expand the SOC initiatives.23 One of the biggest advantages of the SOC model is that numerous studies have examined the efficacy of the SOC, which focused on evaluating the effectiveness of the SOC in improving children’s mental health.24,29 The earliest of such studies (Fort Bragg evaluation project, North Carolina) did not find significant differences of improvement in adaptive behaviour between the SOC experimental groups and traditional service groups.24,26 Several studies which followed, on the other hand, reported positive findings of the SOC.2830,32 For example, the Dawn Project demonstrated a significant reduction in the utilisation of restrictive setting and a decrease in recidivism among participating youth.31 More recently, the American Academy of Child and Adolescent Psychiatry (AACAP) Committee has published a review of the SOC,33 which summarised the availability and quality of SOC-related resources reported to be highly variable across different states and communities.34 However, the report did not explicitly propose standard criteria or methodologies to evaluate the implementation of the SOC. Studies introduced above led to the development of various instruments which evaluate the implementation of the SOC. First, the System of Care Implementation Survey (SOCIS) is one of the representative measures used to assess the implementation of SOC, which has been applied in various studies,34,36 including a national study in the USA.37 The SOCIS is a 70-item based questionnaire that measures 14 factors (eg, Theory of Change, Management and Governance, Provider Accountability). Second, the System of Care Readiness and Implementation Measurement Scale (SOC-RIMS)38 has also been identified as a comprehensive assessment tool for the implementation of the SOC. This scale is mainly used for care providers of the SOC community services to evaluate the readiness of the community to implement the SOC (eg,‘There is a felt need for services within the community by the stakeholders’). Third, the Wraparound Fidelity Index (WFI)39 40 was used to evaluate ‘wraparound care coordination’, which is claimed to be an element of the SOC,22 though several reports referred to the difference of these two approaches.41 42 Fourth, the Georgetown University Centre for Child and Human Development released a ‘Rating Tool for Implementation of the System of Care Approach for Children, Youth and Young Adults with Behavioural Health Challenges and Their Families43’, which includes 101 items including checklist of comprehensive approaches (ask the implementation of components, eg, ‘outpatient group therapy’, ‘Substance use treatment’ and ‘inpatient hospitalisation’). Finally, several other studies reported different assessment tools.44 45 We propose that these instruments have the potential to evaluate integrated care systems globally, extending beyond the USA, given the shared need for children and adolescents across the globe. To achieve this goal, it is essential to systematically summarise, compare and examine the characteristics of these scales, to identify which is the most appropriate for different research purposes. This review provides a comprehensive overview of existing instruments evaluating the implementation of the SOC, aiming at identifying gaps and informing the development of a more standardised framework to enhance accessibility, continuity and responsiveness in services for children and adolescents with severe emotional problems outside, as well as in, the USA. Objective Our objective is to systematically identify the existing instruments for evaluating the implementation of the SOC. We aim to identify the availability, progress and cultural adaptability of instruments for assessing the performance across varying contexts. Therefore, the proposed systematic review will aim as follows: To review existing instruments used to evaluate the implementation of the SOC. To review existing instruments used to evaluate the implementation of the SOC. To review the strengths and weaknesses of the instruments regarding availability, progress (including examination of reliability and validity) and cultural adaptation. To review the strengths and weaknesses of the instruments regarding availability, progress (including examination of reliability and validity) and cultural adaptation. The Population, Exposure, Comparison, Outcome (PECO) format of the current systematic review is shown in table 1. Table 1PECO formatPECOPopulationChildren and adolescents, families, caregivers, facilitators and communities evaluated or assessed by the implementation of the system-of-Care (SOC) approach in mental health settingExposureSOC implementation effortComparisonNo comparison groupOutcomeAvailability, progress and cultural adaptability of SOC-related measurement tools PECO format Methods This systematic review will be reported according to the Preferred Reporting Items for Systematic reviews and Meta-Analysis Protocol (PRISMA-P) guidelines.46 Any important amendments will be documented with date-stamped version updates in the PROSPERO registration record and clearly reported in the final systematic review publication. Eligible criteria The eligible criteria of the current systematic review were delineated according to the PECO format (table 1). Detailed inclusion and exclusion criteria are outlined in table 2. Briefly, peer-reviewed articles or official reports which describe or evaluate measurement tools related to SOC implementation will be included. There will be no language restrictions, and efforts will be made to translate studies reported in languages other than English. Table 2Inclusion/exclusion criteriaInclusion criteriaExclusion criteriaPeer-reviewed articles or official reportsDescribing or evaluating measurement tools related to SOC implementationStudies that involve no measurement tool related to SOC implementationStudies that assess SOC principles, structure, process or outcomesStudies that do not assess SOC principles, structure, process or outcomesStudies that assess SOC including integrated care for children and adolescents (age <18)Studies only focused on clinical outcome without integrated careStudies focused on adultsThe scale explicitly indicates that it assesses support for children with serious emotional disturbance or similar conditionsThe scale indicates limits to assess support for general health check-up or specific mental illnessThe evaluation must involve individuals who have sufficient understanding of the community care system (eg, local government officials)SOC, system of care. Inclusion/exclusion criteria SOC, system of care. Data source and search strategy To identify relevant studies, we will perform a comprehensive search in five databases: PubMed (via NCBI), PsycINFO (via EBSCOhost), ERIC (via Institute of Education Sciences), Social Work Abstracts (via EBSCOhost) and Web of Science (via Clarivate). To maintain uniform search conditions, each of the five databases will be queried using syntax adapted to its specific requirements. As a supplementary search, Google Scholar will be used to identify relevant grey literature. Data review will start from 1 December 2025 and is scheduled to be completed by 31 March 2026. Database-specific search strategies will be developed and medical subject headings (MeSH) terms47 will be included for PubMed. Additional searches will be performed via citation tracking of the identified article’s references. The search strategy is shown in figure 1. Search keywords and detailed queries are shown in online supplemental table. The screening process. Flow chart illustrating the study process. Screening and data extraction Retrieved titles from all databases will be merged and duplicate titles will be excluded. All data of searched publications will be imported into Rayyan V.1.6.4, which is the AI-powered platform for systematic review. In the first step, the title and abstracts of all searched articles will be screened independently by two reviewers (MK and AN) according to the eligible criteria. In the second step, the full-text screening of the identified articles will be performed by at least two independent reviewers (MK, AN, MS, SK, MSR and TI) to identify studies. The extracted data for screening will include author(s) name(s), publication year, publication type, journal title, study design and organisation. Finally, a last search of studies and relevant articles will be summarised. Data synthesis A narrative synthesis of the extracted data will be performed according to study characteristics and findings. Extracted data will be summarised with the author name, study quality, strengths and limitations. Sensitivity, specificity and cut-off points will be extracted quantitatively. Variations across studies will be examined in terms of methodological quality, population characteristics and operational definitions. Patterns in test performance will be discussed with attention to consistent findings, ranges of estimates and contextual factors that may account for differences. Quality assessment To evaluate the methodological rigour, risk of bias and results of the instruments, we will use the JBI critical appraisal tool48 by two assessors (MK and AN) independently. Any discrepancies between two evaluators will be resolved through constructive discussions. Discussion To the our knowledge, the systematic review will be the first to evaluate the existing literature on the instruments which evaluate the implementation of the SOC. Through a comprehensive analysis, this study identifies the key characteristics, strengths and limitations of currently available instruments used to assess and support effective implementation of the SOC. The SOC originated in the USA; however, the fundamental needs underlying the SOC framework are considered to represent universal challenges. Because the SOC framework is inherently tied to national mental health systems in the USA, its direct adaptation to other contexts may be challenging. Nonetheless, our findings could provide internationally valuable insights, such as for cross-national comparisons and for understanding what components of the US SOC model are distinctive or absent in other countries. The systematic review will have several limitations. First, capturing relevant literature from outside the USA may be challenging, as the term similar to ‘SOC’ might be conceptualised or referred to differently in other countries. Second, some instruments may have been originally designed for regional comparisons within the USA. While these tools could be useful for assessing SOC in various settings (eg, in low-resource areas), applying their findings to culturally adapted contexts requires careful consideration. Finally, considerable variation among SOC implementation tools is anticipated due to the lack of consistency in defining the primary outcome of the SOC. Therefore, when comparing different SOC instruments, it is essential to carefully account for potential discrepancies in their main outcomes, as such differences may influence the interpretation and comparability of findings across studies. Notwithstanding these considerations, the findings of this review will elucidate the current state of the availability of such instruments and offer valuable insights for their refinement and future development of such instruments. As we discussed above, the research outcomes will also help advance studies on the construction of integrated service systems for child mental health across the globe, which could inform the global efforts to design and establish effective integrated support systems for children and adolescents. Patient and public involvement Patients or the public were not involved in the design, conduct, reporting or dissemination plans of this research. Ethics and dissemination The current systematic review will be solely based on previously reported data and will not involve new data collection. There are no concerns that require ethical vetting. Findings of the review will be disseminated through conference presentations and publication in a peer-reviewed journal. Supplementary material Footnotes Funding: The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors. Prepublication history and additional supplemental material for this paper are available online. To view these files, please visit the journal online (https://doi.org/10.1136/bmjopen-2025-107748). Provenance and peer review: Not commissioned; externally peer reviewed. Patient consent for publication: Not applicable. Patient and public involvement: Patients and/or the public were not involved in the design, or conduct, or reporting or dissemination plans of this research. References Supplementary Materials