Evaluating a transitional housing program for people who use substances (PWUS) who experience homelessness and live with a mental health issue: a mixed-methods study protocol in Sudbury Ontario Authors: Kristen A Morin, Daniel Molke, Natalie Aubin, Shannon Knowlan, Tara Leary Published: 2024 ABSTRACT AbstractBackgroundA new transitional housing program was established in Sudbury, Ontario, Canada, in response to the escalating global prevalence of substance use and homelessness, and the specific challenges faced in Northern Ontario. This protocol outlines a comprehensive program evaluation to assess its impact on patient outcomes, healthcare utilization, and client perspectives.MethodsWe will conduct a parallel mixed-method study that includes the analysis of single-center-level administrative health data and primary data collection. This includes a longitudinal observational study (target n = 1,200), pre- and post-admission quantitative interviews (target n = 40), and qualitative interviews (target n = 40). We will implement a participatory approach to this evaluation collaborating with people who use substances, frontline staff, and decision-makers. Data analysis methods include a range of statistical techniques, including logistic regression models, Cox proportional hazards models, Kaplan-Meier curves, Generalized Estimating Equations, and thematic qualitative analysis, ensuring a robust evaluation of patient outcomes and healthcare utilization.DiscussionThis protocol underpins a comprehensive assessment aimed at providing insights into the program’s effectiveness in addressing substance use-related challenges, reducing healthcare disparities, and improving patient outcomes, such as stable housing and increased social capital. All study procedures adhere to the ethical principles outlined in the Tri-Council Policy Statement: Ethical Conduct for Research Involving Humans. Findings will be disseminated progressively through established committees and working groups and subsequently published in peer-reviewed journals. Anticipated outcomes include informing evidence-based healthcare decision-making and driving improvements in addiction treatment practices within healthcare settings.Keywords: Mixed methods study, Transitional housing , Housing instability, Housing first, Addiction medicine, Program evaluation, Substance use disorders, Psychiatry Background Homelessness presents a multifaceted societal challenge, profoundly impacting both health and well-being. Individuals experiencing homelessness, experience high rates of substance use disorders, mental disorders, infectious diseases, cardiovascular and metabolic issues, and cognitive impairments [1, 2]. Regrettably, this demographic also faces a heightened risk of premature mortality compared to the general population, with substance use-related factors prominently contributing to the high mortality rates [3, 4]. While the transition out of homelessness can positively influence health outcomes [5], conventional housing approaches often impose prerequisites, such as abstinence from substance use or adherence to psychotropic medications, as part of a “treatment first” housing approach [6]. However, these strategies inadvertently exacerbate the social exclusion experienced by many people experiencing homelessness living with mental health issues or substance use disorders (SUD). In contrast, the “housing first” approach prioritizes addressing the primary unmet need of people experiencing homelessness, without requiring them to demonstrate preparedness for housing, such as being abstinent from using substances or adherence to treatment, thus minimizing the risk of prolonged homelessness or increased housing instability [6]. Despite the merits of this approach, there remains a dearth of research on the ongoing importance of harm reduction principles within housing support strategies, creating a notable gap between research evidence and practical implementation, consequently impeding the provision of patient-centred care [7]. Furthermore, assertive community treatment (ACT) teams, introduced in the early 1970s in response to the deinstitutionalization of psychiatric hospitals, adopt a team-oriented methodology to keep individuals living with mental health issues connected to services [8]. With the primary objective of reducing hospital admissions, these teams play a crucial role in improving outcomes for vulnerable populations, notably enhancing social functioning and overall quality of life. In January 2022, Health Sciences North (HSN) and the City of Greater Sudbury collaboratively launched a transitional housing program aimed at addressing the complex challenges faced by people experiencing homelessness living with mental health issues and substance use disorders. Drawing upon the principles of the ACT model, Housing First philosophy, and harm reduction strategies, the program offers immediate access to apartments within the transitional housing facility as its cornerstone. This program offers specialized support aimed at enhancing well-being and preparing people for independent living while integrating harm reduction principles throughout. Given the growing interest in developing treatment and continuing care strategies for people who use substances (PWUS) and experience homelessness, there is a pressing need to assess the effectiveness of transitional housing models in improving health and social outcomes. This protocol delineates a comprehensive evaluation of the transitional housing program in Sudbury, which was designed to provide evidence-based insights into its effectiveness at improving health and social outcomes and integration within the healthcare system. The protocol was developed to provide a rigorous methodology to perform a comprehensive evaluation of a transitional housing program for PWUS who experience homelessness and live with a mental health issue. The protocol outlines ways in which to measure the effectiveness of a transitional housing program and gain insight into client satisfaction and program integration within the healthcare system. More specifically, the objectives of the evaluation aim to answer the following questions: How does the transitional housing program influence health system metrics over time? How does the transitional housing program influence health system metrics over time? What benefits and barriers do clients identify while participating in the program? What benefits and barriers do clients identify while participating in the program? How does the program influence physical, mental, and emotional well-being? How does the program influence physical, mental, and emotional well-being? Does the program effectively contribute to housing stability? Does the program effectively contribute to housing stability? Addressing these inquiries provides feedback for potential improvement, contributing to the goal of better outcomes for individuals grappling with substance use and related challenges. Evaluating the transitional housing program in Sudbury allows us to build upon current knowledge of best practices for implementing harm-reduction principles, the ACT model, and the Housing First philosophy within the community setting. This program aims to support a particularly vulnerable population, and by implementing a structured research protocol as part of standard operations, the program will have evidence-based metrics to reference for decision-making. Methods Design This study is a mixed-methods observational design centred on a cohort of clients accessing the transitional housing program in Sudbury, Ontario, Canada. The estimated evaluation period will span from September 1, 2024, to August 31, 2026. We adhered to STROBE guidelines to design the observation objectives of the protocol. Setting Functioning as an integral part of community mental health and addiction care in Sudbury, Ontario, the transitional housing program is paired with support services aimed at skill development for independent living. The program also addresses health care, substance use, mental disorders, and social issues. Admission to the program mandates a referral through the by-names list process implemented in all Ontario municipalities. Eligibility is determined by a high vulnerability score with the VI-SPDAT [9] subsequently requiring acceptance by the transitional housing program through an initial program interview. The program accepts clients irrespective of whether they continue to use substances or not. Clients admitted to the program receive transitional housing for up to 18 months, with the overarching goal of developing the wellness and skills needed for stable housing. The program actively assists clients in overcoming individual barriers. While in the program, clients are provided with tools to help them connect them with various social, judicial, and medical services, such as the Local Food Bank, dental and psychiatry appointments, addiction treatment, court proceedings, vocational programs, and more, to support their journey toward stable housing. Operating under HSN’s harm reduction philosophy, the transitional housing program is dedicated to providing specialized care for individuals at different stages of stability in substance use. The comprehensive approach addresses medical and psychosocial needs, offering wrap-around care through a dedicated team of specialized professionals, including nurses, addiction workers, social workers, primary care physicians, and psychiatrists. This collaborative team works to improve clients’ abilities and skills to sustain permanent, independent housing. Participants The research includes all individuals residing within the transitional housing facility in Sudbury, Ontario. Participation in the program requires that clients meet specific criteria, including experiencing homelessness for over six months, living with mental health issues, and having a co-occurring substance use disorder. These clients are registered on the Greater City of Sudbury By-Name list [10] and exhibit “high” or “very high” scores on Vulnerability-Index Service Prioritization Decision Assistance Tool (VI-SPDAT) [9]. Considering the current rate of program attrition, we anticipate that around 188 clients will be part of the data collection process over two years. Data collection will begin March 2025.`. Data sources, collection, and measurement Objective 1: Determine the regional health system-level impact of the program We will use retrospective administrative data to conduct a longitudinal cohort study to determine the regional health system-level impact of the program implementation. Data collection All clients who accessed the program will be included in the analysis. We will perform an ongoing retrospective cohort study utilizing routinely collected data at HSN. The data will be gathered through the hospital’s electronic medical record (EMR) system. All patients affiliated with the program will be included, eliminating the need for recruitment or consent due to the inherent nature of the data. The administrative data will be pulled from January 2022, when the program launched. This data will continue to be collected until the end of the data collection period of this study, August 31, 2026. We put forth a simulated dataset comprising variables derived from routinely collected data. Subsequently, we collaborated with the HSN Decision Support personnel to extract the correct data in a format conducive to generating quality improvement reports and facilitating trend analysis. The HSN Decision Support team continues to generate these reports, which are accessible exclusively to authorized research personnel. The reports are subject to analysis every quarter, utilizing the SAS for Academics version, and presented to the program leadership. Statistical analysis We will use the administrative data to describe the client profile, report on health service use (HSU) metrics, and analyze the association between client factors and frequent users of HSN. The HSU metrics that will be reported include rates of hospital admissions, emergency department presentations, 30-day emergency department revisits, discharges against medical advice, and inpatient length of stay. To assess the HSU metrics of the cohort before and after program participation, we will employ an interrupted time series (ITS) analysis [11]. The ITS baseline period corresponds to the clients’ HSU before joining the program, the ITS intervention period covers HSU during residency in the program, and the ITS post-intervention period encompasses HSU after leaving the program. Utilizing ITS analysis to compare pre- and post-HSU trends will allow us to quantify the significance of any changes and determine whether these changes can be attributed to participation in the program. Given the small cohort size and variations in starting points and exposure durations, conducting ITS at the patient level may lack the necessary statistical power. Consequently, we will aggregate the data to the group level, aiming to provide insights into the program’s overall impact on the entire group’s HSU trends. Additionally, in a concurrent project with HSN, we have access to a larger regional cohort of individuals who use substances who present to the hospital (identified by substance use being their primary or secondary reason for their acute care visit). Conducting a comparative analysis can shed light on how HSU trends in the program compare to those of other PWUS. This approach provides additional context regarding who the program is serving in the community and its potential system-level impact. Furthermore, there is an opportunity to integrate the program cohort into an ongoing study that assesses the impact of various addiction services on HSU metrics, allowing for a comparison of different standards of care. Objective 2: Determine the community-level impact of the program We will use program-level data to conduct a longitudinal cohort study to determine the community-level impact of the program implementation. Data collection Every client in the program will be included in this part of the study. We will perform an ongoing observational analysis utilizing program-level data. Data will be gathered through a standardized data collection process. All patients affiliated with the program will be included, thereby eliminating the need for recruitment or consent due to the inherent nature of the data. The program-level data will be pulled from January 2022, when the program launched. This data will continue to be collected until the end of the data collection period of this study, August 31, 2026. Variables We will work with the program staff to track service provision and program supports based on some predetermined categories: case management, community accompaniment, income support, opioid antagonist treatment, medication support, referrals to internal programs, referrals to external programs, referrals to primary care, naloxone and harm reduction kit distribution, Hepatitis-C testing, alcohol pours, activities of daily living activity education, and participation in group activities. We will also collect data on outcomes, including overdoses in the program, length of stay in the program, receipt of financial assistance, connection to a primary care provider, and stable housing after discharge. Data analysis We will use administrative data to describe the program client profile and report service provision and community-level outcomes. Given the small cohort size and variations in starting points and exposure durations, conducting inferential analysis at the patient level may lack the necessary statistical power. Consequently, we will aggregate the data to the group level, aiming to provide insights into the program’s overall impact on the entire group’s community-level outcomes. We will also be conducting a comparative analysis with a larger regional cohort of individuals who use substances who present to HSN (identified by substance use being their primary or secondary reason for their acute care visit). This can shed light on how the program compares to those of other PWUS. This approach provides additional context regarding who the program is serving in the community and its potential system-level impact. Objective 3: Assess clients’ perceived barriers and key components of the program We will conduct qualitative interviews with clients to gain insight into their experience while in the program. These interviews will be collected over the two-year primary data collection period. Clients will be interviewed within 60 days of their exit from the program. They will be limited to one interview. Recruitment and data collection All clients, while participating in the program, will have the opportunity to participate in the research study and will be informed by the on-site research staff. Participants will be compensated for their time. The research staff will obtain participant-signed consent and fill in to collect basic demographic information before proceeding with the interview. The interviews will be conducted in person by a peer research assistant, in an on-site private room available for the research team to use. The questions have been thoughtfully developed and revised after piloting with current clients and incorporating feedback from PWUS. Interviews will be audio-recorded. Study procedures The Standing Operating Procedures will be designed together with program staff to provide the high-level protocol for our research team, encompassing the workflow and responsibilities from recruiting participants up to the point of analysis. Thematic analysis All clients who provide consent will be included in the analysis. Data will be analyzed quarterly. The transcripts will be reorganized into a Microsoft Excel workbook, grouping all responses for each question into a worksheet. We will identify key concepts and their frequencies for each interview question. The findings will be summarized, and in consultation with established academic literature, the research team will delve into the overarching themes articulated through the client’s voice. People with lived experience will be integral to discussing the results to achieve this objective. Their unique knowledge and perspectives on inequitable barriers and navigating the system as an oppressed group, stemming from lived experiences, distinguish their insights from individuals who have not undergone similar challenges. Objective 4: Measure the client-level outcomes of the program We will use three validated tools to measure the clients’ physical, mental, and emotional well-being and compare their results at admission and discharge. These interviews will be conducted within 60 days after admission (within month 1–2 of 18) and 60 days before discharge (within month 17–18 of 18) or at the time of discharge if the client’s stay is shorter than 18 months. Data collection can be divided into three sessions, one for each tool, to accommodate client needs. Recruitment and data collection All clients will have the chance to participate in the study during their stay and will be informed by the on-site research staff of the opportunity. Interested clients can book three dates and times with the on-site research staff at their convenience, prescribing a session for each tool. Compensation for participation will include $10 cash for each tool ($30 cash total. The research staff will obtain participant written consent using the Information Sheet and collect basic demographic information before proceeding with the first session. The data will be collected in person, in an interview format, with our peer researcher articulating the questions on the tool verbatim and completing the tool to reflect participant responses. Each data collection session will occur in a private room on-site that is available to our research team. The three measurement tools that will be used include the Recovery Capital Index (RCI) [12], the Maudsley Addiction Profile (MAP) [13] and the VI-SPDAT) [9]. These three measures have been chosen due to their long history of being utilized and validated in social, health services, and academic contexts. Study data will be collected and managed using (Research Electronic Data Capture (REDCap) electronic data capture tools hosted at Laurentian University [14]. REDCap is a secure, web-based software platform designed to support data capture for research studies, providing (1) an intuitive interface for validated data capture; (2) audit trails for tracking data manipulation and export procedures; (3) automated export procedures for seamless data downloads to common statistical packages; and (4) procedures for data integration and interoperability with external sources. Measures The Recovery Capital Index (RCI). The term “recovery capital” of the RCI refers to the collection of personal, social, and environmental resources an individual can draw upon to initiate and sustain recovery from addiction. This expands recovery potential beyond personal motivation and internal factors, recognizing the impact of external resources and support networks. It underscores the importance of addressing both internal motivation and external factors in the recovery process [12, 15]. Since then, researchers have developed several instruments to measure recovery capital [15]. The RCI® developed at Face it TOGETHER (FIT) by David Whitesock is built on three domains, nine indicators, and 22 components, which have been proven to positively affect recovery and provide a comprehensive assessment of intervention effectiveness. It also focuses on longitudinal health outcomes, providing insights into an individual’s progress, growth, and well-being over time [16]. The MAP is a multidimensional instrument developed to assess treatment outcomes for people with substance abuse disorders [13]. It measures sixty items across the four domains of substance use, health risk, physical and psychological health, and personal/social functioning. The tool is implemented as a brief interviewer-administered questionnaire [14]. The VI-SPDAT for Individuals [9] is an assessment tool for frontline workers in agencies dealing with homeless clients. Its purpose is to aid in prioritizing clients to determine who should receive assistance first, thereby guiding effective case management and enhancing housing stability outcomes. The VI-SPDAT [9], designed for busy communities lacking resources for a comprehensive VI-SPDAT assessment for each client, acts as a pre-screening tool. This less time-intensive survey efficiently categorizes clients into high, moderate, or low acuity levels [9]. Paired samples analysis Unpaired data will be excluded; admission data for participants without corresponding discharge data (and vice versa) will be removed. A paired samples t-test (or a Wilcoxon signed-rank test if assumptions are not met) will be employed to compare patients’ admission and discharge measures. A repeated measures analysis of covariance (ANCOVA) will be utilized, incorporating client factors as covariates. Additionally, a regression analysis will explore the relationship between the time spent in the program and the three outcome measures (three tools). Independent samples / unpaired analysis All data will be included; admission data for participants who did not complete discharge data (and vice versa) will be retained. Instead of comparing data points for individuals, a broader comparison between the admission and discharge groups will be made using an independent samples t-test (or a Mann-Whitney U test if assumptions are not met). A regression model may also be employed to understand the impact of multiple factors on the three outcome measures (the three tools), identifying significant covariates in the model. Data quality assurance The data we can gather will naturally have limitations based on the participants’ flow through the program. The analyses in this protocol were outlined with the assumption that each admission and discharge within the program would correspond to a distinct client. Nevertheless, the possibility of repeat clients exists. We will treat each unique stay as an individual participant in data collection, focusing on gathering all pertinent information for our outcome measures over two years. After the data collection phase, we will assess the extent to which repeat clients feature in our data and make necessary adjustments to our analyses. Research staff training All research team members will undergo mandatory training to ensure competency in all aspects of the study protocol before engaging in recruitment or participant enrollment. The comprehensive training encompasses the Tri-Council Policy Statement: Ethical Conduct for Research Involving Humans (TCPS-2) [17], relevant required orientation before working with specific assessment tools (e.g., VI-SPDAT), and protocol-specific training related to data collection and management. Research staff will practice the protocol through mock interviews with program staff — providing an opportunity for invaluable feedback by staff to assure engagement with clients is being done in the best way possible. Data governance, peer review, and ethical considerations We will respect the Tri-Council Policy Statement, Chap. 1, which highlights the importance of respect for persons, concern for welfare, and justice throughout all phases of the research process. The collected anonymized data will be linked in accordance with data governance protocols. This protocol underwent peer review during both the funding and ethics approval processes for this project by the Northern Ontario Academic Medicine Association review committee and Health Sciences North Research and Ethics Board, respectively. Co-design, knowledge integration, and translation Codesign stands as a fundamental principle within this evaluation. An embedded researcher at HSN takes the lead in collaborating directly with leadership to co-create the evaluation framework and research designs. The involvement of staff and peer workers is pivotal, as they contributed to the development of data collection tools, with peers leading the data collection efforts for this project. Regular input from the leadership team, frontline staff, and peer support workers, who possess practice-based knowledge, will ensure the contextualization of research findings, facilitate the successful implementation of these findings, and provide timely feedback on the process. The knowledge translation of research findings will focus on disseminating the program’s activities to stakeholders and making research findings accessible to the public. HSN’s research and communications team will devise knowledge translation strategies to share research outcomes with the public. These strategies encompass leveraging social media platforms to distribute infographics and concise one-page research summaries. An interactive dashboard will also be developed to provide live updates on ongoing HSU trends. Discussion The protocol outlines a comprehensive framework for the evaluation of a transitional housing program for PWUS who are experiencing homelessness and living with mental health issues in Northern Ontario, Canada. The program addresses the pressing challenges posed by PWUS, offering specialized care that bridges the gap between medical treatment, addiction, social support and housing. The evaluation aims to provide a holistic understanding of the program’s impact on patients’ health outcomes, healthcare utilization, and overall well-being. The protocol emphasizes the significance of patient-centred perspectives by including qualitative interviews and PWUS as researchers, ensuring that the voices and experiences of those utilizing the services are integral to the evaluation process. Simultaneously, quantitative analyses will robustly assess the program’s effects on healthcare utilization trends, contributing valuable insights for evidence-based healthcare decision-making. This evaluation seeks to illuminate the role of the transitional program in addressing issues related to substance use, reducing healthcare disparities, and improving patient outcomes. The insights garnered from this study will have direct implications for the specific healthcare institution of the HSN but also have the potential to provide more generalizable implications for addiction medicine practices elsewhere. Ultimately, this protocol sets the stage for a comprehensive and valuable evaluation that can drive positive changes in addiction treatment and healthcare delivery. Strengths and limitations Through evaluating a transitional housing program, the study has the potential to make a real-world impact by informing the development and enhancement of similar initiatives aimed at supporting PWUS experiencing homelessness and living with mental health issues while providing a foundation for developing a comprehensive evaluation of those similar initiatives. This study could significantly advance evidence-based practices in the field. However, there are important limitations to consider. The study’s restricted time frame (18 months) may hinder changes in participants’ experiences and outcomes over time, emphasizing the importance of follow-up research to assess long-term program impacts. The key limitation of objectives 1 and 2 is that the analyses are inherently limited by the data being collected; administrative data collected through the hospital and program, respectively. This highlights the importance of objectives 3 and 4 components, to round out the study as a proper mixed-method design. The interview / self-report data and qualitative interview data provide the richness and detail that the administrative data lacks. Moreover, the study’s findings may have limited generalizability beyond the specific context of Sudbury, Ontario, Canada, due to regional variations in housing policies, healthcare systems, and social services. Research ethics All experiments will be performed in accordance with the relevant guidelines and regulations of the Declaration of Helsinki. Written consent will be obtained from all participants participating in the following sections of the study (Objectives 3 and 4). Objectives 1 and 2 use administrative, already collected, anonymized data only and do not require consent. Acknowledgements We want to express our gratitude to everyone who contributed to this protocol. Our most profound appreciation goes to the patients and staff who have and continue to share their experiences and insights. Thank you to Neil St. Jean for this manuscript’s final editing and grammar checks. Special recognition goes to the peer researcher Jada Carter, who actively participated in the design and development of the qualitative questions and the pilot phase of this project. We acknowledge the support of decision-makers and HSN leadership, including Natalie Aubin. We also thank the City of Greater Sudbury, including Tyler Campbell, Gail Spencer, and Ayesha Ifatt, for their guidance in this research, as well as the contributions of various committees and working groups. Authors’ contributions KP contributed to the project design and coordination and co-wrote the original draft of this manuscript. DM contributed to the project design and supervision, and provided manuscript revisions. TL contributed to the project design and supervision, acquisition of funding, and provided manuscript revisions. SK contributed to the project supervision, and provided manuscript revisions. KAM contributed to the project design and coordination, acquisition of funding, co-wrote the original draft of this manuscript, and provided the final review and oversight on the manuscript version submitted. Funding This research is supported by the Northern Ontario Academic Medicine Association (NOAMA) Clinical Innovation Opportunities Fund award grant number C-23-14. Data availability No datasets were generated or analysed during the current study. Declarations Ethics approval and consent to participate The study was approved by the Health Sciences North Research Ethics Board (project number 23–024). Consent for publication Not applicable. Competing interests The authors declare no competing interests. Footnotes Publisher’s Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. References Associated Data Data Availability Statement No datasets were generated or analysed during the current study.