Quality assessment with diverse studies (QuADS): an appraisal tool for methodological and reporting quality in systematic reviews of mixed- or multi-method studies.

Harrison R, Jones B, Gardner P, Lawton R. · 2021

Read Quality assessment with diverse studies (QuADS): an appraisal tool for methodological and reporting quality in systematic reviews of mixed- or multi-method studies. by Harrison R, Jones B, Gardner P, Lawton R. (2021) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/europmc-pmc7885606.txt.

Introduction

Harrison and co-authors report a study that built a quality appraisal tool for mixed-methods research. Systematic reviews in health services often include studies of many designs. Judging the quality of such mixed sets is hard with tools made for one design alone. The team set out to fix an earlier tool.

The authors reviewed all papers that had cited their earlier QATSDD tool, 197 in all. They also surveyed first authors of those papers for hands-on feedback. The feedback pointed to unclear scope and hard-to-use criteria. The revised tool, called QuADS, was then tested for agreement between raters. It reached a reliability score of k=0.66, with good face and content validity. The study gives reviewers a tested way to judge diverse studies.

Catalog record

CREATORHarrison R, Jones B, Gardner P, Lawton R.
YEAR2021
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
LICENSECC BY 4.0
RIGHTSCC BY
SOURCEEurope PMC

Source: https://europepmc.org/articles/PMC7885606

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Quality assessment with diverse studies (QuADS): an appraisal tool for methodological and reporting quality in systematic reviews of mixed- or multi-method studies By Reema Harrison, Benjamin Jones, Peter Gardner, Rebecca Lawton Published in: BMC Health Services Research DOI: 10.1186/s12913-021-06122-y ABSTRACT Abstract Background In the context of the volume of mixed- and multi-methods studies in health services research, the present study sought to develop an appraisal tool to determine the methodological and reporting quality of such studies when included in systematic reviews. Evaluative evidence regarding the design and use of our existing Quality Assessment Tool for Studies with Diverse Designs (QATSDD) was synthesised to enhance and refine it for application across health services research. Methods Secondary data were collected through a literature review of all articles identified using Google Scholar that had cited the QATSDD tool from its inception in 2012 to December 2019. First authors of all papers that had cited the QATSDD (n=197) were also invited to provide further evaluative data via a qualitative online survey. Evaluative findings from the survey and literature review were synthesised narratively and these data used to identify areas requiring refinement. The refined tool was subject to inter-rater reliability, face and content validity analyses. Results Key limitations of the QATSDD tool identified related to a lack of clarity regarding scope of use of the tool and in the ease of application of criteria beyond experimental psychological research. The Quality Appraisal for Diverse Studies (QuADS) tool emerged as a revised tool to address the limitations of the QATSDD. The QuADS tool demonstrated substantial inter-rater reliability (k=0.66), face and content validity for application in systematic reviews with mixed, or multi-methods health services research. Conclusion Our findings highlight the perceived value of appraisal tools to determine the methodological and reporting quality of studies in reviews that include heterogeneous studies. The QuADS tool demonstrates strong reliability and ease of use for application to multi or mixed-methods health services research. Supplementary Information The online version contains supplementary material available at 10.1186/s12913-021-06122-y. Keywords: Quality appraisal, Mixed-methods research, Multi-methods research, Systematic review, Health services research WHAT IS KNOWN - Many tools exist for assessing the quality of studies in systematic reviews of either quantitative or qualitative work. - There is a paucity of tools that assess the quality of studies within systematic reviews that include a diverse group of study designs, and mixed or multi-methods studies in particular. - The Quality Assessment Tool for Studies with Diverse Designs (QATSDD) published in 2012 was developed to assess the quality of studies with heterogenous designs primarily for use in the discipline of Psychology. WHAT THIS STUDY ADDS - The Quality Assessment for Diverse Studies (QuADS) tool is a refined version of the QATSDD tool. The aim was to use survey and literature review data to enhance the applicability of the tool to health services research, and more specifically, to multi or mixed-methods research. - The QuADS tool demonstrates substantial inter-rater reliability and content and face validity. BACKGROUND The inclusion of diverse types of evidence, such as qualitative and mixed or multi-methods research, is well-established in systematic reviews of health services research [1–3]. This is important because these methods can address the complexities within healthcare that cannot often be readily measured through a single method. Qualitative methods, when used alone, offer explanatory power to enhance understanding of multi-faceted and complex phenomena such as experiences of healthcare and systems [3]. When partnered with quantitative methods, qualitative data can support and add depth of understanding [4, 5]. The appraisal of the methodological quality, evidence quality and quality of reporting of individual studies and of studies included in a review collectively is firmly established for reviews of quantitative studies. There are more than 60 tools currently available to assess the quality of randomised controlled trials alone [6]. Appraisal of the quality of evidence is often used to assess bias, particularly in randomised controlled trials. More recently, quality appraisal tools have extended to tools for appraising qualitative research, with the emergence of multiple tools in this space [7] creating a topic of extended debate [7–10]. As a result, reviews that include both qualitative and quantitative research often utilise separate quality appraisal tools for the quantitative and qualitative studies within the review, often citing the lack of a standard, empirically grounded tools suitable to assess methodological quality, evidence quality and/or quality of reporting with a variety of study designs [11]. The use of a parallel approach to all aspects of quality appraisal offers strength in the ability to acknowledge the unique nature of qualitative research and its epistemological distinction from quantitative approaches. Yet, a dual approach does not facilitate the appraisal of methodological, evidence or reporting quality for mixed-methods research, and creates challenges in appraising these aspects of multi-methods work. Thus, acknowledging that the underlying assumptions of quantitative and qualitative research are substantially different, a tool to appraise methodological quality, evidence quality and/or quality of reporting mixed- or multi-methods research is valuable in enabling researchers to consider the transparency and reporting of key elements of these approaches [12]. Moreover, a tool that is relevant to mixed- and multi-method approaches is significant in the context of growing recognition of the value of these methodologies in health systems and services research [4]. A single tool that can be used to evaluate methodological quality, evidence quality, and quality of reporting across a body of diverse evidence facilitates reviewers to reflect on the extent to which there is apparent transparency and congruency in the research purpose and its reporting and the implications for evidence quality. This is currently not available for mixed- and multi-methods work, with study heterogeneity as a key obstacle to evidence appraisal. Given the complexities of multiple individuals evaluating a diverse set of studies, a supporting tool may also provide an underpinning method to develop a shared understanding of what constitutes quality in research methods, evidence and reporting. The authors published in 2012, a pragmatic approach to facilitate reviewers to appraise the methodological quality, evidence quality, and quality of reporting in reviews that included qualitative, quantitative, mixed- and multi-methods research using a single tool (QATSDD) [13]. The QATSDD has been cited more than 270 times to date and has been used in more than 80 reviews. The tool provides a framework for exploring the congruency, transparency and organised reporting of the research process for research grounded in post-positivistic or positivist methodology that informs multiple-methods or mixed-methods designs. The tool was not proposed as a basis for determining studies to be excluded from a review given that any cut-off points to indicate high or low quality would be arbitrary. The QATSDD tool was originally developed for application in Psychology but has demonstrated wider relevance through its application in a broad range of health services research. Its wide use suggests that researchers value the ability to appraise quality of evidence from studies that employ or combine a range of methods. Yet the QATSDD tool has some limitations in its ease of use beyond the discipline of Psychology. We therefore aimed to revise, enhance and adapt the current QATSDD tool into an updated version; Quality Assessment for Diverse Studies (QuADS), for greater applicability to health services researchers appraising quality of methods, evidence and

Read the complete text