Skip to main content
Top
Published in: Systematic Reviews 1/2024

Open Access 01-12-2024 | Commentary

Tools for assessing the methodological limitations of a QES—a short note

Authors: Heid Nøkleby, Heather Melanie R. Ames, Lars Jørun Langøien, Christine Hillestad Hestevik

Published in: Systematic Reviews | Issue 1/2024

Login to get access

Abstract

The increasing prevalence and application of qualitative evidence syntheses (QES) in decision-making processes underscore the need for robust tools to assess the methodological limitations of a completed QES. This commentary discusses the limitations of three existing tools and presents the authors’ efforts to address this gap. Through a simple comparative analysis, the three tools are examined in terms of their coverage of essential topic areas. The examination finds that existing assessment tools lack comprehensive coverage, clarity, and grounding in qualitative research principles. The authors advocate for the development of a new collaboratively developed evidence-based tool rooted in qualitative methodology and best practice methods. The conclusion emphasizes the necessity of a tool that can provide a comprehensive judgement on the methodological limitations of a QES, addressing the needs of end-users, and ultimately enhancing the trustworthiness of QES findings in decision-making processes.
Literature
1.
go back to reference Lewin S, Glenton C, Lawrie TA, Downe S, Finlayson KW, Rosenbaum S, et al. Qualitative evidence synthesis (QES) for guidelines: paper 2–using qualitative evidence synthesis findings to inform evidence-to-decision frameworks and recommendations. Health Res Policy Syst. 2019;17(1):1–18.CrossRef Lewin S, Glenton C, Lawrie TA, Downe S, Finlayson KW, Rosenbaum S, et al. Qualitative evidence synthesis (QES) for guidelines: paper 2–using qualitative evidence synthesis findings to inform evidence-to-decision frameworks and recommendations. Health Res Policy Syst. 2019;17(1):1–18.CrossRef
2.
go back to reference Downe S, Finlayson KW, Lawrie TA, Lewin SA, Glenton C, Rosenbaum S, et al. Qualitative evidence synthesis (QES) for guidelines: paper 1–using qualitative evidence synthesis to inform guideline scope and develop qualitative findings statements. Health Res Policy Syst. 2019;17(1):1–12.CrossRef Downe S, Finlayson KW, Lawrie TA, Lewin SA, Glenton C, Rosenbaum S, et al. Qualitative evidence synthesis (QES) for guidelines: paper 1–using qualitative evidence synthesis to inform guideline scope and develop qualitative findings statements. Health Res Policy Syst. 2019;17(1):1–12.CrossRef
3.
go back to reference Glenton C, Lewin S, Lawrie TA, Barreix M, Downe S, Finlayson KW, et al. Qualitative evidence synthesis (QES) for guidelines: paper 3–using qualitative evidence syntheses to develop implementation considerations and inform implementation processes. Health Res Policy Syst. 2019;17(1):1–14.CrossRef Glenton C, Lewin S, Lawrie TA, Barreix M, Downe S, Finlayson KW, et al. Qualitative evidence synthesis (QES) for guidelines: paper 3–using qualitative evidence syntheses to develop implementation considerations and inform implementation processes. Health Res Policy Syst. 2019;17(1):1–14.CrossRef
4.
go back to reference Flemming K, Noyes J. Qualitative evidence synthesis: where are we at? Int J Qual Methods. 2021;20:1609406921993276.CrossRef Flemming K, Noyes J. Qualitative evidence synthesis: where are we at? Int J Qual Methods. 2021;20:1609406921993276.CrossRef
5.
go back to reference Munthe-Kaas HM, Glenton C, Booth A, Noyes J, Lewin S. Systematic mapping of existing tools to appraise methodological strengths and limitations of qualitative research: first stage in the development of the CAMELOT tool. BMC Med Res Methodol. 2019;19(1):1–13.CrossRef Munthe-Kaas HM, Glenton C, Booth A, Noyes J, Lewin S. Systematic mapping of existing tools to appraise methodological strengths and limitations of qualitative research: first stage in the development of the CAMELOT tool. BMC Med Res Methodol. 2019;19(1):1–13.CrossRef
7.
go back to reference Whiting P, Wolff R, Mallett S, Simera I, Savović J. A proposed framework for developing quality assessment tools. Syst Rev. 2017;6:1–9.CrossRef Whiting P, Wolff R, Mallett S, Simera I, Savović J. A proposed framework for developing quality assessment tools. Syst Rev. 2017;6:1–9.CrossRef
8.
go back to reference Lewin S. Criteria for assessing how well a qualitative evidence syntheses (systematic reviews of qualitative studies) was conducted; EPOC resources for review authors. Oslo: Norwegian Institute of Public Health; 2018. Lewin S. Criteria for assessing how well a qualitative evidence syntheses (systematic reviews of qualitative studies) was conducted; EPOC resources for review authors. Oslo: Norwegian Institute of Public Health; 2018.
9.
go back to reference Lewin S, Bosch-Capblanch X, Oliver S, Akl EA, Vist GE, Lavis JN, et al. Guidance for evidence-informed policies about health systems: assessing how much confidence to place in the research evidence. PLoS Med. 2012;9(3):e1001187.CrossRefPubMedPubMedCentral Lewin S, Bosch-Capblanch X, Oliver S, Akl EA, Vist GE, Lavis JN, et al. Guidance for evidence-informed policies about health systems: assessing how much confidence to place in the research evidence. PLoS Med. 2012;9(3):e1001187.CrossRefPubMedPubMedCentral
10.
go back to reference Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ. 2017;358:j4008.CrossRefPubMedPubMedCentral Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ. 2017;358:j4008.CrossRefPubMedPubMedCentral
11.
go back to reference Booth AftCQaIMG. Prototype - Measurement Appraisal Checklist to Assess Qualitative Evidence Syntheses (QES) (MACAQuES). Sheffield: University of Sheffield; 2019. Booth AftCQaIMG. Prototype - Measurement Appraisal Checklist to Assess Qualitative Evidence Syntheses (QES) (MACAQuES). Sheffield: University of Sheffield; 2019.
12.
go back to reference Tong A, Flemming K, McInnes E, Oliver S, Craig J. Enhancing transparency in reporting the synthesis of qualitative research: ENTREQ. BMC Med Res Methodol. 2012;12(1):1–8.CrossRef Tong A, Flemming K, McInnes E, Oliver S, Craig J. Enhancing transparency in reporting the synthesis of qualitative research: ENTREQ. BMC Med Res Methodol. 2012;12(1):1–8.CrossRef
13.
go back to reference SBU. Granskningsmall för kvalitativa evidenssynteser (QES) (Tool to assess methodological limitations of qualitative evidence synthesis). Stockholm: SBU; 2023. SBU. Granskningsmall för kvalitativa evidenssynteser (QES) (Tool to assess methodological limitations of qualitative evidence synthesis). Stockholm: SBU; 2023.
14.
go back to reference Munthe-Kaas H, Bohren MA, Glenton C, Lewin S, Noyes J, Tunçalp Ö, et al. Applying GRADE-CERQual to qualitative evidence synthesis findings—paper 3: how to assess methodological limitations. Implement Sci. 2018;13(1):25–32. Munthe-Kaas H, Bohren MA, Glenton C, Lewin S, Noyes J, Tunçalp Ö, et al. Applying GRADE-CERQual to qualitative evidence synthesis findings—paper 3: how to assess methodological limitations. Implement Sci. 2018;13(1):25–32.
15.
go back to reference Munthe-Kaas HM, Sommer I, Noyes J, Cooper S, Garside R, Hannes K, et al. Development of the CAMELOT approach for considering methodological limitations of qualitative research in the context of GRADE-CERQual and qualitative evidence syntheses – protocol (version 1). Geneve: Zenodo; 2023. Munthe-Kaas HM, Sommer I, Noyes J, Cooper S, Garside R, Hannes K, et al. Development of the CAMELOT approach for considering methodological limitations of qualitative research in the context of GRADE-CERQual and qualitative evidence syntheses – protocol (version 1). Geneve: Zenodo; 2023.
16.
go back to reference Munthe-Kaas AHB, Sommer I, Cooper S, Garside R, Hannes K, Noyes J. Developing CAMELOT for assessing methodological limitations of qualitative research for inclusion in qualitative evidence syntheses. Submitted to Cochrane Evidence Synthesis and Methods. 2024. Munthe-Kaas AHB, Sommer I, Cooper S, Garside R, Hannes K, Noyes J. Developing CAMELOT for assessing methodological limitations of qualitative research for inclusion in qualitative evidence syntheses. Submitted to Cochrane Evidence Synthesis and Methods. 2024.
17.
go back to reference Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2020;2021:372. Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2020;2021:372.
18.
go back to reference Page MJ, Moher D, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews. BMJ. 2021;372:n160.CrossRefPubMedPubMedCentral Page MJ, Moher D, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews. BMJ. 2021;372:n160.CrossRefPubMedPubMedCentral
19.
go back to reference Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. Updating guidance for reporting systematic reviews: development of the PRISMA 2020 statement. J Clin Epidemiol. 2021;134:103–12.CrossRefPubMed Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. Updating guidance for reporting systematic reviews: development of the PRISMA 2020 statement. J Clin Epidemiol. 2021;134:103–12.CrossRefPubMed
20.
go back to reference France E, Ring N, Noyes J, Maxwell M, Jepson R, Duncan E, et al. Protocol-developing meta-ethnography reporting guidelines (eMERGe). BMC Med Res Methodol. 2015;15:1–14.CrossRef France E, Ring N, Noyes J, Maxwell M, Jepson R, Duncan E, et al. Protocol-developing meta-ethnography reporting guidelines (eMERGe). BMC Med Res Methodol. 2015;15:1–14.CrossRef
21.
go back to reference France EF, Cunningham M, Ring N, Uny I, Duncan EA, Jepson RG, et al. Improving reporting of meta-ethnography: the eMERGe reporting guidance. BMC Med Res Methodol. 2019;19(1):1–13.CrossRef France EF, Cunningham M, Ring N, Uny I, Duncan EA, Jepson RG, et al. Improving reporting of meta-ethnography: the eMERGe reporting guidance. BMC Med Res Methodol. 2019;19(1):1–13.CrossRef
22.
go back to reference Cunningham M, France EF, Ring N, Uny I, Duncan EA, Roberts RJ, et al. Developing a reporting guideline to improve meta-ethnography in health research: the eMERGe mixed-methods study. Health Serv Deliv Res. 2019;7(4):1–116.CrossRef Cunningham M, France EF, Ring N, Uny I, Duncan EA, Roberts RJ, et al. Developing a reporting guideline to improve meta-ethnography in health research: the eMERGe mixed-methods study. Health Serv Deliv Res. 2019;7(4):1–116.CrossRef
23.
go back to reference Svendsen C, Whaley P, Vist GE, Husøy T, Beronius A, Di Consiglio E, et al. Protocol for designing INVITES-IN, a tool for assessing the internal validity of in vitro studies. Evid Based Toxicol. 2023;1(1):2232415.CrossRef Svendsen C, Whaley P, Vist GE, Husøy T, Beronius A, Di Consiglio E, et al. Protocol for designing INVITES-IN, a tool for assessing the internal validity of in vitro studies. Evid Based Toxicol. 2023;1(1):2232415.CrossRef
Metadata
Title
Tools for assessing the methodological limitations of a QES—a short note
Authors
Heid Nøkleby
Heather Melanie R. Ames
Lars Jørun Langøien
Christine Hillestad Hestevik
Publication date
01-12-2024
Publisher
BioMed Central
Published in
Systematic Reviews / Issue 1/2024
Electronic ISSN: 2046-4053
DOI
https://doi.org/10.1186/s13643-024-02511-6

Other articles of this Issue 1/2024

Systematic Reviews 1/2024 Go to the issue