Top

Published in:

Open Access 01-01-2018 | Method

Applying GRADE-CERQual to qualitative evidence synthesis findings—paper 3: how to assess methodological limitations

Authors: Heather Munthe-Kaas, Meghan A. Bohren, Claire Glenton, Simon Lewin, Jane Noyes, Özge Tunçalp, Andrew Booth, Ruth Garside, Christopher J. Colvin, Megan Wainwright, Arash Rashidian, Signe Flottorp, Benedicte Carlsen

Published in: Implementation Science | Special Issue 1/2018

Abstract

Background

The GRADE-CERQual (Confidence in Evidence from Reviews of Qualitative research) approach has been developed by the GRADE (Grading of Recommendations Assessment, Development and Evaluation) Working Group. The approach has been developed to support the use of findings from qualitative evidence syntheses in decision-making, including guideline development and policy formulation.

CERQual includes four components for assessing how much confidence to place in findings from reviews of qualitative research (also referred to as qualitative evidence syntheses): (1) methodological limitations, (2) coherence, (3) adequacy of data and (4) relevance. This paper is part of a series providing guidance on how to apply CERQual and focuses on CERQual’s methodological limitations component.

Methods

We developed the methodological limitations component by searching the literature for definitions, gathering feedback from relevant research communities and developing consensus through project group meetings. We tested the CERQual methodological limitations component within several qualitative evidence syntheses before agreeing on the current definition and principles for application.

Results

When applying CERQual, we define methodological limitations as the extent to which there are concerns about the design or conduct of the primary studies that contributed evidence to an individual review finding. In this paper, we describe the methodological limitations component and its rationale and offer guidance on how to assess methodological limitations of a review finding as part of the CERQual approach. This guidance outlines the information required to assess methodological limitations component, the steps that need to be taken to assess methodological limitations of data contributing to a review finding and examples of methodological limitation assessments.

Conclusions

This paper provides guidance for review authors and others on undertaking an assessment of methodological limitations in the context of the CERQual approach. More work is needed to determine which criteria critical appraisal tools should include when assessing methodological limitations. We currently recommend that whichever tool is used, review authors provide a transparent description of their assessments of methodological limitations in a review finding. We expect the CERQual approach and its individual components to develop further as our experiences with the practical implementation of the approach increase.

Available only for authorised users

Lewin S, Glenton C, Munthe-Kaas H, Carlsen B, Colvin CJ, Gulmezoglu M, Noyes J, Booth A, Garside R, Rashidian A. Using qualitative evidence in decision making for health and social interventions: an approach to assess confidence in findings from qualitative evidence syntheses (GRADE-CERQual). PLoS Med. 2015;12(10):e1001895.CrossRefPubMedPubMedCentral

Guyatt GH, Oxman AD, Vist G, Kunz R, Brozek J, Alonso-Coello P, Montori V, Akl EA, Djulbegovic B, Falck-Ytter Y, et al. GRADE guidelines: 4. Rating the quality of evidence—study limitations (risk of bias). J Clin Epidemiol. 2011;64(4):407–15.CrossRefPubMed

Higgins JP, Altman DG, Gotzsche PC, Juni P, Moher D, Oxman AD, Savovic J, Schulz KF, Weeks L, Sterne JA, et al. The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ. 2011;343:d5928.CrossRefPubMedPubMedCentral

Colvin CJ, Garside R, Wainwright M, Lewin S, Bohren M, Glenton C, Munthe-Kaas HM, Carlsen B, Tuncalp Ö, Noyes J, et al. Applying GRADE-CERQual to qualitative evidence synthesis findings—paper 4: how to assess coherence. Implement Sci. 2018;13(Suppl 1): https://doi.org/10.1186/s13012-017-0691-8.

Glenton C, Carlsen B, Lewin S, Munthe-Kaas HM, Colvin CJ, Tuncalp Ö, Bohren M, Noyes J, Booth A, Garside R, et al. Applying GRADE-CERQual to qualitative evidence synthesis findings—paper 5: how to assess adequacy of data. Implement Sci. 2018;13(Suppl 1): https://doi.org/10.1186/s13012-017-0692-7.

Noyes J, Booth A, Lewin S, Carlsen B, Glenton C, Munthe-Kaas HM, Colvin CJ, Garside R, Bohren M, Rashidian A, et al. Applying GRADE-CERQual to qualitative evidence synthesis findings—paper 6: how to assess relevance of the data. Implement Sci. 2018;13(Suppl 1): https://doi.org/10.1186/s13012-017-0693-6.

Lewin S, Bohren M, Rashidian A, Glenton C, Munthe-Kaas HM, Carlsen B, Colvin CJ, Tuncalp Ö, Noyes J, Booth A, et al. Applying GRADE-CERQual to qualitative evidence synthesis findings—paper 2: how to make an overall CERQual assessment of confidence and create a Summary of Qualitative Findings Table. Implement Sci. 2018;13(Suppl 1): https://doi.org/10.1186/s13012-017-0689-2.

Ames HMR, Glenton C, Lewin S. Parents’ and informal caregivers’ views and experiences of communication about routine childhood vaccination: a synthesis of qualitative evidence. Cochrane Database Syst Rev. 2017;(2) Art. No.: CD011787.

Aslam RW, Hendry M, Carter B, Noyes J, Rycroft Malone J, Booth A, Pasterfield D, Charles JM, Craine N, Tudor Edwards R, et al. Interventions for preventing unintended repeat pregnancies among adolescents (protocol). Cochrane Database Syst Rev. 2015;(1) Art. No.: CD011477.

10.

Bohren MA, Hunter EC, Munthe-Kaas HM, Souza JP, Vogel JP, Gulmezoglu AM. Facilitators and barriers to facility-based delivery in low- and middle-income countries: a qualitative evidence synthesis. Reprod Health. 2014;11(1):71.CrossRefPubMedPubMedCentral

11.

Bohren MA, Vogel JP, Hunter EC, Lutsiv O, Makh SK, Souza JP, Aguiar C, Saraiva Coneglian F, Diniz AL, Tuncalp O, et al. The mistreatment of women during childbirth in health facilities globally: a mixed-methods systematic review. PLoS Med. 2015;12(6):e1001847. discussion e1001847CrossRefPubMedPubMedCentral

12.

Colvin CJ, de Heer J, Winterton L, Mellenkamp M, Glenton C, Noyes J, Lewin S, Rashidian A. A systematic review of qualitative evidence on barriers and facilitators to the implementation of task-shifting in midwifery services. Midwifery. 2013;29(10):1211–21.CrossRefPubMed

13.

Glenton C, Colvin CJ, Carlsen B, Swartz A, Lewin S, Noyes J, Rashidian A. Barriers and facilitators to the implementation of lay health worker programmes to improve access to maternal and child health: qualitative evidence synthesis. Cochrane Database Syst Rev. 2013;10:CD010414.

14.

Munabi-Babigumira S, Glenton C, Lewin S, Fretheim A, Nabudere H. Factors that influence the provision of intrapartum and postnatal care by skilled birth attendants in low- and middle-income countries: a qualitative evidence synthesis. Cochrane Database Syst Rev. 2017;(11) Art. No.: CD011558.

15.

Munthe-Kaas HM, Hammerstrøm KT, et al. Effekt av og erfaringer med kontinuitetsfremmende tiltak i barnevernsinstitusjoner. Oslo: Norwegian Knowledge Centre for the Health Services; 2013.

16.

O’Brien TD, Noyes J, Spencer LH, Kubis HP, Hastings RP, Edwards RT, Bray N, Whitaker R. ‘Keep fit’ exercise interventions to improve health, fitness and well-being of children and young people who use wheelchairs: mixed-method systematic review protocol. J Adv Nurs. 2014;70(12):2942–51.CrossRefPubMed

17.

Rashidian A, Shakibazadeh E, Karimi-Shahanjarini A, Glenton C, Noyes J, Lewin S, Colvin C, Laurant M. Barriers and facilitators to the implementation of doctor-nurse substitution strategies in primary care: qualitative evidence synthesis (protocol). Cochrane Database Syst Rev. 2013;2:CD010412.

18.

Whitaker R, Hendry M, Booth A, Carter B, Charles J, Craine N, Edwards RT, Lyons M, Noyes J, Pasterfield D, et al. Intervention now to eliminate repeat unintended pregnancy in teenagers (INTERUPT): a systematic review of intervention effectiveness and cost-effectiveness, qualitative and realist synthesis of implementation factors and user engagement. BMJ Open. 2014;4(4):e004733.CrossRefPubMedPubMedCentral

19.

Downe S, Finlayson K, Tuncalp, Metin Gulmezoglu A. What matters to women: a systematic scoping review to identify the processes and outcomes of antenatal care provision that are important to healthy pregnant women. BJOG. 2016;123(4):529–39.CrossRefPubMed

20.

Odendaal WA, Goudge J, Griffiths F, Tomlinson M, Leon N, Daniels K. Healthcare workers’ perceptions and experience on using mHealth technologies to deliver primary healthcare services: qualitative evidence synthesis (protocol). Cochrane Database Syst Rev. 2015;(11) Art. No.: CD011942.

21.

Lewin S, Booth A, Glenton C, Munthe-Kaas HM, Rashidian A, Wainwright M, Bohren MA, Tuncalp Ö, Colvin CJ, Garside R, et al. Applying GRADE-CERQual to qualitative evidence synthesis findings: introduction to the series. Implement Sci. 2018;13(Suppl 1): https://doi.org/10.1186/s13012-017-0688-3.

22.

Denzin N. Qualitative inquiry under fire: toward a new paradigm dialogue. USA: Left Coast Press; 2009.

23.

Hammersley M. The issue of quality in qualitative research. Int J Res Method Educ. 2007;30(3):287–305.CrossRef

24.

Smith J. The problem of criteria for judging interpretive inquiry. Educ Eval Policy Anal. 1984;6(4):379–91.CrossRef

25.

Smith J, Deemer D. The problem of criteria in the age of relativism. In: Denzin N, Lincoln Y, editors. Handbook of Qualitative Research. London: Sage; 2000.

26.

Saini M, Shlonsky A. Systematic synthesis of qualitative research. Oxford: Oxford University Press; 2012.CrossRef

27.

Santiago-Delefosse M, Gavin A, Bruchez C, Roux P, Stephen SL. Quality of qualitative research in the health sciences: analysis of the common criteria present in 58 assessment guidelines by expert users. Soc Sci Med. 2016;148:142–51.CrossRefPubMed

28.

Hannes K. Chapter 4: critical appraisal of qualitative research. In: Noyes J, Booth A, Hannes K, Harden A, Lewin S, Lockwood C, editors. Supplementary Guidance for Inclusion of Qualitative Research in Cochrane Systematic Reviews of Interventions Version 1 (Updated August 2011). London: Cochrane Qualitative and Implementation Methods Group. http://cqrmg.cochrane.org/supplemental-handbook-guidance; 2011.

29.

Noyes J, Booth A, Flemming K, Garside R, Harden A, Lewin S, Pantoja T, Hannes K, Harris J, Cargo M, et al. Cochrane Qualitative and Implementation Methods Group Guidance paper 2: Methods for assessing methodological limitations, data extraction and synthesis, and confidence in synthesized qualitative findings. J Clin Epidemiol. 2017. In press. https://doi.org/10.1016/j.jclinepi.2017.06.020.

30.

CASP. Qualitative appraisal checklist for qualitative research. Available at: http://www.casp-uk.net/#!casp-tools-checklists/c18f8: Critical Appraisal Skills Programme; 2011.

31.

Walsh D, Downe S. Appraising the quality of qualitative research. Midwifery. 2006;22(2):108–19.CrossRefPubMed

Title: Applying GRADE-CERQual to qualitative evidence synthesis findings—paper 3: how to assess methodological limitations
Authors: Heather Munthe-Kaas
Meghan A. Bohren
Claire Glenton
Simon Lewin
Jane Noyes
Özge Tunçalp
Andrew Booth
Ruth Garside
Christopher J. Colvin
Megan Wainwright
Arash Rashidian
Signe Flottorp
Benedicte Carlsen
Publication date: 01-01-2018
Publisher: BioMed Central
Published in: Implementation Science / Issue Special Issue 1/2018
Electronic ISSN: 1748-5908
DOI: https://doi.org/10.1186/s13012-017-0690-9

At a glance: The STEP trials

Springer Medicine

Applying GRADE-CERQual to qualitative evidence synthesis findings—paper 3: how to assess methodological limitations

Abstract

Background

Methods

Results

Conclusions

At a glance: The STEP trials

Springer Medicine

Abstract

Background

Methods

Results

Conclusions

Please log in to get access to this content

Other articles of this Special Issue 1/2018

Applying GRADE-CERQual to qualitative evidence synthesis findings—paper 4: how to assess coherence

Applying GRADE-CERQual to qualitative evidence synthesis findings–paper 7: understanding the potential impacts of dissemination bias

Applying GRADE-CERQual to qualitative evidence synthesis findings–paper 6: how to assess relevance of the data

Applying GRADE-CERQual to qualitative evidence synthesis findings: introduction to the series

Applying GRADE-CERQual to qualitative evidence synthesis findings—paper 2: how to make an overall CERQual assessment of confidence and create a Summary of Qualitative Findings table

Applying GRADE-CERQual to qualitative evidence synthesis findings—paper 5: how to assess adequacy of data