Skip to main content
Key Specialties
Cardiology Diabetology Endocrinology Gastroenterology Geriatrics and Gerontology Gynecology and Obstetrics Hematology Infectious Disease Internal Medicine Nephrology Neurology Oncology Pediatrics Respiratory Medicine Rheumatology Surgery
Congress Coverage
2024 ACC Congress 2023 ESMO Congress 2023 EASD Congress 2023 ERS Congress 2023 ESC Congress
Clinical Collections
Advances in Alzheimer’s

At a glance: The STEP trials

A round-up of the STEP phase 3 clinical trials evaluating semaglutide for weight loss in people with overweight or obesity.
ADVANCED SEARCH
Log in
Top
Implementation Science
Published in: Implementation Science 1/2018

Open Access 01-01-2018 | Method

Applying GRADE-CERQual to qualitative evidence synthesis findings–paper 6: how to assess relevance of the data

Authors: Jane Noyes, Andrew Booth, Simon Lewin, Benedicte Carlsen, Claire Glenton, Christopher J. Colvin, Ruth Garside, Meghan A. Bohren, Arash Rashidian, Megan Wainwright, Özge Tunςalp, Jacqueline Chandler, Signe Flottorp, Tomas Pantoja, Joseph D. Tucker, Heather Munthe-Kaas

Published in: Implementation Science | Special Issue 1/2018

Login to get access

Abstract

Background

The GRADE-CERQual (Confidence in Evidence from Reviews of Qualitative research) approach has been developed by the GRADE (Grading of Recommendations Assessment, Development and Evaluation) Working Group. The approach has been developed to support the use of findings from qualitative evidence syntheses in decision-making, including guideline development and policy formulation.
CERQual includes four components for assessing how much confidence to place in findings from reviews of qualitative research (also referred to as qualitative evidence syntheses): (1) methodological limitations, (2) coherence, (3) adequacy of data and (4) relevance. This paper is part of a series providing guidance on how to apply CERQual and focuses on CERQual’s relevance component.

Methods

We developed the relevance component by searching the literature for definitions, gathering feedback from relevant research communities and developing consensus through project group meetings. We tested the CERQual relevance component within several qualitative evidence syntheses before agreeing on the current definition and principles for application.

Results

When applying CERQual, we define relevance as the extent to which the body of data from the primary studies supporting a review finding is applicable to the context (perspective or population, phenomenon of interest, setting) specified in the review question. In this paper, we describe the relevance component and its rationale and offer guidance on how to assess relevance in the context of a review finding. This guidance outlines the information required to assess relevance, the steps that need to be taken to assess relevance and examples of relevance assessments.

Conclusions

This paper provides guidance for review authors and others on undertaking an assessment of relevance in the context of the CERQual approach. Assessing the relevance component requires consideration of potentially important contextual factors at an early stage in the review process. We expect the CERQual approach, and its individual components, to develop further as our experiences with the practical implementation of the approach increase.
Appendix
Available only for authorised users
Literature
1.
Lewin S, Glenton C, Munthe-Kaas H, Carlsen B, Colvin CJ, Gulmezoglu M, Noyes J, Booth A, Garside R, Rashidian A. Using qualitative evidence in decision making for health and social interventions: an approach to assess confidence in findings from qualitative evidence syntheses (GRADE-CERQual). PLoS Med. 2015;12(10):e1001895.CrossRefPubMedPubMedCentral
2.
Guyatt GH, Oxman AD, Kunz R, Woodcock J, Brozek J, Helfand M, Alonso-Coello P, Falck-Ytter Y, Jaeschke R, Vist G, et al. GRADE guidelines: 8. Rating the quality of evidence––indirectness. J Clin Epidemiol. 2011;64(12):1303–10.CrossRefPubMed
3.
Colvin CJ, Garside R, Wainwright M, Lewin S, Bohren M, Glenton C, Munthe-Kaas HM, Carlsen B, Tuncalp Ö, Noyes J, et al. Applying GRADE-CERQual to qualitative evidence synthesis findings––paper 4: how to assess coherence. Implement Sci. 2018;13(Suppl 1) https://​doi.​org/​10.​1186/​s13012-017-0691-8.
4.
Glenton C, Carlsen B, Lewin S, Munthe-Kaas HM, Colvin CJ, Tuncalp Ö, Bohren M, Noyes J, Booth A, Garside R, et al. Applying GRADE-CERQual to qualitative evidence synthesis findings––paper 5: how to assess adequacy of data. Implement Sci. 2018;13(Suppl 1) https://​doi.​org/​10.​1186/​s13012-017-0692-7.
5.
Munthe-Kaas HM, Bohren M, Carlsen B, Glenton C, Lewin S, Colvin CJ, Tuncalp Ö, Noyes J, Booth A, Garside R, et al. Applying GRADE-CERQual to qualitative evidence synthesis findings––paper 3: how to assess methodological limitations. Implement Sci. 2018;13(Suppl 1) https://​doi.​org/​10.​1186/​s13012-017-0690-9.
6.
Lewin S, Bohren M, Rashidian A, Glenton C, Munthe-Kaas HM, Carlsen B, Colvin CJ, Tuncalp Ö, Noyes J, Booth A, et al. Applying GRADE-CERQual to qualitative evidence synthesis findings––paper 2: how to make an overall CERQual assessment of confidence and create a Summary of Qualitative Findings table. Implement Sci. 2018;13(Suppl 1) https://​doi.​org/​10.​1186/​s13012-017-0689-2.
7.
Ames HMR, Glenton C, Lewin S. Parents’ and informal caregivers’ views and experiences of communication about routine childhood vaccination: a synthesis of qualitative evidence. Cochrane Database Syst Rev. 2017;2:CD011787.PubMed
8.
Aslam RW, Hendry M, Carter B, Noyes J, Rycroft Malone J, Booth A, Pasterfield D, Charles JM, Craine N, Tudor Edwards R, et al. Interventions for preventing unintended repeat pregnancies among adolescents (protocol). Cochrane Database Syst Rev. 2015;(1):Art. No.: CD011477.
9.
Bohren MA, Hunter EC, Munthe-Kaas HM, Souza JP, Vogel JP, Gulmezoglu AM. Facilitators and barriers to facility-based delivery in low- and middle-income countries: a qualitative evidence synthesis. Reprod Health. 2014;11(1):71.CrossRefPubMedPubMedCentral
10.
Bohren MA, Vogel JP, Hunter EC, Lutsiv O, Makh SK, Souza JP, Aguiar C, Saraiva Coneglian F, Diniz AL, Tuncalp O, et al. The mistreatment of women during childbirth in health facilities globally: a mixed-methods systematic review. PLoS Med. 2015;12(6):e1001847. discussion e1001847CrossRefPubMedPubMedCentral
11.
Colvin CJ, de Heer J, Winterton L, Mellenkamp M, Glenton C, Noyes J, Lewin S, Rashidian A. A systematic review of qualitative evidence on barriers and facilitators to the implementation of task-shifting in midwifery services. Midwifery. 2013;29(10):1211–21.CrossRefPubMed
12.
Glenton C, Colvin CJ, Carlsen B, Swartz A, Lewin S, Noyes J, Rashidian A. Barriers and facilitators to the implementation of lay health worker programmes to improve access to maternal and child health: qualitative evidence synthesis. Cochrane Database Syst Rev. 2013;10:CD010414.
13.
Munabi-Babigumira S, Glenton C, Lewin S, Fretheim A, Nabudere H. Factors that influence the provision of intrapartum and postnatal care by skilled birth attendants in low- and middle-income countries: a qualitative evidence synthesis. Cochrane Database Syst Rev. 2017;(11):Art. No.: CD011558.
14.
Munthe-Kaas HM, Hammerstrøm KT, et al. Effekt av og erfaringer med kontinuitetsfremmende tiltak i barnevernsinstitusjoner. Norwegian Knowledge Centre for the Health Services: Oslo; 2013.
15.
O'Brien TD, Noyes J, Spencer LH, Kubis HP, Hastings RP, Edwards RT, Bray N, Whitaker R. ‘Keep fit’ exercise interventions to improve health, fitness and well-being of children and young people who use wheelchairs: mixed-method systematic review protocol. J Adv Nurs. 2014;70(12):2942–51.CrossRefPubMed
16.
Rashidian A, Shakibazadeh E, Karimi- Shahanjarini A, Glenton C, Noyes J, Lewin S, Colvin C, Laurant M. Barriers and facilitators to the implementation of doctor-nurse substitution strategies in primary care: qualitative evidence synthesis (protocol). Cochrane Database Syst Rev. 2013;2:CD010412.
17.
Whitaker R, Hendry M, Booth A, Carter B, Charles J, Craine N, Edwards RT, Lyons M, Noyes J, Pasterfield D, et al. Intervention Now To Eliminate Repeat Unintended Pregnancy in Teenagers (INTERUPT): a systematic review of intervention effectiveness and cost-effectiveness, qualitative and realist synthesis of implementation factors and user engagement. BMJ Open. 2014;4(4):e004733.CrossRefPubMedPubMedCentral
18.
Downe S, Finlayson K, Tuncalp O, Metin Gulmezoglu A. What matters to women: a systematic scoping review to identify the processes and outcomes of antenatal care provision that are important to healthy pregnant women. BJOG. 2016;123(4):529–39.CrossRefPubMed
19.
Odendaal WA, Goudge J, Griffiths F, Tomlinson M, Leon N, Daniels K. Healthcare workers’ perceptions and experience on using mHealth technologies to deliver primary healthcare services: qualitative evidence synthesis (protocol). Cochrane Database Syst Rev. 2015;(11):Art. No.: CD011942.
20.
Lewin S, Booth A, Glenton C, Munthe-Kaas HM, Rashidian A, Wainwright M, Bohren MA, Tuncalp Ö, Colvin CJ, Garside R, et al. Applying GRADE-CERQual to qualitative evidence synthesis findings: introduction to the series. Implement Sci. 2018;13(Suppl 1) https://​doi.​org/​10.​1186/​s13012-017-0688-3.
21.
Lavis JN, Oxman AD, Lewin S, Fretheim A. SUPPORT Tools for evidence-informed health Policymaking (STP). Health Res Policy Syst/BioMed Central. 2009;7(Suppl 1):I1.CrossRef
22.
Bambra C. Real world reviews: a beginner’s guide to undertaking systematic reviews of public health policy interventions. J Epidemiol Community Health. 2011;65(1):14–9.CrossRefPubMed
23.
Hannes K, Harden A. Multi-context versus context-specific qualitative evidence syntheses: combining the best of both. Res Synth Methods. 2011;2(4):271–8.CrossRefPubMed
24.
25.
Gardner B, Whittington C, McAteer J, Eccles MP, Michie S. Using theory to synthesise evidence from behaviour change interventions: the example of audit and feedback. Soc Sci Med. 2010;70(10):1618–25.CrossRefPubMed
26.
Main C, Thomas S, Ogilvie D, Stirk L, Petticrew M, Whitehead M, Sowden A. Population tobacco control interventions and their effects on social inequalities in smoking: placing an equity lens on existing systematic reviews. BMC Public Health. 2008;8:178.CrossRefPubMedPubMedCentral
27.
Noyes J, Hendry M, Booth A, Chandler J, Lewin S, Glenton C, Garside R. Current use was established and Cochrane guidance on selection of social theories for systematic reviews of complex interventions was developed. J Clin Epidemiol. 2016;75:78–92.CrossRefPubMed
28.
Barroso J, Sandelowski M, Voils CI. Research results have expiration dates: ensuring timely systematic reviews. J Eval Clin Pract. 2006;12(4):454–62.CrossRefPubMedPubMedCentral
29.
Evidence for Policy and Practice Information and Co-ordinating Centre. EPPI-Centre Methods for Conducting Systematic Reviews. London: University of London; 2007.
30.
Toews I, Booth A, Berg RC, Lewin S, Glenton C, Munthe-Kaas HM, Noyes J, Schroter S, Meerpohl JJ. Further exploration of dissemination bias in qualitative research required to facilitate assessment within qualitative evidence syntheses. J Clin Epidemiol. 2017;88:133–9.
31.
Toews I, Glenton C, Lewin S, Berg RC, Noyes J, Booth A, Marusic A, Malicki M, Munte-Kaas HM, Meerpohl JJ. Extent, awareness and perception of dissemination bias in qualitative research: an explorative survey. PLoS One. 2016;11(8):e0159290.CrossRefPubMedPubMedCentral
32.
Booth A, Lewin S, Glenton C, Munthe-Kaas HM, Meerpohl J, Rees R, Noyes J, Rashidian A, Berg R, Nyakango B, et al. Applying GRADE-CERQual to qualitative evidence synthesis findings––paper 7: understanding the potential impacts of dissemination bias. Implement Sci. 2018;13(Suppl 1) https://​doi.​org/​10.​1186/​s13012-017-0694-5.
33.
Kavanagh J, Oliver S, Lorenc T. Reflections on developing and using PROGRESS-Plus. Equity Update. 2008;2(1):1–3.
34.
Burford BJ, Welch V, Waters E, Tugwell P, Moher D, O'Neill J, Koehlmoos T, Petticrew M. Testing the PRISMA-Equity 2012 reporting guideline: the perspectives of systematic review authors. PLoS One. 2013;8(10):e75122.CrossRefPubMedPubMedCentral
35.
Pfadenhauer LM, Mozygemba K, Gerhardus A, Hofmann B, Booth A, Lysdahl KB, Tummers M, Burns J, Rehfuess EA. Context and implementation: a concept analysis towards conceptual maturity. Z Evid Fortbild Qual Gesundhwes. 2015;109(2):103–14.CrossRefPubMed
36.
Lewin S, Hendry M, Chandler J, Oxman AD, Michie S, Shepperd S, Reeves BC, Tugwell P, Hannes K, Rehfuess EA, et al. Assessing the complexity of interventions within systematic reviews: development, content and use of a new tool (iCAT_SR). BMC Med Res Methodol. 2017;17(1):76.CrossRefPubMedPubMedCentral
37.
Hoffmann TC, Glasziou PP, Boutron I, Milne R, Perera R, Moher D, Altman DG, Barbour V, Macdonald H, Johnston M, et al. Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide. BMJ. 2014;348:g1687.CrossRefPubMed
Metadata
Title
Applying GRADE-CERQual to qualitative evidence synthesis findings–paper 6: how to assess relevance of the data
Authors
Jane Noyes
Andrew Booth
Simon Lewin
Benedicte Carlsen
Claire Glenton
Christopher J. Colvin
Ruth Garside
Meghan A. Bohren
Arash Rashidian
Megan Wainwright
Özge Tunςalp
Jacqueline Chandler
Signe Flottorp
Tomas Pantoja
Joseph D. Tucker
Heather Munthe-Kaas
Publication date
01-01-2018
Publisher
BioMed Central
Published in
Implementation Science / Issue Special Issue 1/2018
Electronic ISSN: 1748-5908
DOI
https://doi.org/10.1186/s13012-017-0693-6

Other articles of this Special Issue 1/2018

Implementation Science 1/2018 Go to the issue

Method

Applying GRADE-CERQual to qualitative evidence synthesis findings—paper 5: how to assess adequacy of data

Method

Applying GRADE-CERQual to qualitative evidence synthesis findings—paper 2: how to make an overall CERQual assessment of confidence and create a Summary of Qualitative Findings table

Introduction

Applying GRADE-CERQual to qualitative evidence synthesis findings: introduction to the series

Method

Applying GRADE-CERQual to qualitative evidence synthesis findings—paper 4: how to assess coherence

Method

Applying GRADE-CERQual to qualitative evidence synthesis findings—paper 3: how to assess methodological limitations

Method

Applying GRADE-CERQual to qualitative evidence synthesis findings–paper 7: understanding the potential impacts of dissemination bias