Skip to main content
Top
Published in: Systematic Reviews 1/2012

Open Access 01-12-2012 | Commentary

Clarifying differences between review designs and methods

Authors: David Gough, James Thomas, Sandy Oliver

Published in: Systematic Reviews | Issue 1/2012

Login to get access

Abstract

This paper argues that the current proliferation of types of systematic reviews creates challenges for the terminology for describing such reviews. Terminology is necessary for planning, describing, appraising, and using reviews, building infrastructure to enable the conduct and use of reviews, and for further developing review methodology. There is insufficient consensus on terminology for a typology of reviews to be produced and any such attempt is likely to be limited by the overlapping nature of the dimensions along which reviews vary. It is therefore proposed that the most useful strategy for the field is to develop terminology for the main dimensions of variation. Three such main dimensions are proposed: (1) aims and approaches (including what the review is aiming to achieve, the theoretical and ideological assumptions, and the use of theory and logics of aggregation and configuration in synthesis); (2) structure and components (including the number and type of mapping and synthesis components and how they relate); and (3) breadth and depth and the extent of ‘work done’ in addressing a research issue (including the breadth of review questions, the detail with which they are addressed, and the amount the review progresses a research agenda). This then provides an overarching strategy to encompass more detailed descriptions of methodology and may lead in time to a more overarching system of terminology for systematic reviews.
Appendix
Available only for authorised users
Literature
1.
go back to reference Cooper H, Hedges L: The Handbook of Research Synthesis. 1994, Russell Sage Foundation, New York Cooper H, Hedges L: The Handbook of Research Synthesis. 1994, Russell Sage Foundation, New York
2.
go back to reference Gough D: Dimensions of difference in evidence reviews (Overview; I. Questions, evidence and methods; II.Breadth and depth; III. Methodological approaches; IV. Quality and relevance appraisal; V. Communication, interpretation and application. Series of six posters presented at National Centre for Research Methods meeting, Manchester. January 2007, EPPI-Centre, London,http://eppi.ioe.ac.uk/cms/Default.aspx?tabid=1919, Gough D: Dimensions of difference in evidence reviews (Overview; I. Questions, evidence and methods; II.Breadth and depth; III. Methodological approaches; IV. Quality and relevance appraisal; V. Communication, interpretation and application. Series of six posters presented at National Centre for Research Methods meeting, Manchester. January 2007, EPPI-Centre, London,http://​eppi.​ioe.​ac.​uk/​cms/​Default.​aspx?​tabid=​1919,
3.
go back to reference Gough D, Thomas J: Commonality and diversity in reviews. Introduction to Systematic Reviews. Edited by: Gough D, Oliver S, Thomas J. 2012, Sage, London, 35-65. Gough D, Thomas J: Commonality and diversity in reviews. Introduction to Systematic Reviews. Edited by: Gough D, Oliver S, Thomas J. 2012, Sage, London, 35-65.
4.
go back to reference Chalmers I, Hedges L, Cooper H: A brief history of research synthesis. Eval Health Professions. 2002, 25: 12-37. 10.1177/0163278702025001003.CrossRef Chalmers I, Hedges L, Cooper H: A brief history of research synthesis. Eval Health Professions. 2002, 25: 12-37. 10.1177/0163278702025001003.CrossRef
5.
go back to reference Bohlin I: Formalising syntheses of medical knowledge: the rise of meta-analysis and systematic reviews. Perspect Sci. in press, in press Bohlin I: Formalising syntheses of medical knowledge: the rise of meta-analysis and systematic reviews. Perspect Sci. in press, in press
6.
go back to reference Noblit G: Hare RD: Meta-ethnography: synthesizing qualitative studies. 1988, Sage Publications, Newbury Park NY Noblit G: Hare RD: Meta-ethnography: synthesizing qualitative studies. 1988, Sage Publications, Newbury Park NY
7.
go back to reference Noyes J, Popay J, Pearson A, Hannes K, Booth A: Qualitative research and Cochrane reviews. Cochrane Handbook for Systematic Reviews of Interventions. Edited by: Higgins JPT, Green S. Version 5.1.0 (updated March 2011). The Cochrane Collaboration. www.cochrane-handbook.org Noyes J, Popay J, Pearson A, Hannes K, Booth A: Qualitative research and Cochrane reviews. Cochrane Handbook for Systematic Reviews of Interventions. Edited by: Higgins JPT, Green S. Version 5.1.0 (updated March 2011). The Cochrane Collaboration. www.​cochrane-handbook.​org
8.
go back to reference Dixon-Woods M, Cavers D, Agarwal S, Annandale E, Arthur A, Harvey J, Hsu R, Katbamna S, Olsen R, Smith L, Riley R, Sutton AJ: Conducting a critical interpretive synthesis of the literature on access to healthcare by vulnerable groups. BMC Med Res Methodol. 2006, 6: 35-10.1186/1471-2288-6-35.CrossRefPubMedPubMedCentral Dixon-Woods M, Cavers D, Agarwal S, Annandale E, Arthur A, Harvey J, Hsu R, Katbamna S, Olsen R, Smith L, Riley R, Sutton AJ: Conducting a critical interpretive synthesis of the literature on access to healthcare by vulnerable groups. BMC Med Res Methodol. 2006, 6: 35-10.1186/1471-2288-6-35.CrossRefPubMedPubMedCentral
9.
go back to reference Pawson R: Evidenced-based policy: a realist perspective. 2006, Sage, LondonCrossRef Pawson R: Evidenced-based policy: a realist perspective. 2006, Sage, LondonCrossRef
10.
go back to reference Shepperd S, Lewin S, Struas S, Clarke M, Eccles M, Fitzpatrick R, Wong G, Sheikh A: Can we systematically review studies that evaluate complex interventions?. PLoS Med. 2009, 6: 8-10.1371/journal.pmed.1000008.CrossRef Shepperd S, Lewin S, Struas S, Clarke M, Eccles M, Fitzpatrick R, Wong G, Sheikh A: Can we systematically review studies that evaluate complex interventions?. PLoS Med. 2009, 6: 8-10.1371/journal.pmed.1000008.CrossRef
11.
go back to reference Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O, Peacock R: Storylines of research in diffusion of innovation: a meta-narrative approach to systematic review. Soc Sci Med. 2005, 61: 417-430. 10.1016/j.socscimed.2004.12.001.CrossRefPubMed Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O, Peacock R: Storylines of research in diffusion of innovation: a meta-narrative approach to systematic review. Soc Sci Med. 2005, 61: 417-430. 10.1016/j.socscimed.2004.12.001.CrossRefPubMed
12.
go back to reference Barnett-Page E, Thomas J: Methods for the synthesis of qualitative research: a critical review. BMC Med Res Methodol. 2009, 9: 59-10.1186/1471-2288-9-59.CrossRefPubMedPubMedCentral Barnett-Page E, Thomas J: Methods for the synthesis of qualitative research: a critical review. BMC Med Res Methodol. 2009, 9: 59-10.1186/1471-2288-9-59.CrossRefPubMedPubMedCentral
13.
go back to reference Moher D, Liberati A, Tetzlaff J, Altman DG, The PRISMA Group: Preferred reporting items for systematic reviews and meta-analyses: the PRISMA Statement. PLoS Med. 2009, 6: 6-10.1371/journal.pmed.1000006.CrossRef Moher D, Liberati A, Tetzlaff J, Altman DG, The PRISMA Group: Preferred reporting items for systematic reviews and meta-analyses: the PRISMA Statement. PLoS Med. 2009, 6: 6-10.1371/journal.pmed.1000006.CrossRef
14.
go back to reference PLoS Medicine Editors: Best practice in systematic reviews: The importance of protocols and registration. PLoS Med. 2011, 8: 2- PLoS Medicine Editors: Best practice in systematic reviews: The importance of protocols and registration. PLoS Med. 2011, 8: 2-
15.
go back to reference Thomas G: Introduction: evidence and practice. Evidence-based Practice in Education. Edited by: Pring R, Thomas G. 2004, Open University Press, Buckingham, 44-62. Thomas G: Introduction: evidence and practice. Evidence-based Practice in Education. Edited by: Pring R, Thomas G. 2004, Open University Press, Buckingham, 44-62.
16.
go back to reference Gough D, Oliver S, Newman M, Bird K: Transparency in planning, warranting and interpreting research. Teaching and Learning Research Briefing 78. 2009, Teaching and Learning Research Programme, London Gough D, Oliver S, Newman M, Bird K: Transparency in planning, warranting and interpreting research. Teaching and Learning Research Briefing 78. 2009, Teaching and Learning Research Programme, London
17.
go back to reference Strauss A, Corbin J: Basics of qualitative research, grounded theory procedures and techniques. 1990, Sage, London Strauss A, Corbin J: Basics of qualitative research, grounded theory procedures and techniques. 1990, Sage, London
18.
go back to reference Miles M, Huberman A: Qualitative Data Analysis. 1994, Sage, London Miles M, Huberman A: Qualitative Data Analysis. 1994, Sage, London
19.
go back to reference Voils CI, Sandelowski M, Barroso J, Hasselblad V: Making sense of qualitative and quantitative findings in mixed research synthesis studies. Field Methods. 2008, 20: 3-25. 10.1177/1525822X07307463.CrossRefPubMedPubMedCentral Voils CI, Sandelowski M, Barroso J, Hasselblad V: Making sense of qualitative and quantitative findings in mixed research synthesis studies. Field Methods. 2008, 20: 3-25. 10.1177/1525822X07307463.CrossRefPubMedPubMedCentral
20.
go back to reference Sandelowski M, Voils CJ, Leeman J, Crandlee JL: Mapping the Mixed Methods-Mixed Research Synthesis Terrain. Journal of Mixed Methods Research. 2011, 10.1177/1558689811427913. Sandelowski M, Voils CJ, Leeman J, Crandlee JL: Mapping the Mixed Methods-Mixed Research Synthesis Terrain. Journal of Mixed Methods Research. 2011, 10.1177/1558689811427913.
21.
go back to reference Pawson R, Boaz A, Grayson L, Long A, Barnes C: Types and Quality of Knowledge in Social Care. 2003, Social Care Institute for Excellence, London Pawson R, Boaz A, Grayson L, Long A, Barnes C: Types and Quality of Knowledge in Social Care. 2003, Social Care Institute for Excellence, London
22.
go back to reference Oancea A, Furlong J: Expressions of excellence and the assessment of applied and practice-based research. Res Pap Educ. 2007, 22: 119-137. 10.1080/02671520701296056.CrossRef Oancea A, Furlong J: Expressions of excellence and the assessment of applied and practice-based research. Res Pap Educ. 2007, 22: 119-137. 10.1080/02671520701296056.CrossRef
23.
go back to reference Harden A, Gough D: Quality and relevance appraisal. Introduction to Systematic Reviews. Edited by: Gough D, Oliver S, Thomas J. 2012, Sage, London, 153-178. Harden A, Gough D: Quality and relevance appraisal. Introduction to Systematic Reviews. Edited by: Gough D, Oliver S, Thomas J. 2012, Sage, London, 153-178.
24.
go back to reference Thomas J, Harden A: Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Med Res Methodol. 2008, 8: 45-10.1186/1471-2288-8-45.CrossRefPubMedPubMedCentral Thomas J, Harden A: Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Med Res Methodol. 2008, 8: 45-10.1186/1471-2288-8-45.CrossRefPubMedPubMedCentral
25.
go back to reference Oliver S, Rees RW, Clarke-Jones L, Milne R, Oakley AR, Gabbay J, Stein K, Buchanan P, Gyte G: A multidimensional conceptual framework for analyzing public involvement in health services research. Heal Expect. 2008, 11: 72-84. 10.1111/j.1369-7625.2007.00476.x.CrossRef Oliver S, Rees RW, Clarke-Jones L, Milne R, Oakley AR, Gabbay J, Stein K, Buchanan P, Gyte G: A multidimensional conceptual framework for analyzing public involvement in health services research. Heal Expect. 2008, 11: 72-84. 10.1111/j.1369-7625.2007.00476.x.CrossRef
26.
go back to reference Carroll C, Booth A, Cooper K: A worked example of “best fit” framework synthesis: a systematic review of views concerning the taking of some potential chemopreventive agents. BMC Med Res Methodol. 2011, 11: 29-10.1186/1471-2288-11-29.CrossRefPubMedPubMedCentral Carroll C, Booth A, Cooper K: A worked example of “best fit” framework synthesis: a systematic review of views concerning the taking of some potential chemopreventive agents. BMC Med Res Methodol. 2011, 11: 29-10.1186/1471-2288-11-29.CrossRefPubMedPubMedCentral
27.
go back to reference Rees R, Oliver S: Stakeholder perspectives and participation in reviews. Introduction to Systematic Reviews. Edited by: Gough D, Oliver S, Thomas J. 2012, Sage, London, 17-34. Rees R, Oliver S: Stakeholder perspectives and participation in reviews. Introduction to Systematic Reviews. Edited by: Gough D, Oliver S, Thomas J. 2012, Sage, London, 17-34.
28.
go back to reference Oliver S, Dickson K, Newman M: Getting started with a review. Introduction to Systematic Reviews. Edited by: Gough D, Oliver S, Thomas J. 2012, Sage, London, 66-82. Oliver S, Dickson K, Newman M: Getting started with a review. Introduction to Systematic Reviews. Edited by: Gough D, Oliver S, Thomas J. 2012, Sage, London, 66-82.
29.
go back to reference Spencer L, Ritchie J, Lewis J, Dillon L: Quality in Qualitative Evaluation: a Framework for Assessing Research Evidence. 2003, Government Chief Social Researcher’s Office, London Spencer L, Ritchie J, Lewis J, Dillon L: Quality in Qualitative Evaluation: a Framework for Assessing Research Evidence. 2003, Government Chief Social Researcher’s Office, London
30.
go back to reference Weiss C: The many meanings of research utilisation. Public Adm Rev. 1979, 29: 426-431.CrossRef Weiss C: The many meanings of research utilisation. Public Adm Rev. 1979, 29: 426-431.CrossRef
31.
go back to reference Peersman G: A Descriptive Mapping of Health Promotion Studies in Young People EPPI Research Report. 1996, EPI-Centre, London Peersman G: A Descriptive Mapping of Health Promotion Studies in Young People EPPI Research Report. 1996, EPI-Centre, London
32.
go back to reference Arksey H, O’Malley L: Scoping Studies: towards a methodological framework. Int J Soc Res Methodol. 2005, 8: 19-32. 10.1080/1364557032000119616.CrossRef Arksey H, O’Malley L: Scoping Studies: towards a methodological framework. Int J Soc Res Methodol. 2005, 8: 19-32. 10.1080/1364557032000119616.CrossRef
34.
go back to reference Thomas J, Harden A, Oakley A, Oliver S, Sutcliffe K, Rees R, Brunton G, Kavanagh J: Integrating qualitative research with trials in systematic reviews: an example from public health. Brit Med J. 2004, 328: 1010-1012. 10.1136/bmj.328.7446.1010.CrossRefPubMedPubMedCentral Thomas J, Harden A, Oakley A, Oliver S, Sutcliffe K, Rees R, Brunton G, Kavanagh J: Integrating qualitative research with trials in systematic reviews: an example from public health. Brit Med J. 2004, 328: 1010-1012. 10.1136/bmj.328.7446.1010.CrossRefPubMedPubMedCentral
35.
go back to reference Harden A, Thomas J: Mixed methods and systematic reviews: examples and emerging issues. Handbook of Mixed Methods in the Social and Behavioral Sciences. Edited by: Tashakkori A, Teddlie C. 2010, Sage, London, 749-774. 2CrossRef Harden A, Thomas J: Mixed methods and systematic reviews: examples and emerging issues. Handbook of Mixed Methods in the Social and Behavioral Sciences. Edited by: Tashakkori A, Teddlie C. 2010, Sage, London, 749-774. 2CrossRef
36.
go back to reference Leontien M, van der Knaap , Leeuw FL, Bogaerts S, Laura TJ: Nijssen Combining campbell standards and the realist evaluation approach: the best of two worlds?. J Eval. 2008, 29: 48-57. 10.1177/1098214007313024.CrossRef Leontien M, van der Knaap , Leeuw FL, Bogaerts S, Laura TJ: Nijssen Combining campbell standards and the realist evaluation approach: the best of two worlds?. J Eval. 2008, 29: 48-57. 10.1177/1098214007313024.CrossRef
37.
go back to reference Smith V, Devane D, Begley CM, Clarke M: Methodology in conducting a systematic review of systematic reviews of healthcare interventions. BMC Med Res Methodol. 2011, 11: 15-10.1186/1471-2288-11-15.CrossRefPubMedPubMedCentral Smith V, Devane D, Begley CM, Clarke M: Methodology in conducting a systematic review of systematic reviews of healthcare interventions. BMC Med Res Methodol. 2011, 11: 15-10.1186/1471-2288-11-15.CrossRefPubMedPubMedCentral
38.
go back to reference Oliver S, Bagnall AM, Thomas J, Shepherd J, Sowden A, White I, Dinnes J, Rees R, Colquitt J, Oliver K, Garrett Z: RCTs for policy interventions: a review of reviews and meta-regression. Health Technol Assess. 2010, 14: 16-CrossRef Oliver S, Bagnall AM, Thomas J, Shepherd J, Sowden A, White I, Dinnes J, Rees R, Colquitt J, Oliver K, Garrett Z: RCTs for policy interventions: a review of reviews and meta-regression. Health Technol Assess. 2010, 14: 16-CrossRef
39.
go back to reference Scriven M: An introduction to meta-evaluation. Educational Products Report. 1969, 2: 36-38. Scriven M: An introduction to meta-evaluation. Educational Products Report. 1969, 2: 36-38.
40.
go back to reference Carroll C, Patterson M, Wood S, Booth A, Rick J, Balain S: A conceptual framework for implementation fidelity. Implement Sci. 2007, 2: 40-10.1186/1748-5908-2-40.CrossRefPubMedPubMedCentral Carroll C, Patterson M, Wood S, Booth A, Rick J, Balain S: A conceptual framework for implementation fidelity. Implement Sci. 2007, 2: 40-10.1186/1748-5908-2-40.CrossRefPubMedPubMedCentral
41.
42.
go back to reference Hattie J: Visible Learning: A Synthesis of Over 800 Meta-Analyses Relating to Achievement. 2008, Routledge, London Hattie J: Visible Learning: A Synthesis of Over 800 Meta-Analyses Relating to Achievement. 2008, Routledge, London
43.
go back to reference Cook TD, Cooper H, Cordray DS, Hartmann H, Hedges LV, Light RJ, Louis TA, Mosteller F: Meta-analysis for Explanation: A Casebook. 1992, Russell Sage Foundation, New York Cook TD, Cooper H, Cordray DS, Hartmann H, Hedges LV, Light RJ, Louis TA, Mosteller F: Meta-analysis for Explanation: A Casebook. 1992, Russell Sage Foundation, New York
44.
go back to reference Thompson SG, Sharp SJ: Explaining heterogeneity in meta-analysis: a comparison of methods. Stat Med. 1999, 18: 2693-2708. 10.1002/(SICI)1097-0258(19991030)18:20<2693::AID-SIM235>3.0.CO;2-V.CrossRefPubMed Thompson SG, Sharp SJ: Explaining heterogeneity in meta-analysis: a comparison of methods. Stat Med. 1999, 18: 2693-2708. 10.1002/(SICI)1097-0258(19991030)18:20<2693::AID-SIM235>3.0.CO;2-V.CrossRefPubMed
45.
go back to reference Stewart R, Oliver S: Making a difference with systematic reviews. Introduction to Systematic Reviews. Edited by: Gough D, Oliver S, Thomas J. 2012, Sage, London, 227-244. Stewart R, Oliver S: Making a difference with systematic reviews. Introduction to Systematic Reviews. Edited by: Gough D, Oliver S, Thomas J. 2012, Sage, London, 227-244.
47.
go back to reference Abrami PC, Borokhovski E, Bernard RM, Wade CA, Tamim R, Persson T, Surkes MA: Issues in conducting and disseminating brief reviews of evidence. Evidence & Policy: A Journal of Research, Debate and Practice. 2010, 6: 371-389. 10.1332/174426410X524866.CrossRef Abrami PC, Borokhovski E, Bernard RM, Wade CA, Tamim R, Persson T, Surkes MA: Issues in conducting and disseminating brief reviews of evidence. Evidence & Policy: A Journal of Research, Debate and Practice. 2010, 6: 371-389. 10.1332/174426410X524866.CrossRef
48.
go back to reference Brunton J, Thomas J: Information management in reviews. Introduction to Systematic Reviews. Edited by: Gough D, Oliver S, Thomas J. 2012, Sage, London, 83-106. Brunton J, Thomas J: Information management in reviews. Introduction to Systematic Reviews. Edited by: Gough D, Oliver S, Thomas J. 2012, Sage, London, 83-106.
Metadata
Title
Clarifying differences between review designs and methods
Authors
David Gough
James Thomas
Sandy Oliver
Publication date
01-12-2012
Publisher
BioMed Central
Published in
Systematic Reviews / Issue 1/2012
Electronic ISSN: 2046-4053
DOI
https://doi.org/10.1186/2046-4053-1-28

Other articles of this Issue 1/2012

Systematic Reviews 1/2012 Go to the issue