Skip to main content
Key Specialties
Cardiology Diabetology Endocrinology Gastroenterology Geriatrics and Gerontology Gynecology and Obstetrics Hematology Infectious Disease Internal Medicine Nephrology Neurology Oncology Pediatrics Respiratory Medicine Rheumatology Surgery
Congress Coverage
2024 ACC Congress 2023 ESMO Congress 2023 EASD Congress 2023 ERS Congress 2023 ESC Congress
Clinical Collections
Advances in Alzheimer’s

At a glance: The STEP trials

A round-up of the STEP phase 3 clinical trials evaluating semaglutide for weight loss in people with overweight or obesity.
ADVANCED SEARCH
Log in
Top
Systematic Reviews
Published in: Systematic Reviews 1/2014

Open Access 01-12-2014 | Research

The impact of Cochrane Systematic Reviews: a mixed method evaluation of outputs from Cochrane Review Groups supported by the UK National Institute for Health Research

Authors: Frances Bunn, Daksha Trivedi, Phil Alderson, Laura Hamilton, Alice Martin, Steve Iliffe

Published in: Systematic Reviews | Issue 1/2014

Login to get access

Abstract

Background

There has been a growing emphasis on evidence-informed decision-making in health care. Systematic reviews, such as those produced by the Cochrane Collaboration, have been a key component of this movement. The UK National Institute for Health Research (NIHR) Systematic Review Programme currently supports 20 Cochrane Review Groups (CRGs). The aim of this study was to identify the impacts of Cochrane reviews published by NIHR-funded CRGs during the years 2007–2011.

Methods

We sent questionnaires to CRGs and review authors, interviewed guideline developers and used bibliometrics and documentary review to get an overview of CRG impact and to evaluate the impact of a sample of 60 Cochrane reviews. We used a framework with four categories (knowledge production, research targeting, informing policy development and impact on practice/services).

Results

A total of 1,502 new and updated reviews were produced by the 20 NIHR-funded CRGs between 2007 and 2011. The clearest impacts were on policy with a total of 483 systematic reviews cited in 247 sets of guidance: 62 were international, 175 national (87 from the UK) and 10 local. Review authors and CRGs provided some examples of impact on practice or services, for example, safer use of medication, the identification of new effective drugs or treatments and potential economic benefits through the reduction in the use of unproven or unnecessary procedures. However, such impacts are difficult to objectively document, and the majority of reviewers were unsure if their review had produced specific impacts. Qualitative data suggested that Cochrane reviews often play an instrumental role in informing guidance, although a poor fit with guideline scope or methods, reviews being out of date and a lack of communication between CRGs and guideline developers were barriers to their use.

Conclusions

Health and economic impacts of research are generally difficult to measure. We found that to be the case with this evaluation. Impacts on knowledge production and clinical guidance were easier to identify and substantiate than those on clinical practice. Questions remain about how we define and measure impact, and more work is needed to develop suitable methods for impact analysis.
Appendix
Available only for authorised users
Literature
1.
2.
Dawes M, Summerskill W, Glasziou P, Cartabellotta A, Martin J, Hopayian K, Porzsolt F, Burls A, Osborne J: Sicily statement on evidence-based practice. BMC Med Educ. 2005, 5 (1): 1-10.1186/1472-6920-5-1.CrossRefPubMedPubMedCentral
3.
Sackett DL, Rosenberg WM, Gray JA, Haynes RB, Richardson WS: Evidence based medicine: what it is and what it isn’t. BMJ. 1996, 312 (7023): 71-72. 10.1136/bmj.312.7023.71.CrossRefPubMedPubMedCentral
4.
5.
HEFC: Research Excellence Framework: Second Consultation on the Assessment and Funding of Research. 2009, London: Higher Education Funding Council for England
6.
Sheldon TA: Making evidence synthesis more useful for management and policy-making. J Health Serv Res Policy. 2005, 10 (Suppl 1): 1-5. 10.1258/1355819054308521.CrossRefPubMed
7.
8.
Lomas J: Using ‘linkage and exchange’ to move research into policy at a Canadian foundation. Health Aff (Millwood). 2000, 19 (3): 236-240. 10.1377/hlthaff.19.3.236.CrossRef
9.
Weiss C: Using research in the policy process: potential and constraints. Pol Stud J. 1976, 4: 224-228. 10.1111/j.1541-0072.1976.tb01557.x.CrossRef
10.
Olsen O, Middleton P, Ezzo J, Gotzsche PC, Hadhazy V, Herxheimer A, Kleijnen J, McIntosh H: Quality of Cochrane reviews: assessment of sample from 1998. BMJ. 2001, 323 (7317): 829-832. 10.1136/bmj.323.7317.829.CrossRefPubMedPubMedCentral
11.
Moseley AM, Elkins MR, Herbert RD, Maher CG, Sherrington C: Cochrane reviews used more rigorous methods than non-Cochrane reviews: survey of systematic reviews in physiotherapy. J Clin Epidemiol. 2009, 62 (10): 1021-1030. 10.1016/j.jclinepi.2008.09.018.CrossRefPubMed
12.
Jorgensen A, Hilden J, Gotzsche P: Cochrane reviews compared with industry supported meta-analyses and other meta-analyses of the same drugs: systematic review. BMJ. 2006, 333 (7572): 782-10.1136/bmj.38973.444699.0B.CrossRefPubMedPubMedCentral
13.
Huberman M: Linking the practitioner and researcher communities for school improvement. Sch Eff Sch Improv. 1993, 4: 1-16. 10.1080/0924345930040101.CrossRef
14.
Nutley S, Percy-Smith J, Solesbury W: Models of Research Impact: A Cross-Sector Review of Literature and Practice. 2003, London: Learning and Skills Research Centre
15.
Estabrooks CE: The conceptual structure of research utilization. Res Nurs Health. 1999, 22: 203-216. 10.1002/(SICI)1098-240X(199906)22:3<203::AID-NUR3>3.0.CO;2-9.CrossRefPubMed
16.
Buxton M, Hanney S: How can payback from health services research be assessed?. J Health Serv Res Policy. 1996, 1 (1): 35-43.PubMed
17.
Hanney S, Packwood T, Buxton M: Evaluating the benefits from health research and development centres: a categorization, a model and examples of application. Evaluation: Int J Theory Res Pract. 2000, 6 (2): 137-160.CrossRef
18.
Kuruvilla S, Mays N, Pleasant A, Walt G: Describing the impact of health research: a Research Impact Framework. BMC Health Serv Res. 2006, 6: 134-10.1186/1472-6963-6-134.CrossRefPubMedPubMedCentral
19.
Kuruvilla S, Mays N, Walt G: Describing the impact of health services and policy research. J Health Serv Res Policy. 2007, 12 (Suppl 1): S1-10.1258/135581907780318400. -23-31CrossRefPubMed
20.
Boaz A, Fitzpatrick S, Shaw B: Assessing the Impact of Research on Policy: A Review of the Literature for a Project on Bridging Research and Policy Through Outcome Evaluation. 2008, London: University of London
21.
Hanney SR, Grant J, Wooding S, Buxton MJ: An assessment of the impact of NHS Health Technology Assessment Programme. Health Technol Assess. 2007, 11 (53): 1-180.CrossRef
22.
Lavis J, Ross S, McLeod C, Gildiner A: Measuring the impact of health research. J Health Serv Res Policy. 2003, 8 (3): 165-170. 10.1258/135581903322029520.CrossRefPubMed
23.
Hanney S, Grant J, Wooding S, Buxton M: Proposed methods for reviewing the outcomes of health research: the impact of funding by the UK’s ‘Arthritis Research Campaign’. Health Res Pol Syst. 2004, 2 (1): 4-10.1186/1478-4505-2-4.CrossRef
24.
Coffey A, Atkinson P: Making Sense of Qualitative Data: Complementary Research Strategies. 1996, New York: Sage
25.
26.
27.
McInnes E, Bell-Syer SE, Dumville JC, Legood R, Cullum NA: Support surfaces for pressure ulcer prevention. Cochrane Database Syst Rev. 2008, 4: CD001735-PubMed
28.
Burnett J, Clift M, Sprange K: Buyers Guide: Pressure-Redistribution Mattresses and Overlays CEP08042. 2008, London: NHS Purchasing and Supply Agency, Centre for Evidence-Based Purchasing
29.
Cates CJ, Lasserson TJ: Regular treatment with formoterol and an inhaled corticosteroid versus regular treatment with salmeterol and an inhaled corticosteroid for chronic asthma: serious adverse events. Cochrane Database Syst Rev. 2009, 1: CD007694-
30.
Cates CJ, Lasserson TJ, Jaeschke R: Regular treatment with salmeterol and inhaled steroids for chronic asthma: serious adverse events. Cochrane Database Syst Rev. 2009, 3: CD006922-PubMed
31.
Cates CJ, Cates MJ: Regular treatment with salmeterol for chronic asthma: serious adverse events. Cochrane Database Syst Rev. 2008, 3: CD006363-PubMed
32.
Perel P, Roberts I: Colloids versus crystalloids for fluid resuscitation in critically ill patients. Cochrane Database Syst Rev. 2011, 3: CD000567-PubMed
33.
Perel P, Roberts I: Colloids versus crystalloids for fluid resuscitation in critically ill patients. Cochrane Database Syst Rev. 2007, 4: CD000567-PubMed
34.
35.
Lockhart P, Daly F, Pitkethly M, Comerford N, Sullivan F: Antiviral treatment for Bell’s palsy (idiopathic facial paralysis). Cochrane Database Syst Rev. 2009, 4: CD001869-PubMed
36.
Roberts I, Shakur H, Ker K, Coats T: Antifibrinolytic drugs for acute traumatic injury. Cochrane Database Syst Rev. 2011, 1: CD004896-PubMed
37.
Roberts I, Shakur H, Ker K, Coats T: Antifibrinolytic drugs for acute traumatic injury. Cochrane Database Syst Rev. 2009, 12: CD004896-
38.
Gillespie LD, Robertson MC, Gillespie WJ, Lamb SE, Gates S, Cumming RG, Rowe BH: Interventions for preventing falls in older people living in the community. Cochrane Database Syst Rev. 2009, 2: CD007146-PubMed
39.
Cameron ID, Murray GR, Gillespie LD, Robertson MC, Hill KD, Cumming RG, Kerse N: Interventions for preventing falls in older people in nursing care facilities and hospitals. Cochrane Database Syst Rev. 2010, 1: CD005465-PubMed
40.
Meho LI: The rise and rise of citation analysis. Phys World. 2007, 20 (1): 32-36.CrossRef
41.
Bunn F: The impact of systematic reviews on health care policy in England. 2010, PhD dissertation: University of Hertfordshire
42.
43.
Falagas ME, Pitsouni EI, Malietzis GA, Pappas G: Comparison of PubMed, Scopus, Web of Science, and Google Scholar: strengths and weaknesses. FASEB J. 2008, 22 (2): 338-342.CrossRefPubMed
44.
Cozzens SE: The knowledge pool: measurement challenges in evaluating fundamental research programs. Eval Program Plann. 1997, 20 (1): 77-89. 10.1016/S0149-7189(96)00038-9.CrossRef
45.
Comroe J, Dripps R: Scientific basis for the support of biomedical science. Biomedical Scientists and Public Policy. Edited by: Hugh Fudenberg H, Melnick VL. 1978, US: Springer, 15-33.CrossRef
46.
Becker L, Oxman AD: Chapter 22: overview of reviews. Cochrane Handbook for Systematic Reviews of Interventions Version 5.10 [updated March 2011]. Edited by: Higgins J, Churchill R, Cumpston M, Chandler J. 2011, Oxford: The Cochrane Collaboration, [http://​www.​cochrane-handbook.​org],
47.
Peckham S, Willmott M, Allen P, Anderson S, Goodwin N: Assessing the impact of the NHS Service Delivery and Organisation Research and Development Programme. Evid Pol: J Res Debate Pract. 2008, 4: 313-330. 10.1332/174426408X338857.CrossRef
48.
Bunn F, Sworn K: Strategies to promote the impact of systematic reviews on healthcare policy: a systematic review of the literature. Evid Pol: J Res Debate Pract. 2011, 7 (4): 403-428. 10.1332/174426411X603434.CrossRef
49.
Oliver K, Innvar S, Lorenc T, Woodman J, Thomas J: A systematic review of barriers to and facilitators of the use of evidence by policymakers. BMC Health Serv Res. 2014, 14: 2-10.1186/1472-6963-14-2.CrossRefPubMedPubMedCentral
50.
51.
Ismail S, Nason E, Marjanovic S, Grant J: Bibliometrics as a Tool for Supporting Prospective R&D Decision-Making in the Health Sciences: Strengths, Weaknesses and Options for Future Development. 2009, RAND: Santa Monica
52.
van Leeuwen TN, Moed HF, Reedijk J: Critical comments on Institute for Scientific Information impact factors: a sample of inorganic molecular chemistry journals. J Inf Sci. 1999, 25 (6): 489-498. 10.1177/016555159902500605.CrossRef
53.
Sabatier PA: An advocacy coalition framework of policy change and the role of policy-oriented learning therein. Pol Sci. 1988, 21 (2–4): 129-168.CrossRef
54.
Sheldon TA, Cullum N, Dawson D, Lankshear A, Lowson K, Watt I, West P, Wright D, Wright J: What’s the evidence that NICE guidance has been implemented? Results from a national evaluation using time series analysis, audit of patients’ notes, and interviews. BMJ. 2004, 329 (7473): 999-10.1136/bmj.329.7473.999.CrossRefPubMedPubMedCentral
55.
Ryan J, Piercy J, James P: Assessment of NICE guidance on two surgical procedures. Lancet. 2004, 363 (9420): 1525-1526. 10.1016/S0140-6736(04)16180-1.CrossRefPubMed
56.
Thomas L, Cullum N, McColl E, Rousseau N, Soutter J, Steen N: Guidelines in professions allied to medicine. Cochrane Database Syst Rev. 2000, 2: CD000349-PubMed
57.
Grimshaw JM, Russell IT: Effect of clinical guidelines on medical practice: a systematic review of rigorous evaluations. Lancet. 1993, 342 (8883): 1317-1322. 10.1016/0140-6736(93)92244-N.CrossRefPubMed
58.
Grimshaw J, Freemantle N, Wallace S, Russell I, Hurwitz B, Watt I, Long A, Sheldon T: Developing and implementing clinical practice guidelines. Qual Health Care. 1995, 4 (1): 55-64. 10.1136/qshc.4.1.55.CrossRefPubMedPubMedCentral
59.
Martin HT, Mark JD, Jamie MF, Corey GR, Vivian HC, David JC, Peter BL: Impact of the NICE guideline recommending cessation of antibiotic prophylaxis for prevention of infective endocarditis: before and after study. BMJ. 2011, 342: d2392-10.1136/bmj.d2392.CrossRef
60.
Grimshaw J, Thomas R, MacLennan G, Fraser C, Ramsay C, Vale L, Whitty P, Eccles M, Matowe L, Shirran L: Effectiveness and efficiency of guideline dissemination and implementation strategies. Health Technol Assess. 2004, 8 (6): 1-72.CrossRef
61.
Metadata
Title
The impact of Cochrane Systematic Reviews: a mixed method evaluation of outputs from Cochrane Review Groups supported by the UK National Institute for Health Research
Authors
Frances Bunn
Daksha Trivedi
Phil Alderson
Laura Hamilton
Alice Martin
Steve Iliffe
Publication date
01-12-2014
Publisher
BioMed Central
Published in
Systematic Reviews / Issue 1/2014
Electronic ISSN: 2046-4053
DOI
https://doi.org/10.1186/2046-4053-3-125

Other articles of this Issue 1/2014

Systematic Reviews 1/2014 Go to the issue

Protocol

Instruments to measure patient experience of health care quality in hospitals: a systematic review protocol

Protocol

Clinical manifestations of tension pneumothorax: protocol for a systematic review and meta-analysis

Protocol

Comparative safety of anti-epileptic drugs among infants and children exposed in utero or during breastfeeding: protocol for a systematic review and network meta-analysis

Protocol

The therapeutic potential of C-peptide in kidney disease: a protocol for a systematic review and meta-analysis

Research

Do randomized clinical trials with inadequate blinding report enhanced placebo effects for intervention groups and nocebo effects for placebo groups?

Protocol

Homeopathy for allergic rhinitis: protocol for a systematic review