Top

Published in:

Open Access 01-12-2018 | Research

Enhancing the uptake of systematic reviews of effects: what is the best format for health care managers and policy-makers? A mixed-methods study

Authors: Christine Marquez, Alekhya Mascarenhas Johnson, Sabrina Jassemi, Jamie Park, Julia E. Moore, Caroline Blaine, Gertrude Bourdon, Mark Chignell, Moriah E. Ellen, Jacques Fortin, Ian D. Graham, Anne Hayes, Jemila Hamid, Brenda Hemmelgarn, Michael Hillmer, Bev Holmes, Jayna Holroyd-Leduc, Linda Hubert, Brian Hutton, Monika Kastner, John N. Lavis, Karen Michell, David Moher, Mathieu Ouimet, Laure Perrier, Andrea Proctor, Thomas Noseworthy, Victoria Schuckel, Sharlene Stayberg, Marcello Tonelli, Andrea C. Tricco, Sharon E. Straus

Published in: Implementation Science | Issue 1/2018

Abstract

Background

Systematic reviews are infrequently used by health care managers (HCMs) and policy-makers (PMs) in decision-making. HCMs and PMs co-developed and tested novel systematic review of effects formats to increase their use.

Methods

A three-phased approach was used to evaluate the determinants to uptake of systematic reviews of effects and the usability of an innovative and a traditional systematic review of effects format. In phase 1, survey and interviews were conducted with HCMs and PMs in four Canadian provinces to determine perceptions of a traditional systematic review format. In phase 2, systematic review format prototypes were created by HCMs and PMs via Conceptboard©. In phase 3, prototypes underwent usability testing by HCMs and PMs.

Results

Two hundred two participants (80 HCMs, 122 PMs) completed the phase 1 survey. Respondents reported that inadequate format (Mdn = 4; IQR = 4; range = 1–7) and content (Mdn = 4; IQR = 3; range = 1–7) influenced their use of systematic reviews. Most respondents (76%; n = 136/180) reported they would be more likely to use systematic reviews if the format was modified. Findings from 11 interviews (5 HCMs, 6 PMs) revealed that participants preferred systematic reviews of effects that were easy to access and read and provided more information on intervention effectiveness and less information on review methodology. The mean System Usability Scale (SUS) score was 55.7 (standard deviation [SD] 17.2) for the traditional format; a SUS score < 68 is below average usability. In phase 2, 14 HCMs and 20 PMs co-created prototypes, one for HCMs and one for PMs. HCMs preferred a traditional information order (i.e., methods, study flow diagram, forest plots) whereas PMs preferred an alternative order (i.e., background and key messages on one page; methods and limitations on another). In phase 3, the prototypes underwent usability testing with 5 HCMs and 7 PMs, 11 out of 12 participants co-created the prototypes (mean SUS score 86 [SD 9.3]).

Conclusions

HCMs and PMs co-created prototypes for systematic review of effects formats based on their needs. The prototypes will be compared to a traditional format in a randomized trial.

Available only for authorised users

HCMs were identified through the Strategic Clinical Networks (SCNs) in Alberta, Council of Academic Hospitals of Ontario (CAHO) in Ontario, and Regional Health Authorities (RHA) and Association Québecoise d’éstablissements de Santé and de Services Sociaux (AQESSS) in Quebec, and the regional health authorities via Michael Smith Foundation for Health Research in British Columbia. PMs were identified through Alberta Health, the Ontario Ministry of Health and Long-Term Care, the Quebec Ministry of Health and Social Services, and the British Columbia Ministry of Health

Straus SE, Tetroe J, Graham ID, Zwarenstein M, Bhattacharyya O, Shepperd S. Monitoring use of knowledge and evaluating outcomes. CMAJ. 2010;182(2):E94–8. https://doi.org/10.1503/cmaj.081335.CrossRefPubMedPubMedCentral

Macleod MR, Michie S, Roberts I, Dirnagl U, Chalmers I, Ioannidis JP, Al-Shahi Salman R, Chan AW, Glasziou P. Biomedical research: increasing value, reducing waste. Lancet. 2014;383(9912):101–4. https://doi.org/10.1016/S0140-6736(13)62329-6.CrossRefPubMed

Brownlee S, Chalkidou K, Doust J, Elshaug AG, Glasziou P, Heath I, Nagpal S, Saini V, Srivastava D, Chalmers K, Korenstein D. Evidence for underuse of effective medical services around the world. Lancet. 2017;390(10090):169–77. https://doi.org/10.1016/s0140-6736(16)32585-5.CrossRefPubMed

Grol R. Successes and failures in the implementation of evidence-based guidelines for clinical practice. Med Care. 2001;39(8 Suppl 2):Ii46–54.PubMed

Bastian H, Glasziou P, Chalmers I. Seventy-five trials and eleven systematic reviews a day: how will we ever keep up? PLoS Med. 2010;7:e1000326. https://doi.org/10.1371/journal.pmed.1000326.CrossRefPubMedPubMedCentral

Cochrane Library About Cochrane Reviews. What is a systematic review? http://www.cochranelibrary.com/about/about-cochrane-systematic-reviews.html Accessed 30 July 2013.

Chalmers I, Glasziou P. Systematic reviews and research waste. Lancet. 2016;387(10014):122–3. https://doi.org/10.1016/s0140-6736(15)01353-7.CrossRefPubMed

Clarke M, Brice A, Chalmers I. Accumulating research: a systematic account of how cumulative meta-analyses would have provided knowledge, improved health, reduced harm and saved resources. PLoS One. 2014;9(7):e102670. https://doi.org/10.1371/journal.pone.0102670.CrossRefPubMedPubMedCentral

Petkovic J, Welch V, Jacob MH, Yoganathan M, Ayala AP, Cunningham H, Tugwell P. The effectiveness of evidence summaries on health policymakers and health system managers use of evidence from systematic reviews: a systematic review. Implement Sci. 2016;11(1):162. https://doi.org/10.1186/s13012-016-0530-3. CrossRefPubMedPubMedCentral

10.

Tricco AC, Cardoso R, Thomas SM, Motiwala S, Sullivan S, Kealey MR, Hemmelgarn B, Ouimet M, Hillmer MP, Perrier L, Shepperd S, Straus SE. Barriers and facilitators to uptake of systematic reviews by policy makers and health care managers: a scoping review. Implement Sci. 2016;11(1):4. https://doi.org/10.1186/s13012-016-0370-1. CrossRefPubMedPubMedCentral

11.

Wallace J, Byrne C, Clarke M. Improving the uptake of systematic reviews: a systematic review of intervention effectiveness and relevance. BMJ Open. 2014;4(10):e005834. https://doi.org/10.1136/bmjopen-2014-005834.CrossRefPubMedPubMedCentral

12.

Wallace J, Byrne C, Clarke M. Making evidence more wanted: a systematic review of facilitators to enhance the uptake of evidence from systematic reviews and meta-analyses. Int J Evid Based Healthc. 2012;10(4):338–46. https://doi.org/10.1111/j.1744-1609.2012.00288.x.CrossRefPubMed

13.

Oliver K, Innvar S, Lorenc T, Woodman J, Thomas J. A systematic review of barriers to and facilitators of the use of evidence by policymakers. BMC Health Serv Res. 2014;14:2. https://doi.org/10.1186/1472-6963-14-2.CrossRefPubMedPubMedCentral

14.

Murthy L, Shepperd S, Clarke MJ, Garner SE, Lavis JN, Perrier L, Roberts NW, Straus SE. Interventions to improve the use of systematic reviews in decision-making by health system managers, policy makers and clinicians. Cochrane Database Syst Rev. 2012;(9):Cd009401. https://doi.org/10.1002/14651858.CD009401.pub2.

15.

Dillman DA. Why choice of survey mode makes a difference. Public Health Rep. 2006;121(1):11–3. https://doi.org/10.1177/003335490612100106.CrossRefPubMedPubMedCentral

16.

Brooke J. SUS—a quick and dirty usability scale. 1986. https://hell.meiert.org/core/pdf/sus.pdf. Accessed 7 Feb 2014.

17.

System Usability Scale. http://www.usability.gov/how-to-and-tools/methods/system-usability-scale.html . Accessed 7 Feb 2014.

18.

Elo S, Kyngas H. The qualitative content analysis process. J Adv Nurs. 2008;62(1):107–15. https://doi.org/10.1111/j.1365-2648.2007.04569.x.CrossRefPubMed

19.

Kushniruk AW, Patel VL, Cimino JJ. Usability testing in medical informatics: cognitive approaches to evaluation of information systems and user interfaces. In: Proc AMIA Annu Fall Symp; 1997. p. 218–22.

20.

Tricco AC, Antony J, Ivers NM, Ashoor HM, Khan PA, Blondal E, Ghassemi M, MacDonald H, Chen MH, Ezer LK, Straus SE. Effectiveness of quality improvement strategies for coordination of care to reduce use of health care services: a systematic review and meta-analysis. CMAJ. 2014;186:E568–78. https://doi.org/10.1503/cmaj.140289.CrossRefPubMedPubMedCentral

21.

Creswell JW. Qualitative inquiry and research design: Choosing among five approaches. Thousand Oaks: Sage Publications Incorporated; 2013.

22.

Johnson RB, Onwuegbuzie AJ, Turner LA. Toward a definition of mixed methods research. J Mixed Methods Res. 2007;1(2):112–333.CrossRef

23.

Denzin NK, Lincoln YS. Introduction: the discipline and practice of qualitative research. In: Denzin NK, Lincoln YS, editors. The sage handbook of qualitative research. 3rd ed. Thousand Oaks: Sage; 2005.

24.

Miles MB, Huberman AM, Saldana J. Qualitative data analysis: A methods sourcebook. Thousand Oaks: Sage Publications, Incorporated; 2013.

25.

Gupta S, Wan FT, Newton D, Bhattacharyya O, Chignell M, Straus SE. WikiBuild: a new online collaboration process for multi-stakeholder tool development and consensus building. J Med Internet Res. 2011;13:e108. https://doi.org/10.2196/jmir.1833.CrossRefPubMedPubMedCentral

26.

Perrier L, Kealey MR, Straus SE. A usability study of two formats of a shortened systematic review for clinicians. BMJ Open. 2014;4(12):e005919. https://doi.org/10.1136/bmjopen-2014-005919.CrossRefPubMedPubMedCentral

27.

Perrier L, Persaud N, Ko A, Kastner M, Grimshaw J, McKibbon KA, Straus SE. Development of two shortened systematic review formats for clinicians. Implement Sci. 2013;8:68. https://doi.org/10.1186/1748-5908-8-68.CrossRefPubMedPubMedCentral

28.

Kastner M, Lottridge D, Marquez C, Newton D, Straus SE. Usability evaluation of a clinical decision support tool for osteoporosis disease management. Implement Sci. 2010;5:–96. https://doi.org/10.1186/1748-5908-5-96.

29.

Lavis J, Davies H, Oxman A, Denis JL, Golden-Biddle K, Ferlie E. Towards systematic reviews that inform health care management and policy-making. J Health Serv Res Policy. 2005;10(Suppl 1):35–48. https://doi.org/10.1258/1355819054308549.CrossRefPubMed

30.

Ellen ME, Horowitz E, Vaknin S, Lavis JN. Views of health system policymakers on the role of research in health policymaking in Israel. Isr J Health Policy Res. 2016;5:24. https://doi.org/10.1186/s13584-016-0088-1.CrossRefPubMedPubMedCentral

31.

Ellen ME, Lavis JN, Wilson MG, Grimshaw J, Haynes RB, Ouimet M, Raina P, Gruen R. Health system decision makers’ feedback on summaries and tools supporting the use of systematic reviews: a qualitative study. Evidence Policy. 2014;10:337–59. https://doi.org/10.1332/174426514X672362.CrossRef

32.

El-Jardali F, Lavis JN, Ataya N, Jamal D, Ammar W, Rauof S. Use of health systems evidence by policymakers in eastern Mediterranean countries: views, practices, and contextual influences. BMC Health Serv Res. 2012;12:200. https://doi.org/10.1186/1472-6963-12-200.CrossRefPubMedPubMedCentral

33.

Vogel JP, Oxman AD, Glenton C, Rosenbaum S, Lewin S, Gulmezoglu AM, Souza JP. Policymakers’ and other stakeholders’ perceptions of key considerations for health system decisions and the presentation of evidence to inform those considerations: an international survey. Health Res Policy Syst. 2013;11:–19. https://doi.org/10.1186/1478-4505-11-19.

34.

Halcomb EJ, Davidson PM. Is verbatim transcription of interview data always necessary? Appl Nurs Res. 2006;19(1):38–42. https://doi.org/10.1016/j.apnr.2005.06.001.CrossRefPubMed

35.

Lewis JR. Sample sizes for usability studies: additional considerations. Hum Factors. 1994;36(2):368–78. https://doi.org/10.1177/001872089403600215.CrossRefPubMed

36.

Canadian Institutes of Health Research (CIHR): Guide to knowledge translation planning at CIHR: integrated and end-of-grant approaches; 2015. http://www.cihr-irsc.gc.ca/e/45321.html#a3 Accessed 7 Feb, 2014.

Title: Enhancing the uptake of systematic reviews of effects: what is the best format for health care managers and policy-makers? A mixed-methods study
Authors: Christine Marquez
Alekhya Mascarenhas Johnson
Sabrina Jassemi
Jamie Park
Julia E. Moore
Caroline Blaine
Gertrude Bourdon
Mark Chignell
Moriah E. Ellen
Jacques Fortin
Ian D. Graham
Anne Hayes
Jemila Hamid
Brenda Hemmelgarn
Michael Hillmer
Bev Holmes
Jayna Holroyd-Leduc
Linda Hubert
Brian Hutton
Monika Kastner
John N. Lavis
Karen Michell
David Moher
Mathieu Ouimet
Laure Perrier
Andrea Proctor
Thomas Noseworthy
Victoria Schuckel
Sharlene Stayberg
Marcello Tonelli
Andrea C. Tricco
Sharon E. Straus
Publication date: 01-12-2018
Publisher: BioMed Central
Published in: Implementation Science / Issue 1/2018
Electronic ISSN: 1748-5908
DOI: https://doi.org/10.1186/s13012-018-0779-9

At a glance: The STEP trials

Springer Medicine

Enhancing the uptake of systematic reviews of effects: what is the best format for health care managers and policy-makers? A mixed-methods study

Abstract

Background

Methods

Results

Conclusions

At a glance: The STEP trials

Springer Medicine

Abstract

Background

Methods

Results

Conclusions

Please log in to get access to this content

Other articles of this Issue 1/2018

Supporting teams to optimize function and independence in Veterans: a multi-study program and mixed methods protocol

“I had to somehow still be flexible”: exploring adaptations during implementation of brief cognitive behavioral therapy in primary care

Implementation of a facilitation intervention to improve postpartum care in a low-resource suburb of Dar es Salaam, Tanzania

What drives adoption of a computerised, multifaceted quality improvement intervention for cardiovascular disease management in primary healthcare settings? A mixed methods analysis using normalisation process theory

Sustaining alcohol and opioid use disorder treatment in primary care: a mixed methods study

Implementation between text and work—a qualitative study of a readmission prevention program targeting elderly patients