Top

Published in:

Open Access 01-12-2018 | Research

Development of measurable indicators to enhance public health evidence-informed policy-making

Authors: Valentina Tudisca, Adriana Valente, Tommaso Castellani, Timo Stahl, Petru Sandu, Diana Dulf, Hilde Spitters, Ien Van de Goor, Christina Radl-Karimi, Mohamed Ahmed Syed, Natasa Loncarevic, Cathrine Juel Lau, Susan Roelofs, Maja Bertram, Nancy Edwards, Arja R. Aro, on behalf of the REPOPA Consortium

Published in: Health Research Policy and Systems | Issue 1/2018

Abstract

Background

Ensuring health policies are informed by evidence still remains a challenge despite efforts devoted to this aim. Several tools and approaches aimed at fostering evidence-informed policy-making (EIPM) have been developed, yet there is a lack of availability of indicators specifically devoted to assess and support EIPM. The present study aims to overcome this by building a set of measurable indicators for EIPM intended to infer if and to what extent health-related policies are, or are expected to be, evidence-informed for the purposes of policy planning as well as formative and summative evaluations.

Methods

The indicators for EIPM were developed and validated at international level by means of a two-round internet-based Delphi study conducted within the European project ‘REsearch into POlicy to enhance Physical Activity’ (REPOPA). A total of 82 researchers and policy-makers from the six European countries (Denmark, Finland, Italy, the Netherlands, Romania, the United Kingdom) involved in the project and international organisations were asked to evaluate the relevance and feasibility of an initial set of 23 indicators developed by REPOPA researchers on the basis of literature and knowledge gathered from the previous phases of the project, and to propose new indicators.

Results

The first Delphi round led to the validation of 14 initial indicators and to the development of 8 additional indicators based on panellists’ suggestions; the second round led to the validation of a further 11 indicators, including 6 proposed by panellists, and to the rejection of 6 indicators. A total of 25 indicators were validated, covering EIPM issues related to human resources, documentation, participation and monitoring, and stressing different levels of knowledge exchange and involvement of researchers and other stakeholders in policy development and evaluation.

Conclusion

The study overcame the lack of availability of indicators to assess if and to what extent policies are realised in an evidence-informed manner thanks to the active contribution of researchers and policy-makers. These indicators are intended to become a shared resource usable by policy-makers, researchers and other stakeholders, with a crucial impact on fostering the development of policies informed by evidence.

Available only for authorised users

With ‘tacit knowledge’ we mean implicit and intuitive knowledge that is difficult to communicate, e.g. know-how acquired during practical experience, but we also include explicit knowledge that was not formalised in scientific papers or reports.

For policy-makers, we considered the following definition: “people taking decisions about the proposal and/or implementation of a program, project or activity aimed to an institutional goal, and having responsibility on it” [58, 91‐93].

The first questionnaire comprised a further section concerning some multi-faceted aspects which influence EIPM, but too wide to be translated in terms of measurable indicators, which were presented under the label of ‘Towards complex indicators’ and were rated on their relevance. They will be the subject of a separate paper.

Indicator 6. Procedures for ensuring a review of scientific literature relevant to the policy.

Indicator 8. Citation of peer-reviewed research articles in policy documents; Indicator 9. Citation of reports and other documents containing evidence in policy documents.

Indicator 7. Published scientific articles based on policy results.

Indicator 10. Available evidence briefs for policy; Indicator 11. Available reports on policy results from policy-making organisations of different municipalities/regions/countries.

Indicator 4. Training courses on research issues and on EIPM for the staff working on the policy; Indicator 13. Initiatives to inform researchers during the policy; Indicator 16. Engagement and consultation methodologies to gather knowledge from researchers during the policy.

Indicator 1. Staff with research experience working on the policy; Indicator 25. Researchers working on the policy evaluation.

Indicator 12. Initiatives to inform stakeholders during the policy; Indicator 14. Communication methods tailored for vulnerable groups likely to be impacted by the policy; Indicator 15. Engagement and consultation methodologies to gather knowledge from stakeholders during the policy; Indicator 17. Engagement and consultation methodologies to gather knowledge from vulnerable groups during the policy.

Indicator 19. Communication competences among the staff who interacts with stakeholders; Indicator 20. Initiatives for fostering knowledge sharing between different stakeholders; Indicator 21. Initiatives for consulting target groups to get their perspectives.

Indicator b. Budget for scientific advice; Indicator c. Administrative procedures allowing timely employment of research staff and scientific advisors; Indicator d. Budget for producing/acquiring scientific publications; Indicator f. Budget for external evaluation of the policy in Additional file 6.

Bowen S, Zwi AB. Pathways to “evidence-informed” policy and practice: a framework for action. PLoS Med. 2005;2(7):e166. https://doi.org/10.1371/journal.pmed.0020166.CrossRefPubMedPubMedCentral

Majone G. Evidence, Argument, and Persuasion in the Policy Process. New Haven: Yale University Press; 1989.

Collin J, Johnson E, Hill S. Government support for alcohol industry: promoting exports, jeopardising global health? BMJ. 2014;348:g3648.CrossRefPubMedPubMedCentral

Volmink J. Evidence-informed policy making: challenges and opportunities. BMJ Glob Health. 2017;2:A3.CrossRef

Orton L, Lloyd-Williams F, Taylor-Robinson D, O’Flaherty M, Capewell S. The use of research evidence in public health decision making processes: systematic review. PLoS One. 2011;6(7):e21704. https://doi.org/10.1371/journal.pone.0021704.CrossRefPubMedPubMedCentral

Lomas J. Connecting research and policy. Can J Policy Res. 2000;1(1):140–4.

Gough D, Boaz A. Applying the rational model of evidence-informed policy and practice in the real world. Evid Policy. 2017;13:3–6.CrossRef

Knai C, Petticrew M, Durand MA, Eastmure E, James L, Mehotra A, Scott C, Mays N. Has a public-private partnership resulted in action on healthier diets in England? An analysis of the Public Health Responsibility Deal food pledges. Food Policy. 2015;54:1–10.CrossRef

Bes-Rastrollo M, Schulze MB, Ruiz-Canela M, Martinez-Gonzalez MA. Financial conflicts of interest and reporting bias regarding the association between sugar-sweetened beverages and weight gain: a systematic review of systematic reviews. PLoS Med. 2013;10(12):e1001578. https://doi.org/10.1371/journal.pmed.1001578.CrossRefPubMedPubMedCentral

10.

Bellagio Report. Improving health through better governance – Strengthening the governance of diet and nutrition partnerships for the prevention of chronic diseases. 2016. https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&ved=0ahUKEwja1c2skIrbAhUDzaQKHTecBDoQFggoMAA&url=http%3A%2F%2Fwww.ukhealthforum.org.uk%2FEasysiteWeb%2Fgetresource.axd%3FAssetID%3D58296%26servicetype%3DAttachment&usg=AOvVaw2_UDC9FzBlPTwIL5uJ1yFP. Accessed 16 May 2018.

11.

Ciliska, TH, Buffett C. A Compendium of Critical Appraisal Tools for Public Health Practice. 2008. http://www.nccmt.ca/uploads/media/media/0001/01/b331668f85bc6357f262944f0aca38c14c89c5a4.pdf. Accessed 16 May 2018.

12.

Kiefer L, Frank J, Di Ruggiero E, Dobbins M, Doug M, Gully PR, Mowat D. Fostering evidence-based decision-making in Canada: examining the need for a Canadian population and public health evidence centre and research network. Can J Public Health. 2005;96(3):I1–I19.PubMed

13.

Yost J, Dobbins M, Traynor R, DeCorby K, Workentine S, Greco L. Tools to support evidence-informed public health decision making. BMC Public Health. 2014;14(1):1.CrossRef

14.

Kothari A, Edwards N, Hamel N, Judd M. Is research working for you? Validating a tool to examine the capacity of health organizations to use research. Implement Sci. 2009;4:46. https://doi.org/10.1186/1748-5908-4-46.CrossRefPubMedPubMedCentral

15.

Makkar SR, Turner T, Williamson A, Louviere J, Redman S, Haynes A, Green S, Brennan S. The development of ORACLe: a measure of an organisation’s capacity to engage in evidence-informed health policy. Health Res Policy Syst. 2015;14:4. https://doi.org/10.1186/s12961-015-0069-9.CrossRef

16.

Wilson MG, Moat KA, Hammill AC, Boyko JA, Grimshaw JM, Flottorp S. Developing and refining the methods for a ‘one-stop shop’for research evidence about health systems. Health Res Policy Syst. 2015;13:10.CrossRefPubMedPubMedCentral

17.

Lavis JN, Oxman AD, Lewin S, Fretheim ASUPPORT. Tools for Evidence-Informed Health Policymaking (STP). Health Res Policy Syst. 2009;7(Suppl 1):I1. https://doi.org/10.1186/1478-4505-7-S1-I1.CrossRefPubMedPubMedCentral

18.

Stoker G, Evans M. Evidence-Based Policy Making in the Social Sciences: Methods that Matter. Bristol: Policy Press; 2016.CrossRef

19.

Jacobs JA, Clayton PF, Dove C, Funchess T, Jones E, Perveen G, Skidmore B, Sutton V, Worthington S, Baker EA, Deshpande AD, Brownson RC. A survey tool for measuring evidence-based decision making capacity in public health agencies. BMC Health Serv Res. 2012;12:57.CrossRefPubMedPubMedCentral

20.

Brennan SE, McKenzie TT, Redman S, Makkar S, Williamson A, Haynes A, Green SE. Development and validation of SEER (Seeking, Engaging with and Evaluating Research): a measure of policymakers’ capacity to engage with and use research. Health Res Policy Syst. 2017;15:1.CrossRefPubMedPubMedCentral

21.

Oliver K, Innvaer S, Lorenc T, Woodman J, Thomas J. A systematic review of barriers to and facilitators of the use of evidence by policymakers. BMC Health Serv Res. 2014;14(1):2.CrossRefPubMedPubMedCentral

22.

Oliver K, Lorenc T, Innvaer S. New directions in evidence-based policy research: a critical analysis of the literature. Health Res Policy Syst. 2014;12:34.CrossRefPubMedPubMedCentral

23.

Pülzl H, Rametsteiner E. Indicator development as ‘boundary spanning’ between scientists and policy-makers. Sci Public Policy. 2009;36(10):743–52.CrossRef

24.

Kothari A, MacLean L, Edwards N, Hobbs A. Indicators at the interface: managing policymaker-researcher collaboration. Knowl Manag Res Pract. 2011;9:203–14. https://doi.org/10.1057/kmrp.2011.16.CrossRef

25.

World Health Organization. WHO Global Reference List of 100 Core Health Indicators. Geneva: WHO; 2015.

26.

Aro AR., Bertram M, Hämäläinen RM, Van De Goor I, Skovgaard T, Valente A, Castellani T, et al. Integrating Research Evidence and Physical Activity Policy Making-REPOPA Project. Health Promot Int. 2015;31(2):430–9. https://doi.org/10.1093/heapro/dav002.

27.

Valente A, Tudisca V, Castellani T, Cori L, Bianchi F, Aro AR, Syed A, Radl-Karimi C, Bertram M, Skovgaard T, Loncarevic N, van de Goor LAM, Spitters HPEM, Jansen J, Swinkels W, Ståhl T, Chereches RM, Rus D, Bozdog E, Sandu P, Edwards N, Roelofs S, Viehbeck S, Glümer C, Lau CJ, Jørgensen T, on behalf of the REPOPA Consortium. Delphi-based Implementation and Guidance Development: WP4 Final Report of the REsearch into POlicy to Enhance Physical Activity (REPOPA) Project. 2016 REPOPA website. http://repopa.eu/sites/default/files/latest/D4.2.Delphi-based-implementation-guidance.pdf. Accessed 16 May 2018.

28.

Linstone HA, Murray T. The Delphi Method. Murray T, Linstone HA, editors. Techniques and Applications 53. 2002. https://web.njit.edu/~turoff/pubs/delphibook/delphibook.pdf. Accessed 18 May 2018.

29.

Gupta G, Clarke RE. Theory and applications of the Delphi technique: A bibliography (1975–1994). Technol Forecast Soc Chang. 1996;53(2):185–211. https://doi.org/10.1016/S0040-1625(96)00094-7.

30.

Fletcher AJ, Marchildon GP. Using the Delphi method for qualitative, participatory action research in health leadership. Int J Qual Methods. 2014;13:1–18.CrossRef

31.

Okoli C, Pawlowski SD. The Delphi method as a research tool: an example, design considerations and applications. Inf Manag. 2004;42(1):15–29. https://doi.org/10.1016/j.im.2003.11.002.

32.

Castellani T, Valente A. Democrazia e Partecipazione: La Metodologia Delphi. IRPPS Working Papers 46. Rome: CNR-IRPPS e-publishing; 2012.

33.

Bossel H, International Institute for Sustainable Development. Indicators for Sustainable Development: Theory, Method, Applications: A Report to the Balaton Group. Winnipeg: International Institute for Sustainable Development; 1999.

34.

Turnhout E, Hisschemöller M, Eijsackers H. Ecological indicators: between the two fires of science and policy. Ecol Indic. 2007;7(2):215–28. https://doi.org/10.1016/j.ecolind.2005.12.003.

35.

Funtowicz S. Why knowledge assessment? Interfaces Sci Soc. 2006;1(48):137–45.CrossRef

36.

Weiss CH. The many meanings of research utilization. Public Adm Rev. 1979;39:426–31.CrossRef

37.

Nutley SM, Isabel W, Huw TOD. Using Evidence: How Research Can Inform Public Services. Bristol: Policy Press; 2007.CrossRef

38.

Satterfield JM, Spring B, Brownson Ross C, Mullen EJ, Robin Newhouse P, Walker BB, Whitlock EP. Toward a transdisciplinary model of evidence-based practice: a transdisciplinary model of evidence-based practice. Milbank Q. 2009;87(2):368–90. https://doi.org/10.1111/j.1468-0009.2009.00561.x.

39.

Straus SE, Tetroe JM, Graham ID. Knowledge translation is the use of knowledge in health care decision making. J Clin Epidemiol. 2011;64(1):6–10.CrossRefPubMed

40.

Landry R, Amara N, Lamari M. Utilization of social science research knowledge in Canada. Res Policy. 2001;30(2):333–49.CrossRef

41.

Landry R, Lamari M, Amara N. The extent and determinants of the utilization of university research in government agencies. Public Adm Rev. 2003;63(2):192–205.CrossRef

42.

Traynor R, Dobbins M, DeCorby K. Challenges of partnership research: insights from a collaborative partnership in evidence-informed public health decision making. Evid Policy: A Journal of Research, Debate and Practice. 2015;11(1):99–109. https://doi.org/10.1332/174426414X14043807774174.

43.

Langlois EV, Becerril Montekio V, Young T, Song K, Alcalde-Rabanal J, Tran N. Enhancing evidence informed policymaking in complex health systems: lessons from multi-site collaborative approaches. Health Res Policy Syst. 2016;14:20. https://doi.org/10.1186/s12961-016-0089-0.

44.

Amara N, Ouimet M, Landry R. New evidence on instrumental, conceptual, and symbolic utilization of university research in government agencies. Sci Commun. 2006;26(1):75–106. https://doi.org/10.1177/1075547004267491.

45.

Belkhodja O, Amara N, Landry RM. The extent and organizational determinants of research utilization in Canadian health services organizations. Sci Commun. 2007;28(3):377–417. https://doi.org/10.1177/1075547006298486.

46.

Knott J, Wildavsky A. If dissemination is the solution, what is the problem? Sci Commun. 1980;1(4):537–78. https://doi.org/10.1177/107554708000100404

47.

Andermann A, Tikki P, Newton JN, Davis A, Panisset U. Evidence for Health II: Overcoming Barriers to Using Evidence in Policy and Practice. Health Res Policy Syst. 2016;14:17. https://doi.org/10.1186/s12961-016-0086-3.

48.

Innvaer S, Vist G, Trommald M, Oxman A. Health policy-makers’ perceptions of their use of evidence: a systematic review. J Health Serv Res Policy. 2002;7(4):239–44.CrossRefPubMed

49.

Oxman AD, Lavis JN, Lewin S, Fretheim A. SUPPORT Tools for evidence-informed health Policymaking (STP) 1: What is evidence-informed policymaking? Health Res Policy Syst. 2009;7(Suppl 1):S1.CrossRefPubMedPubMedCentral

50.

Hämäläinen RM, Aro A, van de Goor I, Lau CJ, Jakobsen MW, Chereches RM, Syed AM. Exploring the use of research evidence in health-enhancing physical activity policies. Health Res Policy Syst. 2015;13:43. https://doi.org/10.1186/s12961-015-0047-2.

51.

Ellen ME, Léon G, Bouchard G, Lavis JN, Ouimet M, Grimshaw JM. What supports do health system organizations have in place to facilitate evidence-informed decision-making? A qualitative study. Implement Sci. 2013;8(1):1.CrossRef

52.

Larsen M, Gulis G, Pedersen KM. Use of evidence in local public health work in Denmark. Int J Public Health. 2017;121(3):273–81. https://doi.org/10.1007/s00038-011-0324-y

53.

Owen R, Macnaghten P, Stilgoe J. Responsible research and innovation: From science in society to science for society, with society. Sci Public Policy. 2012;39(6):751–60. https://doi.org/10.1093/scipol/scs093.

54.

Rametsteiner E, Pülzl H, Alkan-Olsson J, Frederiksen P. Sustainability indicator development - science or political negotiation? Ecol Indic. 2011;11(1):67–70. https://doi.org/10.1016/j.ecolind.2009.06.009.

55.

Niemeijer D, de Groot RS. A Conceptual Framework for Selecting Environmental Indicator Sets. Ecol Indic. 2008;8(1):14–25. https://doi.org/10.1016/j.ecolind.2006.11.012.

56.

Lehtonen M. Indicators as an Appraisal Technology: Framework for Analysing the Policy Influence of the UK Energy Sector Indicators. Sustainable development, evaluation and policy-making: theory, practise and quality assurance. 2012. https://www.elgaronline.com/view/9780857932549.00020.xml. Accessed 18 May 2018.

57.

Vargiu A. Indicators for the evaluation of public engagement of higher education institutions. J Knowl Econ. 2014;5(3):562–84. https://doi.org/10.1007/s13132-014-0194-7.

58.

Aro AR, Radl-Karimi C, Loncarevic N, Bertram M, Joshi R, Thøgersen M., Pettersen CLH, Skovgaard T, Van de Goor LAM, Spitters HPEM, Valente A, Castellani T, Cori L, Jansen J, Dorgelo A, Pos S, on behalf of the REPOPA Consortium. Stewardship-based Intervention. WP3 Final Report of the REsearch into POlicy to Enhance Physical Activity (REPOPA) Project. 2015.http://www.repopa.eu/sites/default/files/D3.2.Report_Stewardship%20based%20intervention.pdf. Accessed 18 May 2018.

59.

Hämäläinen RM, Villa T, Aro AR, Fredsgaard MW, Larsen M, Skovgaard T, van de Goor LAM, Spitters, HPEM, Chereches R, Rus, D, Sandu P, Bianchi F, Castellani T, Cori, L, Valente A, Edwards N, Viehbeck S, Glümer C, Lau CJ, Jørgensen T, Wichbold C, Cavill N, Dorgelo A, Jansen J. Evidence-informed Policy Making to Enhance Physical Activity in Six European Countries. WP1 Final Report of the REsearch into POlicy to Enhance Physical Activity (REPOPA) Project. 2013. http://repopa.eu/sites/default/files/latest/D1.1_Role_of_evidence_in_pm_14062013.pdf. Accessed 18 May 2018.

60.

van de Goor LAM, Quanjel M, Spitters HPEM, Swinkels W, Boumans J, Eklund Karlsson L, Aro AR, Jakobsen MW, Koudenburg OA, Chereches R, Sandu P, Rus D, Roelofs S, Lau CJ, Glümer C, Jørgensen T Jansen J, Dorgelo A, Pos S. In2Action: Development and Evaluation of a Policy Game Intervention to Enhance Evidence-use in HEPA Policy Making. WP2 Final Report of the REsearch into POlicy to Enhance Physical Activity (REPOPA) Project. 2015. http://www.repopa.eu/sites/default/files/D2.2.%20Report_Game%20simulation%20intervention.pdf. Accessed 18 May 2018.

61.

van de Goor LAM, Hämäläinen RM, Syed A, Lau CJ, Sandu P, Spitters HPEM, Eklund Karlsson L, Dulf D, Valente A, Castellani T, Aro AR, on behalf of REPOPA consortium. Determinants of evidence use in public health policy making: results from a study across six EU countries. Health Policy. 2017;121(3):273–81.CrossRefPubMedPubMedCentral

62.

Bertram M, Loncarevic N, Castellani T, Valente A, Gulis G, Aro AR. How could we start to develop indicators for evidence-informed policy making in public health and health promotion? Health Syst Policy Res. 2015;2(2):14. http://www.hsprj.com/health-maintanance/how-could-we-start-to-develop-indicators-for-evidenceinformed-policy-making-in-public-health-and-health-promotion.pdf.

63.

Bertram M, Radl-Karimi C, Loncarevic N, Thøgersen M, Skovgaard T, Jansen J, Castellani T, et al. Planning locally tailored interventions on evidence informed policy making a needs assessment, design and methods. Health Syst Policy Res. 2016;3(2)15. http://www.hsprj.com/health-maintanance/planning-locally-tailored-interventions-on-evidence-informed-policy-making--needs-assessment-design-and-methods.pdf.

64.

Valente A, Castellani T, Larsen M, Aro AR. Models and visions of science-policy interaction: remarks from a Delphi study in Italy. Sci Public Policy. 2015;42(2):228–41. https://doi.org/10.1093/scipol/scu039.

65.

Castellani T, Valente A, Cori L, Bianchi F. Detecting the use of evidence in a meta-policy. Evid Policy. 2016;12(1)):91–107.CrossRef

66.

Lau CJ, Glümer C, Spitters HPEM, Sandu P, Rus D, Eklund Karlsson L, van de Goor LAM. Impact of policy game on insight and attitude to inter sectoral policy processes - EU country cases. Eur J Pub Health. 2015;25(3):333.

67.

Nonaka L, Takeuchi H, Umemoto K. A theory of organizational knowledge creation. Int J Technol Manag. 1996;11(7-8). https://doi.org/10.1504/IJTM.1996.025472.

68.

Litman T. Developing indicators for comprehensive and sustainable transport planning. Transp Res Rec. 2017;1:10–5.

69.

Hyder AA, Corluka A, Winch PJ, El-Shinnawy A, Ghassany H, Malekafzali H, Lim MK, Mfutso-Bengo J, Segura E, Ghaffar A. National policy-makers speak out: are researchers giving them what they need? Health Policy Plan. 2011;26(1):73–82. https://doi.org/10.1093/heapol/czq020.

70.

Lavis JN, Guindon GE, Cameron D, Boupha B, Dejman M, Osei EJA, Sadana R, for the Research to Policy and Practice Study Team. Bridging the gaps between research, policy and practice in low- and middle-income countries: a survey of researchers. Can Med Assoc J. 2010;182(9):E350–61. https://doi.org/10.1503/cmaj.081164.

71.

World Health Organization Regional Office for Europe. Towards an Accelerated Roadmap for Strengthening Evidence-informed Policy-making in the European Region. Report of the First Technical Expert Meeting, 20–30 January 2015. Vilnius: World Health Organization; 2015. http://www.euro.who.int/__data/assets/pdf_file/0019/291061/EIP-Report-1st-Technical-Expert-Meeting-en.pdf?ua=1. Accessed 18 May 2018.

72.

Lavis JN, Ross SE, Hurley JE, Hohenadel JM, Stoddart GL, Woodward CA, Abelson J. Examining the Role of Health Services Research in Public Policymaking. Milbank Q. 2002;80(1):125–54.CrossRefPubMedPubMedCentral

73.

Pelz DC. Some Expanded Perspectives on Use of Social Science in Public Policy. In: Yinger JM, Cutler SJ, editors. Major Social Issues: A Multidisciplinary View. New York: Free Press; 1978. p. 346–57.

74.

Lavis JN, Robertson D, Woodside JM, McLeod CB, Abelson J. How can research organizations more effectively transfer research knowledge to decision makers? Milbank Q. 2003;81(2):221–48.CrossRefPubMedPubMedCentral

75.

Sigurdson K, Sa CM, Kretz A. Looking under the street light: limitations of mainstream technology transfer indicators. Sci Public Policy. 2015;42(5):632–45. https://doi.org/10.1093/scipol/scu080.

76.

Spitters HPEM, Lau CJ, Sandu P, Quanjel M, Dulf D, Glümer C, van Oers HAM, van de Goor IAM. Unravelling networks in local public health policymaking in three European countries – a systems analysis. Health Res Policy Syst. 2017;15:5. https://doi.org/10.1186/s12961-016-0168-2.

77.

Lavis JN, Govin Permanand A, Oxman D, Lewin S, Fretheim A. SUPPORT Tools for evidence-informed health Policymaking (STP) 13: Preparing and using policy briefs to support evidence-informed policymaking. Health Res Policy Syst. 2009;7(Suppl 1):S13. https://doi.org/10.1186/1478-4505-7-S1-S13.

78.

Hanney SR, Gonzalez-Block MA, Buxton MJ, Kogan M. The utilisation of health research in policy-making: concepts, examples and methods of assessment. Health Res Policy Syst. 2003;1(1):2.CrossRefPubMedPubMedCentral

79.

El-Jardali F, Lavis JN, Moat K, Pantoja T, Ataya N. Capturing lessons learned from evidence-to-policy initiatives through structured reflection. Health Res Policy Syst. 2014;12(1):1.CrossRef

80.

Jones H. Promoting Evidence-Based Decision-Making in Development Agencies. Overseas Development Institute Background Note. 2012. https://www.odi.org/sites/odi.org.uk/files/odi-assets/publications-opinion-files/7575.pdf. Accessed 16 May 2018.

81.

Wehrens R, Bekker M, Bal R. Coordination of research, policy and practice: a case study of collaboration in the field of public health. Sci Public Policy. 2011;38(10):755–66.CrossRef

82.

Rosella LC, Kumanan W, Crowcroft NS, Chu A, Upshur R, Willison D, Deeks SL, et al. Pandemic H1N1 in Canada and the use of evidence in developing public health policies – a policy analysis. Soc Sci Med. 2013;83:1–9. https://doi.org/10.1016/j.socscimed.2013.02.009.

83.

Hofmeyer A, Scott C, Lagendyk L. Researcher-decision-maker partnerships in health services research: Practical challenges, guiding principles. BMC Health Serv Res. 2012;12(1):280.CrossRefPubMedPubMedCentral

84.

Mitchell P, Pirkis J, Hall J, Haas M. Partnerships for knowledge exchange in health services research, policy and practice. J Health Serv Res Policy. 2009;14(2):104–11.CrossRefPubMed

85.

Eklund KL, Winge JM, Winblad HM, Aro AR. Involvement of external stakeholders in local health policymaking process: a case study from Odense Municipality, Denmark. Evid Policy. 2016;13(3):433–54. http://www.ingentaconnect.com/contentone/tpp/ep/2017/00000013/00000003/art00004.

86.

Michaels S. Matching knowledge brokering strategies to environmental policy problems and settings. Environ Sci Pol. 2009;12(7):994–1011.CrossRef

87.

Jones H. A Guide to Monitoring and Evaluating Policy Influence. Overseas Development Institute Background Note. 2011. https://www.odi.org/sites/odi.org.uk/files/odi-assets/publications-opinion-files/6453.pdf. Accessed 16 May 2018.

88.

Hung HL, Altschuld JW, Lee YF Methodological and conceptual issues confronting a cross-country Delphi study of educational program evaluation. Eval Program Plann. 2008;31(2):191–8. https://www.sciencedirect.com/science/article/pii/S014971890800013X.

89.

Boulkedid R, Abdoul H, Loustau M, Sibony O, Alberti C. Using and reporting the Delphi method for selecting healthcare quality indicators: a systematic review. PLoS One. 2011;6(6):e20476. https://doi.org/10.1371/journal.pone.0020476.

90.

Stamatakis KA, Hino AAF, Allen P, McQueen A, Jacob RR, Baker EA, Brownson RC. Results from a psychometric assessment of a new tool for measuring evidence-based decision making in public health organizations. Eval Program Plann. 2017;60:17–23. https://doi.org/10.1016/j.evalprogplan.2016.08.002.

91.

Anderson JE. Public Policymaking. Boston: Cengage Learning; 2014.

92.

Haines M. Towards a broader definition of policy making and its evaluation. Agricultural Administration. 1980;8(2):125–35. https://www.sciencedirect.com/science/article/pii/0309586X81900042.

93.

Lippi A. La Valutazione delle Politiche Pubbliche. Bologna: il Mulino; 2007.

94.

Edwards N, Viehbeck S, Hämäläinen RM, Rus D, Skovgaard T, van de Goor LAM, Valente A, Syed A, Aro AR. Challenges of ethical clearance in international health policy and social sciences research: experiences and recommendations from a multi-country research programme. Public Health Rev. 2012;34:11.CrossRef

Title: Development of measurable indicators to enhance public health evidence-informed policy-making
Authors: Valentina Tudisca
Adriana Valente
Tommaso Castellani
Timo Stahl
Petru Sandu
Diana Dulf
Hilde Spitters
Ien Van de Goor
Christina Radl-Karimi
Mohamed Ahmed Syed
Natasa Loncarevic
Cathrine Juel Lau
Susan Roelofs
Maja Bertram
Nancy Edwards
Arja R. Aro
on behalf of the REPOPA Consortium
Publication date: 01-12-2018
Publisher: BioMed Central
Published in: Health Research Policy and Systems / Issue 1/2018
Electronic ISSN: 1478-4505
DOI: https://doi.org/10.1186/s12961-018-0323-z

At a glance: The ONWARDS insulin icodec trials

Springer Medicine

Development of measurable indicators to enhance public health evidence-informed policy-making

Abstract

Background

Methods

Results

Conclusion

At a glance: The ONWARDS insulin icodec trials

Springer Medicine

Abstract

Background

Methods

Results

Conclusion

Please log in to get access to this content

Other articles of this Issue 1/2018

Assessing policy-makers’, academics’ and experts’ satisfaction with the performance of the Palestinian health research system: a qualitative study

A bibliometric analysis of the published road traffic injuries research in India, post-1990

Changing roles of universities in the era of SDGs: rising up to the global challenge through institutionalising partnerships with governments and communities

The AGING Initiative experience: a call for sustained support for team science networks

The potential role of cost-utility analysis in the decision to implement major system change in acute stroke services in metropolitan areas in England

Deconstructing knowledge brokering for commissioned rapid reviews: an observational study