Skip to main content
Key Specialties
Cardiology Diabetology Endocrinology Gastroenterology Geriatrics and Gerontology Gynecology and Obstetrics Hematology Infectious Disease Internal Medicine Nephrology Neurology Oncology Pediatrics Respiratory Medicine Rheumatology Surgery
Congress Coverage
2024 ACC Congress 2023 ESMO Congress 2023 EASD Congress 2023 ERS Congress 2023 ESC Congress 2023 EHA Congress 2023 ASCO Annual Meeting
Clinical Collections
Advances in Alzheimer’s

At a glance: The ONWARDS insulin icodec trials

A round-up of the ONWARDS phase 3 clinical trials evaluating the novel weekly insulin icodec in people with diabetes.
ADVANCED SEARCH
Log in
Top
Health Research Policy and Systems
Published in: Health Research Policy and Systems 1/2015

Open Access 01-12-2015 | Research

Using conjoint analysis to develop a system of scoring policymakers’ use of research in policy and program development

Authors: Steve R Makkar, Anna Williamson, Tari Turner, Sally Redman, Jordan Louviere

Published in: Health Research Policy and Systems | Issue 1/2015

Login to get access

Abstract

Background

The importance of utilising the best available research evidence in the development of health policies, services, and programs is increasingly recognised, yet few standardised systems for quantifying policymakers’ research use are available. We developed a comprehensive measurement and scoring tool that assesses four domains of research use (i.e. instrumental, conceptual, tactical, and imposed). The scoring tool breaks down each domain into its key subactions like a checklist. Our aim was to develop a tool that assigned appropriate scores to each subaction based on its relative importance to undertaking evidence-informed health policymaking. In order to establish the relative importance of each research use subaction and generate this scoring system, we conducted conjoint analysis with a sample of knowledge translation experts.

Methods

Fifty-four experts were recruited to undertake four choice surveys. Respondents were shown combinations of research use subactions called profiles, and rated on a 1 to 9 scale whether each profile represented a limited (1–3), moderate (4–6), or extensive (7–9) example of research use. Generalised Estimating Equations were used to analyse respondents’ choice data, which calculated a utility coefficient for each subaction. A large utility coefficient indicated that a subaction was particularly influential in guiding experts’ ratings of extensive research use.

Results

Utility coefficients were calculated for each subaction, which became the points assigned to the subactions in the scoring system. The following subactions yielded the largest utilities and were regarded as the most important components of each research use domain: using research to directly influence the core of the policy decision; using research to inform alternative perspectives to deal with the policy issue; using research to persuade targeted stakeholders to support a predetermined decision; and using research because it was a mandated requirement by the policymaker’s organisation.

Conclusions

We have generated an empirically derived and context-sensitive means of measuring and scoring the extent to which policymakers used research to inform the development of a policy document. The scoring system can be used by organisations to not only quantify the extent of their research use, but also to provide them with insights into potential strategies to improve subsequent research use.
Appendix
Available only for authorised users
Literature
1.
Hausman A. Implications of evidence-based practice for community health. Am J Community Psychol. 2002;30(3):453–67.PubMedCrossRef
2.
3.
Lemay MA, Sa C. The use of academic research in public health policy and practice. Res Evaluat. 2014;23:79–88.CrossRef
4.
5.
Lavis JN, Ross SE, Hurley JE, Hohenadel JM, Stoddart GL, Woodward CA, et al. Examining the role of health services research in public policymaking. Milbank Q. 2002;80(1):125–54.PubMedPubMedCentralCrossRef
6.
LaRocca R, Yost J, Dobbins M, Ciliska D, Butt M. The effectiveness of knowledge translation strategies used in public health: a systematic review. BMC Public Health. 2012;12:751.PubMedPubMedCentralCrossRef
7.
El-Jardali F, Lavis JN, Moat KA, Pantoja T, Ataya N. Capturing lessons learned from evidence-to-policy initiatives through structured reflection. Health Res Policy Syst. 2014;12:2.PubMedPubMedCentralCrossRef
8.
9.
Hanney SR, Gonzalez-Block MA, Buxton MJ, Kogan M. The utilisation of health research in policy-making: concepts, examples and methods of assessment. Health Res Policy Syst. 2003;1:2.PubMedPubMedCentralCrossRef
10.
Buchan H. Gaps between best evidence and practice: causes for concern. Med J Australia. 2004;180:S48–9.PubMed
11.
Bowen S, Erickson T, Martens PJ, Crockett S. More than “using research”: the real challenges in promoting evidence-informed decision-making. Health Policy. 2009;4(3):87–102.
12.
Fielding JE, Briss PA. Promoting evidence-based public health policy: can we have better evidence and more action? Health Aff (Millwood). 2006;4:969–78.CrossRef
13.
Andre FE, Booy R, Bock HL, Clemens J, Datta SK, John TJ, et al. Vaccination greatly reduces disease, disability, death and inequity worldwide. Bull World Health Organ. 2008;86(2):140–6.PubMedCrossRef
14.
Hanna JN, Hills SL, Humphreys JL. Impact of hepatitis A vaccination of Indigenous children on notifications of hepatitis A in north Queensland. Med J Aust. 2004;181(9):482–5.PubMed
15.
Morrato EH, Elias M, Gericke CA. Using population-based routine data for evidence-based health policy decisions: lessons from three examples of setting and evaluating national health policy in Australia, the UK and the USA. J Public Health. 2007;19(4):463–71.CrossRef
16.
Milat A, Laws R, King L, Newson R, Rychetnik L, Rissel C, et al. Policy and practice impacts of applied research: a case study analysis of the New South Wales Health promotion Demonstration Research Grants Scheme 2000-2006. Health Res Policy Syst. 2013;11:5.PubMedPubMedCentralCrossRef
17.
Chagnon F, Poullot L, Malo C, Gervais MJ, Pigeon ME. Comparison of determinants of research knowledge utilization by practitioners and administrators in the field of child and family social services. Implement Sci. 2010;5:41.PubMedPubMedCentralCrossRef
18.
19.
Amara N, Ouimet M, Landry R. New evidence on instrumental, conceptual, and symbolic utilization of university research in government agencies. Sci Commun. 2004;26(1):75–106.CrossRef
20.
Campbell DM, Redman S, Jorm L, Cooke M, Zwi AB, Rychetnik L. Increasing the use of evidence in health policy: practice and views of policy makers and researchers. Aust New Zealand Health Policy. 2009;6:21.PubMedPubMedCentralCrossRef
21.
Dobbins M, Cockerill R, Barnsley J. Factors affecting the utilization of systematic reviews: a study of public health decision makers. Int J Technol Assess Health Care. 2001;17:203–14.PubMedCrossRef
22.
El-Jardali F, Lavis JN, Ataya N, Jamal D. Use of health systems and policy research evidence in the health policymaking in eastern Mediterranean countries: views and practices of researchers. Implement Sci. 2012;7:2.PubMedPubMedCentralCrossRef
23.
24.
Elshaug AG, Hiller JE, Tunis SR, Moss JR. Challenged in Australian policy processes for disinvestment from existing, ineffective health care practices. Aust New Zealand Health Policy. 2007;4:23.PubMedPubMedCentralCrossRef
25.
Jackson Jr WL, Sales JF. Potentially ineffective care: time for earnest reexamination. Crit Care Res Pract. 2014;2014:1–6.CrossRef
26.
Erixon F, van der Marel E. What is driving the rise in health care expenditures? An inquiry into the nature and causes of the cost disease. In: ECIPE working papers, vol. 5. Brussels, Belgium: European Centre for International Political Economy; 2011.
27.
Hansen J. Health services research in Europe: evaluating and improving its contribution to health care policy. J Health Serv Res Policy. 2011;16 Suppl 2:1.PubMedCrossRef
28.
Redman S, Turner T, Davies H, Haynes A, Williamson A, Milat A, et al. The SPIRIT Action Framework: a structured approach to selecting and testing strategies to increase the use of research in policy. Soc Sci Med. 2015;136–137:147–55.PubMedCrossRef
29.
Brownson RC, Fielding JE, Maylan CM. Evidence-based public health: a fundamental concept for public health practice. Annu Rev Public Health. 2009;30:175–201.PubMedCrossRef
30.
Dobbins M, Ciliska D, Cockerill R, Barnsley J, DiCenso A. A framework for the dissemination and utilization of research for health-care policy and practice. Online J Knowl Synth Nurs. 2002;9:149–60.
31.
de Goede J, van Bon-Martens MJ, Putters K, van Oers HA. Looking for interaction: quantitative measurement of research utilization by Dutch local health officials. Health Res Policy Syst. 2012;10:9.PubMedPubMedCentralCrossRef
32.
Landry R, Amara N, Lamari M. Utilization of social science research knowledge in Canada. Res Policy. 2001;30:333–49.CrossRef
33.
Landry R, Amara N, Lamari M. Climbing the ladder of research utilization. Sci Commun. 2001;22(4):396–422.CrossRef
34.
Landry R, Lamari M, Amara N. The extent and determinants of the utilization of university research in government agencies. Public Adm Rev. 2003;63(2):192–205.CrossRef
35.
Squires JE, Estabrooks CA, Newburn-Cook CV, Gierl M. Validation of the conceptual research utilization scale: an application of the standards for educational and psychological testing in healthcare. BMC Health Serv Res. 2011;11:107.PubMedPubMedCentralCrossRef
36.
Squires JE, Estabrooks CA, O’Rourke HM, Gustavsson P, Newburn-Cook CV, Wallin L. A systematic review of the psychometric properties of self-report research utilization measures used in healthcare. Implement Sci. 2011;6:83.PubMedPubMedCentralCrossRef
37.
Zardo P, Collie A. Measuring use of research evidence in public health policy: a policy content analysis. Implement Sci. 2014;14:496.
38.
Walker I, Hulme C. Concrete words are easier to recall than abstract words: evidence for a semantic contribution to short-term serial recall. J Exper Psychol. 1999;25(5):1256–71.
39.
Wattenmaker WD, Shoben EJ. Context and the recallability of concrete and abstract sentences. J Exper Psychol. 1987;13(1):140–50.
40.
Weiss C. Knowledge creep and decision accretion. Sci Commun. 1980;1(3):381–404.CrossRef
41.
Beyer JM, Trice HM. The utilization process: a conceptual framework and synthesis of empirical findings. Adm Sci Q. 1982;27(4):591–622.CrossRef
42.
Haynes A, Turner T, Redman S, Milat AJ, Moore G. Developing definitions for a knowledge exchange intervention in health policy and program agencies: reflections on process and value. Int J Soc Res Meth. 2014;18(2):145–59.CrossRef
43.
Weiss C, Bucuvalas MJ. Social science research and decision-making. New York: Columbia University Press; 1980.
44.
Beyer JM. Research utilisation: bridging a gap between communities. J Manage Inquiry. 1997;20:385.CrossRef
45.
Sumner A, Crichton J, Theobald S, Zulu E, Parkhurst J. What shapes research impact on policy? Understanding research uptake in sexual and reproductive health policy processes in resource poor contexts. Health Res Policy Syst. 2011;9 Suppl 1:53.CrossRef
46.
Weiss CH. The many meanings of research utilization. Public Adm Rev. 1979;39(5):426–31.CrossRef
47.
Liverani M, Hawkins B, Parkhurst JO. Political and institutional influences on the use of evidence in public health policy. PLoS ONE. 2013;8(10):e77404.PubMedPubMedCentralCrossRef
48.
Makkar SR, Williamson A, Turner T, Redman S, Louviere J. Using conjoint analysis to develop a system to score research engagement actions by health decision makers. Health Res Policy Syst. 2015;13:22.PubMedPubMedCentralCrossRef
49.
50.
Hair JF, Black WC, Babin BJ, Anderson RE, Tatham RL. Multivariate data analysis. 6th ed. Pearson: New Delhi; 2006.
51.
Ryan M, McIntosh E, Shackley P. Methodological issues in the application of conjoint analysis in health care. Health Econ. 1998;7:373–8.PubMedCrossRef
52.
Ryan M. Using conjoint analysis to take account of patient preferences and go beyond health outcomes: an application to in vitro fertilisation. Soc Sci Med. 1999;48:535–46.PubMedCrossRef
53.
Ryan M. A role for conjoint analysis in technology assessment in health care? Int J Technol Assess Health Care. 1999;15(3):443–57.PubMed
54.
Bridges JF, Hauber AB, Marshall D, Lloyd A, Prosser LA, Regier DA, et al. Conjoint analysis applications in health – a checklist: a report of the ISPOR Good Research Practices for Conjoint Analysis Task Force. Value Health. 2011;14(4):403–13.PubMedCrossRef
55.
Carson RT, Louviere J. A common nomenclature for stated preference elicitation approaches. Environ Resource Econ. 2011;49:539–59.CrossRef
56.
57.
Ryan M, Hughes J. Using conjoint analysis to assess preferences for miscarriage management. Health Econ. 1997;6:216–73.CrossRef
58.
San Miguel F, Ryan M, McIntosh E. Applying conjoint analysis in economic evaluations: an application to menorrhagia. Appl Econ. 2010;32:823–33.CrossRef
59.
Farrar S, Ryan M. Response-ordering effects: a methodological issue of conjoint analysis. Health Econ. 1999;8:75–9.PubMedCrossRef
60.
Lancsar E, Louviere J. Conducting discrete choice experiments to inform healthcare decision making: a user’s guide. Pharmacoeconomics. 2008;26(8):661–77.PubMedCrossRef
61.
Oppewal H, Louviere JJ, Timmermans HJP. Modeling hierarchical conjoint processes with integrated choice experiments. J Market Res. 1994;31(1):91–105.CrossRef
62.
Louviere J. Hierarchical information integration: a new method for the design and analysis of complex multiattribute judgment problems. Adv Consum Res. 1984;11:148–55.
63.
Haynes A, Turner T, Redman S, Milat A, Moore G. Developing definitions for a knowledge exchange intervention in health policy and program agencies: reflections on process and value. Int J Soc Res Meth. 2015;18(2):145–59.CrossRef
64.
Bak A, Bartlomowicz T. Conjoint analysis method and its implementation in conjoint R package. Wroclaw: University of Economics; 2009.
65.
66.
Norusis MJ. SPSS 15.0 Advanced statistical procedures companion. Prentice Hall: Upper Saddle River, NJ; 2007.
67.
Ghisletta P, Spini D. An introduction to generalized estimating equations and an application to assess selectivity effects in a longitudinal study on very old individuals. J Educ Behav Stat. 2004;29(4):421–37.CrossRef
68.
Lipsitz SR, Fitzmaurice GM, Orav EJ, Laird NM. Performance of generalized estimating equations in practical situations. Biometrics. 1994;50(1):270–8.PubMedCrossRef
69.
Orme D. Getting started with conjoint analysis: strategies for product design and pricing research. 2nd ed. Madison, WI: Research Publishers LLC; 2010.
70.
Elliott H, Popay J. How are policy makers using evidence? Models of research utilisation and local NHS policy making. J Epidemiol Commun Health. 2000;54:461–8.CrossRef
71.
Lavis JN, Davies H, Oxman A, Denis JL, Golden-Biddle K, Ferlie E. Towards systematic reviews that inform health care management and policy-making. J Health Serv Res Policy. 2005;10 Suppl 1:35–48.PubMedCrossRef
72.
Weiss CH, Bucuvalas MJ. Truth tests and utility tests: decision-makers’ frames of reference for social science research. Am Sociol Rev. 1980;45(2):302–13.CrossRef
73.
Anderson M, Cosby J, Swan B, Moore H, Broekhoven M. The use of research in local health service agencies. Soc Sci Med. 1999;49(8):1007–19.PubMedCrossRef
74.
Orton L, Lloyd-Williams F, Taylor-Robinson D, O’Flaherty M, Capewell S. The use of research evidence in public health decision making processes: systematic review. PLoS ONE. 2011;6(7):e21704.PubMedPubMedCentralCrossRef
75.
Hennink M, Stephenson R. Using research to inform health policy: barriers and strategies in developing countries. J Health Commun. 2006;10:163–80.CrossRef
76.
Buse K, Mays N, Walt G. Making health policy. Glasgow: Open University Press; 2005.
77.
Ellen ME, Lavis JN, Ouimet M, Grimshaw J, Bedard PO. Determining research knowledge infrastructure for healthcare systems: a qualitative study. Implement Sci. 2011;6:60.PubMedPubMedCentralCrossRef
78.
Ettelt S, Mays N. Health services research in Europe and its use for informing policy. J Health Serv Res Policy. 2011;16 Suppl 2:48–60.PubMedCrossRef
79.
Hyder AA, Corluka A, Winch PJ, El-Shinnawy A, Ghassany H, Malekafzali H, et al. National policymakers speak out: are researchers giving them what they need? Health Policy Plann. 2011;26:73–82.CrossRef
80.
81.
Green J, Thorogood N. Qualitative methods for health research. London: SAGE; 2005.
82.
Brook RH, Chassin MR, Fink A, Solomon DH, Kosecoff J, Park RE. A method for the detailed assessment of the appropriateness of medical technologies. Int J Technol Assess Health Care. 1986;2(1):53–3.PubMedCrossRef
83.
Hsu CC, Sanford BA. The Delphi technique: making sense of consensus. Pract Assess Res Evaluation. 2007;12:10.
84.
85.
Wortman PM, Smyth JM, Langenbrunner JC, Yeaton WH. Consensus among experts and research synthesis: a comparison of methods. Int J Technol Assess Health Care. 1998;14(1):109–22.PubMedCrossRef
86.
Investigators CIPHER. Supporting policy in health with research: an intervention trial (SPIRIT)-protocol for a stepped wedge trial. BMJ Open. 2014;4(7):e005293.CrossRef
87.
Oliver K, Lorenc T, Innvaer S. New directions in evidence-based policy research: a critical analysis of the literature. Health Res Policy Syst. 2014;12:34.PubMedPubMedCentralCrossRef
88.
Marshall D, Bridges JF, Hauber AB, Cameron R, Donnalley L, Fyie K, et al. Conjoint analysis applications in health – how are studies being designed and reported? Patient. 2010;3(4):249–56.PubMedCrossRef
Metadata
Title
Using conjoint analysis to develop a system of scoring policymakers’ use of research in policy and program development
Authors
Steve R Makkar
Anna Williamson
Tari Turner
Sally Redman
Jordan Louviere
Publication date
01-12-2015
Publisher
BioMed Central
Published in
Health Research Policy and Systems / Issue 1/2015
Electronic ISSN: 1478-4505
DOI
https://doi.org/10.1186/s12961-015-0022-y

Other articles of this Issue 1/2015

Health Research Policy and Systems 1/2015 Go to the issue

Research

Seizing the strategic opportunities of emerging technologies by building up innovation system: monoclonal antibody development in China

Research

An analysis of national collaboration with Spanish researchers abroad in the health sciences

Research

A process evaluation accompanying an attempted randomized controlled trial of an evidence service for health system policymakers

Research

Introduction of male circumcision for HIV prevention in Uganda: analysis of the policy process

Editorial

Health research improves healthcare: now we have the evidence and the chance to help the WHO spread such benefits globally

Research

From market access to patient access: overview of evidence-based approaches for the reimbursement and pricing of pharmaceuticals in 36 European countries