Skip to main content
Top
Published in: Health Research Policy and Systems 1/2017

Open Access 01-12-2017 | Opinion

Understanding relevance of health research: considerations in the context of research impact assessment

Authors: Mark J. Dobrow, Fiona A. Miller, Cy Frank, Adalsteinn D. Brown

Published in: Health Research Policy and Systems | Issue 1/2017

Login to get access

Abstract

Background

With massive investment in health-related research, above and beyond investments in the management and delivery of healthcare and public health services, there has been increasing focus on the impact of health research to explore and explain the consequences of these investments and inform strategic planning. Relevance is reflected by increased attention to the usability and impact of health research, with research funders increasingly engaging in relevance assessment as an input to decision processes. Yet, it is unclear whether relevance is a synonym for or predictor of impact, a necessary condition or stage in achieving it, or a distinct aim of the research enterprise. The main aim of this paper is to improve our understanding of research relevance, with specific objectives to (1) unpack research relevance from both theoretical and practical perspectives, and (2) outline key considerations for its assessment.

Approach

Our approach involved the scholarly strategy of review and reflection. We prepared a draft paper based on an exploratory review of literature from various fields, and gained from detailed and insightful analysis and critique at a roundtable discussion with a group of key health research stakeholders. We also solicited review and feedback from a small sample of expert reviewers.

Conclusions

Research relevance seems increasingly important in justifying research investments and guiding strategic research planning. However, consideration of relevance has been largely tacit in the health research community, often depending on unexplained interpretations of value, fit and potential for impact. While research relevance seems a necessary condition for impact – a process or component of efforts to make rigorous research usable – ultimately, relevance stands apart from research impact. Careful and explicit consideration of research relevance is vital to gauge the overall value and impact of a wide range of individual and collective research efforts and investments. To improve understanding, this paper outlines four key considerations, including how research relevance assessments (1) orientate to, capture and compare research versus non-research sources, (2) consider both instrumental versus non-instrumental uses of research, (3) accommodate dynamic temporal-shifting perspectives on research, and (4) align with an intersubjective understanding of relevance.
Literature
2.
go back to reference Chalmers I, Bracken MB, Djulbegovic B, Garattini S, Grant J, Gülmezoglu AM, Howells DW, Ioannidis JPA, Oliver S. How to increase value and reduce waste when research priorities are set. Lancet. 2014;383(9912):156–65.CrossRefPubMed Chalmers I, Bracken MB, Djulbegovic B, Garattini S, Grant J, Gülmezoglu AM, Howells DW, Ioannidis JPA, Oliver S. How to increase value and reduce waste when research priorities are set. Lancet. 2014;383(9912):156–65.CrossRefPubMed
3.
go back to reference Chalmers I, Glasziou P. Avoidable waste in the production and reporting of research evidence. Lancet. 2009;374(9683):86–9.CrossRefPubMed Chalmers I, Glasziou P. Avoidable waste in the production and reporting of research evidence. Lancet. 2009;374(9683):86–9.CrossRefPubMed
4.
go back to reference Panel on the Return on Investments in Health Research. Making an impact: a preferred framework and indicators to measure returns on investment in health research. Ottawa: Canadian Academy of Health Sciences; 2009. Panel on the Return on Investments in Health Research. Making an impact: a preferred framework and indicators to measure returns on investment in health research. Ottawa: Canadian Academy of Health Sciences; 2009.
5.
go back to reference Banzi R, Moja L, Pistotti V, Facchini A, Liberati A. Conceptual frameworks and empirical approaches used to assess the impact of health research: an overview of reviews. Health Res Policy Syst. 2011;9:26.CrossRefPubMedPubMedCentral Banzi R, Moja L, Pistotti V, Facchini A, Liberati A. Conceptual frameworks and empirical approaches used to assess the impact of health research: an overview of reviews. Health Res Policy Syst. 2011;9:26.CrossRefPubMedPubMedCentral
7.
go back to reference Buxton M, Hanney S. How can payback from health services research be assessed? J Health Serv Res Policy. 1996;1:35–43.PubMed Buxton M, Hanney S. How can payback from health services research be assessed? J Health Serv Res Policy. 1996;1:35–43.PubMed
8.
go back to reference Graham KER, Chorzempa HL, Valentine PA, J M. Evaluating health research impact: Development and implementation of the Alberta Innovates – Health Solutions impact framework. Res Eval. 2012;21:354–67 Graham KER, Chorzempa HL, Valentine PA, J M. Evaluating health research impact: Development and implementation of the Alberta Innovates – Health Solutions impact framework. Res Eval. 2012;21:354–67
9.
go back to reference Nowotny H, Scott P, Gibbons M. Introduction: ‘Mode 2’ Revisited: The New Production of Knowledge. Minerva. 2003;41(3):179–94.CrossRef Nowotny H, Scott P, Gibbons M. Introduction: ‘Mode 2’ Revisited: The New Production of Knowledge. Minerva. 2003;41(3):179–94.CrossRef
11.
go back to reference Lomas J. Preface: The first ones over the barricade. In: Potvin L, Armstrong P, editors. Shaping Academia for the Public Good: Critical Reflections on the CHSRF/CIHR Chair Program. Toronto: University of Toronto Press; 2013. Lomas J. Preface: The first ones over the barricade. In: Potvin L, Armstrong P, editors. Shaping Academia for the Public Good: Critical Reflections on the CHSRF/CIHR Chair Program. Toronto: University of Toronto Press; 2013.
12.
go back to reference Ontario Ministry of Health and Long-Term Care – Community and Health Promotion Branch. Ontario Support for People and Patient-Oriented Research and Trials (SUPPORT) Unit: Business Plan. Toronto, ON: MOHLTC; 2013. Ontario Ministry of Health and Long-Term Care – Community and Health Promotion Branch. Ontario Support for People and Patient-Oriented Research and Trials (SUPPORT) Unit: Business Plan. Toronto, ON: MOHLTC; 2013.
13.
go back to reference Buxton M. The payback of ‘Payback’: challenges in assessing research impact. Res Eval. 2011;20(3):259–60.CrossRef Buxton M. The payback of ‘Payback’: challenges in assessing research impact. Res Eval. 2011;20(3):259–60.CrossRef
14.
go back to reference Floridi L. Understanding epistemic relevance. Erkenntnis. 2008;69(1):69–92.CrossRef Floridi L. Understanding epistemic relevance. Erkenntnis. 2008;69(1):69–92.CrossRef
15.
go back to reference Cohen LJ. Some steps towards a general theory of relevance. Synthese. 1994;101(2):171–85.CrossRef Cohen LJ. Some steps towards a general theory of relevance. Synthese. 1994;101(2):171–85.CrossRef
16.
17.
go back to reference Schlesinger GN. Relevance Theoria. 1986;57(1):57–67. Schlesinger GN. Relevance Theoria. 1986;57(1):57–67.
18.
go back to reference Keynes JM. A treatise on probability. London: MacMillan and Co. Limited; 1921. Keynes JM. A treatise on probability. London: MacMillan and Co. Limited; 1921.
19.
go back to reference Bush V. Science - the endless frontier. A report to the President on a program of postwar scientific research. Washington: National Science Foundation; 1945. Bush V. Science - the endless frontier. A report to the President on a program of postwar scientific research. Washington: National Science Foundation; 1945.
20.
21.
go back to reference Stokes DE. Pasteur's quadrant - basic science and technological innovation. Washington: Brookings Institution Press; 1997. Stokes DE. Pasteur's quadrant - basic science and technological innovation. Washington: Brookings Institution Press; 1997.
22.
go back to reference Tushman M, O'Reilly C. Research and relevance: implications of Pasteur's Quadrant for doctoral programs and faculty development. Acad Manag J. 2007;50(4):769–74.CrossRef Tushman M, O'Reilly C. Research and relevance: implications of Pasteur's Quadrant for doctoral programs and faculty development. Acad Manag J. 2007;50(4):769–74.CrossRef
23.
go back to reference Graham ID, Logan J, Harrison MB, Straus SE, Tetroe J, Caswell W, Robinson N. Lost in knowledge translation: time for a map? J Contin Educ Health Prof. 2006;26(1):13–24.CrossRefPubMed Graham ID, Logan J, Harrison MB, Straus SE, Tetroe J, Caswell W, Robinson N. Lost in knowledge translation: time for a map? J Contin Educ Health Prof. 2006;26(1):13–24.CrossRefPubMed
24.
go back to reference Berger ML, Martin BC, Husereau D, Worley K, Allen JD, Yang W, Quon NC, Mullins CD, Kahler KH, Crown W. A questionnaire to assess the relevance and credibility of observational studies to inform health care decision making: an ISPOR-AMCP-NPC Good Practice Task Force report. Value Health. 2014;17(2):143–56.CrossRefPubMedPubMedCentral Berger ML, Martin BC, Husereau D, Worley K, Allen JD, Yang W, Quon NC, Mullins CD, Kahler KH, Crown W. A questionnaire to assess the relevance and credibility of observational studies to inform health care decision making: an ISPOR-AMCP-NPC Good Practice Task Force report. Value Health. 2014;17(2):143–56.CrossRefPubMedPubMedCentral
25.
go back to reference Jansen JP, Trikalinos T, Cappelleri JC, Daw J, Andes S, Eldessouki R, Salanti G. Indirect treatment comparison/network meta-analysis study questionnaire to assess relevance and credibility to inform health care decision making: an ISPOR-AMCP-NPC Good Practice Task Force report. Value Health. 2014;17(2):157–73.CrossRefPubMed Jansen JP, Trikalinos T, Cappelleri JC, Daw J, Andes S, Eldessouki R, Salanti G. Indirect treatment comparison/network meta-analysis study questionnaire to assess relevance and credibility to inform health care decision making: an ISPOR-AMCP-NPC Good Practice Task Force report. Value Health. 2014;17(2):157–73.CrossRefPubMed
27.
go back to reference Pope C, Ziebland S, Mays N. Qualitative research in health care: analysing qualitative data. BMJ. 2000;2000(320):114–6.CrossRef Pope C, Ziebland S, Mays N. Qualitative research in health care: analysing qualitative data. BMJ. 2000;2000(320):114–6.CrossRef
28.
go back to reference Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, Alonso-Coello P, Schunemann HJ, for the GRADE Working Group. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ. 2008;336(7650):924–6.CrossRefPubMedPubMedCentral Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, Alonso-Coello P, Schunemann HJ, for the GRADE Working Group. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ. 2008;336(7650):924–6.CrossRefPubMedPubMedCentral
30.
go back to reference Burchett H, Umoquit M, Dobrow M. How do we know when research from one setting can be useful in another? A review of external validity, applicability and transferability. J Health Serv Res Policy. 2011;16(4):238–44.CrossRefPubMed Burchett H, Umoquit M, Dobrow M. How do we know when research from one setting can be useful in another? A review of external validity, applicability and transferability. J Health Serv Res Policy. 2011;16(4):238–44.CrossRefPubMed
35.
go back to reference Giacomini M. One of these things is not like the others: the idea of precedence in health technology assessment and coverage decisions. Milbank Q. 2005;83(2):193–223.CrossRefPubMedPubMedCentral Giacomini M. One of these things is not like the others: the idea of precedence in health technology assessment and coverage decisions. Milbank Q. 2005;83(2):193–223.CrossRefPubMedPubMedCentral
36.
go back to reference Karğın S. The Impact of IFRS on the Value Relevance of Accounting Information: Evidence from Turkish Firms. Int J Econ Finance. 2013;5(4):71–80. Karğın S. The Impact of IFRS on the Value Relevance of Accounting Information: Evidence from Turkish Firms. Int J Econ Finance. 2013;5(4):71–80.
37.
go back to reference Abbott S. Relevance. In: Abbott S, editor. The glossary of education reform. Portland: Great Schools Partnership; 2013. Abbott S. Relevance. In: Abbott S, editor. The glossary of education reform. Portland: Great Schools Partnership; 2013.
41.
go back to reference Google. Search Quality Evaluator Guidelines. 2016. Accessed 28 Mar 2016. Google. Search Quality Evaluator Guidelines. 2016. Accessed 28 Mar 2016.
42.
go back to reference Pariser E. The Filter Bubble: What the Internet is Hiding from You. New York: Penguin Group Inc.; 2011. Pariser E. The Filter Bubble: What the Internet is Hiding from You. New York: Penguin Group Inc.; 2011.
43.
go back to reference Weiss C. The many meanings of research utilization. Public Adm Rev. 1979;39(5):426–31.CrossRef Weiss C. The many meanings of research utilization. Public Adm Rev. 1979;39(5):426–31.CrossRef
Metadata
Title
Understanding relevance of health research: considerations in the context of research impact assessment
Authors
Mark J. Dobrow
Fiona A. Miller
Cy Frank
Adalsteinn D. Brown
Publication date
01-12-2017
Publisher
BioMed Central
Published in
Health Research Policy and Systems / Issue 1/2017
Electronic ISSN: 1478-4505
DOI
https://doi.org/10.1186/s12961-017-0188-6

Other articles of this Issue 1/2017

Health Research Policy and Systems 1/2017 Go to the issue