Societal output and use of research performed by health research groups

The last decade has seen the evaluation of health research pay more and more attention to societal use and benefits of research in addition to scientific quality, both in qualitative and quantitative ways. This paper elaborates primarily on a quantitative approach to assess societal output and use of research performed by health research groups (societal quality of research). For this reason, one of the Dutch university medical centres (i.e. the Leiden University Medical Center (LUMC)) was chosen as the subject of a pilot study, because of its mission to integrate top patient care with medical, biomedical and healthcare research and education. All research departments were used as units of evaluation within this university medical centre.

The method consisted of a four-step process to reach a societal quality score per department, based on its (research) outreach to relevant societal stakeholders (the general public, healthcare professionals and the private sector). For each of these three types of stakeholders, indicators within four modes of communication were defined (knowledge production, knowledge exchange, knowledge use and earning capacity). These indicators were measured by a bottom-up approach in a qualitative way (i.e. all departments of the LUMC were asked to list all activities they would consider to be of societal relevance), after which they were converted into quantitative scores. These quantitative scores could then be compared to standardised scientific quality scores that are based on scientific publications and citations of peer-reviewed articles.

Based on the LUMC pilot study, only a weak correlation was found between societal and scientific quality. This suggests that societal quality needs additional activities to be performed by health research groups and is not simply the consequence of high scientific quality. Therefore we conclude that scientific and societal evaluation should be considered to be synergistic in terms of learning for the future, accountability and advocacy.

This quantitative approach to assess societal quality in a quantitative sense is based on indicators that function as proxies for society quality on different levels, based on the communication of researchers with their societal stakeholders (i.e. knowledge production, knowledge exchange and knowledge use). The methodology presented is just a first attempt to compare scientific quality scores (publication and citation scores) with societal quality scores in a quantitative way. This comparison can be used by organisations (e.g. university medical centres) in their planning and control cycle.