Abstract
The expression "third mission" is generally used to refer to universities’ direct and indirect contribution to society. Some authors maintain the idea that a relevant aspect of third mission concerns public engagement of universities. Relevance and visibility of institutions’ as well as scholars’ public engagement is connected with the possibility of accounting for it. The debate about the evaluation of teaching and research is quite advanced and so are assessment instruments and techniques (although far from producing generalized consensus). Confrontation on the assessment of public engagement lags behind, although some significant advancements exist. The paper presents and discusses possible indicators for the evaluation of public engagement of universities, on the basis of comparison between three reports that were chosen after analysis of both mainstream publishing and grey literature. Indicators for institutional public engagement proposed by those three reports are subsumed under a common framework which encompasses them within six domains, such as: mission, governance and overarching institutionalized strategies for public engagement; research; student engagement and educational outreach; dissemination; accessibility and use of facilities; community partnerships, stakeholders’ relations and participation in external activities. Conclusions identify a shortlist of indicators based on validity and feasibility. Some integration will also be proposed in the light of critical aspects pointed out in discussion.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
Throughout the present paper, impact evaluation will be often mentioned, notably referring to a wide societal context. Obviously enough, such widely diffused expressions within the academic community as “citation impact” and “impact factor” have completely different semantics from the ones hereby attached to the term “impact”.
Some relevant coordination initiatives can be recalled here, such as the Living Knowledge Network (based in Germany, but which also extends beyond the borders of Europe), Sciences Citoyennes in France, Popular Education Network in Scotland, Community-Based Research in Canada, Community-University Partnership for Health, and the Service Learning movement in the United States as well as the National CBR Networking Initiative or the Community Partnership for Social Action Network. Other similar initiatives are found in Asia where we can find the Society for Participatory Research and the Academe Civil Society Network in South East Asia (ACSN), in Europe (Global University Network for Innovation in Barcelona; Developing Research on Citizenship Network University of Sussex, Community University Partnership Program at the University of Brighton), Africa (Sub-Sahara African Participatory Research Network in Senegal, Gambia, Mauritania and Chad), Australia (Australian University Community Engagement Association and the Australian Universities Community Engagement Alliance). Many of these come together in larger networks (such as, for example, GACER—Global Alliance on Community Engaged Research, created in 2008) trying to play a role not only as a link between the different experiences to share practices and ideas but also to advocate and formulate political proposals for a more adequate recognition of participatory research practices geared to the needs of the community.
An “Inventory of Tools for Assessing University Capacity, Support for and Outcomes of Community/Civic-Engaged Scholarship”, prepared by L. Wenger and A. MacInnis in June 2011, listed no less than 53 resources dealing with those themes that were identified by exploring grey literature and peer-reviewed articles. Nine of those resources are articles in journals, two are book’s chapters and two are books. Wenger and MacInnis also report 24 journal articles and one book which make reference to those resources, although, authors say, “search is not complete and is significantly challenged by shifting names, versions of tools, modifications, etc.” Most of the tools identified within the inventory are working papers, recommendations, reports, assessment and classification instruments, rubrics and the like. The inventory shows that options are diversified and practical experiences are quite advanced.
See acknowledgement sections.
The search for documentation has been mainly done on the Internet on major scientific publication repositories and resource lists, thus extending beyond Wenger and MacInnis’ inventory. A particular effort has been done to reduce the limitations imposed by the use of the Internet, notably the dominance of documents in English and/or which refer to English-speaking countries, whereas experiences are definitively geographically diversified (see above, note one). Nonetheless, some limitations persist. The documentation analysed for the present paper not only mainly refers to the North American continent, Europe and Australia but also concerns other significant national and regional experiences in other parts of the world, such as, for instance, the Brazilian experience where the concept of university outreach (extensão universitária) is provided for in article 207 of the 1988 Federal Constitution. See, for instance, Forum de Pró-Reitores de extensão das Universidades Públicas Brasileiras, Avaliação Nacional da Extensão Universitária, Brasilia, 2001.
Molas-Gallart et al. discuss about the opportunity of assessing activities rather than outputs or impacts. Their decision for the first option (assessing activities) is maintained by pointing out the difficulties in assessing outputs or impacts. In fact, authors argue (pp. vi–vii and 12–14), complexities arise as to: identification of additionality, timing, halo effect and skewed impact distribution, serendipity and influence of factors beyond the university’s control.
As it has been noticed, indicators proposed by Molas-Gallart et al. are not developed by specifically referring to the civic mission, but rather to third stream at large.
References
Aranguren, M. J., & Larrea, M. (2011). Regional innovation policy processes: linking learning to action. Journal of Knowledge Economy, 2(3), 569–585.
Bauer, M. W., & Jensen, P. (2011). The mobilization of scientists for public engagement. Public Understanding of Science, 20(1), 3–11.
Borofsky, R. (2000). Public anthropology. Where to? What next? Anthropology News, 41(5), 9–10.
Boyte, H. C., & Hollander, E. L. (1999). Wingspread declaration on renewing the civic mission of the American research university. Campus Compact.
Bozeman, B. (2000). Technology transfer and public policy: a review of research and theory. Research Policy, 29, 627–635.
Brady, D. (2004). Why public sociology may fail. Social Forces, 82(4), 1629–1638.
Burawoy, M. (2005). For public sociology. American Sociological Review, 70, 4–28.
Carayannis, E., & Campbell, D. (2009). ‘Mode 3’ and ‘Quadruple Helix’: toward a 21st century fractal innovation ecosystem. International Journal of Technology Management, 46(3–4), 201–234.
Carnegie Foundation for the Advancement of Learning. (2007). Elective classification: community engagement 2008 documentation framework. New York: Carnegie Foundation. http://consensus.fsu.edu/bog-fcrc/pdfs2/Carnegie_Community_Engagement_08_Documentation.pdf. Accessed 27 June 2012.
Checkoway, B. (2001). Renewing the civic mission of the American Research University. The Journal of Higher Education, 72(2), 125–147.
Calhoun C. (2011). Who needs knowledge? Communication, Nov. 4th 2011. Retrieved from http://www.ssrc.org/calhoun/2011/11/04/who-needs-knowledge/. Accessed 27 June 2012.
Chessa, S., & Vargiu, A. (2011). Valutazione universitaria e mutamenti istituzionali in Europa. Studi di Sociologia, 1(XLIX), 3–34.
Department of Trade and Industry. (2000). Excellence and opportunity: a science and innovation policy for the 21st century. London: Department of Trade and Industry, Office of Science and Technology, HM Government.
Duke, C. (2008). University engagement. Avoidable confusion and inescapable contradiction. Higher Education Management and Policy, 20(2), 1–11. doi:10.1787/hemp-v20-art17-en. http://dx.doi.org/10.1787/hemp-v20-art12-en.
E3M (2011). Final report of Delphi Study. The E3M Project—European Indicators and Ranking Methodology for University Third Mission. http://www.e3mproject.eu/docs/Delphi-E3M-project.pdf. Accessed 27 June 2012.
Ellis, A., Bianchi, G., & Shoop, K. (2008). Service-learning in American higher education: an analysis. Higher Education Forum, 5, 141–150.
Furco, A., & Miller, W. (2009). Issues in benchmarking and assessing institutional engagement. New Directions for Higher Education, 147, 47–54.
Gleeson, R. E. (2010). The third mission and the history of reform in American higher education. In P. Inman & H. G. Schuetze (Eds.), The community engagement and service mission of universities (pp. 121–137). Leicester: Niace.
Garlick, S., & Langworthy, A. (2008). Benchmarking university community engagement: developing a national approach in Australia. Higher Education Management and Policy, 20(2), 1–12. http://dx.doi.org/10.1787/hemp-v20-art17-en.
Gastil, E., & Levine, P. (Eds.). (2005). The deliberative democracy handbook. Strategies for effective civic engagement in the 21st century. San Francisco: Jossey Bass.
Habermas, J. (1968). Erkenntniss und Interesse. Suhrkamp: Frankfut am Main.
Hart A., Northmore S., & Gerhardt C. (2009). Briefing paper: auditing, benchmarking and evaluating public engagement. Bristol, UK: National Co-ordinating Centre for Public Engagement Research Synthesis n° 1. http://talloiresnetwork.tufts.edu/wp-content/uploads/AuditingBenchmarkingandEvaluatingPublicEngagement.pdf. Accessed 27 June 2012.
Hazelkorn, E. (2011). Rankings and the reshaping of higher education: The battle for world-class excellence. Basingstoke: Palgrave Macmillan.
Hill, M., & Williamson Hill, L.-V. (2010). Faculty reward system and third mission of colleges and universities. In P. Inman & H. G. Schuetze (Eds.), The community engagement and service mission of universities (pp. 249–260). Leicester: Niace.
Holland, B. (2009). Will it last? Evidence of institutionalization at Carnegie classified community engagement institutions. New Directions for Higher Education, 147, 85–98.
Jaeger, C. C., Renn, O., Rosa, E. A., & Webler, T. (2001). Risk, uncertainty, and rational action. London: Earthscan.
Jasanoff, S. (Ed.). (2004). States of knowledge: the co-production of science and the social order. London: Routledge.
Joss, S., & Bellucci, S. (Eds.). (2002). Participatory technology assessment. European perspective. London: Centre for the Study of Democracy.
Kehm, B., & Stensaker, B. (Eds.). (2008). University rankings, diversity, and the new landscape of higher education. Rotterdam: Sense.
Kellogg Commission. (1999). Taking charge of change: renewing the promise of state and land-grant universities. Washington: W.K. Kellogg Foundation.
Martin, B. R. (1996). Technology foresight: capturing the benefits from science-related technologies. Research Evaluation, 6(2), 158–168.
Mejlgaard, N., Bloch, C., Degn, L., & Ravn, T. (2012). Monitoring policy and research activities on science in society in Europe (MASIS) final synthesis report. Brussels: European Commission.
Mohrman, K. (2010). Public universities and regional development. In P. Inman & H. G. Schuetze (Eds.), The community engagement and service mission of universities (pp. 139–163). Leicester: Niace.
Molas-Gallart J., Salter A., Patel P., Scott A., & Duran X. (2002). Measuring third stream activities. Final report to the Russell Group of University, Brighton: UK, SPRU—Science and Technology Policy Research, University of Sussex. http://www2.lse.ac.uk/economicHistory/Research/CCPN/pdf/russell_report_thirdStream.pdf. Accessed 27 June 2012.
Neresini, F., & Bucchi, M. (2011). Which indicators for the new public engagement activities? An exploratory study of European research institutions. Public Understanding of Science, 20, 64–79. doi:10.1177/0963662510388363.
Nichols, L. (Ed.). (2007). Public sociology: the contemporary debate. New Brunswick: Transaction.
Nielsen, F. (2004). The vacant ‘we’: remarks on public sociology. Social Forces, 82(4), 1619–1627.
Oecd. (1983). The university and the community. Paris: Oecd.
Oecd. (1997). Proposed guidelines for collecting and interpreting technological innovation data: the ‘Oslo Manual’. Paris: Oecd.
Oecd. (1999). The response of higher education institution to regional needs. Paris: Oecd.
Oecd. (2001a). Innovation and the strategic use of IPRs: Issues Paper, DSTI/STP/TIP (2001) 4. Paris: Oecd.
Oecd. (2001b). Workshop on the management of intellectual property rights from public research, DSTI/STP/TIP (2001) 11. Paris: Oecd.
Oecd. (2001c). Cities and regions in the learning economy. Paris: Oecd.
Oecd. (2007). Higher education and regions: globally competitive, locally engaged. Paris: Oecd.
Owen, R., Macnaghten, P., & Stilgoe, J. (2012). Responsible research and innovation: from science in society to science for society, with society. Science and Public Policy, 39(6), 751–760. doi:10.1093/scipol/scs093.
Polanyi, K. (1944). The great transformation. New York: Holt, Rinehart & Winston.
Polt W., Rammer C., Gassler H., Schibany A., & Schartinger D. (2001). Benchmarking industry-science relations: the role of framework conditions. Vienna/Mannheim: AT/DE, Report to the Federal Ministry of Economy and Labour, Austria and to the European Commission (Enterprise DG).
Rask, M., Maciukaite-Zviniene, S., & Petrauskiene, J. (2012). Innovations in public engagement and participatory performance of the nations. Science and Public Policy, 39(6), 710–721. doi:10.1093/scipol/scs089.
Saltmarsh, J., Giles, D. E., Ward, E., & Bulione, S. M. (2009). Rewarding community-engaged scholarship. New Directions for Higher Education, 147, 25–35.
Schoonmaker, M. G., & Carayannis, E. G. (2012). Mode 3: a proposed classification scheme for the knowledge economy and society. Journal of Knowledge Economy. doi:10.1007/s13132-012-0097-4.
Schuetze, H. G. (2010). The ‘third mission’ of universities: engagement and service. In P. Inman & H. G. Schuetze (Eds.), The community engagement and service mission of universities (pp. 13–31). Leicester: Niace.
Simon, L. A. K. (2009). Embracing the world grand ideal: affirming the Morrill Act for the twenty-first century global society. East Lansing: Michigan State University.
Strand, K., Marullo, S., Cutforth, N., Stoeker, R., & Domhue, P. (2003). Community-based research and higher education. Principles and practices. San Francisco: Jossey-Bass.
Varga, A. (Ed.). (2009). Universities, knowledge transfer and regional development. Cheltenham: Edward Elgar.
Vargiu, A. (2008). Le diversità nella ricerca-azione partecipata: alcune implicazioni metodologiche e procedurali. Studi di Sociologia, 2(XLVI), 205–232.
Wachelder, J. (2003). Democratizing science: various routes and visions of Dutch science shops. Science, Technology and Human Values, 28(2), 244–273.
Walshok, M. (1995). Knowledge without boundaries: what America’s research universities can do for the economy, the workplace, and the community. San Francisco: Jossey-Bass.
Zaal, R., & Leydersdorff, L. (1987). Amsterdam science shop and its influence on university research: the effects of ten years of dealing with non-academic questions. Science and Public Policy, 14(6), 310–316.
Acknowledgments
I especially wish to acknowledge fruitful work and discussion with colleagues working in the PERARES Project, and in particular Henk Mulder and Henny van de Windt from Rijksuniversiteit Groningen and Brian Trench from Dublin City University. My work would not be possible without the continuous confrontation and idea sharing with fellow researchers at the FOIST Laboratory for Social Policies and Educational Training Processes, in the Department of Humanities and Social Sciences at the University of Sassari (Italy). Responsibility for any error or misinterpretation within this paper is, of course, mine.
Funding acknowledgement
The work leading to the writing of this paper has been funded by PERARES (Public Engagement with Research and Research Engagement with Society): a project run under the Science and Society programme in European Union’s FP7; Grant agreement number 244264, call SiS-2009-1.2.1.1.
Work Package 9 of the PERARES project is specifically dedicated to evaluation of public engagement. The following partners are involved in WP9: Dublin City University (Ireland), Rijksuniversiteit Groningen (Netherlands), Wageningen Universiteit (Netherlands), University of Cambridge (England), and University of Sassari (Italy).
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Vargiu, A. Indicators for the Evaluation of Public Engagement of Higher Education Institutions. J Knowl Econ 5, 562–584 (2014). https://doi.org/10.1007/s13132-014-0194-7
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s13132-014-0194-7