Top

Systematic Reviews

Published in:

Open Access 01-12-2018 | Research

Can automated content analysis be used to assess and improve the use of evidence in mental health policy? A systematic review

Authors: Kristel Alla, Florin Oprescu, Wayne D. Hall, Harvey A. Whiteford, Brian W. Head, Carla S. Meurk

Published in: Systematic Reviews | Issue 1/2018

Abstract

Background

This review assesses the utility of applying an automated content analysis method to the field of mental health policy development. We considered the possibility of using the Wordscores algorithm to assess research and policy texts in ways that facilitate the uptake of research into mental health policy.

Methods

The PRISMA framework and the McMaster appraisal tools were used to systematically review and report on the strengths and limitations of the Wordscores algorithm. Nine electronic databases were searched for peer-reviewed journal articles published between 2003 and 2016. Inclusion criteria were (1) articles had to be published in public health, political science, social science or health services disciplines; (2) articles had to be research articles or opinion pieces that used Wordscores; and (3) articles had to discuss both strengths and limitations of using Wordscores for content analysis.

Results

The literature search returned 118 results. Twelve articles met the inclusion criteria. These articles explored a range of policy questions and appraised different aspects of the Wordscores method.

Discussion

Following synthesis of the material, we identified the following as potential strengths of Wordscores: (1) the Wordscores algorithm can be used at all stages of policy development; (2) it is valid and reliable; (3) it can be used to determine the alignment of health policy drafts with research evidence; (4) it enables existing policies to be revised in the light of research; and (5) it can determine whether changes in policy over time were supported by the evidence. Potential limitations identified were (1) decreased accuracy with short documents, (2) words constitute the unit of analysis and (3) expertise is needed to choose ‘reference texts’.

Conclusions

Automated content analysis may be useful in assessing and improving the use of evidence in mental health policies. Wordscores is an automated content analysis option for comparing policy and research texts that could be used by both researchers and policymakers.

Available only for authorised users

Grimshaw JM, Eccles MP, Lavis JN, Hill SJ, Squires JE. Knowledge translation of research findings. Implement Sci. 2012;7:1–17. https://doi.org/10.1186/1748-5908-7-50.CrossRef

Head B. Three lenses of evidence-based policy. Aust J Publ Admin. 2008;67:1–11. https://doi.org/10.1111/j.1467-8500.2007.00564.x.CrossRef

Zardo P, Collie A, Livingstone C. External factors affecting decision-making and use of evidence in an Australian public health policy environment. Soc Sci Med. 2014;108:120–7. https://doi.org/10.1016/j.socscimed.2014.02.046.CrossRefPubMed

Gibson J, Kelvin R, Goodyer I. A call for greater transparency in health policy development: observations from an analysis of child and adolescent mental health policy. Evid Policy. 2015;11:7–18. https://doi.org/10.1332/174426414X13940168597987.CrossRef

Milat AJ, Bauman AE, Redman S. A narrative review of research impact assessment models and methods. Health Res Policy Syst. 2015;13:1–7. https://doi.org/10.1186/s12961-015-0003-1.CrossRef

Brownson RC, Chriqui JF, Stamatakis KA. Understanding evidence-based public health policy. Am J Pub Health. 2009;99:1576–83. https://doi.org/10.2105/ajph.2008.156224.CrossRef

Katikireddi SV, Higgins M, Bond L, Bonell C, Macintyre S. How evidence based is English public health policy? Brit Med J. 2011;343:1–7. https://doi.org/10.1136/bmj.d7310.CrossRef

Australian Institute of Health and Welfare (AIHW). Mental health services in Australia 2006–07, mental health series no.11. Cat no. HSE 74. Canberra: AIHW; 2009.

Townsend C, Whiteford H, Baingana F, Gulbinat W, Jenkins R, Baba A, Lieh Mak F, Manderscheid R, Mayeya J, Minoletti A, et al. The mental health policy template: domains and elements for mental health policy formulation. Int Rev Psychiatry. 2004;16:18–23. https://doi.org/10.1080/09540260310001635069.CrossRefPubMed

10.

Hanney SR, Gonzalez-Block MA, Buxton MJ, Kogan M. The utilisation of health research in policy-making: concepts, examples and methods of assessment. Health Res Policy Syst. 2003;1:2.CrossRef

11.

Hyder A, Corluka A, Winch PJ, El-Shinnawy A, Ghassany H, Malekafzali H, Lim M-K, Mfutso-Bengo J, Segura E, Ghaffar A. National policy-makers speak out: are researchers giving them what they need? Health Policy Plan. 2010;26:73–82.CrossRef

12.

Whiteford H, Harris M, Diminic S. Mental health service system improvement: translating evidence into policy. Aust N Z J Psychiatry. 2013;47:703–6. https://doi.org/10.1177/0004867413494867.CrossRefPubMed

13.

Innvaer S, Vist G, Trommald M, Oxman A. Health policymakers’ perceptions of their use of evidence: a systematic review. J Health Serv Res Policy. 2002;7:239–44. https://doi.org/10.1258/13558190232043277.CrossRefPubMed

14.

Grimmer J, Stewart BM. Text as data: the promise and pitfalls of automatic content analysis methods for political texts. Polit Anal. 2013;21:267–97. https://doi.org/10.1093/pan/mps028.CrossRef

15.

Oliver K, Lorenc T, Innvær S. New directions in evidence-based policy research: a critical analysis of the literature. Health Res Policy Syst. 2014;12:34.CrossRef

16.

Orton L, Lloyd-Williams F, Taylor-Robinson D, O’Flaherty M, Capewell S. The use of research evidence in public health decision making processes: systematic review. PLoS One. 2011;6:e21704. https://doi.org/10.1371/journal.pone.0021704.CrossRefPubMedPubMedCentral

17.

Arnold C, Hug S, Schulz T. Cross-validating measurement techniques of party positioning. In: Paper presented at the annual meeting of the American Political Science Association. Toronto; 2009. http://www.unige.ch/ses/spo/static/simonhug/cvmtpp/cvmtpp.pdf.

18.

Gross J, Acree B, Sim Y, Smith NA. Testing the etch-a-sketch hypothesis: a computational analysis of Mitt Romney’s ideological makeover during the 2012 primary vs. general elections. In: Paper presented at the American Political Science Association annual meeting 2013. Chicago; 2013. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2299991.

19.

Odena O. Using software to tell a trustworthy, convincing and useful story. Int J Soc Res Methodol. 2013;16:355–72. https://doi.org/10.1080/13645579.2012.706019.CrossRef

20.

Richards L. Qualitative computing - methods revolution? Int J Soc Res Methodol. 2002;5:263–76. https://doi.org/10.1080/13645570210146302.CrossRef

21.

Ampofo L, Collister S, O’Loughlin B, Chadwick A. Text mining and social media: when quantitative meets qualitative, and software meets humans. In: Halfpenny P, Procter R, editors. Innovations in digital research methods. London: Sage; 2015. p. 161–92.

22.

Angus D, Rintel S, Wiles J. Making sense of big text: a visual-first approach for analysing text data using Leximancer and Discursis. Int J Soc Res Methodol. 2013;16:261–7. https://doi.org/10.1080/13645579.2013.774186.CrossRef

23.

Gjerløw H. What’s right? A construct validation of party policy position measures. Oslo: University of Oslo, Department of Political Science; 2014.

24.

Eriksson J, Giacomello G. Content analysis in the digital age: tools, functions, and implications for security. In: Krüger J, Nickolay B, Gaycken S, editors. The secure information society: ethical, legal and political challenges. London: Springer London; 2013. p. 137–48.CrossRef

25.

Laver M, Benoit K, Garry J. Extracting policy positions from political texts using words as data. Am Polit Sci Rev. 2003;97:311–31.CrossRef

26.

Baturo A, Dasandi N, Mikhaylov SJ. Understanding state preferences with text as data: Introducing the UN General Debate corpus. Res Pol. 2017;1-9. https://doi.org/10.1177/2053168017712821.CrossRef

27.

Benoit K, Herzog A. Text analysis: estimating policy preferences from written and spoken words. http://www.kenbenoit.net/pdfs/HerzogBenoit_bookchapter.pdf. Accessed 2 June 2016.

28.

Costa H, Gilmore AB, Peeters S, McKee M, Stuckler D. Quantifying the influence of the tobacco industry on EU governance: automated content analysis of the EU tobacco products directive. Tob Control. 2014;23:473–8. https://doi.org/10.1136/tobaccocontrol-2014-051822.CrossRefPubMedPubMedCentral

29.

Baek YM, Cappella JN, Bindman A. Automating content analysis of open-ended responses: Wordscores and affective intonation. Commun Methods Meas. 2011;5:275–96. https://doi.org/10.1080/19312458.2011.624489.CrossRefPubMedPubMedCentral

30.

Klemmensen R, Hobolt SB, Hansen ME. Estimating policy positions using political texts: an evaluation of the Wordscores approach. Elect Stud. 2007;26:746–55. https://doi.org/10.1016/j.electstud.2007.07.006.CrossRef

31.

Lowe W. Understanding Wordscores. Polit Anal. 2008;16:356–71. https://doi.org/10.1093/pan/mpn004.CrossRef

32.

Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred Reporting Items for Systematic Reviews and Meta-Analyses: the PRISMA statement. Int J Surg. 2010;8:336–41. https://doi.org/10.1016/j.ijsu.2010.02.007.CrossRefPubMed

33.

Law M, Stewart D, Letts L, Pollock N, Bosch J, Westmorland M. Guidelines for critical review form - quantitative studies. Hamilton: McMaster University Occupational Therapy Evidence-Based Practice Research Group; 1998.

34.

Law M, Stewart D, Letts L, Pollock N, Bosch J, Westmorland M. Critical review form - quantitative studies. Ontario: McMaster University Occupational Therapy Evidence-Based Practice Research Group; 1998.

35.

Debus M. Analysing party politics in Germany with new approaches for estimating policy preferences of political actors. Ger Polit. 2009;18:281–300. https://doi.org/10.1080/09644000903055773.CrossRef

36.

Volkens A. Strengths and weaknesses of approaches to measuring policy positions of parties. Elect Stud. 2007;26:108–20. https://doi.org/10.1016/j.electstud.2006.04.003.CrossRef

37.

Budge I, Pennings P. Do they work? Validating computerised word frequency estimates against policy series. Elect Stud. 2007;26:121–9. https://doi.org/10.1016/j.electstud.2006.04.002.CrossRef

38.

Bernauer J, Bräuninger T. Intra-party preference heterogeneity and faction membership in the 15th German bundestag: a computational text analysis of parliamentary speeches. Ger Polit. 2009;18:385–402. https://doi.org/10.1080/09644000903055823.CrossRef

39.

Coffé H, Da Roit B. Party policy positions in Italy after pre-electoral coalition disintegration. Acta Politica. 2011;46:25–42. https://doi.org/10.1057/ap.2010.16.CrossRef

40.

Hug S, Schulz T. Left-right positions of political parties in Switzerland. Party Polit. 2007;13:305–30. https://doi.org/10.1177/1354068807075938.CrossRef

41.

Baumann M, Debus M, Müller J. Convictions and signals in parliamentary speeches: Dáil Éireann debates on abortion in 2001 and 2013. Ir Pol Stud. 2015;30:199–219. https://doi.org/10.1080/07907184.2015.1022152.CrossRef

42.

The Manifesto Project: Manifesto Corpus. https://manifesto-project.wzb.eu/information/documents/corpus. Accessed 30 Jan 2018.

43.

Armstrong R, Waters E, Roberts H, Oliver S, Popay J. The role and theoretical evolution of knowledge translation and exchange in public health. J Pub Health. 2006;28:384–9. https://doi.org/10.1093/pubmed/fdl072.CrossRef

44.

Hutchinson JR, Huberman M. Knowledge dissemination and use in science and mathematics education: a literature review. J Sci Educ Tech. 1994;3:27–47.CrossRef

45.

Walter I, Nutley S, Davies H. Research impact: a cross sector review. St. Andrews: Research Unit for Research Utilisation, University of St. Andrews; 2003.

Title: Can automated content analysis be used to assess and improve the use of evidence in mental health policy? A systematic review
Authors: Kristel Alla
Florin Oprescu
Wayne D. Hall
Harvey A. Whiteford
Brian W. Head
Carla S. Meurk
Publication date: 01-12-2018
Publisher: BioMed Central
Published in: Systematic Reviews / Issue 1/2018
Electronic ISSN: 2046-4053
DOI: https://doi.org/10.1186/s13643-018-0853-z

At a glance: The ONWARDS insulin icodec trials

Springer Medicine

Can automated content analysis be used to assess and improve the use of evidence in mental health policy? A systematic review

Abstract

Background

Methods

Results

Discussion

Conclusions

At a glance: The ONWARDS insulin icodec trials

Springer Medicine

Abstract

Background

Methods

Results

Discussion

Conclusions

Please log in to get access to this content

Other articles of this Issue 1/2018

The effect of rehabilitation interventions on physical function and immobility-related complications in severe stroke—protocol for a systematic review

Quality of life of women who practice dance: a systematic review protocol

Digital storytelling as a method in health research: a systematic review protocol

Choice of outcomes and measurement instruments in randomised trials on eLearning in medical education: a systematic mapping review protocol

Intra-amniotic inflammation and child neurodevelopment: a systematic review protocol

Functional, motor, and sensory assessment instruments upon nerve repair in adult hands: systematic review of psychometric properties