Top

BMC Medical Informatics and Decision Making

Published in:

Open Access 01-12-2019 | Research article

Standards as applied in reality: a case study on the translation of standards in eHealth evaluation practice

Author: Monika Jurkeviciute

Published in: BMC Medical Informatics and Decision Making | Issue 1/2019

Abstract

Background

Application of standards is a way to increase quality in an evaluation study. However, standards are used insufficiently in eHealth evaluation, affecting the generalization of the knowledge generated.

This study aimed to explore how standards are used in a practical setting of an eHealth evaluation, and to identify the factors that can hinder their use.

Methods

The data were collected in a multi-national and interdisciplinary eHealth evaluation study targeted at the elderly people suffering from mild cognitive impairment and mild dementia. The study was carried out in four countries and funded by the European Union. The collected data included meeting minutes (n = 8) and e-mail correspondence (n = 261) between partners. The chronological sequence of events related to the use of standards was established. Subsequently, the hindering factors related to the use of standards were identified from the sequence.

Results

The use of four standards was described, reflecting a variety of related processes or barriers that emerge during eHealth evaluation. The processes differed based on the type of the standard. Specifically, evaluation frameworks were found to be conceptual standards and they were easy to agree upon, while standardized metrics were more tangible and their use triggered negotiations. Further, the following factors hindered the use of standards in eHealth evaluations: (1) inadequacy of a standard to address a target population or a disease, (2) insufficient resources to use a standard, (3) lack of experience in using a standard, and (4) lack of validation of a standard in a particular location.

Conclusions

Standardization initiatives in eHealth evaluation provide a blueprint for evaluation, but their practical application is problematic. The practical circumstances of an evaluation study can cause deviations in the standards, thus producing heterogeneity in the evaluation methodologies.

Eysenbach G. What is eHealth? J Med Internet Res. 2001;3(2):e20. https://doi.org/10.2196/jmir.3.2.e20.CrossRefPubMedPubMedCentral

Khoja S, Durrani H, Scott RE, Sajwani A, Piryani U. Conceptual framework for development of comprehensive e-health evaluation tool. Telemed E Health. 2013;19(1):48–53. https://doi.org/10.1089/tmj.2012.0073.CrossRef

Ammenwerth E, Brender J, Nykänen P, Prokosch HU, Rigby M, Talmon J. Visions and strategies to improve evaluation of health information systems: reflections and lessons based on the HIS-EVAL workshop in Innsbruck. Int J Med Inform. 2004;73(6):479–91. https://doi.org/10.1016/j.ijmedinf.2004.04.004.CrossRefPubMed

Lilford RJ, Foster J, Pringle M. Evaluating eHealth: how to make evaluation more methodologically robust. PLoS Med. 2009;6(11):e1000186. https://doi.org/10.1371/journal.pmed.1000186.CrossRefPubMedPubMedCentral

Mookherji S, Mehl G, Kaonga N, Mechael P. Unmet need: improving mHealth evaluation rigor to build the evidence base. J Health Commun. 2015;20(10):1224–9. https://doi.org/10.1080/10810730.2015.1018624.CrossRefPubMed

Cowie MR, Bax J, Bruining N, Cleland JG, Koehler F, Malik M, Pinto F, van der Velde E, Vardas P. E-health: a position statement of the European Society of Cardiology. Eur Heart J. 2016;37(1):63. https://doi.org/10.1093/eurheartj/ehv416.CrossRefPubMed

Catwell L, Sheikh A. Evaluating eHealth interventions: the need for continuous systemic evaluation. PLoS Med. 2009;6(8):e1000126. https://doi.org/10.1371/journal.pmed.1000126.CrossRefPubMedPubMedCentral

Ammenwerth E. Evidence-based health informatics: how do we know what we know? Methods Inf Med. 2015;54(04):298–307. https://doi.org/10.3414/ME14-01-0119.CrossRefPubMed

Kaplan B. Evaluating informatics applications—some alternative approaches: theory, social interactionism, and call for methodological pluralism. Int J Med Inform. 2001;64(1):39–56. https://doi.org/10.1016/S1386-5056(01)00184-8.CrossRefPubMed

10.

Talmon J, Ammenwerth E, Brender J, De Keizer N, Nykänen P, Rigby M. STARE-HI—statement on reporting of evaluation studies in health informatics. Int J Med Inform. 2009;78(1):1–9. https://doi.org/10.1016/j.ijmedinf.2008.09.002.CrossRefPubMed

11.

Proudfoot J, Klein B, Barak A, Carlbring P, Cuijpers P, Lange A, Ritterband L, Andersson G. Establishing guidelines for executing and reporting internet intervention research. Cogn Behav Ther. 2011;40(2):82–97. https://doi.org/10.1080/16506073.2011.573807.CrossRefPubMed

12.

Janssen R, Hettinga M, Visser S, Menko R, Prins H, Krediet I, Haaker T, Bodenstaff L. Innovation routes and evidence guidelines for eHealth small and medium-sized enterprises. Int J Adv Life Sc. 2013;5:188–203.

13.

Fraser H, Bailey C, Sinha C, Mehl G, Labrique AB. Call to action on global eHealth evaluation: consensus statement of the WHO Global eHealth Evaluation Meeting, Bellagio, September 2011. Bellagio: The Bellagio eHealth Evaluation Group; 2011. http://www.ghdonline.org/uploads/The_Bellagio_eHealth_Evaluation_Call_to_Action-Release.docx. Accessed 1 Apr 2018

14.

Cusack CM, Byrne C, Hook JM, McGowan J, Poon EG, Zafar A. Health Information Technology Evaluation Toolkit: 2009 Update (Prepared for the AHRQ National Resource Center for Health Information Technology under Contract No. 290-04-0016), AHRQ Publication No. 09-0083-EF. Rockville: Agency for Healthcare Research and Quality; 2009. https://healthit.ahrq.gov/sites/default/files/docs/page/health-information-technology-evaluation-toolkit-2009-update.pdf. Accessed 7 Sept 2019

15.

Nykänen P, Brender J, Talmon J, de Keizer N, Rigby M, Beuscart-Zephir MC, Ammenwerth E. Guideline for good evaluation practice in health informatics (GEP-HI). Int J Med Inform. 2011;80(12):815–27. https://doi.org/10.1016/j.ijmedinf.2011.08.004.CrossRefPubMed

16.

Health Technology Assessment International. HTA Glossary. https://htaglossary.net. Accessed 7 Sept 2019.

17.

Doupi P. Evolving health IT systems evaluation: the convergence of health informatics and HTA. In: Ammenwerth E, Rigby M, editors. Evidence-based health informatics: promoting safety and efficiency through scientific methods and ethical policy. Amsterdam: IOS press BV; 2016. p. 220–36.

18.

Lampe K, Mäkelä M, Garrido MV, Anttila H, Autti-Rämö I, Hicks NJ, Hofmann B, Koivisto J, Kunz R, Kärki P, Malmivaara A. The HTA core model: a novel method for producing and reporting health technology assessments. Int J Technol Assess Health Care. 2009;25(S2):9–20. https://doi.org/10.1017/S0266462309990638.CrossRefPubMed

19.

Kidholm K, Ekeland AG, Jensen LK, Rasmussen J, Pedersen CD, Bowes A, Flottorp SA, Bech M. A model for assessment of telemedicine applications: MAST. Int J Technol Assess Health Care. 2012;28(1):44–51. https://doi.org/10.1017/S0266462311000638.CrossRefPubMed

20.

Regions of Europe Working together for HEALTH, Project homepage. https://cordis.europa.eu/project/rcn/191719/factsheet/en. Accessed 5 Sept 2019.

21.

Narne E. How will decision makers of Azienda Ospedaliera of Padova/ Padova University Hospital use MAST in practice?. 2010. https://slideplayer.com/slide/8080/. Accessed 7 Sept 2019.

22.

Boehler CE, De Graaf G, Steuten L, Yang Y, Abadie F. Development of a web-based tool for the assessment of health and economic outcomes of the European Innovation Partnership on Active and Healthy Ageing (EIP on AHA). BMC Med Inform Decis Mak. 2015;15(3):S4. https://doi.org/10.1186/1472-6947-15-S3-S4.CrossRefPubMedPubMedCentral

23.

EuroQol Group. EuroQol—A new facility for the measurement of health-related quality of life. Health Policy. 1990;16(3):199.CrossRef

24.

Janssen MF, Pickard AS, Golicki D, Gudex C, Niewada M, Scalone L, Swinburn P, Busschbach J. Measurement properties of the EQ-5D-5L compared to the EQ-5D-3L across eight patient groups: a multi-country study. Qual Life Res. 2013;22(7):1717–27. https://doi.org/10.1007/s11136-012-0322-4.CrossRefPubMed

25.

Eysenbach G, Consort-EHEALTH Group. CONSORT-EHEALTH: Improving and standardizing evaluation reports of Web-based and mobile health interventions. J Med Internet Res. 2011;13(4):e126. https://doi.org/10.2196/jmir.1923.CrossRefPubMedPubMedCentral

26.

Glasgow RE. eHealth evaluation and dissemination research. Am J Prev Med. 2007;32(5):S119–26. https://doi.org/10.1016/j.amepre.2007.01.023.CrossRefPubMed

27.

Calhoun WJ, Wooten K, Bhavnani S, Anderson KE, Freeman J, Brasier AR. The CTSA as an exemplar framework for developing multidisciplinary translational teams. Clin Transl Sci. 2013;6(1):60–71. https://doi.org/10.1111/cts.12004.CrossRefPubMed

28.

Greenhalgh T, Russell J. Why do evaluations of eHealth programs fail? An alternative set of guiding principles. PLoS Med. 2010;7(11):e1000360. https://doi.org/10.1371/journal.pmed.1000360.CrossRefPubMedPubMedCentral

29.

Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O. Diffusion of innovations in service organizations: systematic review and recommendations. Milbank Q. 2004;82(4):581–629. https://doi.org/10.1111/j.0887-378X.2004.00325.x.CrossRefPubMedPubMedCentral

30.

Ring PS, Van de Ven AH. Developmental processes of cooperative interorganizational relationships. Acad Manag Rev. 1994;19(1):90–118. https://doi.org/10.5465/amr.1994.9410122009.CrossRef

31.

Latour B. Science in action: How to follow scientists and engineers through society. Cambridge: Harvard University Press; 1987.

32.

Wæraas A, Sataøen HL. Trapped in conformity? Translating reputation management into practice. Scand J Manag. 2014;30(2):242–53. https://doi.org/10.1016/j.scaman.2013.05.002.CrossRef

33.

Andersen H, Røvik KA, Ingebrigtsen T. Lean thinking in hospitals: is there a cure for the absence of evidence? A systematic review of reviews. BMJ Open. 2014;4(1):e003873. https://doi.org/10.1136/bmjopen-2013-003873.CrossRefPubMedPubMedCentral

34.

Røvik KA. Trender og translasjoner: Ideer som former det 21. århundrets organisasjoner. Oslo: Universitetsforl; 2007.

35.

Marshall GN, Hays RD. The patient satisfaction questionnaire short-form (PSQ-18). Santa Monica: RAND; 1994.

36.

Reynolds T, Thornicroft G, Abas M, Woods B, Hoe J, Leese M, Orrell M. Camberwell Assessment of Need for the Elderly (CANE): development, validity and reliability. Br J Psychiatry. 2000;176(5):444–52. https://doi.org/10.1192/bjp.176.5.444.CrossRefPubMed

37.

Orrell M, Hancock G. CANE: Camberwell assessment of need for the elderly. London: Gaskell; 2004.

38.

Jenkinson C, Layte R. Development and testing of the UK SF-12. J Health Serv Res Pol. 1997;2(1):14–8. https://doi.org/10.1177/135581969700200105.CrossRef

39.

Logsdon RG, Gibbons LE, McCurry SM, Teri L. Quality of life in Alzheimer’s disease: patient and caregiver reports. J Ment Health Aging. 1999;5:21–32.

40.

Mitseva A, Peterson C, Dafoulas G, Efthymiou A, Abildgaard A, Bellini S. ISISEMD evaluation framework for impact assessment of ICT pilot services for elderly with mild dementia, living in the community and their relatives. In: Proceedings of the Networking and Electronic Commerce Research Conference (NAEC); 2010. p. 123–48.

41.

Hancock GA, Woods B, Challis D, Orrell M. The needs of older people with dementia in residential care. Int J Geriatr Psychiatry. 2006;21(1):43–9. https://doi.org/10.1002/gps.1421.CrossRefPubMed

Title: Standards as applied in reality: a case study on the translation of standards in eHealth evaluation practice
Author: Monika Jurkeviciute
Publication date: 01-12-2019
Publisher: BioMed Central
Published in: BMC Medical Informatics and Decision Making / Issue 1/2019
Electronic ISSN: 1472-6947
DOI: https://doi.org/10.1186/s12911-019-0975-9

At a glance: The ONWARDS insulin icodec trials

Springer Medicine

Standards as applied in reality: a case study on the translation of standards in eHealth evaluation practice

Abstract

Background

Methods

Results

Conclusions

At a glance: The ONWARDS insulin icodec trials

Springer Medicine

Abstract

Background

Methods

Results

Conclusions

Please log in to get access to this content

Other articles of this Issue 1/2019

The effect of computerized decision support systems on cardiovascular risk factors: a systematic review and meta-analysis

Development process of a mobile electronic medical record for nurses: a single case study

An open access medical knowledge base for community driven diagnostic decision support system development

Evaluating the implementation of a personal health record for chronic primary and secondary care: a mixed methods approach

Effectiveness of decision aids for female BRCA1 and BRCA2 mutation carriers: a systematic review

Improving health information systems during an emergency: lessons and recommendations from an Ebola treatment centre in Sierra Leone