Skip to main content
Top
Published in: BMC Medical Informatics and Decision Making 1/2017

Open Access 01-12-2017 | Research article

Constructive eHealth evaluation: lessons from evaluation of EHR development in 4 Danish hospitals

Authors: Anna Marie Balling Høstgaard, Pernille Bertelsen, Christian Nøhr

Published in: BMC Medical Informatics and Decision Making | Issue 1/2017

Login to get access

Abstract

Background

Information and communication sources in the healthcare sector are replaced with new eHealth technologies. This has led to problems arising from the lack of awareness of the importance of end-user involvement in eHealth development and of the difficulties caused by using traditional summative evaluation methods. The Constructive eHealth evaluation method (CeHEM) provides a solution to these problems by offering an evaluation framework for supporting and facilitating end-user involvement during all phases of eHealth development. The aim of this paper is to support this process by sharing experiences of the eHealth evaluation method used in the introduction of electronic health records (EHR) in the North Denmark Region of Denmark. It is the first time the fully developed method and the experiences on using the CeHEM in all five phases of a full lifecycle framework is presented.

Methods

A case study evaluation of the EHR development process in the North Denmark Region was conducted from 2004 to 2010. The population consisted of clinicians, IT professionals, administrators, and vendors. The study involved 4 hospitals in the region. Data were collected using questionnaires, observations, interviews, and insight gathered from relevant documents.

Results

The evaluation showed a need for a) Early involvement of clinicians, b) The best possible representation of clinicians, and c) Workload reduction for those involved. The consequences of not providing this were a lack of ownership of decisions and negative attitudes towards the clinical benefits related to these decisions.
Further, the result disclosed that by following the above recommendations, and by providing feedback to the 4 actor groups, the physicians’ involvement was improved. As a result they took ownership of decisions and gained a positive attitude to the clinical benefits.

Conclusions

The CeHEM has proven successful in formative evaluation of EHR development and can point at important issues that need to be taken care of by management. The method provides a framework that takes care of feedback and learning during eHealth development. It can thus support successful eHealth development in a broader context while building on a well-known success factor: end-user involvement in eHealth development.
Literature
1.
go back to reference Ash JS, Stavri PZ, Kuperman GJ. A consensus statement on considerations for a successful CPOE implementation. J Am Med Inform Assoc. 2003;10(3):229–34. Ash JS, Stavri PZ, Kuperman GJ. A consensus statement on considerations for a successful CPOE implementation. J Am Med Inform Assoc. 2003;10(3):229–34.
3.
go back to reference Edmondson AC. Framing for learning: lessons in successful technology implementation. Calif Manage Rev [Internet]. 2003;45(2):34–54. Available from: http://search.ebscohost.com/login.aspx?direct=true&db=bsh&AN=9463628&site=ehost-live%5Cn. http://content.ebscohost.com.turing.library.northwestern.edu/ContentServer.asp?T=P&P=AN&K=9463628&S=R&D=bsh&EbscoContent=dGJyMNHr7ESep7c4zdnyOLCmr0yep7BSsqi4TLCWxWXS&Con.CrossRef Edmondson AC. Framing for learning: lessons in successful technology implementation. Calif Manage Rev [Internet]. 2003;45(2):34–54. Available from: http://​search.​ebscohost.​com/​login.​aspx?​direct=​true&​db=​bsh&​AN=​9463628&​site=​ehost-live%5Cn. http://content.ebscohost.com.turing.library.northwestern.edu/ContentServer.asp?T=P&P=AN&K=9463628&S=R&D=bsh&EbscoContent=dGJyMNHr7ESep7c4zdnyOLCmr0yep7BSsqi4TLCWxWXS&Con.CrossRef
4.
go back to reference Pagliari C. Design and evaluation in ehealth: challenges and implications for an interdisciplinary field. J Med Internet Res. 2007;9(2):e15. Pagliari C. Design and evaluation in ehealth: challenges and implications for an interdisciplinary field. J Med Internet Res. 2007;9(2):e15.
5.
go back to reference van Gemert-Pijnen JEWC, Nijland N, van Limburg M, Ossebaard HC, Kelders SM, Eysenbach G, et al. A holistic framework to improve the uptake and impact of eHealth technologies. J Med Internet Res. 2011;13(4):e111. van Gemert-Pijnen JEWC, Nijland N, van Limburg M, Ossebaard HC, Kelders SM, Eysenbach G, et al. A holistic framework to improve the uptake and impact of eHealth technologies. J Med Internet Res. 2011;13(4):e111.
6.
go back to reference Seemann J, Dinesen B, Gustafsson J. Interorganizational innovation in systemic networks: TELEKAT findings. Innov J. 2013;18(3):1-18. Seemann J, Dinesen B, Gustafsson J. Interorganizational innovation in systemic networks: TELEKAT findings. Innov J. 2013;18(3):1-18.
7.
go back to reference Berg M, Aarts J, Van der Lei J. ICT in health care: sociotechnical approaches. Methods Inf Med. 2003;42(4):297–301. Berg M, Aarts J, Van der Lei J. ICT in health care: sociotechnical approaches. Methods Inf Med. 2003;42(4):297–301.
9.
go back to reference Mumford E. Designing human systems for new technology : the ETHICS method. Manchester: Manchester Business School; 1983. p. 108. Mumford E. Designing human systems for new technology : the ETHICS method. Manchester: Manchester Business School; 1983. p. 108.
10.
go back to reference Berg M. Implementing information systems in health care organizations: myths and challenges. Int J Med Inform [Internet]. 2001;64(2–3):143–56. Available from: http://www.ncbi.nlm.nih.gov/pubmed/11734382.CrossRef Berg M. Implementing information systems in health care organizations: myths and challenges. Int J Med Inform [Internet]. 2001;64(2–3):143–56. Available from: http://​www.​ncbi.​nlm.​nih.​gov/​pubmed/​11734382.​CrossRef
11.
go back to reference Lorenzi NM, Riley RT. Organizational Aspects of Health Informatics. New York: Springer Verlag; 1995.CrossRef Lorenzi NM, Riley RT. Organizational Aspects of Health Informatics. New York: Springer Verlag; 1995.CrossRef
12.
go back to reference Lorenzi NM, Riley RT, Blyth AJC, Southon G, Dixon BJ. Antecedents of the people and organizational aspects of medical informatics. J Am Med Informatics Assoc [Internet]. 1997;4(2):79–93. Available from: http://www.jamia.org/cgi/content/abstract/4/2/79.CrossRef Lorenzi NM, Riley RT, Blyth AJC, Southon G, Dixon BJ. Antecedents of the people and organizational aspects of medical informatics. J Am Med Informatics Assoc [Internet]. 1997;4(2):79–93. Available from: http://​www.​jamia.​org/​cgi/​content/​abstract/​4/​2/​79.​CrossRef
13.
go back to reference Robertson A, Cresswell K, Takian A, Petrakaki D, Crowe S, Cornford T, et al. Implementation and adoption of nationwide electronic health records in secondary care in England: qualitative analysis of interim results from a prospective national evaluation. BMJ. 2010;341:c4564.CrossRefPubMedPubMedCentral Robertson A, Cresswell K, Takian A, Petrakaki D, Crowe S, Cornford T, et al. Implementation and adoption of nationwide electronic health records in secondary care in England: qualitative analysis of interim results from a prospective national evaluation. BMJ. 2010;341:c4564.CrossRefPubMedPubMedCentral
14.
go back to reference Bødker K, Kensing F, Simonsen J. Participatory IT design : designing for business and workplace realities. London: MIT Press; 2004. Bødker K, Kensing F, Simonsen J. Participatory IT design : designing for business and workplace realities. London: MIT Press; 2004.
15.
go back to reference Johnson CW. Why did that happen? Exploring the proliferation of barely usable software in healthcare systems. Qual Saf Health Care [Internet]. 2006;15 Suppl 1:i76–81. Available from: http://www.ncbi.nlm.nih.gov/pubmed/17142614%5Cn. http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=PMC2464864.CrossRef Johnson CW. Why did that happen? Exploring the proliferation of barely usable software in healthcare systems. Qual Saf Health Care [Internet]. 2006;15 Suppl 1:i76–81. Available from: http://​www.​ncbi.​nlm.​nih.​gov/​pubmed/​17142614%5Cn. http://​www.​pubmedcentral.​nih.​gov/​articlerender.​fcgi?​artid=​PMC2464864.​CrossRef
16.
go back to reference Paré G, Sicotte C, Jacques H, Jacques H. The effects of creating psychological ownership on physicians’ acceptance of clinical information systems. J Am Med Inform Assoc [Internet]. 2006;13(2):197–205. Available from: http://www.ncbi.nlm.nih.gov/pubmed/16357351.CrossRef Paré G, Sicotte C, Jacques H, Jacques H. The effects of creating psychological ownership on physicians’ acceptance of clinical information systems. J Am Med Inform Assoc [Internet]. 2006;13(2):197–205. Available from: http://​www.​ncbi.​nlm.​nih.​gov/​pubmed/​16357351.​CrossRef
17.
go back to reference Our History - The Tavistock Institute [Internet]. Available from: http://www.tavinstitute.org/who-we-are/our-history/. Our History - The Tavistock Institute [Internet]. Available from: http://​www.​tavinstitute.​org/​who-we-are/​our-history/​.​
18.
go back to reference Trist E. The Evolution of socio-technical systems: a conceptual framework and action research program. In: Conference on Organizational Design and Performance. 1980. p. 1–67. Trist E. The Evolution of socio-technical systems: a conceptual framework and action research program. In: Conference on Organizational Design and Performance. 1980. p. 1–67.
19.
go back to reference Cherns A. The principles of sociotechnical design. Hum Relations. 1976;29(8):783.CrossRef Cherns A. The principles of sociotechnical design. Hum Relations. 1976;29(8):783.CrossRef
20.
go back to reference Mumford E, Weir M. Computersystems in work design: the ETHICS method. New York: John Wiley; 1979. Mumford E, Weir M. Computersystems in work design: the ETHICS method. New York: John Wiley; 1979.
21.
go back to reference Mumford E. The story of socio-technical design: reflections on its successes, failures and potential. Inf Syst J. 2006;16:317–42.CrossRef Mumford E. The story of socio-technical design: reflections on its successes, failures and potential. Inf Syst J. 2006;16:317–42.CrossRef
22.
go back to reference Müller Jens, Kjaer-Rasmussen Jens NC. Proactive technology assessment as a tool in HIS development. In: Barber B, editor. MEDINFO 1989. Singapore: North Holland; 1989. p. 306. Müller Jens, Kjaer-Rasmussen Jens NC. Proactive technology assessment as a tool in HIS development. In: Barber B, editor. MEDINFO 1989. Singapore: North Holland; 1989. p. 306.
23.
go back to reference Protti D, Protti D. What can the American electronic health record (EHR) pioneers tell us about what it takes to be successful? Healthc Manag Forum [Internet]. 2002;15(2):33–5. Available from: http://linkinghub.elsevier.com/retrieve/pii/S0840470410605789, [cited 2016 Nov 2].CrossRef Protti D, Protti D. What can the American electronic health record (EHR) pioneers tell us about what it takes to be successful? Healthc Manag Forum [Internet]. 2002;15(2):33–5. Available from: http://​linkinghub.​elsevier.​com/​retrieve/​pii/​S084047041060578​9, [cited 2016 Nov 2].CrossRef
24.
go back to reference Edwards PK. Conceptual and methodological issues in evaluating emergent programs. Eval Program Plann. 1987;10(1):27–33.CrossRef Edwards PK. Conceptual and methodological issues in evaluating emergent programs. Eval Program Plann. 1987;10(1):27–33.CrossRef
25.
go back to reference Hornik R. Shedding some light on evaluation’s myths. Dev Commun Rep. 1980;29:1–4. Hornik R. Shedding some light on evaluation’s myths. Dev Commun Rep. 1980;29:1–4.
27.
go back to reference Sheikh A, Cornford T, Barber N, Avery A, Takian A, Lichtner V, et al. Implementation and adoption of nationwide electronic health records in secondary care in England: final qualitative results from prospective national evaluation in "early adopter" hospitals. BMJ [Internet]. 2011;343(1):d6054. Available from: http://www.ncbi.nlm.nih.gov/pubmed/22006942, [cited 2016 Nov 2].CrossRef Sheikh A, Cornford T, Barber N, Avery A, Takian A, Lichtner V, et al. Implementation and adoption of nationwide electronic health records in secondary care in England: final qualitative results from prospective national evaluation in "early adopter" hospitals. BMJ [Internet]. 2011;343(1):d6054. Available from: http://​www.​ncbi.​nlm.​nih.​gov/​pubmed/​22006942, [cited 2016 Nov 2].CrossRef
28.
go back to reference Wears RL, Berg M. Computer technology and clinical work: still waiting for Godot. JAMA. 2005;293(10):1261–3.CrossRefPubMed Wears RL, Berg M. Computer technology and clinical work: still waiting for Godot. JAMA. 2005;293(10):1261–3.CrossRefPubMed
29.
go back to reference Ammenwerth E, de Keizer N. An inventory of evaluation studies of information technology in health care trends in evaluation research 1982–2002. Methods Inf Med [Internet]. 2005;44(1):44–56. Available from: http://www.ncbi.nlm.nih.gov/pubmed/15778794. Ammenwerth E, de Keizer N. An inventory of evaluation studies of information technology in health care trends in evaluation research 1982–2002. Methods Inf Med [Internet]. 2005;44(1):44–56. Available from: http://​www.​ncbi.​nlm.​nih.​gov/​pubmed/​15778794.​
30.
go back to reference Douma KFL, Karsenberg K, Hummel MJM, Bueno-de-Mesquita JM, van Harten WH. Methodology of constructive technology assessment in health care. Int J Technol Assess Health Care [Internet]. 2007;23(2):162–8. Available from: http://www.journals.cambridge.org/abstract_S0266462307070262.CrossRef Douma KFL, Karsenberg K, Hummel MJM, Bueno-de-Mesquita JM, van Harten WH. Methodology of constructive technology assessment in health care. Int J Technol Assess Health Care [Internet]. 2007;23(2):162–8. Available from: http://​www.​journals.​cambridge.​org/​abstract_​S026646230707026​2.​CrossRef
31.
go back to reference Dahler-Larsen P, Krogstrup HK. Evaluering og institutionelle standarder: Nyinstitutionelle betragtninger af evaluering som vor tids ritual. Politica. 2000;32(3):283-98. Dahler-Larsen P, Krogstrup HK. Evaluering og institutionelle standarder: Nyinstitutionelle betragtninger af evaluering som vor tids ritual. Politica. 2000;32(3):283-98.
32.
go back to reference Catwell L, Sheikh A. Evaluating eHealth interventions: the need for continuous systemic evaluation. PLoS Medicine. 2009;6(8)e1000126. Catwell L, Sheikh A. Evaluating eHealth interventions: the need for continuous systemic evaluation. PLoS Medicine. 2009;6(8)e1000126.
33.
go back to reference Bijker WE. How is technology made?-That is the question! Cambridge J Econ. 2009;34(1):63–76.CrossRef Bijker WE. How is technology made?-That is the question! Cambridge J Econ. 2009;34(1):63–76.CrossRef
34.
go back to reference Schot J, Rip ARIE. The past and future of constructive technology assessment. Technol Forecast Soc Change. 1997;268(1996):251–68.CrossRef Schot J, Rip ARIE. The past and future of constructive technology assessment. Technol Forecast Soc Change. 1997;268(1996):251–68.CrossRef
35.
go back to reference Dehar MA, Casswell S, Duignan P. Formative and process evaluation of health promotion and disease prevention programs. Eval Rev. 1993;17(2):204–20.CrossRef Dehar MA, Casswell S, Duignan P. Formative and process evaluation of health promotion and disease prevention programs. Eval Rev. 1993;17(2):204–20.CrossRef
36.
go back to reference Stetler CB, Legro MW, Wallace CM, Bowman C, Guihan M, Hagedorn H, et al. The role of formative evaluation in implementation research and the QUERI experience. J Gen Intern Med. 2006;21(Suppl2):S1-S8. Stetler CB, Legro MW, Wallace CM, Bowman C, Guihan M, Hagedorn H, et al. The role of formative evaluation in implementation research and the QUERI experience. J Gen Intern Med. 2006;21(Suppl2):S1-S8.
37.
go back to reference Smits R, Leyten J, Den Hertog P. Technology assessment and technology policy in Europe: new concepts, new goals, new infrastructures. Policy Sci. 1995;28(3):271–99.CrossRef Smits R, Leyten J, Den Hertog P. Technology assessment and technology policy in Europe: new concepts, new goals, new infrastructures. Policy Sci. 1995;28(3):271–99.CrossRef
38.
go back to reference Smits R, Leyten J. Key issues in the institutionalization of technology assessment. Development of technology assessment in five European countries and the USA. Futures. 1988;20(1):19–36.CrossRef Smits R, Leyten J. Key issues in the institutionalization of technology assessment. Development of technology assessment in five European countries and the USA. Futures. 1988;20(1):19–36.CrossRef
39.
go back to reference Kushniruk AW, Patel VL. Cognitive and usability engineering methods for the evaluation of clinical information systems. J Biomed Inform. 2004;37(1):56–76.CrossRefPubMed Kushniruk AW, Patel VL. Cognitive and usability engineering methods for the evaluation of clinical information systems. J Biomed Inform. 2004;37(1):56–76.CrossRefPubMed
40.
go back to reference Brender J, Brender J. Trends in assessment of IT-based solutions in healthcare and recommendations for the future. Int J Med Inform [Internet]. 1998;52(1–3):217–27. Available from: http://linkinghub.elsevier.com/retrieve/pii/S1386505698001403, [cited 2016 Nov 6].CrossRef Brender J, Brender J. Trends in assessment of IT-based solutions in healthcare and recommendations for the future. Int J Med Inform [Internet]. 1998;52(1–3):217–27. Available from: http://​linkinghub.​elsevier.​com/​retrieve/​pii/​S138650569800140​3, [cited 2016 Nov 6].CrossRef
41.
go back to reference Berleur J, NT N, R D. Technology Assessment in Medical and Health Care Informatics: A Clarification of the Concepts. In: EMSJ van G, Talmon J, editors. Assessment and Evaluation of Information technology in Medicine. Amsterdam: IOS Press; 1995. p. 21–34. Berleur J, NT N, R D. Technology Assessment in Medical and Health Care Informatics: A Clarification of the Concepts. In: EMSJ van G, Talmon J, editors. Assessment and Evaluation of Information technology in Medicine. Amsterdam: IOS Press; 1995. p. 21–34.
42.
go back to reference Vissers MC, Talmon JL. Evaluating the MACRO solution. Multimedia application for clinical research in oncology. Stud Health Technol Inform. 1997;43 Pt B:741–5.PubMed Vissers MC, Talmon JL. Evaluating the MACRO solution. Multimedia application for clinical research in oncology. Stud Health Technol Inform. 1997;43 Pt B:741–5.PubMed
43.
go back to reference Nøhr C, Bernstein K. Implementation and diffusion of medical informatics; the case of electronic prescribing communication between practitioners and pharmacies. Prim Heal Care Spec Gr Br Comput Soc. 1992;9:14–8. Nøhr C, Bernstein K. Implementation and diffusion of medical informatics; the case of electronic prescribing communication between practitioners and pharmacies. Prim Heal Care Spec Gr Br Comput Soc. 1992;9:14–8.
44.
go back to reference Nøhr C. Computer-assisted decision support - improved health? A prospective technology assessment. Aalborg: Aalborg University; 1991. Nøhr C. Computer-assisted decision support - improved health? A prospective technology assessment. Aalborg: Aalborg University; 1991.
45.
go back to reference Nøhr C, Bernstein K. Can HIS be developed without organizational change? The creative potential in user participation. In: MEDINFO 89. Amsterdam: Elsevier Science; 1989. p. 5. Nøhr C, Bernstein K. Can HIS be developed without organizational change? The creative potential in user participation. In: MEDINFO 89. Amsterdam: Elsevier Science; 1989. p. 5.
46.
go back to reference Nøhr C. Involving end-users in prospective technology assessment. The case of computer based information technology. In: The sixth meeting in the international society of technology assessment in health Care (ISTAHC). Cambridge; 1990:6(1). Nøhr C. Involving end-users in prospective technology assessment. The case of computer based information technology. In: The sixth meeting in the international society of technology assessment in health Care (ISTAHC). Cambridge; 1990:6(1).
47.
go back to reference Nøhr C. Social Carriers of Technology. An Actor Oriented View on Diffusion. In: The Seventh Meeting in the International Society of Technology Assessment in Health Care. Helsinki Finland; 1991. Nøhr C. Social Carriers of Technology. An Actor Oriented View on Diffusion. In: The Seventh Meeting in the International Society of Technology Assessment in Health Care. Helsinki Finland; 1991.
48.
go back to reference Nøhr C, Wiegmann I. Technology assessment in danish hospitals. A matter og technology agreements. In: Technology and Democracy: The Use and Impact of Technology Assessment in Europe. Cambridge: Cambridge University Press; 1992. Nøhr C, Wiegmann I. Technology assessment in danish hospitals. A matter og technology agreements. In: Technology and Democracy: The Use and Impact of Technology Assessment in Europe. Cambridge: Cambridge University Press; 1992.
49.
go back to reference Edquist C, Edqvist O. Social carriers of techniques for development. J Peace Res. 1979;16:313–31.CrossRef Edquist C, Edqvist O. Social carriers of techniques for development. J Peace Res. 1979;16:313–31.CrossRef
50.
go back to reference Høstgaard A.M. Succeskriterier og barrierer i Sundheds-it udvikling. Case studie af EPHudbudsprocessen i Region Nordjylland med fokus på slutbruger indvolvering (Success factors and barriers in Healthcare IT development. Case study of the EPH tender process in The North Denmark Region with a focus on end-user's involvement). Delft: River Publishers; 2015. Available at: http://www.riverpublishers.com/pdf/ebook/RP_E9788793379343.pdf. Høstgaard A.M. Succeskriterier og barrierer i Sundheds-it udvikling. Case studie af EPHudbudsprocessen i Region Nordjylland med fokus på slutbruger indvolvering (Success factors and barriers in Healthcare IT development. Case study of the EPH tender process in The North Denmark Region with a focus on end-user's involvement). Delft: River Publishers; 2015. Available at: http://​www.​riverpublishers.​com/​pdf/​ebook/​RP_​E9788793379343.​pdf.
51.
go back to reference Müller J, Kjær-Rasmussen J, Nøhr C. Perspektiver for edb-teknologi i sygehusvæsenet. Aalborg: Centertrykkeriet AUC; 1988. ISBN: 87-89088-06-9. Müller J, Kjær-Rasmussen J, Nøhr C. Perspektiver for edb-teknologi i sygehusvæsenet. Aalborg: Centertrykkeriet AUC; 1988. ISBN: 87-89088-06-9.
52.
go back to reference Checkland P. Soft systems methodology: a 30-year retrospective. Syst Res Behav Sci [Internet]. 1999;58(3):11–58. Available from: http://books.google.com/books?id=R2dQAAAAMAAJ&pgis=1. Checkland P. Soft systems methodology: a 30-year retrospective. Syst Res Behav Sci [Internet]. 1999;58(3):11–58. Available from: http://​books.​google.​com/​books?​id=​R2dQAAAAMAAJ&​pgis=​1.​
53.
go back to reference Checkland P. Systems thinking, systems practice. 1st. ed. Wiley J, editor. Systems thinking systems practice. Chichester; 1981. 356 p. Checkland P. Systems thinking, systems practice. 1st. ed. Wiley J, editor. Systems thinking systems practice. Chichester; 1981. 356 p.
54.
go back to reference Checkland P, Scholes J. Soft systems methodology in action. NY: Wiley; 1999. p. 418. Checkland P, Scholes J. Soft systems methodology in action. NY: Wiley; 1999. p. 418.
55.
go back to reference Bertelsen P, Madsen I. “Participatorisk stuegangsanalyse med fokus på journalanvendelse: medarbejderindflydelse på arbejdsorganisering før og efter EPJ” (Participatory ward-analysis with focus on how to use the EHR: the degree of user- influence on the organization of work befor. In: Nøhr C, Høstgaard AM, editors. Håndbog i organisatoriske forandringer i forbindelse med implementering af elektroniske patientjournaler på sygehuse (Handbook in organizational changes when implementing EHR on hospitals). Aalborg: Aalborg University Press; 2004. p. 77–93. Bertelsen P, Madsen I. “Participatorisk stuegangsanalyse med fokus på journalanvendelse: medarbejderindflydelse på arbejdsorganisering før og efter EPJ” (Participatory ward-analysis with focus on how to use the EHR: the degree of user- influence on the organization of work befor. In: Nøhr C, Høstgaard AM, editors. Håndbog i organisatoriske forandringer i forbindelse med implementering af elektroniske patientjournaler på sygehuse (Handbook in organizational changes when implementing EHR on hospitals). Aalborg: Aalborg University Press; 2004. p. 77–93.
56.
go back to reference Nøhr C, Sørensen M, Kushniruk A. Method for testing a CPOE system in the medication process in a cardiology ward. In: Safran C, Reti H, Marin HF, editors. MEDINFO 2010 : Proceedings of the 13th World Congress on Medical Informatics. Cape Town: Ios Press; 2010. p. 183–7. Nøhr C, Sørensen M, Kushniruk A. Method for testing a CPOE system in the medication process in a cardiology ward. In: Safran C, Reti H, Marin HF, editors. MEDINFO 2010 : Proceedings of the 13th World Congress on Medical Informatics. Cape Town: Ios Press; 2010. p. 183–7.
57.
go back to reference Nøhr C, Botin L, Porot M. Hvordan styres operationsprogrammer for patienter. Tidsskr Dansk Sundhedsvaesen. 2006;82(7):244–7. Nøhr C, Botin L, Porot M. Hvordan styres operationsprogrammer for patienter. Tidsskr Dansk Sundhedsvaesen. 2006;82(7):244–7.
58.
go back to reference Nøhr C, Botin L, Botin L. Methodology for analysis of work practice with video observation. Stud Health Technol Inform [Internet]. 2007;130:291–7. Available from: http://www.ncbi.nlm.nih.gov/pubmed/17917202, [cited 2016 Nov 8]. Nøhr C, Botin L, Botin L. Methodology for analysis of work practice with video observation. Stud Health Technol Inform [Internet]. 2007;130:291–7. Available from: http://​www.​ncbi.​nlm.​nih.​gov/​pubmed/​17917202, [cited 2016 Nov 8].
59.
go back to reference Nøhr C, Boye N. Towards computer supported clinical activity: a roadmap based on empirical knowledge and some theoretical reflections. In: Human, social, and organizational aspects of health information systems [Internet]. 2008. p. 67–83. Available from: http://services.igi-global.com/resolvedoi/resolve.aspx?. doi: 10.4018/978-1-59904-792-8. Nøhr C, Boye N. Towards computer supported clinical activity: a roadmap based on empirical knowledge and some theoretical reflections. In: Human, social, and organizational aspects of health information systems [Internet]. 2008. p. 67–83. Available from: http://​services.​igi-global.​com/​resolvedoi/​resolve.​aspx?​.​ doi: 10.​4018/​978-1-59904-792-8.
60.
go back to reference Elliott G. Global business information technology : an integrated systems approach. Exeter: Addison Wesley; 2004. p 503. Elliott G. Global business information technology : an integrated systems approach. Exeter: Addison Wesley; 2004. p 503.
61.
go back to reference Barber N, Cornford T, Klecun E, Lichtner V, Takian A. User-centered healthcare IT: meaningful or meaningless? 2011. Available from: http://eprints.lse.ac.uk/39256/, [cited 2016 Nov 8]. Barber N, Cornford T, Klecun E, Lichtner V, Takian A. User-centered healthcare IT: meaningful or meaningless? 2011. Available from: http://​eprints.​lse.​ac.​uk/​39256/​, [cited 2016 Nov 8].
62.
go back to reference Høstgaard AMB. Constructive eHealth evaluation: In: Reshaping medical practice and care with health information systems [Internet]. IGI Global; 2016 [cited 2016 Nov 8]. p. 44–86. Available from: http://services.igi-global.com/resolvedoi/resolve.aspx?. doi: 10.4018/978-1-4666-9870-3.ch003. Høstgaard AMB. Constructive eHealth evaluation: In: Reshaping medical practice and care with health information systems [Internet]. IGI Global; 2016 [cited 2016 Nov 8]. p. 44–86. Available from: http://​services.​igi-global.​com/​resolvedoi/​resolve.​aspx?​.​ doi: 10.​4018/​978-1-4666-9870-3.​ch003.
63.
go back to reference Høstgaard AMB, Bertelsen P, Petersen LS, Nøhr C, et al. Constructive technology assessment for HIT development: learning, feedback and user involvement. In: Billika G, Bygholm A, Dencker M, Fossum M, Galster G, Hartvigsen G, editors. Scandinavian Conference on Health Informatics 2013 [Internet]. Linköping: Linköping University Electronic Press; 2013. p. 33–7. Available from: http://shi2013.org/, [cited 2016 Nov 8]. Høstgaard AMB, Bertelsen P, Petersen LS, Nøhr C, et al. Constructive technology assessment for HIT development: learning, feedback and user involvement. In: Billika G, Bygholm A, Dencker M, Fossum M, Galster G, Hartvigsen G, editors. Scandinavian Conference on Health Informatics 2013 [Internet]. Linköping: Linköping University Electronic Press; 2013. p. 33–7. Available from: http://​shi2013.​org/​, [cited 2016 Nov 8].
64.
go back to reference Brender J. Handbook of evaluation methods for health informatics [Internet]. 2006. p. 253–320. Available from: http://www.sciencedirect.com/science/article/pii/B9780123704641500113. Brender J. Handbook of evaluation methods for health informatics [Internet]. 2006. p. 253–320. Available from: http://​www.​sciencedirect.​com/​science/​article/​pii/​B978012370464150​0113.​
65.
go back to reference Yusof MM, Papazafeiropoulou A, Paul RJ, Stergioulas LK. Investigating evaluation frameworks for health information systems. Int J Med Inform. 2008;77(6):377–85.CrossRefPubMed Yusof MM, Papazafeiropoulou A, Paul RJ, Stergioulas LK. Investigating evaluation frameworks for health information systems. Int J Med Inform. 2008;77(6):377–85.CrossRefPubMed
66.
go back to reference Kørnøv L. Public participation. In: Kørnøv L, Thrane M, Remmen A, Lund H, editors. Tools for sustainable development. Aalborg: Aalborg Universitets Forlag; 2007. p. 721–38. Kørnøv L. Public participation. In: Kørnøv L, Thrane M, Remmen A, Lund H, editors. Tools for sustainable development. Aalborg: Aalborg Universitets Forlag; 2007. p. 721–38.
67.
go back to reference Mitchell RK, Agle BR, Wood DJ. Toward a theory of stakeholder identification and salience: defining the principle of who and what really counts. Acad Manag Rev. 1997;22(4):853–86. Mitchell RK, Agle BR, Wood DJ. Toward a theory of stakeholder identification and salience: defining the principle of who and what really counts. Acad Manag Rev. 1997;22(4):853–86.
69.
go back to reference Ehn P. Work-oriented design of computer artifacts. Lawrence Erlbaum Assoc: Umeå; 1988. Ehn P. Work-oriented design of computer artifacts. Lawrence Erlbaum Assoc: Umeå; 1988.
70.
71.
go back to reference Brandt E. Designing exploratory design games: a framework for participation in participatory design? In: Participatory design conference. 2006. Brandt E. Designing exploratory design games: a framework for participation in participatory design? In: Participatory design conference. 2006.
72.
go back to reference Bertelsen P, Kanstrup AM. User innovation management a handbook. 1st ed. Aalborg, Denmark: Aalborg University Press; 2011. Bertelsen P, Kanstrup AM. User innovation management a handbook. 1st ed. Aalborg, Denmark: Aalborg University Press; 2011.
73.
go back to reference Nielsen J. 10 Usability heuristics for user interface design [Internet]. 1995 [cited 2016 Nov 20]. Available from: https://www.nngroup.com/articles/ten-usability-heuristics/. Nielsen J. 10 Usability heuristics for user interface design [Internet]. 1995 [cited 2016 Nov 20]. Available from: https://​www.​nngroup.​com/​articles/​ten-usability-heuristics/​.​
74.
go back to reference Nykänen P, Brender J, Talmon J, de Keizer N, Rigby M, Beuscart-Zephir M-C, et al. Guideline for good evaluation practice in health informatics (GEP-HI). Int J Med Inform [Internet]. 2011;80(12):815–27. Available from: http://www.ncbi.nlm.nih.gov/pubmed/21920809, [cited 2016 Nov 8].CrossRef Nykänen P, Brender J, Talmon J, de Keizer N, Rigby M, Beuscart-Zephir M-C, et al. Guideline for good evaluation practice in health informatics (GEP-HI). Int J Med Inform [Internet]. 2011;80(12):815–27. Available from: http://​www.​ncbi.​nlm.​nih.​gov/​pubmed/​21920809, [cited 2016 Nov 8].CrossRef
75.
go back to reference Kensing F, Simonsen J, Bodker K. MUST: a method for participatory design. In: Kensing F, Dykstra-Erickson E, Blomberg J, editors. The fourth beinnal conference on participatory design Cambridge. Massachusetts: CPSR; 1996. p. 129–40. Kensing F, Simonsen J, Bodker K. MUST: a method for participatory design. In: Kensing F, Dykstra-Erickson E, Blomberg J, editors. The fourth beinnal conference on participatory design Cambridge. Massachusetts: CPSR; 1996. p. 129–40.
76.
go back to reference Kaufman D, Roberts WD, Merrill J, Lai TY, Bakken S. Applying an evaluation tramework tor health information system design, development, and implementation. Nurs Res. 2006;55(2):S37–42.CrossRefPubMed Kaufman D, Roberts WD, Merrill J, Lai TY, Bakken S. Applying an evaluation tramework tor health information system design, development, and implementation. Nurs Res. 2006;55(2):S37–42.CrossRefPubMed
78.
go back to reference Greenhalgh T, Russell J. Why do evaluations of eHealth programs fail? An alternative set of guiding principles. PLoS Med. 2010;7(11):e1000360. Greenhalgh T, Russell J. Why do evaluations of eHealth programs fail? An alternative set of guiding principles. PLoS Med. 2010;7(11):e1000360.
79.
go back to reference Harrison MI, Koppel R, Har-Lev S. Unintented consequences of information technologies in health care - an interactive sociotechnical analysis. J Am Med Inform Assoc. 2007;14:542–9.CrossRefPubMedPubMedCentral Harrison MI, Koppel R, Har-Lev S. Unintented consequences of information technologies in health care - an interactive sociotechnical analysis. J Am Med Inform Assoc. 2007;14:542–9.CrossRefPubMedPubMedCentral
80.
go back to reference Cresswell KM, Worth A, Sheikh A. Actor-network theory and its role in understanding the implementation of information technology developments in healthcare. BMC Med Inform Decis Mak. 2010;10:67.CrossRefPubMedPubMedCentral Cresswell KM, Worth A, Sheikh A. Actor-network theory and its role in understanding the implementation of information technology developments in healthcare. BMC Med Inform Decis Mak. 2010;10:67.CrossRefPubMedPubMedCentral
81.
go back to reference Kushniruk AW, Turner P. Who’s users? Participation and empowerment in socio-technical approaches to health IT developments. In: Studies in Health Technology and Informatics. 2011. p. 280–5. Kushniruk AW, Turner P. Who’s users? Participation and empowerment in socio-technical approaches to health IT developments. In: Studies in Health Technology and Informatics. 2011. p. 280–5.
82.
go back to reference European Commission. eHealth task force report – redesigning health in Europe for 2020 [Internet]. Luxembourg: European Commission; 2012. Available from: http://www.e-health-com.eu/fileadmin/user_upload/dateien/Downloads/redesigning_health-eu-for2020-ehtf-report2012_01.pdf. European Commission. eHealth task force report – redesigning health in Europe for 2020 [Internet]. Luxembourg: European Commission; 2012. Available from: http://​www.​e-health-com.​eu/​fileadmin/​user_​upload/​dateien/​Downloads/​redesigning_​health-eu-for2020-ehtf-report2012_​01.​pdf.​
83.
go back to reference Stake R. The Art of Case Study Research. Calif: SAGE publishing; 1995. p. 192. Stake R. The Art of Case Study Research. Calif: SAGE publishing; 1995. p. 192.
84.
go back to reference Nøhr C, Høstgaard AMB, Vingtoft S. Brugererfaringer fra GEPKA-projektet på afdeling M, Århus Sygehus. Aalborg: Virtual Centre for Health Informatics; 2005. Nøhr C, Høstgaard AMB, Vingtoft S. Brugererfaringer fra GEPKA-projektet på afdeling M, Århus Sygehus. Aalborg: Virtual Centre for Health Informatics; 2005.
85.
go back to reference Scott JT, Rundall TG, Vogt TM, Hsu J. Kaiser permanente’s experience of implementing an electronic medical record: a qualitative study. BMJ Br Med J [Internet]. 2005;331(7528):1313–6. Available from: http://www.jstor.org/stable/25455558.CrossRef Scott JT, Rundall TG, Vogt TM, Hsu J. Kaiser permanente’s experience of implementing an electronic medical record: a qualitative study. BMJ Br Med J [Internet]. 2005;331(7528):1313–6. Available from: http://​www.​jstor.​org/​stable/​25455558.​CrossRef
86.
go back to reference Brinkmann S, Kvale S. InterViews: learning the craft of qualitative research interviewing. Thirdth ed. Los Angeles: SAGE Publications; 2014. Brinkmann S, Kvale S. InterViews: learning the craft of qualitative research interviewing. Thirdth ed. Los Angeles: SAGE Publications; 2014.
87.
go back to reference Muhr T. ATLAS TI 5.0 users guide. manag educ. 2004. Muhr T. ATLAS TI 5.0 users guide. manag educ. 2004.
88.
go back to reference Høstgaard A, Bertelsen P, Nøhr C. Methods to identify, study and understand end-user participation in HIT development. BMC Med Inform Decis Mak [Internet]. 2011;11(1):57. Available from: http://www.biomedcentral.com/1472-6947/11/57.CrossRef Høstgaard A, Bertelsen P, Nøhr C. Methods to identify, study and understand end-user participation in HIT development. BMC Med Inform Decis Mak [Internet]. 2011;11(1):57. Available from: http://​www.​biomedcentral.​com/​1472-6947/​11/​57.​CrossRef
89.
go back to reference Høstgaard AM, Bertelsen P, Nøhr C. Evaluering af pilottest af Klinisk Proces “Vieweren”. Aalborg: Virtual Centre for Health Informatics, Aalborg University; 2011. Høstgaard AM, Bertelsen P, Nøhr C. Evaluering af pilottest af Klinisk Proces “Vieweren”. Aalborg: Virtual Centre for Health Informatics, Aalborg University; 2011.
90.
go back to reference Høstgaard AMB, Bertelsen P, Nøhr C. How are clinicians involved in EHR planning? A process analysis case study of a region in Denmark. In: Safran C, Reti S, Marin HF, editors. MEDINFO 2010: Proceedings of the 13th World Congress on Medical Informatics. Cape Town: Ios Press; 2010. p. 121–5. Høstgaard AMB, Bertelsen P, Nøhr C. How are clinicians involved in EHR planning? A process analysis case study of a region in Denmark. In: Safran C, Reti S, Marin HF, editors. MEDINFO 2010: Proceedings of the 13th World Congress on Medical Informatics. Cape Town: Ios Press; 2010. p. 121–5.
91.
go back to reference Høstgaard AMB. End-User Participation in Health IT Development. In: Belkhamza Z, Wafa SA, editors. Measuring Organizational Information Systems Success [Internet]. Hershey: IGI Global; 2012. [cited 2016 Nov 8]. p. 318–40. Available from: http://services.igi-global.com/resolvedoi/resolve.aspx?. doi: 10.4018/978-1-4666-0170-3.ch016 Høstgaard AMB. End-User Participation in Health IT Development. In: Belkhamza Z, Wafa SA, editors. Measuring Organizational Information Systems Success [Internet]. Hershey: IGI Global; 2012. [cited 2016 Nov 8]. p. 318–40. Available from: http://​services.​igi-global.​com/​resolvedoi/​resolve.​aspx?​.​ doi: 10.​4018/​978-1-4666-0170-3.​ch016
92.
go back to reference Avison D, Fitzgerald G. Effective Technical and Human Implementation of Computer-based System (ETHICS). In: Avison DE, Fitzgerald G, editors. Information system development: methodologies, techniques and tools. Berkshire: The McGraw-Hill Companies; 1995. p. 353–64. Avison D, Fitzgerald G. Effective Technical and Human Implementation of Computer-based System (ETHICS). In: Avison DE, Fitzgerald G, editors. Information system development: methodologies, techniques and tools. Berkshire: The McGraw-Hill Companies; 1995. p. 353–64.
93.
go back to reference Simonsen J, Hertzum M. A Regional PD Strategy for EPR Systems: Evidence- Based IT Development. In: Jacucci G, Kensing F, Wagner I, Blomberg J, editors. PDC 2006: Ninth Biennial Conference on Participatory Design, Vol II. CPSR, p.125–128. Simonsen J, Hertzum M. A Regional PD Strategy for EPR Systems: Evidence- Based IT Development. In: Jacucci G, Kensing F, Wagner I, Blomberg J, editors. PDC 2006: Ninth Biennial Conference on Participatory Design, Vol II. CPSR, p.125–128.
94.
go back to reference Simonsen J, Hertzum M. Participative design and the challenges of large-scale systems: extending the iterative PD approach. Proc Tenth Anniv Conf Particip Des. 2008;2008:1–10. Simonsen J, Hertzum M. Participative design and the challenges of large-scale systems: extending the iterative PD approach. Proc Tenth Anniv Conf Particip Des. 2008;2008:1–10.
95.
go back to reference Berg M, Winthereik BR. Waiting for Godot: episodes from the history of patient records. In: Health Information Management: Integrating Information and Communication Technology in Health Care Work. 2003. p. 234.CrossRef Berg M, Winthereik BR. Waiting for Godot: episodes from the history of patient records. In: Health Information Management: Integrating Information and Communication Technology in Health Care Work. 2003. p. 234.CrossRef
97.
go back to reference The Danish Data Protection Agency. The act on processing of personal data [Internet]. Copenhagen; 2014. Available from: https://www.datatilsynet.dk/english/the-act-on-processing-of-personal-data/read-the-act-on-processing-of-personal-data/compiled-version-of-the-act-on-processing-of-personal-data/. The Danish Data Protection Agency. The act on processing of personal data [Internet]. Copenhagen; 2014. Available from: https://​www.​datatilsynet.​dk/​english/​the-act-on-processing-of-personal-data/​read-the-act-on-processing-of-personal-data/​compiled-version-of-the-act-on-processing-of-personal-data/​.​
Metadata
Title
Constructive eHealth evaluation: lessons from evaluation of EHR development in 4 Danish hospitals
Authors
Anna Marie Balling Høstgaard
Pernille Bertelsen
Christian Nøhr
Publication date
01-12-2017
Publisher
BioMed Central
Published in
BMC Medical Informatics and Decision Making / Issue 1/2017
Electronic ISSN: 1472-6947
DOI
https://doi.org/10.1186/s12911-017-0444-2

Other articles of this Issue 1/2017

BMC Medical Informatics and Decision Making 1/2017 Go to the issue