Top

BMC Medical Informatics and Decision Making

Published in:

Open Access 01-12-2020 | Research article

A combination of two methods for evaluating the usability of a hospital information system

Authors: Reza Khajouei, Fatemeh Farahani

Published in: BMC Medical Informatics and Decision Making | Issue 1/2020

Abstract

Background

None of the evaluation methods can identify all the usability problems of information systems. So far, no study has sufficiently investigated the potential of a combination of these methods to identify usability problems. The present study aimed at examining the potential for combining two commonly utilized user-based and expert-based methods to evaluate the usability of a hospital information system.

Methods

Think aloud (TA) and Heuristic evaluation (HE) methods were used to identify the usability problems of two subsystems of the Social Security Electronic System in Iran. To this end, the problems were categorized into five groups based on ISO-Nielsen usability attributes. The Chi-square test was applied to compare the intended methods based on the total number of problems and the number of problems within each group, followed by utilizing the Mann-Whitney U test to compare the mean severity scores of these methods.

Results

The evaluation by combining these methods yielded 423 problems of which 75% varied between the methods. The two methods were significantly different in terms of the total number of problems, the number of problems in each usability group, and the mean severity of two satisfaction and efficiency attributes (P < 0.05). However, no significant difference was observed between the two methods based on the mean severity of problems and severity scores related to three usability attributes i.e., effectiveness, learnability, and error prevention (P > 0.05). In addition, the mean severity of problems identified by each method was at the “Major” level.

Conclusion

Based on the results, although the mean severity scores of the identified problems were not significantly different, these methods identify heterogeneous problems. HE mainly identifies problems related to satisfaction, learnability, and error prevention while TA detects problems related to effectiveness and efficiency attributes. Therefore, using a combination of these two methods can identify a wider range of usability problems.

Iso W. 9241–11. Ergonomic requirements for office work with visual display terminals (VDTs). Int Org Standard. 1998;45:9.

Nielsen J. Usability 101: introduction to usability; 2003.

Patterson ES, Doebbeling BN, Fung CH, Militello L, Anders S, Asch SM. Identifying barriers to the effective use of clinical reminders: bootstrapping multiple methods. J Biomed Inform. 2005;38(3):189–99. https://doi.org/10.1016/j.jbi.2004.11.015.CrossRefPubMed

Kjeldskov J, Skov MB, Stage J. A longitudinal study of usability in health care: does time heal? Int J Med Inform. 2010;79(6):e135–43. https://doi.org/10.1016/j.ijmedinf.2008.07.008.

Peute LW, Spithovenª R, WM PJ. Usability studies on interactive health information systems; where do we stand? Stud Health Technol Inform. 2008;136:327-32. https://doi.org/10.3233/978-1-58603-864-9-327. PMID: 18487752.

Jaspers MW. A comparison of usability methods for testing interactive health technologies: methodological aspects and empirical evidence. Int J Med Inform. 2009;78(5):340–53. https://doi.org/10.1016/j.ijmedinf.2008.10.002.CrossRefPubMed

Karat CM. A comparison of user interface evaluation methods. In: Usability inspection methods. New York: Wiley; 1994. p. 203–33.

Jeffries R, Miller JR, Wharton C, Uyeda K. User interface evaluation in the real world: a comparison of four techniques. In Proceedings of the SIGCHI conference on Human factors in computing systems; 1991 Mar 1; Orleans Louisiana: Association for Computing Machinery; 1991. p. 119-24. https://doi.org/10.1145/108844.108862.

Yen PY, Bakken S. A comparison of usability evaluation methods: heuristic evaluation versus end-user think-aloud protocol–an example from a web-based communication tool for nurse scheduling. In: AMIA annual symposium proceedings, vol. 2009: American Medical Informatics Association; 2009. p. 714. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2815403/PMID: 20351946.

10.

Lewis C. Using the “think aloud” method in cognitive interface design. New York: IBM; 1982.

11.

Nielsen J. Enhancing the explanatory power of usability heuristics. In proceedings of the SIGCHI conference on human factors in computing systems. 1994. Boston Massachusetts: Association for Computing Machinery; 1994. p. 152–8.

12.

Nielsen J. Heuristic evaluation, w: Nielsen J., Mack RL (eds.), usability inspection methods. New York: Wiley; 1994.

13.

Khajouei R, Hasman A, Jaspers MW. Determination of the effectiveness of two methods for usability evaluation using a CPOE medication ordering system. Int J Med Inform. 2011;80(5):341–50. https://doi.org/10.1016/j.ijmedinf.2011.02.005.

14.

Cheung KL, Hiligsmann M, Präger M, Jones T, Józwiak-Hagymásy J, Munoz C, et al. Optimizing usability of an economic decision support tool: prototype of the EQUIPT tool. Int J Technol Assess Health Care. 2018;34(1):68–77. https://doi.org/10.1017/S0266462317004470.CrossRefPubMed

15.

Davids MR, Chikte UM, Halperin ML. An efficient approach to improve the usability of e-learning resources: the role of heuristic evaluation. Adv Physiol Educ. 2013;37(3):242–8. https://doi.org/10.1152/advan.00043.2013.CrossRefPubMed

16.

Choi J, Bakken S. Web-based education for low-literate parents in neonatal intensive care unit: development of a website and heuristic evaluation and usability testing. Int J Med Inform. 2010;79(8):565–75. https://doi.org/10.1016/j.ijmedinf.2010.05.001.CrossRefPubMedPubMedCentral

17.

Thyvalikakath TP, Monaco V, Thambuganipalle H, Schleyer T. Comparative study of heuristic evaluation and usability testing methods. Stud Health Technol Inform. 2009;143:322. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2736678/.

18.

Khajouei R, Farahani F. The evaluation of users’ satisfaction with the social security electronic system in Iran. Health Technol. 2019:1–8. https://doi.org/10.1007/s12553-019-00347-y.

19.

Montazeri M, Khajouei R, Sabermahani F. Evaluation of radiology and pathology subsystems of hospital information systems. J Kerman Univ Med Sci. 2015;22(2):194–204.

20.

Khajouei R, Abbasi R. Evaluating nurses' satisfaction with two nursing information systems. Comput Inform Nurs. 2017;35(6):307–14. https://doi.org/10.1097/CIN.0000000000000319.CrossRefPubMed

21.

Esfahani MZ, Khajouei R, Baneshi MR. Augmentation of the think aloud method with users’ perspectives for the selection of a picture archiving and communication system. J Biomed Inform. 2018;80:43–51. https://doi.org/10.1016/j.jbi.2018.02.017.CrossRef

22.

Atashi A, Khajouei R, Azizi A, Dadashi A. User Interface problems of a nationwide inpatient information system: a heuristic evaluation. Appl Clin Inform. 2016;7(01):89–100.CrossRef

23.

Nabovati E, Vakili-Arki H, Eslami S, Khajouei R. Usability evaluation of laboratory and radiology information systems integrated into a hospital information system. J Med Syst. 2014;38(4):35.CrossRef

24.

Nielsen J. Usability inspection methods. In: Conference companion on human factors in computing systems; 1994 Apr 28; Boston Massachusetts USA: Association for Computing Machinery; 1994:413–14. https://doi.org/10.1145/259963.260531.

25.

Nielsen J. Severity ratings for usability problems. Papers Essays. 1995;54:1–2.

26.

Gray CD, Kinnear PR. IBM SPSS statistics 19 made simple. New York: Psychology Press; 2012.

27.

Khajouei R, Zahiri Esfahani M, Jahani Y. Comparison of heuristic and cognitive walkthrough usability evaluation methods for evaluating health information systems. J Am Med Inform Assoc. 2017;24(e1):e55–60. https://doi.org/10.1093/jamia/ocw100.CrossRefPubMed

28.

Hasan L, Morris A, Probets S. A comparison of usability evaluation methods for evaluating e-commerce websites. Behav Inform Technol. 2012;31(7):707–37. https://doi.org/10.1080/0144929X.2011.596996.CrossRef

29.

Doubleday A, Ryan M, Springett M, Sutcliffe A. A comparison of usability techniques for evaluating design. In: Proceedings of the 2nd conference on designing interactive systems: processes, practices, methods, and techniques; 1997 Aug 1; Amsterdam the Netherlands: Association for Computing Machinery; 1997:101-10. https://doi.org/10.1145/263552.263583.

30.

Lai TY. Iterative refinement of a tailored system for self-care management of depressive symptoms in people living with HIV/AIDS through heuristic evaluation and end user testing. Int J Med Inform. 2007 Oct 1;76:S317–24. https://doi.org/10.1016/j.ijmedinf.2007.05.007.CrossRefPubMedPubMedCentral

31.

Peute LW, Jaspers MW. The significance of a usability evaluation of an emerging laboratory order entry system. Int J Med Inform. 2007;76(2–3):157–68. https://doi.org/10.1016/j.ijmedinf.2006.06.003.

32.

Devine EB, Lee CJ, Overby CL, Abernethy N, McCune J, Smith JW, et al. Usability evaluation of pharmacogenomics clinical decision support aids and clinical knowledge resources in a computerized provider order entry system: a mixed methods approach. Int J Med Inform. 2014;83(7):473–83. https://doi.org/10.1016/j.ijmedinf.2014.04.008.CrossRefPubMedPubMedCentral

33.

Watbled L, Marcilly R, Guerlinger S, Bastien JM, Beuscart-Zéphir MC, Beuscart R. Combining usability evaluations to highlight the chain that leads from usability flaws to usage problems and then negative outcomes. J Biomed Inform. 2018;78:12–23. https://doi.org/10.1016/j.jbi.2017.12.014.CrossRefPubMed

34.

Boland MR, Rusanov A, So Y, Lopez-Jimenez C, Busacca L, Steinman RC, et al. From expert-derived user needs to user-perceived ease of use and usefulness: a two-phase mixed-methods evaluation framework. J Biomed Inform. 2014;52:141–50. https://doi.org/10.1016/j.jbi.2013.12.004.CrossRefPubMed

35.

Bailey RW, Allan RW, Raiello P. Usability testing vs. heuristic evaluation: a head-to-head comparison. In Proceedings of the human factors society annual meeting 1992 (Vol. 36, 4, pp. 409-413). Los Angeles: SAGE Publications https://doi.org/10.1177/154193129203600431.CrossRef

36.

Paz F, Paz FA, Villanueva D, Pow-Sang JA. Heuristic evaluation as a complement to usability testing: a case study in web domain. In: 2015 12th international conference on information technology-new generations; 2015 Apr 13-15; Las Vegas, NV, USA: IEEE; 2015. https://doi.org/10.1109/ITNG.2015.92.

Title: A combination of two methods for evaluating the usability of a hospital information system
Authors: Reza Khajouei
Fatemeh Farahani
Publication date: 01-12-2020
Publisher: BioMed Central
Published in: BMC Medical Informatics and Decision Making / Issue 1/2020
Electronic ISSN: 1472-6947
DOI: https://doi.org/10.1186/s12911-020-1083-6

At a glance: The STEP trials

Springer Medicine

A combination of two methods for evaluating the usability of a hospital information system

Abstract

Background

Methods

Results

Conclusion

At a glance: The STEP trials

Springer Medicine

Abstract

Background

Methods

Results

Conclusion

Please log in to get access to this content

Other articles of this Issue 1/2020

Explainability for artificial intelligence in healthcare: a multidisciplinary perspective

The use of mobile phone functionalities by patients with asthma and their desire to use for self-care purposes

A stacking-based model for predicting 30-day all-cause hospital readmissions of patients with acute myocardial infarction

Effectiveness of PUSH notifications from a mobile app for improving the body composition of overweight or obese women: a protocol of a three-armed randomized controlled trial

Non-health outcomes affecting self-care behaviors and medical decision-making preference in patients with type 2 diabetes: a cross-sectional study

Availability, price, and affordability of WHO priority maternal and child health medicine in public health facilities of Dessie, north-East Ethiopia