Top

BMC Medical Informatics and Decision Making

Published in:

Open Access 01-12-2024 | Research

Which criteria are important in usability evaluation of mHealth applications: an umbrella review

Authors: Zahra Galavi, Mahdieh Montazeri, Reza Khajouei

Published in: BMC Medical Informatics and Decision Making | Issue 1/2024

Abstract

Introduction

Usability plays a critical role in the design of mHealth applications. A well-designed app enhances user experience and contributes to better healthcare outcomes. However, it remains unclear which usability criteria are often neglected, leading to issues in the actual use of these applications. This study aimed to identify and categorize the usability issues of mHealth applications, mapping them to Nielsen’s usability principles to determine the most critical criteria.

Methods

The PRISMA guidelines were followed to report the results. Different databases (PubMed, Scopus, WoS) were searched for systematic reviews and/or meta-analyses about usability evaluation in mHealth applications. Two reviewers independently applied predefined selection criteria, extracted data, and assessed methodological quality using the AMSTAR tool.

Results

Eight studies met the inclusion criteria. The most common method used in studies to evaluate the usability of mHealth applications was the questionnaire. Researchers identified 79 usability issues from the studies. Eleven of the issues were related to the Aesthetic and minimalist design category. The category of Flexibility and efficiency of use was next (n = 10).

Conclusion

This study identified the usability issues that individuals face when using mHealth applications. By mapping these issues to evaluation criteria, developers can systematically address and prevent them. Attention to these issues will lead to better design and more effective use of mHealth applications.

Available only for authorised users

Vo V, Auroy L, Sarradon-Eck A. Patients’ perceptions of mHealth apps: meta-ethnographic review of qualitative studies. JMIR mHealth uHealth. 2019;7:e13817.CrossRefPubMedPubMedCentral

Azhar FA, Bin, Dhillon JS. A systematic review of factors influencing the effective use of mHealth apps for self-care. In: 2016 3rd International Conference on Computer and Information Sciences (ICCOINS). IEEE 2016. 191–6.

Mathers CD, Loncar D. Projections of global mortality and burden of disease from 2002 to 2030. PLoS Med. 2006;3:e442.CrossRefPubMedPubMedCentral

Paré G, Poba-Nzaou P, Sicotte C. Home telemonitoring for chronic disease management: an economic assessment. Int J Technol Assess Health Care. 2013;29:155–61. https://doi.org/10.1017/S0266462313000111CrossRefPubMed

Choe EK, Klasnja P, Pratt W. mHealth and applications. In: Shortliffe EH, Cimino JJ, editors. Biomedical informatics: computer applications in health care and biomedicine. Springer; 2021. pp. 637–66.

Nouri R, Niakan Kalhori R, Ghazisaeedi S. Criteria for assessing the quality of mHealth apps: a systematic review. J Am Med Inf Assoc. 2018;25:1089–98.CrossRef

Marcolino MS, Oliveira JAQ, D’Agostino M, et al. The impact of mhealth interventions: systematic review of systematic reviews. JMIR mHealth uHealth. 2018;6:e23. https://doi.org/10.2196/mhealth.8873CrossRefPubMedPubMedCentral

Wyatt JC. How can clinicians, specialty societies and others evaluate and improve the quality of apps for patient use? BMC Med. 2018;16:1–10.CrossRef

Henson P, David G, Albright K, et al. Deriving a practical framework for the evaluation of health apps. Lancet Digit Heal. 2019;1:e52–4. https://doi.org/10.1016/S2589-7500(19)30013-5CrossRef

10.

BINDHIM NF, Naicker S, Freeman B, et al. Apps promoting illicit drugs—a need for tighter regulation? J Consum Health Internet. 2014;18:31–43.CrossRef

11.

Zapata BC, Fernández-Alemán JL, Idri A, et al. Empirical studies on usability of mHealth apps: a systematic literature review. J Med Syst. 2015;39:1–19. https://doi.org/10.1007/s10916-014-0182-2CrossRefPubMed

12.

Schnall R, Mosley JP, Iribarren SJ, et al. Comparison of a user-centered design, self-management app to existing mhealth apps for persons living with HIV. JMIR mHealth uHealth. 2015;3. https://doi.org/10.2196/mhealth.4882

13.

Patel B, Thind A. Usability of mobile health apps for postoperative care: systematic review. JMIR Perioper Med. 2020;3:e19099. https://doi.org/10.2196/19099CrossRefPubMedPubMedCentral

14.

Nielsen J. Usability engineering. Morgan Kaufmann; 1994.

15.

Standardization IO for ISO 9241-11: Ergonomic requirements for office work with visual display terminals (VDTs): Part 11: Guidance on usability. 1998. https://doi.org/10.3403/02478696

16.

John BE, Marks SJ. Tracking the effectiveness of usability evaluation methods. Behav Inf Technol. 1997;16:188–202.CrossRef

17.

Molich R, Ede MR, Kaasgaard K, et al. Comparative usability evaluation. Behav Inf Technol. 2004;23:65–74.CrossRef

18.

Hartson HR, Andre TS, Williges RC. Criteria for evaluating usability evaluation methods. Int J Hum Comput Interact. 2001;13:373–410.CrossRef

19.

Assila A, Ezzedine H. Standardized usability questionnaires: features and quality focus. Electron J Comput Sci Inf Technol. 2016;6.

20.

Hensher M, Cooper P, Dona SWA, et al. Scoping review: development and assessment of evaluation frameworks of mobile health apps for recommendations to consumers. J Am Med Inf Assoc. 2021;28:1318–29. https://doi.org/10.1093/jamia/ocab041CrossRef

21.

Azad-Khaneghah P, Neubauer N, Miguel Cruz A, et al. Mobile health app usability and quality rating scales: a systematic review. Disabil Rehabil Assist Technol. 2021;16:712–21. https://doi.org/10.1080/17483107.2019.1701103CrossRefPubMed

22.

Inal Y, Wake JD, Guribye F, et al. Usability evaluations of mobile mental health technologies: systematic review. J Med Internet Res. 2020;22:e15337. https://doi.org/10.2196/15337CrossRefPubMedPubMedCentral

23.

Arthurs N, Tully L, O’Malley G, et al. Usability and engagement testing of mHealth apps in paediatric obesity: a narrative review of current literature. Int J Environ Res Public Health. 2022;19. https://doi.org/10.3390/ijerph19031453

24.

Overdijkink SB, Velu AV, Rosman AN, et al. The usability and effectiveness of mobile health technology-based lifestyle and medical intervention apps supporting health care during pregnancy. Syst Rev. 2018;6. https://doi.org/10.2196/mhealth.8834

25.

Jiménez C, Rusu C, Roncagliolo S, et al. Evaluating a methodology to establish usability heuristics. Proc - Int Conf Chil Comput Sci Soc SCCC. 2012:51–9. https://doi.org/10.1109/SCCC.2012.14

26.

Stewart LA, Clarke M, Rovers M, et al. Preferred reporting items for a systematic review and meta-analysis of individual participant data: the PRISMA-IPD statement. JAMA - J Am Med Assoc. 2015;313:1657–65. https://doi.org/10.1001/jama.2015.3656CrossRef

27.

Ouzzani M, Hammady H, Fedorowicz Z, et al. Rayyan-a web and mobile app for systematic reviews. Syst Rev. 2016;5:1–10. https://doi.org/10.1186/s13643-016-0384-4CrossRef

28.

Shea BJ, Hamel C, Wells GA, et al. AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. J Clin Epidemiol. 2009;62:1013–20.CrossRefPubMed

29.

Reis P, Páris C, Gomes A. An experimental activity to develop usability and UX heuristics. Commun Comput Inf Sci. 2020;1334:20–9. https://doi.org/10.1007/978-3-030-66919-5_3CrossRef

30.

Bashir MS, Farooq A. EUHSA: extending usability heuristics for smartphone application. IEEE Access. 2019;7:100838–59. https://doi.org/10.1109/ACCESS.2019.2923720CrossRef

31.

Humayoun SR, Chotala PH, Bashir MS et al. Heuristics for evaluating multi-touch gestures in mobile applications. In: HCI 2017: Digital Make Believe - Proceedings of the 31st International BCS Human Computer Interaction Conference, HCI 2017. 2017. https://doi.org/10.14236/ewic/HCI2017.89

32.

Huang Z. Developing usability heuristics for recommendation systems within the mobile context. Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect Notes Bioinformatics). 2019;11586 LNCS:143–51. https://doi.org/10.1007/978-3-030-23535-2_10CrossRef

33.

Galavi Z, Norouzi S, Khajouei R. Heuristics used for evaluating the usability of mobile health applications: a systematic literature review. Digit Heal. 2024;10:20552076241253540.

34.

Carreiro S, Newcomb M, Leach R, et al. Current reporting of usability and impact of mHealth interventions for substance use disorder: a systematic review. Drug Alcohol Depend. 2020;215. https://doi.org/10.1016/j.drugalcdep.2020.108201

35.

Fu HL, McMahon SK, Gross CR, et al. Usability and clinical efficacy of diabetes mobile applications for adults with type 2 diabetes: a systematic review. DIABETES Res Clin Pract. 2017;131:70–81. https://doi.org/10.1016/j.diabres.2017.06.016CrossRefPubMed

36.

Nathan SS, Hussain A, Hashim NL. Usability evaluation of DEAF mobile application interface: a systematic review. J Eng Appl Sci. 2018;13:291–7. https://doi.org/10.3923/jeasci.2018.291.297CrossRef

37.

Shah UeM, Chiew TK. A systematic literature review of the design approach and usability evaluation of the pain management mobile applications. Symmetry (Basel). 2019;11. https://doi.org/10.3390/sym11030400

38.

Al-Razgan M, Almoaiqel S, Alrajhi N, et al. A systematic literature review on the usability of mobile applications for visually impaired users. PEERJ Comput Sci. 2021;7. https://doi.org/10.7717/peerj-cs.771

39.

Tatham I, Clarke E, Grieve KA, et al. Process and outcome evaluations of smartphone apps for bipolar disorder: scoping review. J Med Internet Res. 2022;24. https://doi.org/10.2196/29114

40.

Farzandipour M, Nabovati E, Zaeimi G-H, et al. Usability evaluation of three admission and medical records subsystems integrated into nationwide hospital information systems: heuristic evaluation. Acta Inf Med. 2018;26:133.CrossRef

41.

Agnisarman S, Narasimha S, Madathil KC, et al. Toward a more usable home-based video telemedicine system: a heuristic evaluation of the clinician user interfaces of home-based video telemedicine systems. JMIR Hum factors. 2017;4:e7293.CrossRef

42.

Dias CR, Pereira MR, Freire AP. Qualitative review of usability problems in health information systems for radiology. J Biomed Inf. 2017;76:19–33.CrossRef

43.

Loy JS, Ali EE, Yap KY-L. Quality assessment of medical apps that target medication-related problems. J Manag care Spec Pharm. 2016;22:1124–40.PubMed

44.

Yasini M, Beranger J, Desmarais P et al. mHealth quality: a process o seal he qualified mobile health apps. Explor Complex Heal An Interdiscip Syst Approach Proc MIE 2016. 2016;228:205.

45.

Weichbroth P. Usability of mobile applications: a systematic literature study. IEEE Access. 2020;8:55563–77. https://doi.org/10.1109/ACCESS.2020.2981892CrossRef

46.

Bakogiannis C, Tsarouchas A, Mouselimis D, et al. A patient-oriented app (ThessHF) to improve self-care quality in heart failure: from evidence-based design to pilot study. JMIR mHealth uHealth. 2021;9:e24271. https://doi.org/10.2196/24271CrossRefPubMedPubMedCentral

47.

Alanzi T, Istepanian R, Philip N. Design and usability evaluation of social mobile diabetes management system in the Gulf Region. JMIR Res Protoc. 2016;5:e93. https://doi.org/10.2196/resprot.4348CrossRefPubMedPubMedCentral

48.

Zhou L, Bao J, Setiawan IMA, et al. The mhealth app usability questionnaire (MAUQ): development and validation study. JMIR mHealth uHealth. 2019;7. https://doi.org/10.2196/11500

49.

Stoyanov SR, Hides L, Kavanagh DJ, et al. Mobile app rating scale: a new tool for assessing the quality of health mobile apps. JMIR mHealth uHealth. 2015;3:e27. https://doi.org/10.2196/mhealth.3422CrossRefPubMedPubMedCentral

50.

Hajesmaeel-Gohari S, Khordastan F, Fatehi F, et al. The most used questionnaires for evaluating satisfaction, usability, acceptance, and quality outcomes of mobile health. BMC Med Inf Decis Mak. 2022;22:22. https://doi.org/10.1186/s12911-022-01764-2CrossRef

51.

Montazeri M, Khajouei R, Montazeri M. Evaluating hospital information system according to ISO 9241 part 12. Digit Heal. 2020;6:2055207620979466.CrossRef

Title: Which criteria are important in usability evaluation of mHealth applications: an umbrella review
Authors: Zahra Galavi
Mahdieh Montazeri
Reza Khajouei
Publication date: 01-12-2024
Publisher: BioMed Central
Published in: BMC Medical Informatics and Decision Making / Issue 1/2024
Electronic ISSN: 1472-6947
DOI: https://doi.org/10.1186/s12911-024-02738-2

Clinical decision making under uncertainty: a bootstrapped counterfactual inference approach

Open Access
Warfarin
Research

A hybrid framework with large language models for rare disease phenotyping

Open Access
Research

Selection of data analytic techniques by using fuzzy AHP TOPSIS from a healthcare perspective

Open Access
Research

Strengths, weaknesses, opportunities, and threats (SWOT) of the electronic prescribing systems executed in Iran from the physician’s viewpoint: a qualitative study

Open Access
Research

Early prediction of sudden cardiac death risk with Nested LSTM based on electrocardiogram sequential features

Open Access
Sudden Cardiac Death
Research

Machine learning-empowered sleep staging classification using multi-modality signals

Open Access
Polysomnography
Research

Keynote series | Spotlight on menopause

Springer Medicine

Which criteria are important in usability evaluation of mHealth applications: an umbrella review

Abstract

Introduction

Methods

Results

Conclusion

Other articles of this Issue 1/2024

Clinical decision making under uncertainty: a bootstrapped counterfactual inference approach

A hybrid framework with large language models for rare disease phenotyping

Selection of data analytic techniques by using fuzzy AHP TOPSIS from a healthcare perspective

Strengths, weaknesses, opportunities, and threats (SWOT) of the electronic prescribing systems executed in Iran from the physician’s viewpoint: a qualitative study

Early prediction of sudden cardiac death risk with Nested LSTM based on electrocardiogram sequential features

Machine learning-empowered sleep staging classification using multi-modality signals

Keynote series | Spotlight on menopause

Springer Medicine

Abstract

Introduction

Methods

Results

Conclusion

Please log in to get access to this content

Clinical decision making under uncertainty: a bootstrapped counterfactual inference approach

A hybrid framework with large language models for rare disease phenotyping

Selection of data analytic techniques by using fuzzy AHP TOPSIS from a healthcare perspective

Strengths, weaknesses, opportunities, and threats (SWOT) of the electronic prescribing systems executed in Iran from the physician’s viewpoint: a qualitative study

Early prediction of sudden cardiac death risk with Nested LSTM based on electrocardiogram sequential features

Machine learning-empowered sleep staging classification using multi-modality signals