Top

Published in:

Open Access 01-12-2020 | Mood Disorders | Research article

Dimensionality of the system usability scale among professionals using internet-based interventions for depression: a confirmatory factor analysis

Authors: Mayke Mol, Anneke van Schaik, Els Dozeman, Jeroen Ruwaard, Christiaan Vis, David D. Ebert, Anne Etzelmueller, Kim Mathiasen, Bárbara Moles, Teresa Mora, Claus D. Pedersen, Mette Maria Skjøth, Luisa Peleteiro Pensado, Jordi Piera-Jimenez, Didem Gokcay, Burçin Ünlü Ince, Alessio Russi, Ylenia Sacco, Enrico Zanalda, Ane Fullaondo Zabala, Heleen Riper, Jan H. Smit

Published in: BMC Psychiatry | Issue 1/2020

Abstract

Background

The System Usability Scale (SUS) is used to measure usability of internet-based Cognitive Behavioural Therapy (iCBT). However, whether the SUS is a valid instrument to measure usability in this context is unclear. The aim of this study is to assess the factor structure of the SUS, measuring usability of iCBT for depression in a sample of professionals. In addition, the psychometric properties (reliability, convergent validity) of the SUS were tested.

Methods

A sample of 242 professionals using iCBT for depression from 6 European countries completed the SUS. Confirmatory Factor Analysis (CFA) was conducted to test whether a one-factor, two-factor, tone-model or bi-direct model would fit the data best. Reliability was assessed using complementary statistical indices (e.g. omega). To assess convergent validity, the SUS total score was correlated with an adapted Client Satisfaction Questionnaire (CSQ-3).

Results

CFA supported the one-factor, two-factor and tone-model, but the bi-factor model fitted the data best (Comparative Fit Index = 0.992, Tucker Lewis Index = 0.985, Root Mean Square Error of Approximation = 0.055, Standardized Root Mean Square Residual = 0.042 (respectively χ²_diff (9) = 69.82, p < 0.001; χ²_diff (8) ₌ 33.04, p < 0.001). Reliability of the SUS was good (ω = 0.91). The total SUS score correlated moderately with the CSQ-3 (CSQ1 r_s = .49, p < 0.001; CSQ2 r_s = .46, p < 0.001; CSQ3 r_s = .38, p < 0.001), indicating convergent validity.

Conclusions

Although the SUS seems to have a multidimensional structure, the best model showed that the total sumscore of the SUS appears to be a valid and interpretable measure to assess the usability of internet-based interventions when used by professionals in mental healthcare.

Available only for authorised users

Andersson G, Titov N, Dear BF, Rozental A, Carlbring P. Internet-delivered psychological treatments: from innovation to implementation. World Psychiatry. 2019;18:20–8. https://doi.org/10.1002/wps.20610.CrossRefPubMedPubMedCentral

Carlbring P, Andersson G, Cuijpers P, Riper H, Hedman-Lagerlöf E. Internet-based vs. face-to-face cognitive behavior therapy for psychiatric and somatic disorders: an updated systematic review and meta-analysis. Cogn Behav Ther. 2018;47:1–18. https://doi.org/10.1080/16506073.2017.1401115.CrossRefPubMed

Thase ME, Wright JH, Eells TD, Barrett MS, Wisniewski SR, Balasubramani GK, et al. Improving the efficiency of psychotherapy for depression: computer-assisted versus standard CBT. Am J Psychiatry. 2017;175:242–50. https://doi.org/10.1176/appi.ajp.2017.17010089.CrossRefPubMedPubMedCentral

Karyotaki E, Ebert DD, Donkin L, Riper H, Twisk J, Burger S, et al. Do guided internet-based interventions result in clinically relevant changes for patients with depression? An individual participant data meta-analysis. Clin Psychol Rev. 2018;63:80–92. https://doi.org/10.1016/j.cpr.2018.06.007.CrossRefPubMed

Folker AP, Mathiasen K, Lauridsen SM, Stenderup E, Dozeman E, Folker MP. Implementing internet-delivered cognitive behavior therapy for common mental health disorders: a comparative case study of implementation challenges perceived by therapists and managers in five European internet services. Internet Interv. 2018;11:60–70. https://doi.org/10.1016/J.INVENT.2018.02.001.CrossRefPubMedPubMedCentral

Vis C, Mol M, Kleiboer A, Bührmann L, Finch T, Smit J, et al. Improving implementation of emental health for mood disorders in routine practice: systematic review of barriers and facilitating factors. JMIR Mental Health. 2018;20. https://doi.org/10.2196/mental.9769.

Kooistra LC, Ruwaard J, Wiersma JE, van Oppen P, van der Vaart R, van Gemert-Pijnen JEWC, et al. Development and initial evaluation of blended cognitive behavioural treatment for major depression in routine specialized mental health care. Internet Interv. 2016;4:61–71. https://doi.org/10.1016/j.invent.2016.01.003.CrossRefPubMedPubMedCentral

Wozney L, Baxter P, Newton AS. Usability evaluation with mental health professionals and young people to develop an internet-based cognitive-behaviour therapy program for adolescents with anxiety disorders. BMC Pediatr. 2015;15:213. https://doi.org/10.1186/s12887-015-0534-1.CrossRefPubMedPubMedCentral

Kleiboer A, Smit J, Bosmans J, Ruwaard J, Andersson G, Topooco N, et al. European COMPARative effectiveness research on blended depression treatment versus treatment-as-usual (E-COMPARED): study protocol for a randomized controlled, non-inferiority trial in eight European countries. Trials. 2016;17. https://doi.org/10.1186/s13063-016-1511-1.

10.

Borsci S, Federici S, Malizia A, De Filippis ML. Shaking the usability tree: why usability is not a dead end, and a constructive way forward. Behav Inform Technol. 2019;38:519–32. https://doi.org/10.1080/0144929X.2018.1541255.CrossRef

11.

ISO I. 9241--11: 1998, Ergonomic requirements for work with visual display terminals (VDTs)-Part 11: Guidance on usability. Brussels: CEN; 1998.

12.

Tractinsky N. The usability construct: a dead end? Hum Comput Interact. 2018;33:131–77. https://doi.org/10.1080/07370024.2017.1298038.CrossRef

13.

Brooke J. SUS-A quick and dirty usability scale. Usability Eval Ind. 1996;189:4–7.

14.

Lewis JR, Sauro J. The factor structure of the system usability scale. Hum Centered Design. 2009:94–103. https://doi.org/10.1007/978-3-642-02806-9_12.

15.

Lewis JR. The system usability scale: past, present, and future. Int J Hum Comput Interact. 2018;34:577–90. https://doi.org/10.1080/10447318.2018.1455307.CrossRef

16.

Dianat I, Ghanbari Z, AsghariJafarabadi M. Psychometric properties of the persian language version of the system usability scale. Health Promot Perspect. 2014;4:82–9. https://doi.org/10.5681/hpp.2014.011.CrossRefPubMedPubMedCentral

17.

Finstad K. The usability metric for user experience. Interact Comput. 2010;22:323–7. https://doi.org/10.1016/j.intcom.2010.04.004.CrossRef

18.

Borsci S, Federici S, Bacci S, Gnaldi M, Bartolucci F. Assessing user satisfaction in the era of user experience: comparison of the SUS, UMUX, and UMUX-LITE as a function of product experience. Int J Hum Comput Interact. 2015;31:484–95. https://doi.org/10.1080/10447318.2015.1064648.CrossRef

19.

Lewis JR, Sauro J. Item benchmarks for the system usability scale. J Usability Stud. 2018;13:158–67.

20.

Sauro J. A practical guide to the system usability scale: background, benchmarks & best practices. Denver: Measuring Usability LLC; 2011.

21.

Bangor A, Kortum P, Miller J. Determining what individual SUS scores mean: adding an adjective rating scale. J Usability Stud. 2009;4:114–23.

22.

Sauro J, Lewis JR. Quantifying user research. In: Quantifying the User Experience: Practical statistics for user research. Morgan Kaufmann; 2016. p. 9–18. https://doi.org/10.1016/B978-0-12-802308-2.00002-3.CrossRef

23.

Borsci S, Federici S, Lauriola M. On the dimensionality of the system usability scale: a test of alternative measurement models. Cogn Process. 2009;10:193–7.CrossRefPubMed

24.

Kortum P, Sorber M. Measuring the usability of Mobile applications for phones and tablets. Int J Hum Comput Interact. 2015;31:518–29.CrossRef

25.

Lewis JR, Brown J, Mayes DK. Psychometric evaluation of the EMO and the SUS in the context of a large-sample Unmoderated usability study. Int J Hum Comput Interact. 2015;31:545–53. https://doi.org/10.1080/10447318.2015.1064665.CrossRef

26.

Lewis JR, Sauro J. Revisiting the factor structure of the system usability scale. J Usability Stud. 2017;12:183–92.

27.

Vis C, Kleiboer A, Prior R, Bønes E, Cavallo M, Clark SA, et al. Implementing and up-scaling evidence-based eMental health in Europe: the study protocol for the MasterMind project. Internet Interv. 2015;2:399–409. https://doi.org/10.1016/j.invent.2015.10.002.CrossRef

28.

MasterMind project website. http://mastermind-project.eu/. Accessed 30 Jul 2019.

29.

Attkisson CC, Greenfield TK. The client satisfaction questionnaire (CSQ) scales. Outcome assessment in clinical practice. Baltimore: Williams & Wilkins; 1995.

30.

Brislin RW. Back-translation for cross-cultural research. J Cross-Cult Psychol. 1970;1:185–216. https://doi.org/10.1177/135910457000100301.CrossRef

31.

Miglietta E, Belessiotis-Richards C, Ruggeri M, Priebe S. Scales for assessing patient satisfaction with mental health care: a systematic review. J Psychiatr Res. 2018;100:33–46. https://doi.org/10.1016/j.jpsychires.2018.02.014.CrossRefPubMed

32.

Boβ L, Lehr D, Reis D, Vis C, Riper H, Berking M, et al. Reliability and validity of assessing user satisfaction with web-based health interventions. J Med Internet Res. 2016;18. https://doi.org/10.2196/jmir.5952.

33.

Rosseel Y. Lavaan: an R package for structural equation modeling. J Stat Softw. 2012;48:1–36.CrossRef

34.

Revelle WR. psych: Procedures for Personality and Psychological Research. Northwestern University, Evanston, Illinois, USA; 2017.

35.

Dai S, Wang X, Svetina D. Subscore: computing subscores in classical test theory and item response theory. R package. Bloomington, Indiana: Indiana University; 2019.

36.

Brown T. Confirmatory factor analysis for applied research. New York, NY: Guilford Press; 2006.

37.

Hu L, Bentler PM. Cutoff criteria for fit indexes in covariance structure analysis: conventional criteria versus new alternatives. Struct Equ Model A Multidiscip J. 1999;6:1–55. https://doi.org/10.1080/10705519909540118.CrossRef

38.

Dueber DM. Bifactor indices calculator: a Microsoft excel-based tool to calculate various indices relevant to bifactor CFA models; 2017. https://doi.org/10.13023/edp.tool.01. [Available at http://sites.education.uky.edu/apslab/resources/].CrossRef

39.

Rodriguez A, Reise SP, Haviland MG. Applying Bifactor statistical indices in the evaluation of psychological measures. J Pers Assess. 2016;98:223–37. https://doi.org/10.1080/00223891.2015.1089249.CrossRefPubMed

40.

Reise SP, Scheines R, Widaman KF, Haviland MG. Multidimensionality and structural coefficient Bias in structural equation modeling: a Bifactor perspective. Educ Psychol Meas. 2013;73:5–26. https://doi.org/10.1177/0013164412449831.CrossRef

41.

Haberman SJ. When can subscores have value? J Educ Behav Stat. 2008;33:204–29. https://doi.org/10.3102/1076998607302636.CrossRef

42.

Swank JM, Mullen PR. Evaluating evidence for conceptually related constructs using bivariate correlations. Meas Eval Couns Dev. 2017;50:270–4. https://doi.org/10.1080/07481756.2017.1339562.CrossRef

43.

Kyriazos TA. Applied psychometrics: sample size and sample power considerations in factor analysis (EFA, CFA) and SEM in general. Psychology. 2018;9:2230. https://doi.org/10.4236/psych.2018.98126.CrossRef

44.

Bandalos DL. Relative performance of categorical diagonally weighted least squares and robust maximum likelihood estimation. Struct Equ Model Multidiscip J. 2014;21:102–16. https://doi.org/10.1080/10705511.2014.859510.CrossRef

45.

Forero CG, Maydeu-Olivares A, Gallardo-Pujol D. Factor analysis with ordinal indicators: a Monte Carlo study comparing DWLS and ULS estimation. Struct Equ Model. 2009;16:625–41. https://doi.org/10.1080/10705510903203573.CrossRef

46.

Bonifay W, Lane SP, Reise SP. Three concerns with applying a Bifactor model as a structure of psychopathology. Clin Psychol Sci. 2017;5:184–6. https://doi.org/10.1177/2167702616657069.CrossRef

47.

Berkman MI, Karahoca D. Re-assessing the usability metric for user experience (UMUX) scale. J Usability Stud. 2016;11:89–109.

48.

Sauro J, Lewis JR. When designing usability questionnaires, does it hurt to be positive? In: Conference on Human Factors in Computing Systems - Proceedings; 2011. p. 2215–23.

49.

Zhang X, Savalei V. Improving the factor structure of psychological scales: the expanded format as an alternative to the Likert scale format. Educ Psychol Meas. 2016;76:357–86. https://doi.org/10.1177/0013164415596421.CrossRefPubMed

50.

Saris WE, Revilla M, Krosnick JA, Shaeffer EM. Comparing questions with agree/disagree response options to questions with item-specific response options. Surv Res Methods. 2010;4:45–59. https://doi.org/10.18148/srm/2010.v4i1.2682.CrossRef

Title: Dimensionality of the system usability scale among professionals using internet-based interventions for depression: a confirmatory factor analysis
Authors: Mayke Mol
Anneke van Schaik
Els Dozeman
Jeroen Ruwaard
Christiaan Vis
David D. Ebert
Anne Etzelmueller
Kim Mathiasen
Bárbara Moles
Teresa Mora
Claus D. Pedersen
Mette Maria Skjøth
Luisa Peleteiro Pensado
Jordi Piera-Jimenez
Didem Gokcay
Burçin Ünlü Ince
Alessio Russi
Ylenia Sacco
Enrico Zanalda
Ane Fullaondo Zabala
Heleen Riper
Jan H. Smit
Publication date: 01-12-2020
Publisher: BioMed Central
Keywords: Mood Disorders
Affective Disorder
Published in: BMC Psychiatry / Issue 1/2020
Electronic ISSN: 1471-244X
DOI: https://doi.org/10.1186/s12888-020-02627-8

2024 ASCO Annual Meeting

Springer Medicine

Dimensionality of the system usability scale among professionals using internet-based interventions for depression: a confirmatory factor analysis

Abstract

Background

Methods

Results

Conclusions

2024 ASCO Annual Meeting

Springer Medicine

Abstract

Background

Methods

Results

Conclusions

Please log in to get access to this content

Other articles of this Issue 1/2020

Rapid and temporary improvement of depression and anxiety observed following niraparib administration: a case report

How do education and experience with mental illness interact with causal beliefs, eligible treatments and stigmatising attitudes towards schizophrenia? A comparison between mental health professionals, psychology students, relatives and patients

Correlates of mental health in occupations at risk for traumatization: a cross-sectional study

Correction to: The Japanese version of the questionnaire about the process of recovery: development and validity and reliability testing

Exploring mediators of the recovery process over time among mental health service users, using a mixed model regression analysis based on cluster RCT data

Psychological and physical effects of breast cancer diagnosis and treatment on young Ghanaian women: a qualitative study