Top

BMC Psychiatry

Published in:

Open Access 01-12-2016 | Research article

Validation of online psychometric instruments for common mental health disorders: a systematic review

Authors: Wouter van Ballegooijen, Heleen Riper, Pim Cuijpers, Patricia van Oppen, Johannes H. Smit

Published in: BMC Psychiatry | Issue 1/2016

Abstract

Background

Online questionnaires for measuring common mental health disorders such as depression and anxiety disorders are increasingly used. The psychometrics of several pen-and-paper questionnaires have been re-examined for online use and new online instruments have been developed and tested for validity as well. This study aims to review and synthesise the literature on this subject and provide a framework for future research.

Methods

We searched Medline and PsycINFO for psychometric studies on online instruments for common mental health disorders and extracted the psychometric data. Studies were coded and assessed for quality by independent raters.

Results

We included 56 studies on 62 online instruments. For common instruments such as the CES-D, MADRS-S and HADS there is mounting evidence for adequate psychometric properties. Further results are scattered over different instruments and different psychometric characteristics. Few studies included patient populations.

Conclusions

We found at least one online measure for each of the included mental health disorders and symptoms. A small number of online questionnaires have been studied thoroughly. This study provides an overview of online instruments to refer to when choosing an instrument for assessing common mental health disorders online, and can structure future psychometric research.

Available only for authorised users

National Institute for Health and Clinical Excellence (NICE). Common mental health disorders: Identification and pathways to care. London: NICE; 2011.

Alfonsson S, Maathz P, Hursti T. Interformat reliability of digital psychiatric self-report questionnaires: a systematic review. J Med Internet Res. 2014;16(12), e268. doi:10.2196/jmir.3395.PubMedCentralCrossRefPubMed

Hedman E, Ljotsson B, Ruck C, Furmark T, Carlbring P, Lindefors N, et al. Internet administration of self-report measures commonly used in research on social anxiety disorder: A psychometric evaluation. Comput Hum Behav. 2010;26(4):736–40.CrossRef

Carlbring P, Brunt S, Bohman S, Austin D, Richards J, Ost LG, et al. Internet vs. paper and pencil administration of questionnaires commonly used in panic/agoraphobia research. Comput Hum Behav. 2007;23(3):1421–34.CrossRef

Cuijpers P, Boluijt P, Van SA. Screening of depression in adolescents through the Internet : sensitivity and specificity of two screening questionnaires. Eur Child AdolescPsychiatry. 2008;17(1):32–8. doi:10.1007/s00787-007-0631-2.CrossRef

Rogers KD, Young A, Lovell K, Campbell M, Scott PR, Kendal S. The British Sign Language versions of the Patient Health Questionnaire, the Generalized Anxiety Disorder 7-item Scale, and the Work and Social Adjustment Scale. J Deaf Stud Deaf Educ. 2013;18(1):110–22. doi:10.1093/deafed/ens040.PubMedCentralCrossRefPubMed

van Ballegooijen W, Riper H, Donker T, Martin Abello K, Marks I, Cuijpers P. Single-item screening for agoraphobic symptoms: validation of a web-based audiovisual screening instrument. PLoS One. 2012;7(7):e38480. doi:10.1371/journal.pone.0038480.PubMedCentralCrossRefPubMed

Farvolden P, McBride C, Bagby RM, Ravitz P. A Web-based screening instrument for depression and anxiety disorders in primary care. J Med Internet Res. 2003;5(3), e23. doi:10.2196/jmir.5.3.e23.PubMedCentralCrossRefPubMed

Moher D, Liberati A, Tetzlaff J, Altman DG, Group P. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ. 2009;339:b2535. doi:10.1136/bmj.b2535.PubMedCentralCrossRefPubMed

10.

Terwee CB, Mokkink LB, Knol DL, Ostelo RW, Bouter LM, de Vet HC. Rating the methodological quality in systematic reviews of studies on measurement properties: a scoring system for the COSMIN checklist. Qual Life Res. 2012;21(4):651–7. doi:10.1007/s11136-011-9960-1.PubMedCentralCrossRefPubMed

11.

Terwee CB, Jansma EP, Riphagen II, de Vet HC. Development of a methodological PubMed search filter for finding studies on measurement properties of measurement instruments. Qual Life Res. 2009;18(8):1115–23. doi:10.1007/s11136-009-9528-5.PubMedCentralCrossRefPubMed

12.

World Health Organization. ICD-10 Version:2015. 2015. http://apps.who.int/classifications/icd10/browse/2015/en. Accessed January 2015.

13.

American Psychiatric Association. Diagnostic and statistical manual of mental disorders, (DSM-5®). American Psychiatric Pub.; 2013.

14.

Mokkink LB, Terwee CB, Patrick DL, Alonso J, Stratford PW, Knol DL, et al. The COSMIN checklist for assessing the methodological quality of studies on measurement properties of health status measurement instruments: an international Delphi study. Qual Life Res. 2010;19(4):539–49. doi:10.1007/s11136-010-9606-8.PubMedCentralCrossRefPubMed

15.

Mokkink LB, Terwee CB, Patrick DL, Alonso J, Stratford PW, Knol DL, et al. The COSMIN study reached international consensus on taxonomy, terminology, and definitions of measurement properties for health-related patient-reported outcomes. J Clin Epidemiol. 2010;63(7):737–45. doi:10.1016/j.jclinepi.2010.02.006.CrossRefPubMed

16.

Mokkink LB, Terwee CB, Knol DL, Stratford PW, Alonso J, Patrick DL, et al. The COSMIN checklist for evaluating the methodological quality of studies on measurement properties: a clarification of its content. BMC Med Res Methodol. 2010;10:22. doi:10.1186/1471-2288-10-22.PubMedCentralCrossRefPubMed

17.

Mokkink LB, Terwee CB, Gibbons E, Stratford PW, Alonso J, Patrick DL, et al. Inter-rater agreement and reliability of the COSMIN (COnsensus-based Standards for the selection of health status Measurement Instruments) checklist. BMC Med Res Methodol. 2010;10:82. doi:10.1186/1471-2288-10-82.PubMedCentralCrossRefPubMed

18.

Donker T, Van SA, Marks I, Cuijpers P. A brief Web-based screening questionnaire for common mental disorders: development and validation. J Med Internet Res. 2009;11(3):e19. doi:10.2196/jmir.1134.PubMedCentralCrossRefPubMed

19.

McCue P, Martin C, Buchanan T, Rodgers J, Scholey A. An investigation into the psychometric properties of the Hospital Anxiety and Depression Scale in individuals with chronic fatigue syndrome. Psychol Health Med. 2003;8(4):425–39. doi:10.1080/1354850310001604568.CrossRefPubMed

20.

Zlomke KR. Psychometric properties of internet administered versions of Penn State Worry Questionnaire (PSWQ) and Depression, Anxiety, and Stress Scale (DASS). Comput Hum Behav. 2009;25(4):841–3.CrossRef

21.

Carlbring P, Forslin P, Ljungstrand P, Willebrand M, Strandlund C, Ekselius L, et al. Is the Internet-administered CIDI-SF equivalent to a clinician-administered SCID interview? Cogn Behav Ther. 2002;31(4):183–9.CrossRef

22.

Andersson G, Kaldo-Sandstrom V, Strom L, Stromgren T. Internet administration of the Hospital Anxiety and Depression Scale in a sample of tinnitus patients. J Psychosom Res. 2003;55(3):259–62.CrossRefPubMed

23.

McCue P, Buchanan T, Martin CR. Screening for psychological distress using internet administration of the Hospital Anxiety and Depression Scale (HADS) in individuals with chronic fatigue syndrome. Br J Clin Psychol. 2006;45(Pt 4):483–98. doi:10.1348/014466505X82379.CrossRefPubMed

24.

Thoren ES, Andersson G, Lunner T. The use of research questionnaires with hearing impaired adults: online vs. paper-and-pencil administration. BMC Ear Nose Throat Disord. 2012;12:12. doi:10.1186/1472-6815-12-12.PubMedCentralCrossRefPubMed

25.

Whitehead L. Methodological issues in Internet-mediated research: a randomized comparison of internet versus mailed questionnaires. J Med Internet Res. 2011;13(4):e109. doi:10.2196/jmir.1593.PubMedCentralCrossRefPubMed

26.

Drake G, Csipke E, Wykes T. Assessing your mood online: acceptability and use of Moodscope. Psychol Med. 2013;43(7):1455–64. doi:10.1017/S0033291712002280.PubMedCentralCrossRefPubMed

27.

Faries DE, Houston JP, Sulcs EM, Swindle RW. A cross-validation of the provisional diagnostic instrument (PDI-4). BMC Fam Pract. 2012;13:104. doi:10.1186/1471-2296-13-104.PubMedCentralCrossRefPubMed

28.

Zimmerman M, Martinez JH. Web-based assessment of depression in patients treated in clinical practice: reliability, validity, and patient acceptance. J Clin Psychiatry. 2012;73(3):333–8. doi:10.4088/JCP.10m06519.CrossRefPubMed

29.

Birnholz JL, Young MA. Differential item functioning for lesbians, bisexual, and heterosexual women in the center for epidemiological studies depression scale. Assessment. 2012;19(4):502–5. doi:10.1177/1073191112440032.CrossRefPubMed

30.

Donker T, Van SA, Marks I, Cuijpers P. Brief self-rated screening for depression on the Internet. J AffectDisord. 2010;122(3):253–9. doi:10.1016/j.jad.2009.07.013.

31.

Fortson BL, Scotti JR, Del Ben KS, Chen YC. Reliability and validity of an Internet traumatic stress survey with a college student sample. J Trauma Stress. 2006;19(5):709–20. doi:10.1002/jts.20165.CrossRefPubMed

32.

Leykin Y, Torres LD, Aguilera A, Munoz RF. Factor structure of the CES-D in a sample of Spanish- and English-speaking smokers on the Internet. Psychiatry Res. 2011;185(1–2):269–74. doi:10.1016/j.psychres.2010.04.056.PubMedCentralCrossRefPubMed

33.

Yu SC, Yu MN. Comparison of Internet-based and paper-based questionnaires in Taiwan using multisample invariance approach. CyberpsycholBehav. 2007;10(4):501–7. doi:10.1089/cpb.2007.9998.CrossRef

34.

Hedman E, Ljotsson B, Blom K, E-A S, Kraepelien M, Ruck C, et al. Telephone versus internet administration of self-report measures of social anxiety, depressive symptoms, and insomnia: psychometric evaluation of a method to reduce the impact of missing data. J Med Internet Res. 2013;15(10):e229. doi:10.2196/jmir.2818.PubMedCentralCrossRefPubMed

35.

Holländare F, Askerlund AM, Nieminen A, Engstrom I. Can BDI-II and MADRS-S be transferred to online use without affecting their psychometric properties? E-J Appl Psychol. 2008;4(2):63–5.

36.

Holländare F, Andersson G, Engstrom I. A comparison of psychometric properties between internet and paper versions of two depression instruments (BDI-II and MADRS-S) administered to clinic patients. J Med Internet Res. 2010;12(5):e49. doi:10.2196/jmir.1392.PubMedCentralCrossRefPubMed

37.

Thorndike FP, Carlbring P, Smyth FL, Magee JC, Gonder-Frederick L, Ost LG, et al. Web-based measurement: Effect of completing single or multiple items per webpage. Comput Hum Behav. 2009;25(2):393–401.CrossRef

38.

Donker T, Van SA, Marks I, Cuijpers P. Quick and easy self-rating of Generalized Anxiety Disorder: validity of the Dutch web-based GAD-7, GAD-2 and GAD-SI. Psychiatry Res. 2011;188(1):58–64. doi:10.1016/j.psychres.2011.01.016.CrossRefPubMed

39.

Austin DW, Carlbring P, Richards JC, Andersson G. Internet administration of three commonly used questionnaires in panic research: Equivalence to paper administration in Australian and Swedish samples of people with panic disorder. Int J Test. 2006;6(1):25–39.CrossRef

40.

Hirai M, Vernon LL, Clum GA, Skidmore ST. Psychometric properties and administration measurement invariance of social phobia symptom measures: Paper-pencil vs. Internet administrations. J Psychopathol Behav Assessment. 2011;33(4):470–9.CrossRef

41.

Lindner P, Martell C, Bergstrom J, Andersson G, Carlbring P. Clinical validation of a non-heteronormative version of the Social Interaction Anxiety Scale (SIAS). Health Qual Life Outcomes. 2013;11:209. doi:10.1186/1477-7525-11-209.PubMedCentralCrossRefPubMed

42.

Skolnick RB, Schare ML, Wyatt KP, Tillman MA. Aviophobia assessment: validating the Flight Anxiety Situations Questionnaire as a clinical identification measure. J Anxiety Disord. 2012;26(8):779–84. doi:10.1016/j.janxdis.2012.07.006.CrossRefPubMed

43.

Coles ME, Cook LM, Blake TR. Assessing obsessive compulsive symptoms and cognitions on the internet: evidence for the comparability of paper and Internet administration. Behav Res Ther. 2007;45(9):2232–40. doi:10.1016/j.brat.2006.12.009.CrossRefPubMed

44.

Storch EA, Khanna M, Merlo LJ, Loew BA, Franklin M, Reid JM, et al. Children’s Florida Obsessive Compulsive Inventory: psychometric properties and feasibility of a self-report measure of obsessive-compulsive symptoms in youth. Child Psychiatry HumDev. 2009;40(3):467–83. doi:10.1007/s10578-009-0138-9.CrossRef

45.

Williams MT, Turkheimer E, Schmidt KM, Oltmanns TF. Ethnic identification biases responses to the Padua Inventory for obsessive-compulsive disorder. Assessment. 2005;12(2):174–85. doi:10.1177/1073191105275620.CrossRefPubMed

46.

Elhai JD, Miller ME, Ford JD, Biehn TL, Palmieri PA, Frueh BC. Posttraumatic stress disorder in DSM-5: estimates of prevalence and symptom structure in a nonclinical sample of college students. J Anxiety Disord. 2012;26(1):58–64. doi:10.1016/j.janxdis.2011.08.013.CrossRefPubMed

47.

Miller MW, Wolf EJ, Kilpatrick D, Resnick H, Marx BP, Holowka DW et al. The Prevalence and Latent Structure of Proposed DSM-5 Posttraumatic Stress Disorder Symptoms in U.S. National and Veteran Samples. Psychological Trauma: Theory, Research, Practice, and Policy. 2012.

48.

Read JP, Farrow SM, Jaanimagi U, Ouimette P. Assessing trauma and traumatic stress via the Internet: Measurement equivalence and participant reactions. Traumatology. 2009;15(1):94–102.CrossRef

49.

Callahan JL, Borja SE, Hynan MT. Modification of the Perinatal PTSD Questionnaire to enhance clinical utility. J Perinatol. 2006;26(9):533–9. doi:10.1038/sj.jp.7211562.CrossRefPubMed

50.

Verkuil B, Brosschot JF. The online version of the Dutch Penn State Worry Questionnaire: factor structure, predictive validity and reliability. J Anxiety Disord. 2012;26(8):844–8. doi:10.1016/j.janxdis.2012.08.002.CrossRefPubMed

51.

Herrero J, Meneses J. Short Web-based versions of the perceived stress (PSS) and Center for Epidemiological Studies-Depression (CESD) Scales: A comparison to pencil and paper responses among Internet users. Comput Hum Behav. 2006;22(5):830–46.CrossRef

52.

Kortge R, Meade T, Tennant A. Interpersonal and intrapersonal functions of deliberate self-harm (DSH): A psychometric examination of the Inventory of Statements About Self-Injury (ISAS) scale. Behaviour Change. 2013;30(1):24–35.CrossRef

53.

Hemelrijk E, van Ballegooijen W, Donker T, van Straten A, Kerkhof A. Internet-based screening for suicidal ideation in common mental disorders. Crisis. 2012;33(4):215–21. doi:10.1027/0227-5910/a000142.CrossRefPubMed

54.

Kline P. Handbook of psychological testing. London: Routledge; 1993.

55.

van Spijker BA, Batterham PJ, Calear AL, Farrer L, Christensen H, Reynolds J, et al. The suicidal ideation attributes scale (SIDAS): Community-based validation study of a new scale for the measurement of suicidal ideation. Suicide Life Threat Behav. 2014;44(4):408–19. doi:10.1111/sltb.12084.CrossRefPubMed

56.

Hirschfeld G, von Brachel R. Multiple-Group confirmatory factor analysis in R–A tutorial in measurement invariance with continuous and ordinal indicators. Pract Assessment, Res Eval. 2014;19(7):2.

57.

Groves RM, Fowler Jr FJ, Couper MP, Lepkowski JM, Singer E, Tourangeau R. Survey methodology (Vol. 561). Hoboken, NJ: John Wiley & Sons; 2011.

58.

Palmier-Claus JE, Ainsworth J, Machin M, Barrowclough C, Dunn G, Barkus E, et al. he feasibility and validity of ambulatory self-report of psychotic symptoms using a smartphone software application. BMC Psychiatry. 2012;12:172. doi:10.1186/1471-244X-12-172.PubMedCentralCrossRefPubMed

Title: Validation of online psychometric instruments for common mental health disorders: a systematic review
Authors: Wouter van Ballegooijen
Heleen Riper
Pim Cuijpers
Patricia van Oppen
Johannes H. Smit
Publication date: 01-12-2016
Publisher: BioMed Central
Published in: BMC Psychiatry / Issue 1/2016
Electronic ISSN: 1471-244X
DOI: https://doi.org/10.1186/s12888-016-0735-7

At a glance: The ONWARDS insulin icodec trials

Springer Medicine

Validation of online psychometric instruments for common mental health disorders: a systematic review

Abstract

Background

Methods

Results

Conclusions

At a glance: The ONWARDS insulin icodec trials

Springer Medicine

Abstract

Background

Methods

Results

Conclusions

Please log in to get access to this content

Other articles of this Issue 1/2016

Five-year outcomes of western mental health training for Traditional Chinese Medicine practitioners

How anorexia nervosa patients with high and low autistic traits respond to group Cognitive Remediation Therapy

Adherence therapy versus routine psychiatric care for people with schizophrenia spectrum disorders: a randomised controlled trial

Association between neutrophilic granulocyte percentage and depression in hospitalized patients with heart failure

Infant sleep hygiene counseling (sleep trial): protocol of a randomized controlled trial

Erratum to: Significant treatment effect of adjunct music therapy to standard treatment on the positive, negative, and mood symptoms of schizophrenic patients: a meta-analysis