Skip to main content
Key Specialties
Cardiology Diabetology Endocrinology Gastroenterology Geriatrics and Gerontology Gynecology and Obstetrics Hematology Infectious Disease Internal Medicine Nephrology Neurology Oncology Pediatrics Respiratory Medicine Rheumatology Surgery
Congress Coverage
2024 ACC Congress 2023 ESMO Congress 2023 EASD Congress 2023 ERS Congress 2023 ESC Congress
Clinical Collections
Advances in Alzheimer’s

Keynote webinar | Spotlight on medication adherence

LIVE: Thursday 27th June 2024, 18:00-19:30 (CEST)
Join our expert panel to discover why you need to understand the drivers of non-adherence in your patients, and how you can optimize medication adherence in your clinics to drastically improve patient outcomes.
ADVANCED SEARCH
Log in
Top
BMC Medical Research Methodology
Published in: BMC Medical Research Methodology 1/2020

Open Access 01-12-2020 | Research article

Psychometric properties of measures of substance use: a systematic review and meta-analysis of reliability, validity and diagnostic test accuracy

Authors: Glenn-Milo Santos, Steffanie A. Strathdee, Nabila El-Bassel, Poonam Patel, Divya Subramanian, Danielle Horyniak, Ryan R. Cook, Charlotte McCullagh, Phillip Marotta, Foram Choksi, Brian Kang, Isabel Allen, Steven Shoptaw

Published in: BMC Medical Research Methodology | Issue 1/2020

Login to get access

Abstract

Background

Synthesis of psychometric properties of substance use measures to identify patterns of use and substance use disorders remains limited. To address this gap, we sought to systematically evaluate the psychometric properties of measures to detect substance use and misuse.

Methods

We conducted a systematic review and meta-analysis of literature on measures of substance classes associated with HIV risk (heroin, methamphetamine, cocaine, ecstasy, alcohol) that were published in English before June 2016 that reported at least one of the following psychometric outcomes of interest: internal consistency (alpha), test-retest/inter-rater reliability (kappa), sensitivity, specificity, positive predictive value, and negative predictive value. We used meta-analytic techniques to generate pooled summary estimates for these outcomes using random effects and hierarchical logistic regression models.

Results

Findings across 387 paper revealed that overall, 65% of pooled estimates for alpha were in the range of fair-to-excellent; 44% of estimates for kappa were in the range of fair-to-excellent. In addition, 69, 97, 37 and 96% of pooled estimates for sensitivity, specificity, positive predictive value, and negative predictive value, respectively, were in the range of moderate-to-excellent.

Conclusion

We conclude that many substance use measures had pooled summary estimates that were at the fair/moderate-to-excellent range across different psychometric outcomes. Most scales were conducted in English, within the United States, highlighting the need to test and validate these measures in more diverse settings. Additionally, the majority of studies had high risk of bias, indicating a need for more studies with higher methodological quality.
Appendix
Available only for authorised users
Literature
1.
United Nations Office on Drugs and Crime. World Drug Report 2017. Vienna: United Nations Office on Drugs and Crime; 2017. p. 2017.CrossRef
2.
3.
G. B. D. Disease Injury Incidence Prevalence Collaborators. Global, regional, and national incidence, prevalence, and years lived with disability for 328 diseases and injuries for 195 countries, 1990–2016: a systematic analysis for the Global Burden of Disease Study 2016. Lancet. 2017;390(10100):1211–59.CrossRef
4.
Degenhardt L, Whiteford HA, Ferrari AJ, Baxter AJ, Charlson FJ, Hall WD, et al. Global burden of disease attributable to illicit drug use and dependence: findings from the global burden of disease study 2010. Lancet. 2013;382(9904):1564–74.PubMedCrossRef
5.
G. B. D. Risk Factors Collaborators. Global, regional, and national comparative risk assessment of 79 behavioural, environmental and occupational, and metabolic risks or clusters of risks, 1990-2015: a systematic analysis for the global burden of disease study 2015. Lancet. 2016;388(10053):1659–724.CrossRef
6.
Shoptaw S, Montgomery B, Williams CT, El-Bassel N, Aramrattana A, Metsch L, et al. Not just the needle: the state of HIV-prevention science among substance users and future directions. J Acquir Immune Defic Syndr. 2013;63(Suppl 2):S174–8.PubMedPubMedCentralCrossRef
7.
Rowe C, Santos GM, McFarland W, Wilson EC. Prevalence and correlates of substance use among trans female youth ages 16-24 years in the San Francisco Bay Area. Drug Alcohol Depend. 2015;147:160–6.PubMedCrossRef
8.
Santos GM, Coffin PO, Das M, Matheson T, DeMicco E, Raiford JL, et al. Dose-response associations between number and frequency of substance use and high-risk sexual behaviors among HIV-negative substance-using men who have sex with men (SUMSM) in San Francisco. J Acquir Immune Defic Syndr. 2013;63(4):540–4.PubMedPubMedCentralCrossRef
9.
Colfax G, Santos GM, Chu P, Vittinghoff E, Pluddemann A, Kumar S, et al. Amphetamine-group substances and HIV. Lancet. 2010;376(9739):458–74.PubMedCrossRef
10.
Santos GM, Das M, Colfax GN. Interventions for non-injection substance use among US men who have sex with men: what is needed. AIDS Behav. 2011;15(Suppl 1):S51–6.PubMedCrossRef
11.
Strathdee SA, Shoptaw S, Dyer TP, Quan VM, Aramrattana A. Substance use scientific committee of the HIVPTN. Towards combination HIV prevention for injection drug users: addressing addictophobia, apathy and inattention. Curr Opin HIV AIDS. 2012;7(4):320–5.PubMedPubMedCentralCrossRef
12.
Ostrow DG, Plankey MW, Cox C, Li X, Shoptaw S, Jacobson LP, et al. Specific sex drug combinations contribute to the majority of recent HIV seroconversions among MSM in the MACS. J Acquir Immune Defic Syndr. 2009;51(3):349–55.PubMedPubMedCentralCrossRef
13.
Koblin BA, Husnik MJ, Colfax G, Huang Y, Madison M, Mayer K, et al. Risk factors for HIV infection among men who have sex with men. AIDS. 2006;20(5):731–9.PubMedCrossRef
14.
Kerr T, Shannon K, Ti L, Strathdee S, Hayashi K, Nguyen P, et al. Sex work and HIV incidence among people who inject drugs. AIDS. 2016;30(4):627–34.PubMedCrossRef
15.
Strathdee SA, Galai N, Safaiean M, Celentano DD, Vlahov D, Johnson L, et al. Sex differences in risk factors for hiv seroconversion among injection drug users: a 10-year perspective. Arch Intern Med. 2001;161(10):1281–8.PubMedCrossRef
16.
Hinkin CH, Barclay TR, Castellon SA, Levine AJ, Durvasula RS, Marion SD, et al. Drug use and medication adherence among HIV-1 infected individuals. AIDS Behav. 2007;11(2):185–94.PubMedPubMedCentralCrossRef
17.
DeLorenze GN, Weisner C, Tsai AL, Satre DD, Quesenberry CP Jr. Excess mortality among HIV-infected patients diagnosed with substance use dependence or abuse receiving care in a fully integrated medical care program. Alcohol Clin Exp Res. 2011;35(2):203–10.PubMedCrossRef
18.
Chander G, Himelhoch S, Moore RD. Substance abuse and psychiatric disorders in HIV-positive patients: epidemiology and impact on antiretroviral therapy. Drugs. 2006;66(6):769–89.PubMedCrossRef
19.
Dhalla S, Zumbo BD, Poole G. A review of the psychometric properties of the CRAFFT instrument: 1999-2010. Curr Drug Abuse Rev. 2011;4(1):57–64.PubMedCrossRef
20.
Berman AH, Bergman H, Palmstierna T, Schlyter F. Evaluation of the drug use disorders identification test (DUDIT) in criminal justice and detoxification settings and in a Swedish population sample. Eur Addict Res. 2005;11(1):22–31.PubMedCrossRef
21.
Berner MM, Kriston L, Bentele M, Harter M. The alcohol use disorders identification test for detecting at-risk drinking: a systematic review and meta-analysis. Journal of studies on alcohol and drugs. 2007;68(3):461–73.PubMedCrossRef
22.
Substance Abuse and Mental Health Services Administration (SAMHSA). The Role of Biomarkers in the Treatment of Alcohol Use Disorders. SAMHSA Advisory. 2012;11(2):1–8. .
23.
Manea L, Gilbody S, McMillan D. A diagnostic meta-analysis of the patient health Questionnaire-9 (PHQ-9) algorithm scoring method as a screen for depression. Gen Hosp Psychiatry. 2015;37(1):67–75.PubMedCrossRef
24.
Stockings E, Degenhardt L, Lee YY, Mihalopoulos C, Liu A, Hobbs M, et al. Symptom screening scales for detecting major depressive disorder in children and adolescents: a systematic review and meta-analysis of reliability, validity and diagnostic utility. J Affect Disord. 2015;174:447–63.PubMedCrossRef
25.
Mitchell AJ, Coyne JC. Do ultra-short screening instruments accurately detect depression in primary care? A pooled analysis and meta-analysis of 22 studies. Br J Gen Pract. 2007;57(535):144–51.PubMedPubMedCentral
26.
Scaini S, Battaglia M, Beidel DC, Ogliari A. A meta-analysis of the cross-cultural psychometric properties of the social phobia and anxiety inventory for children (SPAI-C). J Anx Disord. 2012;26(1):182–8.CrossRef
27.
Newton AS, Soleimani A, Kirkland SW, Gokiert RJ. A systematic review of instruments to identify mental health and substance use problems among children in the emergency department. Acad Emerg Med Off J Soc Acad Emerg Med. 2017;24(5):552–68.CrossRef
28.
Newton AS, Gokiert R, Mabood N, Ata N, Dong K, Ali S, et al. Instruments to detect alcohol and other drug misuse in the emergency department: a systematic review. Pediatrics. 2011;128(1):e180–92.PubMedCrossRef
29.
Mitchell AJ, Bird V, Rizzo M, Hussain S, Meader N. Accuracy of one or two simple questions to identify alcohol-use disorder in primary care: a meta-analysis. Br J Gen Pract. 2014;64(624):e408–18.PubMedPubMedCentralCrossRef
30.
Dhalla S, Kopec JA. The CAGE questionnaire for alcohol misuse: a review of reliability and validity studies. Clin Invest Med. 2007;30(1):33–41.PubMedCrossRef
31.
Allen JP, Reinert DF, Volk RJ. The alcohol use disorders identification test: an aid to recognition of alcohol problems in primary care patients. Prev Med. 2001;33(5):428–33.PubMedCrossRef
32.
Boateng GO, Neilands TB, Frongillo EA, Melgar-Quinonez HR, Young SL. Best practices for developing and validating scales for health, social, and behavioral research: a primer. Front Public Health. 2018;6:149.PubMedPubMedCentralCrossRef
33.
Whiting PF, Rutjes AW, Westwood ME, Mallett S, Deeks JJ, Reitsma JB, et al. QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Ann Intern Med. 2011;155(8):529–36.CrossRefPubMed
34.
Harris RJ, Bradburn MJ, Deeks JJ, Harbord RM, Altman D, Sterne JA. Metan: fixed- and random-effects meta-analysis. Stata J. 2008;8(1):3–28.CrossRef
35.
Macaskill P, Gatsonis C, Deeks JJ, Harbord RM, Takwoingi Y. Chapter 10: Analysing and Presenting Results. In: Deeks JJ, Bossuyt PM, Gatsonis C, editors. Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy Version 10; 2010.
36.
Freeman K, Taylor-Phillips S, Connock M, Court R, Tsertsvadze A, Shyangdan D, et al. Test accuracy of drug and antibody assays for predicting response to antitumour necrosis factor treatment in Crohn's disease: a systematic review and meta-analysis. BMJ Open. 2017;7(6):e014581.PubMedPubMedCentralCrossRef
37.
Harbord RM. Metandi: meta-analysis of diagnostic accuracy using hierarchical logistic regression. Stata J. 2009;9(2):211–29.CrossRef
38.
Harbord RM, Deeks JJ, Egger M, Whiting P, Sterne JA. A unification of models for meta-analysis of diagnostic accuracy studies. Biostatistics. 2007;8(2):239–51.PubMedCrossRef
39.
Ponterotto JG, Ruckdeschel DE. An overview of coefficient alpha and a reliability matrix for estimating adequacy of internal consistency coefficients with psychological research measures. Percept Mot Skills. 2007;105(3 Pt 1):997–1014.PubMedCrossRef
40.
Andrews JA, Lewinsohn PM, Hops H, Roberts RE. Psychometric properties of scales for the measurement of psychosocial variables associated with depression in adolescence. Psychol Rep. 1993;73(3 Pt 1):1019–46.PubMed
41.
42.
Deeks JJ, Macaskill P, Irwig L. The performance of tests of publication bias and other sample size effects in systematic reviews of diagnostic test accuracy was assessed. J Clin Epidemiol. 2005;58(9):882–93.PubMedCrossRef
43.
McCormick C. Countries with Better English Have Better Economies. Harv Bus Rev. 2013;2013(11): 1–4. .
44.
Substance Abuse and Mental Health Services Administration. Results from the 2013 National Survey on Drug Use and Health: Summary of National Findings,. Rockville, MD; 2014. Contract No.: Publication No. (SMA). p. 14–4863.
45.
Robins LN, Helzer JE, Croughan J, Ratcliff KS. National Institute of Mental Health diagnostic interview schedule. Its history, characteristics, and validity. Arch Gen Psychiatry. 1981;38(4):381–9.PubMedCrossRef
46.
Grant BF, Goldstein RB, Smith SM, Jung J, Zhang H, Chou SP, et al. The alcohol use disorder and associated disabilities interview Schedule-5 (AUDADIS-5): reliability of substance use and psychiatric disorder modules in a general population sample. Drug Alcohol Depend. 2015;148:27–33.PubMedCrossRef
47.
Buck AA, Gart JJ. Comparison of a screening test and a reference test in epidemiologic studies. I. Indices of agreement and their relation to prevalence. Am J Epidemiol. 1966;83(3):586–92.PubMedCrossRef
48.
Schmidt RL, Factor RE. Understanding sources of bias in diagnostic accuracy studies. Arch Pathol Lab Med. 2013;137(4):558–65.PubMedCrossRef
49.
Bentley TG, Catanzaro A, Ganiats TG. Implications of the impact of prevalence on test thresholds and outcomes: lessons from tuberculosis. BMC Res Notes. 2012;5:563.PubMedPubMedCentralCrossRef
Metadata
Title
Psychometric properties of measures of substance use: a systematic review and meta-analysis of reliability, validity and diagnostic test accuracy
Authors
Glenn-Milo Santos
Steffanie A. Strathdee
Nabila El-Bassel
Poonam Patel
Divya Subramanian
Danielle Horyniak
Ryan R. Cook
Charlotte McCullagh
Phillip Marotta
Foram Choksi
Brian Kang
Isabel Allen
Steven Shoptaw
Publication date
01-12-2020
Publisher
BioMed Central
Published in
BMC Medical Research Methodology / Issue 1/2020
Electronic ISSN: 1471-2288
DOI
https://doi.org/10.1186/s12874-020-00963-7

Other articles of this Issue 1/2020

BMC Medical Research Methodology 1/2020 Go to the issue

Research article

Factors affecting adolescents’ participation in randomized controlled trials evaluating the effectiveness of healthcare interventions: the case of the STEPSTONES project

Research article

A scoping review of core outcome sets and their ‘mapping’ onto real-world data using prostate cancer as a case study

Research article

Software tools to support title and abstract screening for systematic reviews in healthcare: an evaluation

Research article

How methodological frameworks are being developed: evidence from a scoping review

Research article

Predictive approaches to heterogeneous treatment effects: a scoping review

Debate

Contamination in complex healthcare trials: the falls in care homes (FinCH) study experience