Top

Published in:

01-02-2021 | Care | Original Research Article

Sensitivity and Specificity of Computer-Based Neurocognitive Tests in Sport-Related Concussion: Findings from the NCAA-DoD CARE Consortium

Authors: Lauren L. Czerniak, Spencer W. Liebel, Gian-Gabriel P. Garcia, Mariel S. Lavieri, Michael A. McCrea, Thomas W. McAllister, Steven P. Broglio, CARE Consortium Investigators

Published in: Sports Medicine | Issue 2/2021

Abstract

Background

To optimally care for concussed individuals, a multi-dimensional approach is critical and a key component of this assessment in the athletic environment is computer-based neurocognitive testing. However, there continues to be concerns about the reliability and validity of these testing tools. The purpose of this study was to determine the sensitivity and specificity of three common computer-based neurocognitive tests (Immediate Post-Concussion Assessment and Cognitive Testing [ImPACT], CNS Vital Signs, and CogState Computerized Assessment Tool [CCAT]), to provide guidance on their clinical utility.

Methods

This study analyzed assessments from a cohort of collegiate athletes and non-varsity cadets from the NCAA-DoD CARE Consortium. The data were collected from 2014–2018. Study participants were divided into two testing groups [concussed, n = 1414 (baseline/24–48 h) and healthy, n = 8305 (baseline/baseline)]. For each test type, change scores were calculated for the components of interest. Then, the Normative Change method, which used normative data published in a similar cohort, and the Reliable Change Index (RCI) method were used to determine if the change scores were significant.

Results

Using the Normative Change method, ImPACT performed best with an 87.5%-confidence interval and 1 number of components failed (NCF; sensitivity = 0.583, specificity = 0.625, F1 = 0.308). CNS Vital Signs performed best with a 90%-confidence interval and 1 NCF (sensitivity = 0.587, specificity = 0.532, F1 = 0.314). CCAT performed best when using a 75%-confidence interval and 2 NCF (sensitivity = 0.513, specificity = 0.715, F1 = 0.290). When using the RCI method, ImPACT performed best with an 87.5%-confidence interval and 1 NCF (sensitivity = 0.626, specificity = 0.559, F1 = 0.297).

Conclusion

When considering all three computer-based neurocognitive tests, the overall low sensitivity and specificity results provide additional evidence for the use of a multi-dimensional assessment for concussion diagnosis, including symptom evaluation, postural control assessment, neuropsychological status, and other functional assessments.

Available only for authorised users

About the Cogstate CCAT. Retrieved September 8, 2019, from http://axonsports.ca/index.cfm?pid=65

Aoullay A. What's WRONG with Metrics? Retrieved February 2, 2020. (2018). https://towardsdatascience.com/choosing-the-right-metric-is-a-huge-issue-99ccbe73de61

Aubry M, Cantu R, Dvorak J, Graf-Baumann T, Johnston K, Kelly J, et al. Summary and agreement statement of the first international conference on concussion in sport, Vienna 2001. Br J Sports Med. 2002;36(1):6–10. https://doi.org/10.1136/bjsm.36.1.6.CrossRefPubMedPubMedCentral

Broglio SP, Ferrara MS, Macciocchi SN, Baumgartner TA, Elliott R. Test-retest reliability of computerized concussion assessment programs. J Athl Train. 2007;42(4):509–14.PubMedPubMedCentral

Broglio SP, Harezlak J, Katz B, Zhao S, McAllister T, McCrea M, Investigators CC. Acute sport concussion assessment optimization: a prospective assessment from the CARE Consortium. Am J Sports Med. 2019;49(12):1977–87. https://doi.org/10.1007/s40279-019-01155-0.CrossRef

Broglio SP, Katz BP, Zhao S, McCrea M, McAllister T, CARE Consortium Investigators. Test–retest reliability and interpretation of common concussion assessment tools: findings from the NCAA-DoD CARE consortium. Am J Sports Med. 2018;48(5):1255–68. https://doi.org/10.1007/s40279-017-0813-0.CrossRef

Broglio SP, Macciocchi SN, Ferrara MS. Sensitivity of the concussion assessment battery. J Neurosurg. 2007;60(6):1050–8. https://doi.org/10.1227/01.NEU.0000255479.90999.C0.CrossRef

Broglio SP, McCrea M, McAllister T, Harezlak J, Katz B, Hack D, et al. A National Study on the Effects of Concussion in Collegiate Athletes and US Military Service Academy Members: The NCAA–DoD Concussion Assessment, Research and Education (CARE) Consortium Structure and Methods. Am J Sports Med. 2017;47(7):1437–51. https://doi.org/10.1007/s40279-017-0707-1.CrossRef

Brooks BL, Iverson GL, Sherman EMS, Roberge M. Identifying cognitive problems in children and adolescents with depression using computerized neuropsychological testing. Appl Neuropsychol. 2010;17(1):37–43. https://doi.org/10.1080/09084280903526083.CrossRefPubMed

10.

Bruce J, Echemendia R, Meeuwisse W, Comper P, Sisco A. 1 year test–retest reliability of ImPACT in professional ice hockey players. Clin Neuropsychol. 2014;28(1):14–25. https://doi.org/10.1080/13854046.2013.866272.CrossRefPubMed

11.

Carney N, Ghajar J, Jagoda A, Bedrick S, Davis-OʼReilly C, du Coudray H, Hack D, Helfand N, Huddleston A, Nettleton T, Riggio S. Concussion guidelines step 1: systematic review of prevalent indicators. J Neurosurg. 2014;75:S3–15. https://doi.org/10.1227/NEU.0000000000000433.CrossRef

12.

Cassidy JD, Carroll LJ, Peloso PM, Borg J, von Holst H, Holm L, et al. Incidence, risk factors and prevention of mild traumatic brain injury: Results of the WHO Collaborating Centre Task Force on Mild Traumatic Brain Injury. J Rehabil Med. 2004;43(Suppl):28–60. https://doi.org/10.1080/16501960410023732.CrossRef

13.

Chin EY, Nelson LD, Barr WB, McCrory P, McCrea MA. Reliability and validity of the sport concussion assessment tool-3 (SCAT3) in high school and collegiate athletes. Am J Sports Med. 2016;44(9):2276–85. https://doi.org/10.1177/0363546516648141.CrossRefPubMed

14.

CNS Vital Signs Interpretation Guide. (2014). CNS Vital Signs.

15.

Collins MW, Grindel SH, Lovell MR, Dede DE, Moser DJ, Phalin BR, et al. Relationship between concussion and neuropsychological performance in college football players. JAMA. 1999;282(10):964–70. https://doi.org/10.1001/jama.282.10.964.CrossRefPubMed

16.

Covassin T, Moran T, Wilhelm K. Concussion symptoms and neurocognitive performance of high school and college athletes who incur multiple concussions. Am J Sports Med. 2013;41(12):2885–9. https://doi.org/10.1177/0363546513499230.CrossRefPubMed

17.

Downey RI, Hutchison MG, Comper P. Determining sensitivity and specificity of the Sport Concussion Assessment Tool 3 (SCAT3) components in university athletes. Brain Inj. 2018;32(11):1345–52. https://doi.org/10.1080/02699052.2018.1484166.CrossRefPubMed

18.

Echemendia RJ, Bruce JM, Bailey CM, Sanders JF, Arnett P, Vargas G. The utility of post-concussion neuropsychological data in identifying cognitive change following sports-related MTBI in the absence of baseline data. Clin Neuropsychol. 2012;26(7):1077–91. https://doi.org/10.1080/13854046.2012.721006.CrossRefPubMed

19.

Echemandia RJ, Iverson GL, McCrea M, Macciocchi SN, Gioia GA, Putukian M, Comper P. Advances in neuropsychological assessment of sport-related concussion. Br J Sports Med. 2013;47(5):294–8. https://doi.org/10.1136/bjsports-2013-092186.CrossRef

20.

Elbin RJ, Kontos AP, Kegel N, Johnson E, Burkhart S, Schatz P. Individual and combined effects of LD and ADHD on computerized neurocognitive concussion test performance: evidence for separate norms. Arch Clin Neuropsychol. 2013;28(5):476–84. https://doi.org/10.1093/arclin/act024.CrossRefPubMed

21.

Garcia G-GP, Broglio SP, Lavieri MS, McCrea M, McAllister T, McAllister T. Quantifying the value of multidimensional assessment models for acute concussion: an analysis of data from the NCAA-DoD care consortium. Am J Sports Med. 2018;48(7):1739–49. https://doi.org/10.1007/s40279-018-0880-x.CrossRef

22.

Garcia G-GP, Yang J, Lavieri MS, McAllister TW, McCrea MA, Broglio SP. Optimizing components of the sport concussion assessment tool for acute concussion assessment. J Neurosurg. 2020. https://doi.org/10.1093/neuros/nyaa150.CrossRefPubMed

23.

Gualtieri CT, Johnson LG. Reliability and validity of a computerized neurocognitive test battery, CNS Vital Signs. Arch Clin Neuropsychol. 2006;21:623–43. https://doi.org/10.1016/j.acn.2006.05.007.CrossRefPubMed

24.

Guskiewicz KM, McCrea M, Marshall SW, Cantu RC, Randolph C, Barr W, et al. Cumulative effects associated with recurrent concussion in collegiate football players: The NCAA Concussion Study. JAMA. 2003;290(19):2549–55. https://doi.org/10.1001/jama.290.19.2549.CrossRefPubMed

25.

Hinton-Bayre AD. Choice of reliable change model can alter decisions regarding neuropsychological impairment after sports-related concussion. Clin J Sport Med. 2012;22(2):105–8. https://doi.org/10.1097/JSM.0b013e318248a526.CrossRefPubMed

26.

Houck ZM, Asken BM, Bauer RM, Kontos AP, McCrea MA, McAllister TW, Broglio SP, Clugston JR, Care Consortium Investigators. Multivariate base rates of low scores and reliable decline on ImPACT in healthy collegiate athletes using CARE consortium norms. J Int Neuropsychol Soc. 2019;25(09):961–71. https://doi.org/10.1017/S1355617719000729.CrossRefPubMed

27.

ImPACT Administration and Interpretation Manual. ImPACT Applications, Inc. (2016).

28.

James G, Witten D, Hastie T, Tibshirani R. An introduction to statistical learning, vol. 103. New York : Springer; 2013. https://doi.org/10.1007/978-1-4614-7138-7.CrossRef

29.

Langlois J, Rutland-Brown W, Wald M. The epidemiology and impact of traumatic brain injury: a brief overview. J Head Trauma Rehabil. 2006;21(5):375–8.CrossRef

30.

Lempke LB, Schmidt JD, Lynall RC. Athletic trainers’ concussion-assessment and concussion-management practices: an update. J Athl Train. 2020;55(1):17–26. https://doi.org/10.4085/1062-6050-322-18.CrossRefPubMedPubMedCentral

31.

Louey AG, Cromer JA, Schembri AJ, Darby DG, Maruff P, Makdissi M, McCrory P. Detecting cognitive impairment after concussion: sensitivity of change from baseline and normative data methods using the CogSport/Axon Cognitive Test Battery. Arch Clin Neuropsychol. 2014;29(5):432–41. https://doi.org/10.1093/arclin/acu020.CrossRefPubMed

32.

Lovell MR, Collins MW, Podell K, Powell J, Maroon J. ImPACT: Immediate post-concussion assessment and cognitive testing. Pittsburgh: NeuroHealth Systems LLC; 2000.

33.

Lucini FR, Fogliatto FS, da Silveira GJC, Neyeloff JL, Anzanello MJ, Kuchenbecker RS, Schaan BD. Text mining approach to predict hospital admissions using early medical records from the emergency department. Int J Med Inform. 2017;100:1–8. https://doi.org/10.1016/j.ijmedinf.2017.01.001.CrossRefPubMed

34.

Maruff P, Thomas E, Cysique L, Brew B, Collie A, Snyder P, Pietrzak RH. Validity of the CogState Brief Battery: relationship to standardized tests and sensitivity to cognitive impairment in mild traumatic brain injury, schizophrenia, and AIDS dementia complex. Arch Clin Neuropsychol. 2009;24(2):165–78. https://doi.org/10.1093/arclin/acp010.CrossRefPubMed

35.

McAllister T, McCrea M. Long-term cognitive and neuropsychiatric consequences of repetitive concussion and head-impact exposure. J Athl Train. 2017;52(3):309–17. https://doi.org/10.4085/1062-6050-52.1.14.CrossRefPubMedPubMedCentral

36.

McCrory P, Meeuwisse WH, Aubry M, Cantu RC, Dvorak J, Echemendia RJ, et al. Consensus statement on concussion in sport—the 4th international conference on concussion in sport held in Zurich, November 2012. PM&R. 2013;5(4):255–79. https://doi.org/10.1016/j.pmrj.2013.02.012.CrossRef

37.

McCrory P, Meeuwisse W, Dvorak J, Aubry M, Bailes J, Broglio S, Cantu RC, Cassidy D, Echemendia RJ, Castellani RJ, Davis GA, Ellenbogen R, Emery C, Engebretsen L, Feddermann-Demont N, Giza CC, Guskiewicz KM, Herring S, Iverson GL, et al. Consensus statement on concussion in sport—The 5th international conference on concussion in sport held in Berlin, October 2016. Br J Sports Med. 2017. https://doi.org/10.1136/bjsports-2017-097699.CrossRefPubMedPubMedCentral

38.

Meehan WP, d’Hemecourt P, Collins CL, Taylor AM, Comstock RD. Computerized neurocognitive testing for the management of sport-related concussions. J Pediatr. 2012;129(1):38–44. https://doi.org/10.1542/peds.2011-1972.CrossRef

39.

Nelson LD, LaRoche AA, Pfaller AY, Lerner EB, Hammeke TA, Randolph C, et al. Prospective, head-to-head study of three computerized neurocognitive assessment tools (CNTs): reliability and validity for the assessment of sport-related concussion. J Int Neuropsychol Soc. 2016;22:24–37. https://doi.org/10.1017/S1355617715001101.CrossRefPubMedPubMedCentral

40.

Putukian M, Echemendia R, Dettwiler-Danspeckgruber A, Duliba T, Bruce J, Furtado JL, Murugavel M. Prospective clinical assessment using sideline concussion assessment tool-2 testing in the evaluation of sport-related concussion in college athletes. Clin J Sport Med. 2015;25(1):7.CrossRef

41.

Rabinowitz AR, Levin HS. Cognitive sequelae of traumatic brain injury. Psychiatr Clin N Am. 2014;37(1):1–11. https://doi.org/10.1016/j.psc.2013.11.004.CrossRef

42.

Rahman-Filipiak AAM, Woodard JL. Administration and environment considerations in computer-based sports-concussion assessment. Neuropsychol Rev. 2013;23(4):314–34. https://doi.org/10.1007/s11065-013-9241-6.CrossRefPubMed

43.

Randolph C, McCrea M, Barr WB. Is neuropsychological testing useful in the management of sport-related concussion? J Athl Train. 2005;40(3):139–52.PubMedPubMedCentral

44.

Randolph C, Millis S, Barr WB, McCrea M, Guskiewicz KM, Hammeke TA, Kelly JP. Concussion Symptom Inventory: an empirically derived scale for monitoring resolution of symptoms following sport-related concussion. Arch Clin Neuropsychol. 2009;24(3):219–29. https://doi.org/10.1093/arclin/acp025.CrossRefPubMedPubMedCentral

45.

Reeves DL, Winter KP, Bleiberg J, Kane RL. ANAM^® genogram: historical perspectives, description, and current endeavors. Arch Clin Neuropsychol. 2007;22(1):15–37. https://doi.org/10.1016/j.acn.2006.10.013.CrossRef

46.

Register-Mihalik JK, Guskiewicz KM, Mihalik JP, Schmidt JD, Kerr ZY, McCrea MA. Reliable change, sensitivity, and specificity of a multidimensional concussion assessment battery: implication for caution in clinical practice. J Head Trauma Rehabil. 2013;28(4):274–83. https://doi.org/10.1097/HTR.0b013e3182585d37.CrossRefPubMed

47.

Resch JE, Brown CN, Schmidt J, Macciocchi SN, Blueitt D, Cullum CM, Ferrara MS. The sensitivity and specificity of clinical measures of sport concussion: three tests are better than one. Med BMJ Open Sport Exerc. 2016. https://doi.org/10.1136/bmjsem-2015-000012.CrossRef

48.

Resch JE, McCrea MA, Munro CC. Computerized neurocognitive testing in the management of sport-related concussion: an update. Neuropsychol Rev. 2013;23(4):335–49. https://doi.org/10.1007/211065-013-9242-5.CrossRefPubMed

49.

Robin X. Package “pROC.” Retrieved from https://cran.r-project.org/web/packages/pROC/pROC.pdf. (2019).

50.

Schatz P, Robertshaw S. Comparing post-concussive neurocognitive test data to normative data presents risks for under-classifying “above average” athletes. Arch Clin Neuropsychol. 2014;29(7):625–32. https://doi.org/10.1093/arclin/acu041.CrossRefPubMedPubMedCentral

51.

Schatz P, Sandel N. Sensitivity and specificity of the online version of ImPACT in high school and collegiate athletes. Am J Sports Med. 2013;41(2):321–6. https://doi.org/10.1177/0363546512466038.CrossRefPubMed

52.

Schmidt JD, Register-Mihalik JK, Mihalik JP, Kerr ZY, Guskiewicz KM. Identifying impairments after concussion: normative data versus individualized baselines. Med Sci Sports Exerc. 2012;44(9):1621–8. https://doi.org/10.1249/MSS.0b013e318258a9fb.CrossRefPubMed

53.

Thoma RJ, Cook JA, McGrew C, King JH, Pulsipher DT, Yeo RA, et al. Convergent and discriminant validity of the ImPACT with traditional neuropsychological measures. Cogent Psychol. 2018;5(1):1430199. https://doi.org/10.1080/23311908.2018.1430199.CrossRefPubMedPubMedCentral

54.

Walpole RE, Myers RH, Myers SL, Ye K. Probability & statistics for engineers & scientists. 9th ed. Upper Saddle River: Prentice Hall; 2012.

55.

Wood TA, Hsieh KL, An R, Ballard RA, Sosnoff JJ. Balance and gait alterations observed more than 2 weeks after concussion: a systematic review and meta-analysis. Am J Phys Med Rehabil. 2019;98(7):566–76. https://doi.org/10.1097/PHM.0000000000001152.CrossRefPubMed

Title: Sensitivity and Specificity of Computer-Based Neurocognitive Tests in Sport-Related Concussion: Findings from the NCAA-DoD CARE Consortium
Authors: Lauren L. Czerniak
Spencer W. Liebel
Gian-Gabriel P. Garcia
Mariel S. Lavieri
Michael A. McCrea
Thomas W. McAllister
Steven P. Broglio
CARE Consortium Investigators
Publication date: 01-02-2021
Publisher: Springer International Publishing
Keywords: Care
Traumatic Brain Injuries
Published in: Sports Medicine / Issue 2/2021
Print ISSN: 0112-1642
Electronic ISSN: 1179-2035
DOI: https://doi.org/10.1007/s40279-020-01393-7

Keynote webinar | Spotlight on sleep in brain health

Springer Medicine

Sensitivity and Specificity of Computer-Based Neurocognitive Tests in Sport-Related Concussion: Findings from the NCAA-DoD CARE Consortium

Abstract

Background

Methods

Results

Conclusion

Keynote webinar | Spotlight on sleep in brain health

Springer Medicine

Abstract

Background

Methods

Results

Conclusion

Please log in to get access to this content

Other articles of this Issue 2/2021

Neuromuscular Function of the Knee Joint Following Knee Injuries: Does It Ever Get Back to Normal? A Systematic Review with Meta-Analyses

An Update on Secular Trends in Physical Fitness of Children and Adolescents from 1972 to 2015: A Systematic Review

Transgender Women in the Female Category of Sport: Perspectives on Testosterone Suppression and Performance Advantage

Strategies for Reducing the Impact of Cycling on the Perineum in Healthy Males: Systematic Review and Meta-analysis

Which Physical Exercise Interventions Increase HDL-Cholesterol Levels? A Systematic Review of Meta-analyses of Randomized Controlled Trials

Effects of Bilateral and Unilateral Resistance Training on Horizontally Orientated Movement Performance: A Systematic Review and Meta-analysis