Skip to main content
Key Specialties
Cardiology Diabetology Endocrinology Gastroenterology Geriatrics and Gerontology Gynecology and Obstetrics Hematology Infectious Disease Internal Medicine Nephrology Neurology Oncology Pediatrics Respiratory Medicine Rheumatology Surgery
Congress Coverage
2024 ACC Congress 2023 ESMO Congress 2023 EASD Congress 2023 ERS Congress 2023 ESC Congress
Clinical Collections
Advances in Alzheimer’s

At a glance: The STEP trials

A round-up of the STEP phase 3 clinical trials evaluating semaglutide for weight loss in people with overweight or obesity.
ADVANCED SEARCH
Log in
Top
Sports Medicine
Published in: Sports Medicine 5/2018

Open Access 01-05-2018 | Original Research Article

Test-Retest Reliability and Interpretation of Common Concussion Assessment Tools: Findings from the NCAA-DoD CARE Consortium

Authors: Steven P. Broglio, Barry P. Katz, Shi Zhao, Michael McCrea, Thomas McAllister, CARE Consortium Investigators

Published in: Sports Medicine | Issue 5/2018

Login to get access

Abstract

Background

Concussion diagnosis is typically made through clinical examination and supported by performance on clinical assessment tools. Performance on commonly implemented and emerging assessment tools is known to vary between administrations, in the absence of concussion.

Objective

To evaluate the test-retest reliability of commonly implemented and emerging concussion assessment tools across a large nationally representative sample of student-athletes.

Methods

Participants (n = 4874) from the Concussion Assessment, Research, and Education Consortium completed annual baseline assessments on two or three occasions. Each assessment included measures of self-reported concussion symptoms, motor control, brief and extended neurocognitive function, reaction time, oculomotor/oculovestibular function, and quality of life. Consistency between years 1 and 2 and 1 and 3 were estimated using intraclass correlation coefficients or Kappa and effect sizes (Cohen’s d). Clinical interpretation guidelines were also generated using confidence intervals to account for non-normally distributed data.

Results

Reliability for the self-reported concussion symptoms, motor control, and brief and extended neurocognitive assessments from year 1 to 2 ranged from 0.30 to 0.72 while effect sizes ranged from 0.01 to 0.28 (i.e., small). The reliability for these same measures ranged from 0.34 to 0.66 for the year 1–3 interval with effect sizes ranging from 0.05 to 0.42 (i.e., small to less than medium). The year 1–2 reliability for the reaction time, oculomotor/oculovestibular function, and quality-of-life measures ranged from 0.28 to 0.74 with effect sizes from 0.01 to 0.38 (i.e., small to less than medium effects).

Conclusions

This investigation noted less than optimal reliability for most common and emerging concussion assessment tools. Despite this finding, their use is still necessitated by the absence of a gold standard diagnostic measure, with the ultimate goal of developing more refined and sound tools for clinical use. Clinical interpretation guidelines are provided for the clinician to apply with a degree of certainty in application.
Appendix
Available only for authorised users
Literature
1.
2.
Herring SA, Cantu RC, Guskiewicz KM, Putukian M, Kibler WB, Bergfeld JA, et al. Concussion (mild traumatic brain injury) and the team physician: a consensus statement: 2011 update. Med Sci Sports Exerc. 2011;43(12):2412–22.CrossRefPubMed
3.
Broglio SP, Cantu RC, Gioia GA, Guskiewicz KM, Kutcher J, Palm M, et al. National Athletic Trainers’ Association position statement: management of sport concussion. J Athl Train. 2014;49(2):245–65.
4.
National Collegiate Athletic Association (NCAA). Interassociation consensus: diagnosis and management of sport-related concussion best practices. Indianapolis, IN; 2016.
5.
Resch JE, McCrea MA, Cullum CM. Computerized neurocognitive testing in the management of sport-related concussion: an update. Neuropsychol Rev. 2013;23(4):335–49.CrossRefPubMed
6.
McCrea M. Standardized mental status testing on the sideline after sport-related concussion. J Athl Train. 2001;36(3):274–9.PubMedPubMedCentral
7.
Broglio SP, Zhu W, Sopiarz K, Park Y. Generalizability theory analysis of balance error scoring system reliability in healthy young adults. J Athl Train. 2009;44(5):497–502.
8.
Leong DF, Balcer LJ, Galetta SL, Evans G, Gimre M, Watt D. The King-Devick test for sideline concussion screening in collegiate football. J Optom. 2015;8(2):131–9.
9.
Stocker RP, Khan H, Henry L, Germain A. Effects of sleep loss on subjective complaints and objective neurocognitive performance as measured by the immediate post-concussion assessment and cognitive testing. Arch Clin Neuropsychol. 2017;32(3):349–68.PubMed
10.
Moser RS, Schatz P, Neidzwski K, Ott SD. Group versus individual administration affects baseline neurocognitive test performance. Am J Sport Med. 2011;39(11):2325–50.CrossRef
11.
Jacobson NS, Truax P. Clinical significance: a statistical approach to defining meaningful change in psychotherapy research. J Consult Clin Psychol. 1991;59(1):12–9.CrossRefPubMed
12.
Iverson GL, Lovell MR, Collins MW. Interpreting change on ImPACT following sport concussion. Clin Neuropsychol. 2003;17(4):460–70.CrossRefPubMed
13.
Barr WB, McCrea M. Sensitivity and specificity of standardized neurocognitive testing immediately following sports concussion. J Int Neuropsychol Soc. 2001;7(6):693–702.CrossRefPubMed
14.
Chin EY, Nelson LD, Barr WB, McCrory P, McCrea MA. Reliability and validity of the Sport Concussion Assessment Tool-3 (SCAT3) in high school and collegiate athletes. Am J Sports Med. 2016;44(9):2276–85.CrossRefPubMed
15.
Broglio SP, McCrea M, McAllister T, Harezlak J, Katz B, Hack D, et al. A national study on the effects of concussion in collegiate athletes and US Military Service Academy members: the NCAA–DoD Concussion Assessment, Research and Education (CARE) Consortium structure and methods. Sports Med. 2017;47(7):1437–51.CrossRefPubMedPubMedCentral
16.
McCrea M, Guskiewicz KM, Marshall SW, Barr W, Randolph C, Cantu RC, et al. Acute effects and recovery time following concussion in collegiate football players: the NCAA Concussion Study. JAMA. 2003;290(19):2556–63.CrossRefPubMed
17.
Riemann BL, Guskiewicz KM, Shields EW. Relationship between clinical and forceplate measures of postural stability. J Sport Rehab. 1999;8(2):71–82.CrossRef
18.
McCrory P, Meeuwisse WH, Aubry M, Cantu RC, Dvorak J, Echemendia RJ, et al. Consensus statement on concussion in sport: the 4th International Conference on Concussion in Sport held in Zurich, November 2012. Br J Sports Med. 2013;47(5):250–8.
19.
Meachen SJ, Hanks RA, Milils SR, Rapport LJ. The reliability and validity of the Brief Symptom Inventory-18 in persons with traumatic brain injury. Arch Phys Med Rehabil. 2008;89(5):958–65.CrossRefPubMed
20.
Lovell MR, Iverson GL, Collins MW, Podell K, Johnston KM, Pardini JE, et al. Measurement of symptoms following sports-related concussion: reliability and normative data for the post-concussion scale. Appl Neuropsychol. 2006;13(3):166–74.CrossRefPubMed
21.
Gualtieri CT, Johnson LG. Reliability and validity of a computerized neurocognitive test battery, CNS Vital Signs. Arch Clin Neuropsychol. 2006;21(7):623–43.CrossRefPubMed
22.
Collie A, Maruff P, Darby D, McStephen M. The effects of practice on the cognitive test performance of neurologically normal individuals assessed at brief test-retest intervals. J Int Neuropsychol Soc. 2003;9:419–28.CrossRefPubMed
23.
Eckner JT, Kutcher JS, Broglio SP, Richardson JK. Effect of sport-related concussion on clinically measured simple reaction time. Br J Sports Med. 2014;48(2):112–8.CrossRefPubMed
24.
Mucha A, Collins MW, Elbin RJ, Furman JM, Troutman-Enseki C, DeWolf RM, et al. A brief vestibular/ocular motor screening (VOMS) assessment to evaluate concussions: preliminary findings. Am J Sports Med. 2014;42(10):2479–86.CrossRefPubMedPubMedCentral
25.
Diener E, Emmons R, Larsen R, Griffin S. The satisfaction with life scale. J Pers Assess. 1985;49:1105–17.CrossRef
26.
Pavot W, Diener E. Review of the satisfaction with life scale. Psych Assess. 1993;5:164–72.CrossRef
27.
Shrout PE, Fleiss JL. Intraclass correlations: uses in assessing rater reliability. Psychol Bull. 1979;86(2):420–8.CrossRefPubMed
28.
Portney LG, Watkins MP. Foundations of clinical research: applications to practice. Upper Saddle River: Pearson: Prentice Hall; 2009.
29.
Randolph C, McCrea M, Barr WB. Is neuropsychological testing useful in the management of sport-related concussion? J Athl Train. 2005;40(3):139–54.PubMedPubMedCentral
30.
Cohen J. Statistical power analysis for the behavioral sciences. New York: Academic; 1977.
31.
van der Vaart AW. Asymptotic statistics. Cambridge, UK: Cambridge University Press; 1998.CrossRef
32.
Katz BP, Kudela MP, Harezlak J, McCrea M, McAllister T, Broglio SP. Baseline performance of NCAA athletes on a concussion assessment battery: A report from the CARE consortium. Sports Med (in review).
33.
Register-Mihalik JK, Guskiewicz KM, Mihalik JP, Schmidt JD, Kerr ZY, McCrea MA. Reliable change, sensitivity, and specificity of a multidimensional concussion assessment battery: implications for caution in clinical practice. J Head Trauma Rehabil. 2013;28(4):274–83.
34.
Lancaster MA, McCrea MA, Nelson LD. Psychometric properties and normative data for the Brief Symptom Inventory-18 (BSI-18) in high school and collegiate athletes. Clin Neuropsychol. 2016;30(2):338–50.CrossRefPubMedPubMedCentral
35.
Resch J, Driscoll A, McCaffrey N, Brown C, Ferrara MS, Macciocchi S, et al. ImPact test-retest reliability: reliably unreliable? J Athl Train. 2013;48(4):506–11.
36.
Broglio SP, Ferrara MS, Macciocchi SN, Baumgartner TA, Elliott R. Test-retest reliability of computerized concussion assessment programs. J Athl Train. 2007;42(4):509–14.
37.
Nelson LD, LaRoche AA, Pfaller AY, Lerner EB, Hammeke TA, Randolph C, et al. Prospective, head-to-head study of three Computerized Neurocognitive Assessment Tools (CNTs): reliability and validity for the assessment of sport-related concussion. J Int Neuropsychol Soc. 2016;22(1):24–37.CrossRefPubMedPubMedCentral
38.
Eckner JT, Kutcher JS, Richardson JK. Between-seasons test-retest reliability of clinically measured reaction time in national collegiate athletic association division I athletes. J Athl Train. 2011;46(4):409–14.CrossRefPubMedPubMedCentral
39.
Kontos AP, Sufrinko A, Elbin RJ, Puskar A, Collins MW. Reliability and associated risk factors for performance on the vestibular/ocular motor screening (VOMS) tool in healthy collegiate athletes. Am J Sports Med. 2016;44(6):1400–6.CrossRefPubMed
40.
Eckner JT, Richardson JK, Kim H, Joshi MS, Oh YK, Ashton-Miller JA. Reliability and criterion validity of a novel clinical test of simple and complex reaction time in athletes. Percept Mot Skills. 2015;120(3):841–59.CrossRefPubMedPubMedCentral
41.
Galetta KM, Morganroth J, Moehringer N, Mueller B, Hasanaj L, Webb N, et al. Adding vision to concussion testing: a prospective study of sideline testing in youth and collegiate athletes. J NeuroOphthalmol. 2015;35(3):235–41.CrossRefPubMed
42.
Putukian M, Echemendia R, Dettwiler-Danspeckgruber A, Duliba T, Bruce J, Furtado JL, et al. Prospective clinical assessment using Sideline Concussion Assessment Tool-2 testing in the evaluation of sport-related concussion in college athletes. Clin J Sport Med. 2015;25(1):36–42.CrossRefPubMed
43.
McCrea M, Barr WB, Guskiewicz KM, Randolph C, Marshall SW, Cantu R, et al. Standard regression-based methods for measuring recovery after sport-related concussion. J Int Neuropsychol Soc. 2005;11:58–69.CrossRefPubMed
44.
Broglio SP, Macciocchi SN, Ferrara MS. Sensitivity of the concussion assessment battery. Neurosurgery. 2007;60(6):1050–7 (discussion 7–8).
45.
46.
Byron K, Khazanchi S. A meta-analytic investigation of the relationship of state and trait anxiety to performance on figural and verbal creative tasks. Pers Soc Psychol Bull. 2011;37(2):269–83.CrossRefPubMed
47.
Masson M, East-Richard C, Cellard C. A meta-analysis on the impact of psychiatric disorders and maltreatment on cognition. Neuropsychology. 2016;30(2):143–56.CrossRefPubMed
48.
49.
Broglio SP, Cantu RC, Gioia GA, Guskiewicz KM, Kutcher JS, Palm M, et al. National athletic trainers’ association position statement: management of sport concussion. J Athl Train. 2014;49(2):245–65.CrossRefPubMedPubMedCentral
50.
Harmon KG, Drezner J, Gammons M, Guskiewicz KM, Halstead M, Herring SA, et al. American Medical Society for Sports Medicine position statement: concussion in sport. Clin J Sport Med. 2013;23(1):1–18.CrossRefPubMed
51.
Doros G, Lew R. Design based on intra-class correlation coefficients. Am J Biostat. 2010;1(1):1–8.
52.
Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet. 1986;1(8476):307–10.CrossRefPubMed
53.
Resch JE, Brown CN, Schmidt J, Macciocchi SN, Blueitt D, Cullum CM, et al. The sensitivity and specificity of clinical measures of sport concussion: three tests are better than one. BMJ Open Sport Exerc Med. 2016;2(1):e000012.CrossRefPubMedPubMedCentral
54.
Leahy S. Peyton Manning admits to tanking NFL’s baseline concussion test. USA Today (2011).
55.
Lovell MR, Collins MW, Iverson GL, Field M, Maroon JC, Cantu RC, et al. Recovery from mild concussion in high school athletes. J Neurosurg. 2003;98(2):296–301.CrossRefPubMed
56.
Giedd JN, Blumenthal J, Jefferies NO, Castellanos FX, Liu H, Zijdenbos A, et al. Brain development during childhood and adolescence: a longitudinal MRI study. Nat Neurosci. 1999;2(10):861–3.CrossRefPubMed
57.
Sowell ER, Thompson PM, Holmes CJ, Jernigan TL, Toga AW. In vivo evidence for post-adolescent brain maturation in frontal and striatal regions. Nat Neurosci. 1999;2:859–61.CrossRefPubMed
58.
Fischer KW, Rose S. Dynamic growth cycles of brain and cognitive development. Developmental Neuroimaging. San Diego, CA: Academic Press, Inc.; 1997. p. 263–79.
59.
Yakovlev PI, Lecours AR. The myelogenetic cycles of regional maturation of the brain. In: Minkowski A, editor. Regional development of the brain in early life. Philadelphia, PA: FA Davis; 1967. p. 3–70.
60.
Metadata
Title
Test-Retest Reliability and Interpretation of Common Concussion Assessment Tools: Findings from the NCAA-DoD CARE Consortium
Authors
Steven P. Broglio
Barry P. Katz
Shi Zhao
Michael McCrea
Thomas McAllister
CARE Consortium Investigators
Publication date
01-05-2018
Publisher
Springer International Publishing
Published in
Sports Medicine / Issue 5/2018
Print ISSN: 0112-1642
Electronic ISSN: 1179-2035
DOI
https://doi.org/10.1007/s40279-017-0813-0

Other articles of this Issue 5/2018

Sports Medicine 5/2018 Go to the issue

Correction

Correction to: The Effect of Natural or Simulated Altitude Training on High-Intensity Intermittent Running Performance in Team-Sport Athletes: A Meta-Analysis

Review Article

Running Performance, , and Running Economy: The Widespread Issue of Endogenous Selection Bias

Review Article

Neuromuscular Control Deficits and the Risk of Subsequent Injury after a Concussion: A Scoping Review

Current Opinion

Fuel for the Work Required: A Theoretical Framework for Carbohydrate Periodization and the Glycogen Threshold Hypothesis

Systematic Review

Aerobic Interval vs. Continuous Training in Patients with Coronary Artery Disease or Heart Failure: An Updated Systematic Review and Meta-Analysis with a Focus on Secondary Outcomes

Review Article

Methodological Characteristics and Future Directions for Plyometric Jump Training Research: A Scoping Review