Top

European Journal of Trauma and Emergency Surgery

Published in:

Open Access 01-02-2022 | Original Article

Reliability of the assessment of non-technical skills by using video-recorded trauma resuscitations

Authors: Oscar E. C. van Maarseveen, Wietske H. W. Ham, Roel L. N. Huijsmans, Rianne G. F. Dolmans, Luke P. H. Leenen

Published in: European Journal of Trauma and Emergency Surgery | Issue 1/2022

Abstract

Purpose

Non-technical skills have gained attention, since enhancement of these skills is presumed to improve the process of trauma resuscitation. However, the reliability of assessing non-technical skills is underexposed, especially when using video analysis. Therefore, our primary aim was to assess the reliability of the Trauma Non-Technical Skills (T-NOTECHS) tool by video analysis. Secondarily, we investigated to what extent reliability increased when the T-NOTECHS was assessed by three assessors [average intra-class correlation (ICC)] instead of one (individual ICC).

Methods

As calculated by a pre-study power analysis, 18 videos were reviewed by three research assistants using the T-NOTECHS tool. Average and individual degree of agreement of the assessors was calculated using a two-way mixed model ICC.

Results

Average ICC was ‘excellent’ for the overall score and all five domains. Individual ICC was classified as ‘excellent’ for the overall score. Of the five domains, only one was classified as ‘excellent’, two as ‘good’ and two were even only ‘fair’.

Conclusions

Assessment of non-technical skills using the T-NOTECHS is reliable using video analysis and has an excellent reliability for the overall T-NOTECHS score. Assessment by three raters further improve the reliability, resulting in an excellent reliability for all individual domains.

Celso B, Tepas J, Langland-Orban B, Pracht E, Papa L, Lottenberg L, Flint L. A systematic review and meta-analysis comparing outcome of severely injured patients treated in trauma centers following the establishment of trauma systems. J Trauma Acute Care Surg. 2006;60(2):371–8.CrossRef

MacKenzie EJ, Rivara FP, Jurkovich GJ, Nathens AB, Frey KP, Egleston BL, et al. A national evaluation of the effect of trauma-center care on mortality. N Engl J Med. 2006;354:366–78.CrossRefPubMed

Demetriades D, Martin M, Salim A, Rhee P, Brown C, Chan L, et al. The effect of trauma center designation and trauma volume on outcome in specific severe injuries. Ann Surg. 2005;242:512–9.CrossRefPubMedPubMedCentral

Georgiou A, Lockey DJ. The performance and assessment of hospital trauma teams. Scand J Trauma Resusc Emerg Med. 2010;13(18):66.CrossRef

International ATLS working group. Advanced Trauma Life Support (ATLS®): The Ninth Edition. J Trauma Acute Care Surg. 2013;74(5):1363–1363.

Künzle B, Kolbe M, Grote G. Ensuring patient safety through effective leadership behaviour: a literature review. Safety Sci. 2010;48:1–17.CrossRef

Manser T. Teamwork and patient safety in dynamic domains of healthcare: a review of the literature. Acta Anaesthesiol Scand. 2009;53(2):143–51.CrossRefPubMed

Cole E, Crichton N. The culture of a trauma team in relation to human factors. J Clin Nurs. 2006;15(10):1257–66.CrossRefPubMed

Capella J, Smith S, Philp A, Putnam T, Gilbert C, Fry W, et al. Teamwork training improves the clinical care of trauma patients. J Surg Educ. 2010;67(6):439–43.CrossRefPubMed

10.

Dumas RP, Vella MA, Chreiman KC, Smith BP, Subramanian M, Maher Z, Holena DN. Team assessment and decision making is associated with outcomes: a trauma video review analysis. J Surg Res. 2020;246:544–9.CrossRefPubMed

11.

Bassin B, Murray J, Benner C, Sikavitsas A, Santen S, Nypaver M. Improvement in non-technical skills using an in-situ multidisciplinary simulation based trauma team training (T3) curriculum. Acad Emerg Med. 2014;21(5):S222.

12.

Gjeraa K, Moller TP, Ostergaard D. Efficacy of simulation-based trauma team training of non-technical skills. A systematic review. Acta Anaesthesiol Scand. 2014;58(7):775–87.CrossRefPubMed

13.

Briggs A, Raja AS, Joyce MF, Yule SJ, Jiang W, Lipsitz SR, Havens JM. The role of nontechnical skills in simulated trauma resuscitation. J Surg Educ. 2015;72(4):732–9.CrossRefPubMed

14.

Westli HK, Johnsen BH, Eid J, Rasten I, Brattebø G. Teamwork skills, shared mental models, and performance in simulated trauma teams: an independent group design. Scand J Trauma Resusc Emerg Med. 2010;18:47.CrossRefPubMedPubMedCentral

15.

Murphy M, McCloughen A, Curtis K. The impact of simulated multidisciplinary Trauma Team Training on team performance: a qualitative study. Australas Emerg Care. 2019;22(1):1–7.CrossRefPubMed

16.

Groombridge CJ, Kim Y, Maini A, Fitzgerald MC (2019) Stress and decision-making in resuscitation: a systematic review. Resuscitation.

17.

Rosqvist E, Lauritsalo S, Paloneva J. Short 2-H in situ trauma team simulation training effectively improves non-technical skills of hospital trauma teams. Scand J Surg. 2019;108(2):117–23.CrossRefPubMed

18.

Steinemann S, Berg B, DiTullio A, Skinner A, Terada K, Anzelon K, et al. Assessing teamwork in the trauma bay: introduction of a modified “NOTECHS” scale for trauma. Am J Surg. 2012;203(1):69–75.CrossRefPubMed

19.

Flin R, Martin L, Goeters KM, Hormann HJ, Amalberti R, Valot C, Nijhuis H. Development of the NOTECHS (non-technical skills) system for assessing pilots' CRM skills. Hum Factors Aerosp Saf. 2003;3:97–120.

20.

Sevdalis N, Davis R, Koutantji M, Undre S, Darzi A, Vincent CA. Reliability of a revised NOTECHS scale for use in surgical teams. Am J Surg. 2008;196(2):184–90.CrossRefPubMed

21.

Shieh G. Choosing the best index for the average score intraclass correlation coefficient. Behav Res Methods. 2016;48(3):994–1003.CrossRefPubMed

22.

Tiel Groenestege-Kreb D, Van Maarseveen O, Leenen L. Trauma team. Br J Anaesth. 2014;113(2):258–65.CrossRefPubMed

23.

Walter SD, Eliasziw M, Donner A. Sample size and optimal designs for reliability studies. Stat Med. 1998;17(1):101–10.CrossRefPubMed

24.

Koo TK, Li MY. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J Chiropr Med. 2016;15:155–63.CrossRefPubMedPubMedCentral

25.

Landers RN. Computing intraclass correlations (ICC) as estimates of interrater reliability in SPSS. Winnower. 2015;2:e143518.

26.

Cicchetti DV. Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychol Assess. 1994;6(4):284–90.CrossRef

27.

McGraw KO, Wong SP. Forming inferences about some intraclass correlation coefficients. Psychol Methods. 1996;1:30.CrossRef

28.

Mishra A, Catchpole K, McCulloch P. The Oxford NOTECHS system: reliability and validity of a tool for measuring teamwork behaviour in the operating theatre. Qual Saf Health Care [Internet]. 2009;18(2):104–8.CrossRef

29.

Robertson ER, Hadi M, Morgan LJ, Pickering SP, Collins G, New S, et al. Oxford NOTECHS II: a modified theatre team non-technical skills scoring system. PLoS One. 2014;9(3).

30.

Shrout PE, Fleiss JL. Intraclass correlations: uses in assessing rater reliability. Psychol Bullit. 1979;86(2):420–8.CrossRef

31.

Khan KZ, Gaunt K, Ramachandran S, Pushkar P. The Objective Structured Clinical Examination (OSCE): AMEE Guide No. 81. Part II: organisation and administration. Med Teach. 2013;35(9):e1447-63.PubMed

32.

Wilkinson TJ, Frampton CM, Thompson-Fawcett M, Egan T. Objectivity in objective structured clinical examinations: checklists are no substitute for examiner commitment. Acad Med. 2003;78(2):219–23.CrossRefPubMed

33.

Holmboe ES, Hawkins RE, Huot SJ. Effects of training in direct observation of medical residents' clinical competence: a randomized trial. Ann Intern Med. 2004;140(11):874–8.CrossRefPubMed

34.

Pell G, Homer MS, Roberts TE. Assessor training: its effects on criterion-based assessment in a medical context. Int J Res Methods Educ. 2008;31:143–54.CrossRef

35.

Pradarelli JC, Gupta A, Lipsitz S, Blair PG, Sachdeva AK, Smink DS, Yule S (2020) Assessment of the non‐technical skills for surgeons (NOTSS) framework in the USA. Br J Surg.

36.

El-Shafy IA, Delgado J, Akerman M, Bullaro F, Christopherson NA, Prince JM. Closed-loop communication improves task completion in pediatric trauma resuscitation. J Surg Educ. 2018;75(1):58–64.CrossRefPubMed

37.

Sakran JV, Finneman B, Maxwell C, Sonnad SS, Sarani B, Pascual J, Sims C. Trauma leadership: does perception drive reality? J Surg Educ. 2012;69(2):236–40.CrossRefPubMed

Title: Reliability of the assessment of non-technical skills by using video-recorded trauma resuscitations
Authors: Oscar E. C. van Maarseveen
Wietske H. W. Ham
Roel L. N. Huijsmans
Rianne G. F. Dolmans
Luke P. H. Leenen
Publication date: 01-02-2022
Publisher: Springer Berlin Heidelberg
Published in: European Journal of Trauma and Emergency Surgery / Issue 1/2022
Print ISSN: 1863-9933
Electronic ISSN: 1863-9941
DOI: https://doi.org/10.1007/s00068-020-01401-5

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Reliability of the assessment of non-technical skills by using video-recorded trauma resuscitations

Abstract

Purpose

Methods

Results

Conclusions

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Abstract

Purpose

Methods

Results

Conclusions

Please log in to get access to this content

Other articles of this Issue 1/2022

Evaluation of the altitude impact on a point-of-care thromboelastography analyzer measurement: prerequisites for use in airborne medical evacuation courses

The evolution of emergency general surgery: its time for a dedicated program manager

Operative intervention of unstable paediatric pelvic fracture: radiological and functional assessment

Mortality in relation to presence and type of oral antithrombotic agent among adult trauma patients: a single-center retrospective cohort study

Lisfranc fleck sign: characteristics and clinical outcomes following fixation using a percutaneous position Lisfranc screw

Feasibility of intraoperative angioembolization for trauma patients using C-arm digital subtraction angiography