Top

Surgical Endoscopy

Published in:

01-01-2006

Construct and face validity of MIST-VR, Endotower, and CELTS

Are we ready for skills assessment using simulators?

Authors: S. Maithel, R. Sierra, J. Korndorffer, P. Neumann, S. Dawson, M. Callery, D. Jones, D. Scott

Published in: Surgical Endoscopy | Issue 1/2006

Abstract

Background

Video trainers may best offer visually realistic laparoscopic simulation, whereas virtual reality (VR) modules may best provide multidimensional objective measures of performance. This study compares the construct and face validity of three different laparoscopic simulators.

Methods

Subjects were voluntarily enrolled at the Learning Center during the 2004 SAGES annual meeting. Each subject completed two repetitions of a single task on each of three simulators, MIST-VR, Endotower, and CELTS; performance scores were automatically generated and recorded. Scores of individuals with various levels of experience were compared to determine construct validity for each simulator. Experience was defined according to four parameters: (a) PGY level, (b) fellowship training, (c) basic laparoscopic cases, and (d) advanced laparoscopic cases. Subjects rated each simulator regarding six face validity (realism of simulation) parameters using a 10-point Likert scale (10 = best rating) and participant scores were compared to previously established expert scores (proficiency goals for training).

Results

Ninety-one attendees completed the study. Construct validity was demonstrated for all three simulators; significant differences in scores were detected according to one parameter for MIST-VR, two parameters for Endotower, and all four parameters for CELTS. Face validity was rated as good to excellent for all three simulators (7.0 ± 0.3 for MIST-VR, 7.9 ± 0.3 for Endotower [p < 0.001 vs MIST-VR], and 8.7 ± 0.1 for CELTS [p = 0.001 vs MIST-VR, p = 0.01 vs Endotower]); 6%, 0%, and 36% of “expert” participants obtained expert scores on MIST-VR, Endotower, and CELTS, respectively.

Conclusions

All three simulators demonstrated significant construct and reasonable face validity. Although virtual reality holds great promise to expand the scope of laparoscopic simulation, current interfaces may limit their utility for assessment. Computer-enhanced video trainers may offer an improved interface while incorporating useful multidimensional metrics. Further work is needed to establish standards for appropriate skills assessment methods and performance levels using simulators.

McCormick PH, Tanner WA, Keane FB, Tierney S (2003) Minimally invasive techniques in common surgical procedures: implications for training. Ir J Med Sci 172(1): 27–29PubMedCrossRef

Powers TW, Murayama KM, Toyama M, Murphy S, Denham EW 3rd, Derossis AM, Joehl RJ (2002) Housestaff performance is improved by participation in a laparoscopic skills curriculum. Am J Surg 184(6): 626–629PubMedCrossRef

Ali MR, Mowery Y, Kaplan B, DeMaria EJ (2002) Training the novice in laparoscopy. more challenge is better. Surg Endosc 16(12): 1732–1736PubMedCrossRef

Gonzalez R, Bowers SP, Smith CD Ramshaw BJ (2004) Does setting specific goals and providing feedback during training result in better acquisition of laparoscopic skills? Am J Surg 70(1): 35–39

Scott DJ, Valentine RJ, Bergen PC, Rege RV, Laycock R, Tesfay ST, Jones DB (2000) Evaluating surgical competency with the Amerian Board of Surgery In-Training Examination, skill testing, and intraoperative assessment. Surgery 128(4): 613–622PubMedCrossRef

Hyltander A, Liljegren E, Rhodin PH, Lonroth H (2002) The transfer of basic skills learned in a laparoscopic simulator to the operating room. Surg Endosc 16(9): 1324–1328PubMedCrossRef

Scott DJ, Bergen PC, Rege RV, Laycock R, Tesfay ST, Valentine RJ, Euhus DM, Jeyarajah DR, Thompson WM, Jones DB (2000) Laparoscopic training on bench models: better and more cost effective than operating room experience? J Am Coll Surg 191(3): 272–283PubMedCrossRef

Scott DJ, Young WN, Tesfay ST, Frawley WH, Rege RV, Jones DB (2001) Laparoscopic skills training. Am J Surg 182(2): 137–142PubMedCrossRef

Hamilton EC, Scott DJ, Kapoor A, Nwariaku F, Bergen PC, Rege RV, Tesfay ST, Jones DB (2001) Improving operative performance using a laparoscopic hernia simulator. Am J Surg 182(6): 725–728PubMedCrossRef

10.

Ota D, Loftin B, Saito T, Lea R, Keller J (1995) Virtual reality in surgical education. Comput Bio Med 25(2): 127–137CrossRef

11.

Sutton C, McCloy R, Middlebrook A, Chater P, Wilson M, Stone R (1997) MIST VR: a laparoscopic surgery procedures trainer and evaluator. Stud Health Technol Inform 39: 598–607PubMed

12.

Grantcharov TP, Rosenberg J, Pahle E, Funch-Jensen P (2001) Virtual reality computer simulation. Surg Endosc 15(3): 242–244PubMedCrossRef

13.

Ahlberg G, Heikkinen T, Iselius L, Leijonmarck CE, Rutqvist J, Arvidsson D (2002) Does training in a virtual reality simulator improve surgical perfomance? Surg Endosc 16(1): 126–129PubMedCrossRef

14.

Wilhelm DM, Ogan K, Roehrborn CG, Cadeddu JA, Pearle MS (2002) Assessment of basic endoscopic performance using a virtual reality simulator. J Am Coll Surg 195(5): 675–681PubMedCrossRef

15.

Gallagher AG, McClure N, McGuigan J, Crothers I, Browning J (1999) Virtual reality training in laparoscopic surgery: a preliminary assessment of minimally invasive surgical trainer virtual reality (MIST-VR). Endoscopy 31(4): 310–313PubMedCrossRef

16.

Torkington J, Smith SG, Rees BI, Darzi A (2004) Skill transfer from virtual reality to a real laparoscopic task. Surg Endosc 15(10): 1076–1079CrossRef

17.

Jordan JA, Gallagher AG, McGuigan J, McClure N (2001) Virtual reality training leads to faster adaptation to the novel psychomotor restrictions encountered by laparoscopic surgeons. Surg Endosc 15(10): 1080–1084PubMedCrossRef

18.

Seymour NE, Gallagher AG, Roman SA, O’Brien MK, Bansal VK, Andersen DK, Satava RM (2002) Virtual reality training improves operating room performance: results of a randomized, double-blinded study. Ann Surg 236(4): 458–463PubMedCrossRef

19.

Grantcharov TP, Kristiansen VB, Bendix J, Bardram L, Rosenberg J, Funch-Jensen P (2004) Randomized clinical trial of virtual reality simulation for laparoscopic skills training. Br J Surg 91(2): 146–150PubMedCrossRef

20.

Jordan JA, Gallagher AG, McGuigan J, McGlade K, McClure N (2000) A comparison between randomly alternating imaging, normal laparoscopic imaging, and virtual reality training in laparoscopic psychomotor skill acquisition. Am J Surg 180(3): 208–211PubMedCrossRef

21.

Kothari SN, Kaplan BJ, DeMaria EJ, Broderick TJ, Merrell RC (2002) Training in laparoscopic suturing skills using a new computer-based virtual reality simulator (MIST-VR) provides results comparable to those with an established pelvic trainer system. J Laparoendosc Adv Surg Tech A 12(3): 167–173PubMedCrossRef

22.

Hamilton EC, Scott DJ, Fleming JB, Rege RV, Laycock R, Bergen PC, Tesfay ST, Jones DB (2002) Comparison of video trainer and virtual reality training systems on acquisition of laparoscopic skills. Surg Endosc 16(3): 406–411PubMedCrossRef

23.

Brunner WC, Korndorffer JR, Sierra R, Dunne JB, Yau CL, Corsetti RL, Slakey DP, Townsend MC, Scott DJ (2005) Determining standards for laparoscopic competency using virtual reality. Am Surg 71: 29–35

24.

Haluck RS, Webster RW, Snyder AJ, Melkonian MG, Mohler BJ, Dise ML, Lefever A (2001) A virtual reality surgical trainer for navigation in laparoscopic surgery. Stud Health Technol Inform 81: 171–176PubMed

25.

Stylopoulos N, Cotin S, Maithel SK, Ottensmeyer M, Jackson PG, Bardsley RS, Neumann PF, Rattner DW, Dawson SL (2004) Computer-enhanced laparoscopic training system (CELTS): bridging the gap. Surg Endosc 18(5): 782–789PubMedCrossRef

26.

Torkington J, Smith SG, Rees BI, Darzi A (2001) The role of the basic surgical skills course in the acquisition and retention of laparoscopic skill. Surg Endosc 15(10): 1071–1075PubMedCrossRef

27.

Munz Y, Kumar BD, Moorthy K, Mann S, Darzi A (2004) Laparoscopic virtual reality and box trainers: is one superior to the other? Surg Endosc 18: 485–494PubMedCrossRef

28.

Gallagher AG, Richie K, McClure N, McGuigan J (2001) Objective psychomotor skills assessment of experienced, junior, and novice laparoscopists with virtual reality. World J Surg 25(11): 1478–1483PubMedCrossRef

29.

Gallagher AG, Lederman AB, McGlade K, Satava RM, Smith CD (2004) Discriminative validity of the Minimally Invasive Surgical Trainer in Virtual Reality (MIST-VR) using criteria levels based on expert performance. Surg Endosc 18: 660–665PubMedCrossRef

30.

Taffinder N, Sutton C, Fishwick RJ, McManus IC, Darzi A (1998). Validation of virtual reality to teach and assess psychomotor skills in laparoscopic surgery: results from randomised controlled studies using the MIST VR laparoscopic simulator. Stud Health Technol Inform 50: 124–130PubMed

31.

McNatt SS, Smith CD (2001) A computer-based laparoscopic skills assessment device differentiates experienced from novice laparoscopic surgeons. Surg Endosc 15(10): 1085–1089PubMedCrossRef

32.

Chaudhry A, Sutton C, Wood J, Stone R, McCloy R (1999) Learning rate for laparoscopic surgical skills on MIST-VR, a virtual reality simulator: quality of human-computer interface. Ann R Coll Surg Engl 81(4): 281–286PubMed

33.

Gallagher AG, Satava RM (2002) Virtual reality as a metric for the assessment of laparoscopic psychomotor skills. Learning curves and reliability measures. Surg Endosc 16(12): 1746–1752PubMedCrossRef

34.

Grantcharov TP, Bardram L, Funch-Jensen P, Rosenberg J (2003) Learning curves and impact of previous operative experience on performance on a virtual reality simulator to test laparoscopic surgical skills. Am J Surg 185(2): 146–149PubMedCrossRef

35.

Maithel SK, Villegas L, Stylopoulos N, Dawson S, Jones DB (2005) Simulated laparoscopy using a head-mounted display vs traditional video monitor: an assessment of performance and muscle fatigue. Surg Endosc 19: 406–411PubMedCrossRef

Title: Construct and face validity of MIST-VR, Endotower, and CELTS
Are we ready for skills assessment using simulators?
Authors: S. Maithel
R. Sierra
J. Korndorffer
P. Neumann
S. Dawson
M. Callery
D. Jones
D. Scott
Publication date: 01-01-2006
Publisher: Springer-Verlag
Published in: Surgical Endoscopy / Issue 1/2006
Print ISSN: 0930-2794
Electronic ISSN: 1432-2218
DOI: https://doi.org/10.1007/s00464-005-0054-4

Keynote webinar | Spotlight on sleep in brain health

Springer Medicine

Construct and face validity of MIST-VR, Endotower, and CELTS

Are we ready for skills assessment using simulators?

Abstract

Background

Methods

Results

Conclusions

Keynote webinar | Spotlight on sleep in brain health

Springer Medicine

Abstract

Background

Methods

Results

Conclusions

Please log in to get access to this content

Other articles of this Issue 1/2006

Laparoscopy for abdominal emergencies

Susceptibility of prosthetic biomaterials to infection

Telementoring versus on-site mentoring in virtual reality-based surgical training

Thirty robotic adrenalectomies

International transmission of uncompressed endoscopic surgery images via superfast broadband Internet connections

Laparoscopy-assisted truncal vagotomy with antecolic posterior gastrojejunostomy for benign gastric outlet obstruction