Top

International Journal of Emergency Medicine

Published in:

Open Access 01-12-2014 | Brief Research Report

Challenging script concordance test reference standard by evidence: do judgments by emergency medicine consultants agree with likelihood ratios?

Authors: Seyed-Foad Ahmadi, Shahin Khoshkish, Kamran Soltani-Arabshahi, Peyman Hafezi-Moghadam, Golara Zahmatkesh, Parisa Heidari, Davood Baba-Beigloo, Hamid R Baradaran, Shahram Lotfipour

Published in: International Journal of Emergency Medicine | Issue 1/2014

Abstract

Background

We aimed to compare the clinical judgments of a reference panel of emergency medicine academic physicians against evidence-based likelihood ratios (LRs) regarding the diagnostic value of selected clinical and paraclinical findings in the context of a script concordance test (SCT).

Findings

A SCT with six scenarios and five questions per scenario was developed. Subsequently, 15 emergency medicine attending physicians (reference panel) took the test and their judgments regarding the diagnostic value of those findings for given diseases were recorded. The LRs of the same findings for the same diseases were extracted from a series of published systematic reviews. Then, the reference panel judgments were compared to evidence-based LRs. To investigate the test-retest reliability, five participants took the test one month later, and the correlation of their first and second judgments were quantified using Spearman rank-order coefficient.

In 22 out of 30 (73.3%) findings, the expert judgments were significantly different from the LRs. The differences included overestimation (30%), underestimation (30%), and judging the diagnostic value in an opposite direction (13.3%). Moreover, the score of a hypothetical test-taker was calculated to be 21.73 out of 30 if his/her answers were based on evidence-based LRs.

The test showed an acceptable test-retest reliability coefficient (Spearman coefficient: 0.83).

Conclusions

Although SCT is an interesting test to evaluate clinical decision-making in emergency medicine, our results raise concerns regarding whether the judgments of an expert panel are sufficiently valid as the reference standard for this test.

Available only for authorised users

Boulouffe C, Doucet B, Muschart X, Charlin B, Vanpee D: Assessing clinical reasoning using a script concordance test with electrocardiogram in an emergency medicine clerkship rotation. Emerg Med J 2013, 31: 313–316. 10.1136/emermed-2012-201737CrossRefPubMed

Humbert AJ, Besinger B, Miech EJ: Assessing clinical reasoning skills in scenarios of uncertainty: convergent validity for a script concordance test in an emergency medicine clerkship and residency. Acad Emerg Med 2011, 18: 627–634. 10.1111/j.1553-2712.2011.01084.xCrossRefPubMed

Carriere B, Gagnon R, Charlin B, Downing S, Bordage G: Assessing clinical reasoning in pediatric emergency medicine: validity evidence for a script concordance test. Ann Emerg Med 2009, 53: 647–652. 10.1016/j.annemergmed.2008.07.024CrossRefPubMed

Park AJ, Barber MD, Bent AE, Dooley YT, Dancz C, Sutkin G, Jelovsek JE: Assessment of intraoperative judgment during gynecologic surgery using the script concordance test. Am J Obstet Gynecol 2010,203(240):240. e241-246PubMed

Mathieu S, Couderc M, Glace B, Tournadre A, Malochet-Guinamand S, Pereira B, Dubost J-J, Soubrier M: Construction and utilization of a script concordance test as an assessment tool for dcem3 (5th year) medical students in rheumatology. BMC Med Educ 2013, 13: 166. 10.1186/1472-6920-13-166PubMedCentralCrossRefPubMed

Duggan P, Charlin B: Summative assessment of 5th year medical students' clinical reasoning by script concordance test: requirements and challenges. BMC Med Educ 2012, 12: 29. 10.1186/1472-6920-12-29PubMedCentralCrossRefPubMed

Nouh T, Boutros M, Gagnon R, Reid S, Leslie K, Pace D, Pitt D, Walker R, Schiller D, MacLean A, Hameed M, Fata P, Charlin B, Meterissian SH: The script concordance test as a measure of clinical reasoning: a national validation study. Am J Surg 2012, 203: 530–534. 10.1016/j.amjsurg.2011.11.006CrossRefPubMed

Piovezan RD, Custdio O, Cendoroglo MS, Batista NA, Lubarsky S, Charlin B: Assessment of undergraduate clinical reasoning in geriatric medicine: application of a script concordance test. J Am Geriatr Soc 2012, 60: 1946–1950. 10.1111/j.1532-5415.2012.04152.xCrossRefPubMed

Bursztejn AC, Cuny JF, Adam JL, Sido L, Schmutz JL, de Korwin JD, Latarche C, Braun M, Barbaud A: Usefulness of the script concordance test in dermatology. J Eur Acad Dermatol Venereol 2011, 25: 1471–1475. 10.1111/j.1468-3083.2011.04008.xCrossRefPubMed

10.

Humbert AJ, Johnson MT, Miech E, Friedberg F, Grackin JA, Seidman PA: Assessment of clinical reasoning: a script concordance test designed for pre-clinical medical students. Med Teach 2011, 33: 472–477. 10.3109/0142159X.2010.531157CrossRefPubMed

11.

Kania RE, Verillaud B, Tran H, Gagnon R, Kazitani D, Huy PTB, Herman P, Charlin B: Online script concordance test for clinical reasoning assessment in otorhinolaryngology: the association between performance and clinical experience. Arch Otolaryngol Head Neck Surg 2011, 137: 751–755. 10.1001/archoto.2011.106CrossRefPubMed

12.

Lambert C, Gagnon R, Nguyen D, Charlin B: The script concordance test in radiation oncology: validation study of a new tool to assess clinical reasoning. Radiat Oncol 2009, 4: 7. 10.1186/1748-717X-4-7PubMedCentralCrossRefPubMed

13.

Lubarsky S, Chalk C, Kazitani D, Gagnon R, Charlin B: The script concordance test: a new tool assessing clinical judgement in neurology. Can J Neurol Sci 2009, 36: 326–331.CrossRefPubMed

14.

Meterissian SH: A novel method of assessing clinical reasoning in surgical residents. Surg Innov 2006, 13: 115–119. 10.1177/1553350606291042CrossRefPubMed

15.

Fournier JP, Demeester A, Charlin B: Script concordance tests: guidelines for construction. BMC Med Inform Decis Mak 2008, 8: 18. 10.1186/1472-6947-8-18PubMedCentralCrossRefPubMed

16.

Bowen JL: Educational strategies to promote clinical diagnostic reasoning. N Engl J Med 2006, 355: 2217–2225. 10.1056/NEJMra054782CrossRefPubMed

17.

Oxman AD, Guyatt GH: The science of reviewing research. Ann N Y Acad Sci 1993, 703: 125–133. Discussion 133-124 10.1111/j.1749-6632.1993.tb26342.xCrossRefPubMed

18.

Zehtabchi S, Kline JA: The art and science of probabilistic decision-making in emergency medicine. Acad Emerg Med 2010, 17: 521–523. 10.1111/j.1553-2712.2010.00739.xCrossRefPubMed

19.

Simel DL, Rennie D: Rational clinical examination: Evidence-based clinical diagnosis. McGraw-Hill, Chicago; 2009.

20.

van Laerhoven H, van der Zaag-Loonen HJ, Derkx BHF: A comparison of Likert scale and visual analogue scales as response options in children's questionnaires. Acta Paediatr 2004, 93: 830–835. 10.1111/j.1651-2227.2004.tb03026.xCrossRefPubMed

21.

Guyatt GH, Townsend M, Berman LB, Keller JL: A comparison of Likert and visual analogue scales for measuring change in function. J Chronic Dis 1987, 40: 1129–1133. 10.1016/0021-9681(87)90080-4CrossRefPubMed

22.

Straus SERW, Glasziou P, Haynes RB: Diagnosis and screening. In Evidence-based medicine: how to practice and teach EBM. 3rd edition. Elsevier, London; 2005:67–99.

23.

Hampton JR, Harrison MJ, Mitchell JR, Prichard JS, Seymour C: Relative contributions of history-taking, physical examination, and laboratory investigation to diagnosis and management of medical outpatients. Br Med J 1975, 2: 486–489. 10.1136/bmj.2.5969.486PubMedCentralCrossRefPubMed

24.

Petrucci AM, Nouh T, Boutros M, Gagnon R, Meterissian SH: Assessing clinical judgment using the script concordance test: the importance of using specialty-specific experts to develop the scoring key. Am J Surg 2013, 205: 137–140. 10.1016/j.amjsurg.2012.09.002CrossRefPubMed

25.

Gagnon R, Lubarsky S, Lambert C, Charlin B: Optimization of answer keys for script concordance testing: should we exclude deviant panelists, deviant responses, or neither? Adv Health Sci Educ Theory Pract 2011, 16: 601–608. 10.1007/s10459-011-9279-2CrossRefPubMed

26.

Charlin B, Gagnon R, Lubarsky S, Lambert C, Meterissian S, Chalk C, Goudreau J, van der Vleuten C: Assessment in the context of uncertainty using the script concordance test: more meaning for scores. Teach Learn Med 2010, 22: 180–186. 10.1080/10401334.2010.488197CrossRefPubMed

27.

Charlin B, Gagnon R, Pelletier J, Coletti M, Abi-Rizk G, Nasr C, Sauve E, van der Vleuten C: Assessment of clinical reasoning in the context of uncertainty: the effect of variability within the reference panel. Med Educ 2006, 40: 848–854. 10.1111/j.1365-2929.2006.02541.xCrossRefPubMed

28.

Straus SE, Glasziou P, Richardson WS, Haynes RB: Introduction. In Evidence-based medicine: How to practice and teach it. 4th edition. Churchill Livingstone, Edinburgh; 2010:1–12.

29.

Ramos K, Linscheid R, Schafer S: Real-time information-seeking behavior of residency physicians. Fam Med 2003, 35: 257–260.PubMed

30.

Graber M, Gordon R, Franklin N: Reducing diagnostic errors in medicine: what's the goal? Acad Med 2002, 77: 981–992. 10.1097/00001888-200210000-00009CrossRefPubMed

31.

Norman GR, Eva KW: Diagnostic error and clinical reasoning. Med Educ 2010, 44: 94–100. 10.1111/j.1365-2923.2009.03507.xCrossRefPubMed

32.

Nendaz M, Perrier A: Diagnostic errors and flaws in clinical reasoning: mechanisms and prevention in practice. Swiss Med Wkly 2012, 142: w13706.PubMed

33.

Lineberry M, Kreiter CD, Bordage G: Threats to validity in the use and interpretation of script concordance test scores. Med Educ 2013, 47: 1175–1183. 10.1111/medu.12283CrossRefPubMed

Title: Challenging script concordance test reference standard by evidence: do judgments by emergency medicine consultants agree with likelihood ratios?
Authors: Seyed-Foad Ahmadi
Shahin Khoshkish
Kamran Soltani-Arabshahi
Peyman Hafezi-Moghadam
Golara Zahmatkesh
Parisa Heidari
Davood Baba-Beigloo
Hamid R Baradaran
Shahram Lotfipour
Publication date: 01-12-2014
Publisher: Springer Berlin Heidelberg
Published in: International Journal of Emergency Medicine / Issue 1/2014
Print ISSN: 1865-1372
Electronic ISSN: 1865-1380
DOI: https://doi.org/10.1186/s12245-014-0034-3

At a glance: The STEP trials

Springer Medicine

Challenging script concordance test reference standard by evidence: do judgments by emergency medicine consultants agree with likelihood ratios?

Abstract

Background

Findings

Conclusions

At a glance: The STEP trials

Springer Medicine

Abstract

Background

Findings

Conclusions

Please log in to get access to this content

Other articles of this Issue 1/2014

Serum lactate as a predictor of early outcomes among trauma patients in Uganda

Fracture prevalence during an unusual period of snow and ice in the Netherlands

Prospective external validation of an accelerated (2-h) acute coronary syndrome rule-out process using a contemporary troponin assay

Lifesaving pericardiocentesis due to purulent pericarditis with growth of Gram-negative rods in an immune-competent Inuit male

Retrospective study of elderly frequent attenders presenting with chest pain at emergency department

Teaching and evaluating multitasking ability in emergency medicine residents - what is the best practice?