The script concordance test: an adequate tool to assess clinical reasoning?

Excerpt

In this issue of Perspectives on Medical Education, Lubarsky et al. investigate the possibility that test takers can obtain artificially inflated test scores on the Script Concordance Test (SCT) by strategically selecting options, rather than by using their knowledge to arrive at the correct answer [1]. The Script Concordance Test is a test to assess clinical reasoning, consisting of items that require students to judge the change in likelihood of a diagnostic hypothesis (provided to them) given a new piece of information (also provided to them) on a five-point Likert scale, ranging from −2 ([almost] rules out the hypothesis) through 0 (does not change its likelihood) to +2 (hypothesis now [almost] certain) [2]. For each item, the test takers’ response is compared with the composite judgment of a panel of experts, and credit points are assigned in accordance with the proportion of experts selecting the same response when test norms are constructed. If the testee selects the same response as the majority of experts, he/she obtains the maximum number of credit points. If the testee sides with a minority of experts, points are detracted. Otherwise, no points are awarded at all. By running a simulation, Lubarsky and colleagues [1] demonstrate that test takers who consistently choose the ‘0’ option (no change in likelihood) can artificially inflate their scores from −2.1 SD below the average of an actual sample of respondents—the score obtained by random guessing,—to −0.7 SD below this average. It may be argued that this form of ‘gaming the test’ by consistently selecting the ‘0’ option is contrived. In fact, it is an inevitable consequence of using a computer simulation, which only enables investigation of formal, knowledge-free gaming strategies. However, it makes sense to assume a tendency among human test takers to select ‘0’ on items for which they lack the knowledge, or do not feel confident, because this appears to be the least risky option, which will fairly often yield some points. Participants with relatively little knowledge can exploit this ‘strategy’ by answering only the few questions they are confident they know, selecting the ‘0’ on all other questions. This way, they might succeed in elevating their score to well above the −0.7 SD level. Thus, the authors’ recommendation to restrict the number of SCT items for which ‘0’ is the correct answer seems to be defensible, even if no human test taker will ever consistently choose the ‘0’ on all SCT items. …