Reply

Our study [1] suggests that the surgeon’s estimation of risk is complementary to (not “over and above”) objective data. The surgeon’s estimation of risk was no better than other individual risk factors for predicting major complications but was able to improve both the discrimination and goodness of fit of an established multifactorial model for estimating the risk of surgery. This is probably because the surgeon was able to capture information that was not measured in the multifactorial model. An example is the impact of important but low-frequency events, such as a recent myocardial infarct or cirrhosis of the liver.

The black box analogy implies that “nothing” is known about how good the surgeon is at estimating risk, that the surgeons risk estimation is “so subjective that nothing can be known about it,” and that there are no quantifiable data available. Our study clearly demonstrates that the surgeon’s risk estimation is useful and that quantifiable data can be obtained using a visual analogue scale (VAS). Work exploring how the surgeon predicts risk and how this process can be improved is needed. We therefore suggest that the box of the surgeon’s estimation of risk is gray!

Opening the box on “how” surgeons predict risk. We agree that risk estimation is a “complex cognitive task.” The work of Sevdalis et al. in the area of judgment analysis is a helpful way to approach this problem. This gives a “weighted prediction” by measuring the weight given to a number of risk factors in different clinical scenarios. Modeling in this way produces useful feedback to clinicians, and predictions can be compared with a theoretical gold standard or even with multifactorial models such as POSSUM [2]. This may help individual surgeons improve their risk prediction, and we would encourage the use of judgment analysis to further this end. Our own continuing work on “how” risk is predicted uses VASs to compare the surgeons’ overall prediction with an estimation of risk for six main risk factors (functional status, cardiorespiratory status, severity of pathology, complexity of surgery, acuteness of surgery, “other”). We are also providing clinical feedback to individual surgeons and will assess if there is a subsequent improvement in risk prediction.

Although we fully support vigorous scientific analysis of the decision-making process, we believe there will always be some aspects of the decision-making process that will defy analysis. One problem when “measuring human behavior” is that we all do things differently. A second problem is that some aspects of the surgeon’s risk analysis are difficult to capture or quantify (e.g., how “unwell” the patient “looks” and how “well” the surgeon “feels”.) What our study highlights is that despite the tangibles and the intangibles the surgeon’s assessment of surgical risk has its own unique contribution to predicting outcome. Were this not so we would be forced to ponder the role of the surgeon at the bedside in counseling the patient about what might be expected from the surgery and whether to proceed.