Calibration strategies to validate predictive models: is new always better?

Excerpt

Calibration along with discrimination is an important measure of accuracy to validate predictive logistic regression models. Most predictive models in intensive care such as Simplified Acute Physiology Score (SAPS) II [1] and SAPS 3 [2, 3] consider the binary outcome whether a patient will be alive or dead at hospital discharge. Discrimination measures how well the model can distinguish between patients who die and those who survive. Discrimination is usually assessed by the area under the receiver operating characteristic curve (AU-ROC) [4]. This statistic evaluates each pair of observations that have different outcomes and calculates the proportion of times when the patient who died had a higher predicted mortality than did the survivor. The AU-ROC ranges from 0.50 (no discrimination: complete binary random of 50 % similar to flipping a coin) to 1.00 (100 % correct discrimination of the model) [4]. …