How to Assess Prognostic Models for Survival Data: A Case Study in Oncology

 

 Buy Article Permissions and Reprints

Summary

Objectives: A lack of generally applicable tools for the assessment of predictions for survival data has to be recognized. Prediction error curves based on the Brier score that have been suggested as a sensible approach are illustrated by means of a case study.

Methods: The concept of predictions made in terms of conditional survival probabilities given the patient’s covariates is introduced. Such predictions are derived from various statistical models for survival data including artificial neural networks. The idea of how the prediction error of a prognostic classification scheme can be followed over time is illustrated with the data of two studies on the prognosis of node positive breast cancer patients, one of them serving as an independent test data set.

Results and Conclusions: The Brier score as a function of time is shown to be a valuable tool for assessing the predictive performance of prognostic classification schemes for survival data incorporating censored observations. Comparison with the prediction based on the pooled Kaplan Meier estimator yields a benchmark value for any classification scheme incorporating patient’s covariate measurements. The problem of an overoptimistic assessment of prediction error caused by data-driven modelling as it is, for example, done with artificial neural nets can be circumvented by an assessment in an independent test data set.

• References

• 1 Abu-Hanna A, Lucas PYF. Editorial. Prognostic models in medicine: AI and statistical approaches. Methods Inf Med 2001; 40: 1-5.
  Article in Thieme ConnectPubMedGoogle Scholar
• 2 Christakis NA, Lamont EB. Extent and determinants of error in doctors’ prognoses in terminally ill patients: prospective cohort study. BMJ 2000; 320: 469-73.
  CrossrefPubMedGoogle Scholar
• 3 Muers M, Shevlin P, Brown J. et al. Prognosis in lung cancer: Physicians’ opinions compared with outcome and a predictive model. Thorax 1996; 51: 894-902.
  CrossrefPubMedGoogle Scholar
• 4 Viganó A, Dorgan M, Bruera E, Suarez-Alma-zor ME. The relative accuracy of the clinical estimation of the duration of life for patients with end of life cancer. Cancer 1999; 86: 170-6.
  CrossrefPubMedGoogle Scholar
• 5 Brier GW. Verification of forecasts expressed in terms of probability. Monthly Weather Rev 1950; 78: 1-3.
  CrossrefPubMedGoogle Scholar
• 6 Mackillop WJ, Quirt CF. Measuring the accuracy of prognostic judgements in oncology. J Clin Epidemiol 1997; 50: 21-9.
  CrossrefPubMedGoogle Scholar
• 7 Begg CB, Cramer LD, Venkatraman ES, Rosai J. Comparing tumour staging and grading systems: a case study and a review of the issues, using thyoma as a model. Stat Med. 2000 19: 1997 2014.
  PubMedGoogle Scholar
• 8 Graf E, Schmoor C, Sauerbrei W, Schumacher M. Assessment and comparison of prognostic classification schemes for survival data. Stat Med 1999; 18: 2529-45.
  CrossrefPubMedGoogle Scholar
• 9 Graf E, Schumacher M. An investigation on measures of explained variation in survival analysis. The Statistician 1995; 44: 497-507.
  CrossrefPubMedGoogle Scholar
• 10 Sauerbrei W, Royston P, Bojar H, Schmoor C, Schumacher M. Modelling the effects of standard prognostic factors in node-positive breast cancer. German breast cancer study group (GBSG). Brit J Cancer 1999; 79: 1752-60.
  CrossrefPubMedGoogle Scholar
• 11 Schumacher M, Holländer N, Schwarzer G, Sauerbrei W. Prognostic Factor Studies. In: Crowley J. ed. Handbook of Statistics in Clinical Oncology. New York: Marcel Dekker; 2001: 331-78.
  Google Scholar
• 12 Schmoor C, Olschewski M, Schumacher M. Randomized and non-randomized patients in clinical trials: experiences with comprehensive cohort studies. Stat Med 1996; 15: 263-71.
  CrossrefPubMedGoogle Scholar
• 13 Pfisterer J, Kommoss F, Sauerbrei W, Menzel D, Kiechle M, Giese E, Hilgarth M, Pfleiderer A. DNA flow cytometry in node positive breast cancer: prognostic value and correlation to morphological and clinical factors. Anal Quant Cytol Histol 1995; 17: 406-12.
  PubMedGoogle Scholar
• 14 Bloom HJG, Richardson WW. Histological grading and breast cancer. Brit J Cancer 1957; 2: 359-77.
  PubMedGoogle Scholar
• 15 Cox DR. Regression models and life tables (with discussion). J R Stat Soc (Series B) 1972; 74: 187-200.
  PubMedGoogle Scholar
• 16 Breiman L, Friedman JH, Olsen RA, Stone CJ. Classification and Regression Trees. Monterey: Wadsworth; 1984
  Google Scholar
• 17 Haybittle JL, Blamey RW, Elston CW, Johnson J, Doyle PJ, Campbell FC, Nicholøson RI, Griffiths K. A prognostic index in primary breast cancer. Brit J Cancer 1982; 45: 361-6.
  CrossrefPubMedGoogle Scholar
• 18 Galea MH, Blamey RW, Elston CE, Ellis IO. The Nottingham Prognostic Index in primary breast cancer. Breast Cancer Res Treat 1992; 22: 207-19.
  CrossrefPubMedGoogle Scholar
• 19 Faraggi D, Simon R. A neural network model for survival data. Stat Med 1995; 14: 73-82.
  CrossrefPubMedGoogle Scholar
• 20 Kaplan EL, Meier P. Nonparametric estimation from incomplete observations. JASA 1958; 53: 457-81.
  CrossrefPubMedGoogle Scholar
• 21 Wang Y, Lim LL-Y, Levi C, Heller RF, Fischer J. A prognostic index for 30-day mortality after stroke. J Clin Epidemiol 2001; 54: 766-73.
  CrossrefPubMedGoogle Scholar
• 22 Loprinzi CL, Thomé SD. Understanding the utility of adjuvant systemic therapy for primary breast cancer. J Clin Oncol 2001; 19: 9729.
  PubMedGoogle Scholar
• 23 Ravdin PM, Siminoff LA, Greg JD, Mercer MB, Hewlett J, Gerson N, Parker HL. Computer program to assist in making decisions about adjuvant therapy for women with early breast cancer. J Clin Oncol 2001; 19: 980-99.
  CrossrefPubMedGoogle Scholar
• 24 Liou TG, Adler FR, Fitzsimmons SC, Cahill BC, Hibbs JR, Marshall BC. Predictive 5-year survivorship model of cystic fibrosis. Am J Epidemiol 2001; 156: 345-52.
  PubMedGoogle Scholar
• 25 Dawid AP. Probability forecasting. In: Enyclopedia of Statistical Sciences,. Volume 7, pp. 210-218. New York: John Wiley; 1986
  Google Scholar
• 26 Dybowski R, Gant V. Artificial neural networks in pathology and medical laboratories. Lancet 1995; 346: 1203-7.
  CrossrefPubMedGoogle Scholar
• 27 Ikeda M, Itoh S, Ishigaki T, Yamauchi K. Application of resampling to the statistical analysis of the Brier score. Methods Inf Med 2001; 40: 259-64.
  Article in Thieme ConnectPubMedGoogle Scholar
• 28 Henderson R, Jones M, Stare J. Accuracy of point predictions in survival analysis. Stat Med 2001; 20: 3083-96.
  CrossrefPubMedGoogle Scholar
• 29 Zengh B, Agresti A. Summarizing the predictive power of a generalized linear model. Stat Med 2000; 19: 1771-81.
  CrossrefPubMedGoogle Scholar
• 30 Mittlböck M, Heinzl H. A note on R² measures for Poisson and logistic regression models when both models are applicable. J Clin Epidemiol 2001; 54: 99-103.
  CrossrefPubMedGoogle Scholar
• 31 Schwarzer G, Vach W, Schumacher M. On the misuses of artificial neural networks for prognostic and diagnostic classification in oncology. Stat Med 2000; 19: 541-61.
  CrossrefPubMedGoogle Scholar