Skip to main content
Key Specialties
Cardiology Diabetology Endocrinology Gastroenterology Geriatrics and Gerontology Gynecology and Obstetrics Hematology Infectious Disease Internal Medicine Nephrology Neurology Oncology Pediatrics Respiratory Medicine Rheumatology Surgery
Congress Coverage
2024 ACC Congress 2023 ESMO Congress 2023 EASD Congress 2023 ERS Congress 2023 ESC Congress
Clinical Collections
Advances in Alzheimer’s

At a glance: The STEP trials

A round-up of the STEP phase 3 clinical trials evaluating semaglutide for weight loss in people with overweight or obesity.
ADVANCED SEARCH
Log in
Top
BMC Medical Research Methodology
Published in: BMC Medical Research Methodology 1/2019

Open Access 01-12-2019 | Breast Cancer | Research article

Facilitating validation of prediction models: a comparison of manual and semi-automated validation using registry-based data of breast cancer patients in the Netherlands

Authors: Cornelia D. van Steenbeek, Marissa C. van Maaren, Sabine Siesling, Annemieke Witteveen, Xander A. A. M. Verbeek, Hendrik Koffijberg

Published in: BMC Medical Research Methodology | Issue 1/2019

Login to get access

Abstract

Background

Clinical prediction models are not routinely validated. To facilitate validation procedures, the online Evidencio platform (https://​www.​evidencio.​com) has developed a tool partly automating this process. This study aims to determine whether semi-automated validation can reliably substitute manual validation.

Methods

Four different models used in breast cancer care were selected: CancerMath, INFLUENCE, Predicted Probability of Axillary Metastasis, and PREDICT v.2.0. Data were obtained from the Netherlands Cancer Registry according to the inclusion criteria of the original development population. Calibration (intercepts and slopes) and discrimination (area under the curve (AUC)) were compared between semi-automated and manual validation.

Results

Differences between intercepts and slopes of all models using semi-automated validation ranged from 0 to 0.03 from manual validation, which was not clinically relevant. AUCs were identical for both validation methods.

Conclusions

This easy to use semi-automated validation option is a good substitute for manual validation and might increase the number of validations of prediction models used in clinical practice. In addition, the validation tool was considered to be user-friendly and to save a lot of time compared to manual validation. Semi-automated validation will contribute to more accurate outcome predictions and treatment recommendations in the target population.
Appendix
Available only for authorised users
Literature
1.
Bieber C, Gschwendtner K, Muller N, Eich W. Shared decision making (SDM) - patient and physician as a team. Psychother Psychosom Med Psychol. 2016;66(5):195–207.CrossRef
2.
Engelhardt EG, Garvelink MM, de Haes JH, van der Hoeven JJ, Smets EM, Pieterse AH, et al. Predicting and communicating the risk of recurrence and death in women with early-stage breast cancer: a systematic review of risk prediction models. J Clin Oncol. 2014;32(3):238–50.CrossRef
3.
Kinnier CV, Asare EA, Mohanty S, Paruch JL, Rajaram R, Bilimoria KY. Risk prediction tools in surgical oncology. J Surg Oncol. 2014;110(5):500–8.CrossRef
4.
Hajage D, de Rycke Y, Bollet M, Savignoni A, Caly M, Pierga JY, et al. External validation of adjuvant! Online breast cancer prognosis tool. Prioritising recommendations for improvement. PLoS One. 2011;6(11):e27446.CrossRef
5.
Qiu SQ, Zeng HC, Zhang F, Chen C, Huang WH, Pleijhuis RG, et al. A nomogram to predict the probability of axillary lymph node metastasis in early breast cancer patients with positive axillary ultrasound. Sci Rep. 2016;6:21196.CrossRef
6.
Xie X, Tan W, Chen B, Huang X, Peng C, Yan S, et al. Preoperative prediction nomogram based on primary tumor miRNAs signature and clinical-related features for axillary lymph node metastasis in early-stage invasive breast cancer. Int J Cancer. 2017. https://​doi.​org/​10.​1002/​ijc.​31208.CrossRef
7.
Jiang Y, Xu H, Zhang H, Ou X, Xu Z, Ai L, et al. Nomogram for prediction of level 2 axillary lymph node metastasis in proven level 1 node-positive breast cancer patients. Oncotarget. 2017;8(42):72389–99.CrossRef
8.
Chen K, Liu J, Li S, Jacobs L. Development of nomograms to predict axillary lymph node status in breast cancer patients. BMC Cancer. 2017;17(1):561.CrossRef
9.
Barco I, Garcia Font M, Garcia-Fernandez A, Gimenez N, Fraile M, Lain JM, et al. A logistic regression model predicting high axillary tumour burden in early breast cancer patients. Clin Transl Oncol. 2017;19(11):1393–9.CrossRef
10.
Zhang J, Li X, Huang R, Feng WL, Kong YN, Xu F, et al. A nomogram to predict the probability of axillary lymph node metastasis in female patients with breast cancer in China: a nationwide, multicenter, 10-year epidemiological study. Oncotarget. 2017;8(21):35311–25.PubMed
11.
van den Hoven I, van Klaveren D, Voogd AC, Vergouwe Y, Tjan-Heijnen V, Roumen RM. A Dutch prediction tool to assess the risk of additional axillary non-sentinel lymph node involvement in sentinel node-positive breast Cancer patients. Clin breast cancer. 2016;16(2):123–30.CrossRef
12.
Witteveen A, Vliegen IM, Siesling S, MJ IJ. A validated prediction model and nomogram for risk of recurrence in early breast Cancer patients. Value Health. 2014;17(7):A619–20.CrossRef
13.
Wadasadawala T, Kannan S, Gudi S, Rishi A, Budrukkar A, Parmar V, et al. Predicting loco-regional recurrence risk in T1, T2 breast cancer with 1-3 positive axillary nodes postmastectomy: development of a predictive nomogram. Indian J Cancer. 2017;54(1):352–7.CrossRef
14.
Cheng SH, Horng CF, Clarke JL, Tsou MH, Tsai SY, Chen CM, et al. Prognostic index score and clinical prediction model of local regional recurrence after mastectomy in breast cancer patients. Int J Rad Oncol Biol Phys. 2006;64(5):1401–9.CrossRef
15.
van Nes JG, Putter H, van Hezewijk M, Hille ET, Bartelink H, Collette L, et al. Tailored follow-up for early breast cancer patients: a prognostic index that predicts locoregional recurrence. Eur J Surg Oncol. 2010;36(7):617–24.CrossRef
16.
Matsuda N, Hayashi N, Ohde S, Yagata H, Kajiura Y, Yoshida A, et al. A nomogram for predicting locoregional recurrence in primary breast cancer patients who received breast-conserving surgery after neoadjuvant chemotherapy. J Surg Oncol. 2014;109(8):764–9.CrossRef
17.
Candido Dos Reis FJ, Wishart GC, Dicks EM, Greenberg D, Rashbass J, Schmidt MK, et al. An updated PREDICT breast cancer prognostication and treatment benefit prediction model with independent validation. Breast Cancer Res. 2017;19(1):58.CrossRef
18.
Chen LL, Nolan ME, Silverstein MJ, Mihm MC Jr, Sober AJ, Tanabe KK, et al. The impact of primary tumor size, lymph node status, and other prognostic factors on the risk of cancer death. Cancer. 2009;115(21):5071–83.CrossRef
19.
Haybittle JL, Blamey RW, Elston CW, Johnson J, Doyle PJ, Campbell FC, et al. A prognostic index in primary breast cancer. Br J Cancer. 1982;45(3):361–6.CrossRef
20.
Ravdin PM, Siminoff LA, Davis GJ, Mercer MB, Hewlett J, Gerson N, et al. Computer program to assist in making decisions about adjuvant therapy for women with early breast cancer. J Clin Oncol. 2001;19(4):980–91.CrossRef
21.
Pleijhuis RG, Kwast AB, Jansen L, de Vries J, Lanting R, Bart J, et al. A validated web-based nomogram for predicting positive surgical margins following breast-conserving surgery as a preoperative tool for clinical decision-making. Breast. 2013;22(5):773–9.CrossRef
22.
van Giessen A, Peters J, Wilcher B, Hyde C, Moons C, de Wit A, et al. Systematic review of health economic impact evaluations of risk prediction models: stop developing, start evaluating. Value Health. 2017;20(4):718–26.CrossRef
23.
Balch CM, Jacobs LK. Mastectomies on the rise for breast cancer: “the tide is changing”. Ann Surg Oncol. 2009;16(10):2669–72.CrossRef
24.
Michaelson JS, Chen LL, Bush D, Fong A, Smith B, Younger J. Improved web-based calculators for predicting breast carcinoma outcomes. Breast Cancer Res Treat. 2011;128(3):827–35.CrossRef
25.
Qiu SQ, Aarnink M, van Maaren MC, Dorrius MD, Bhattacharya A, Veltman J, Klazen CAH, Korte JH, Estourgie SH, Ott P, Kelder W, Zeng HC, Koffijberg H, Zhang GJ, van Dam GM, Siesling S. Validation and update of a lymph node metastasis prediction model for breast cancer. Eur J Surg Oncol. 2018. https://​doi.​org/​10.​1016/​j.​ejso.​2017.​12.​008. Epub ahead of print.CrossRef
26.
van Maaren MC, van Steenbeek CD, Pharoah PDP, Witteveen A, Sonke GS, Strobbe LJA, et al. Validation of the online prediction tool PREDICT v. 2.0 in the Dutch breast cancer population. Eur J Cancer. 2017;86:364–72.CrossRef
27.
Royston P, Altman DG. External validation of a cox prognostic model: principles and methods. BMC Med Res Methodol. 2013;13:33.CrossRef
28.
Hajian-Tilaki K. Receiver operating characteristic (ROC) curve analysis for medical diagnostic test evaluation. Caspian J Internal Med. 2013;4(2):627–35.
29.
Olivotto IA, Bajdik CD, Ravdin PM, Speers CH, Coldman AJ, Norris BD, et al. Population-based validation of the prognostic model ADJUVANT! For early breast cancer. J Clin Oncol. 2005;23(12):2716–25.CrossRef
30.
Mook S, Schmidt MK, Rutgers EJ, van de Velde AO, Visser O, Rutgers SM, et al. Calibration and discriminatory accuracy of prognosis calculation for breast cancer with the online adjuvant! Program: a hospital-based retrospective cohort study. Lancet Oncol. 2009;10(11):1070–6.CrossRef
31.
de Glas NA, Bastiaannet E, Engels CC, de Craen AJ, Putter H, van de Velde CJ, et al. Validity of the online PREDICT tool in older patients with breast cancer: a population-based study. Br J Cancer. 2016;114(4):395–400.CrossRef
32.
Collins GS, de Groot JA, Dutton S, Omar O, Shanyinde M, Tajar A, et al. External validation of multivariable prediction models: a systematic review of methodological conduct and reporting. BMC Med Res Methodol. 2014;14:40.CrossRef
33.
Wilkinson MD, Dumontier M, Aalbersberg IJ, Appleton G, Axton M, Baak A, et al. The FAIR guiding principles for scientific data management and stewardship. Sci Data. 2016;15(3):160018.CrossRef
34.
Steyerberg EW, Vickers AJ, Cook NR, Gerds T, Gonen M, Obuchowski N, et al. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology. 2010;21(1):128–38.CrossRef
35.
Steyerberg EW, Vergouwe Y. Towards better clinical prediction models: seven steps for development and an ABCD for validation. Eur Heart J. 2014;35(29):1925–31.CrossRef
Metadata
Title
Facilitating validation of prediction models: a comparison of manual and semi-automated validation using registry-based data of breast cancer patients in the Netherlands
Authors
Cornelia D. van Steenbeek
Marissa C. van Maaren
Sabine Siesling
Annemieke Witteveen
Xander A. A. M. Verbeek
Hendrik Koffijberg
Publication date
01-12-2019
Publisher
BioMed Central
Published in
BMC Medical Research Methodology / Issue 1/2019
Electronic ISSN: 1471-2288
DOI
https://doi.org/10.1186/s12874-019-0761-5

Other articles of this Issue 1/2019

BMC Medical Research Methodology 1/2019 Go to the issue

Research article

Acceptability and barriers to implementation of N-of-1 tests in Ethiopia - a qualitative study

Research article

The power of the group: comparison of interviews and group concept mapping for identifying patient-important outcomes of care

Research article

Obstacles to recruitment in paediatric studies focusing on mental health in a physical health context: the experiences of clinical gatekeepers in an observational cohort study

Research article

Sensitivity and specificity of an algorithm based on medico-administrative data to identify hospitalized patients with major bleeding presenting to an emergency department

Technical advance

Simulation and minimization: technical advances for factorial experiments designed to optimize clinical interventions

Research article

The Danish health and morbidity surveys: study design and participant characteristics