Skip to main content
Top
Published in: BMC Medical Research Methodology 1/2012

Open Access 01-12-2012 | Technical advance

Incorporating published univariable associations in diagnostic and prognostic modeling

Authors: Thomas P A Debray, Hendrik Koffijberg, Difei Lu, Yvonne Vergouwe, Ewout W Steyerberg, Karel G M Moons

Published in: BMC Medical Research Methodology | Issue 1/2012

Login to get access

Abstract

Background

Diagnostic and prognostic literature is overwhelmed with studies reporting univariable predictor-outcome associations. Currently, methods to incorporate such information in the construction of a prediction model are underdeveloped and unfamiliar to many researchers.

Methods

This article aims to improve upon an adaptation method originally proposed by Greenland (1987) and Steyerberg (2000) to incorporate previously published univariable associations in the construction of a novel prediction model. The proposed method improves upon the variance estimation component by reconfiguring the adaptation process in established theory and making it more robust. Different variants of the proposed method were tested in a simulation study, where performance was measured by comparing estimated associations with their predefined values according to the Mean Squared Error and coverage of the 90% confidence intervals.

Results

Results demonstrate that performance of estimated multivariable associations considerably improves for small datasets where external evidence is included. Although the error of estimated associations decreases with increasing amount of individual participant data, it does not disappear completely, even in very large datasets.

Conclusions

The proposed method to aggregate previously published univariable associations with individual participant data in the construction of a novel prediction models outperforms established approaches and is especially worthwhile when relatively limited individual participant data are available.
Appendix
Available only for authorised users
Literature
1.
go back to reference Moons KGM, Kengne AP, Woodward M, Royston P, Vergouwe Y, Altman DG, Grobbee DE: Risk prediction models: I. Development, internal validation, and assessing the incremental value of a new (bio)marker. Heart. 2012, 683-690. [doi:10.1136/heartjnl-2011-301246] Moons KGM, Kengne AP, Woodward M, Royston P, Vergouwe Y, Altman DG, Grobbee DE: Risk prediction models: I. Development, internal validation, and assessing the incremental value of a new (bio)marker. Heart. 2012, 683-690. [doi:10.1136/heartjnl-2011-301246]
2.
go back to reference Moons KGM, Altman DG, Vergouwe Y, Royston P: Prognosis and prognostic research: application and impact of prognostic models in clinical practice. Br Med J. 2009, 338: b606-10.1136/bmj.b606.CrossRef Moons KGM, Altman DG, Vergouwe Y, Royston P: Prognosis and prognostic research: application and impact of prognostic models in clinical practice. Br Med J. 2009, 338: b606-10.1136/bmj.b606.CrossRef
3.
go back to reference Wasson JH, Sox HC, Neff RK, Goldman L: Clinical prediction rules. Applications and methodological standards. New England J Med. 1985, 313 (13): 793-799. 10.1056/NEJM198509263131306.CrossRef Wasson JH, Sox HC, Neff RK, Goldman L: Clinical prediction rules. Applications and methodological standards. New England J Med. 1985, 313 (13): 793-799. 10.1056/NEJM198509263131306.CrossRef
4.
go back to reference Reilly BM, Evans AT: Translating clinical research into clinical practice: impact of using prediction rules to make decisions. Ann Internal Med. 2006, 144 (3): 201-209.CrossRef Reilly BM, Evans AT: Translating clinical research into clinical practice: impact of using prediction rules to make decisions. Ann Internal Med. 2006, 144 (3): 201-209.CrossRef
5.
go back to reference Steyerberg EW: Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating. 2009, New York: SpringerCrossRef Steyerberg EW: Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating. 2009, New York: SpringerCrossRef
6.
go back to reference Stewart LA: Practical methodology of meta-analyses (overviews) using updated individual patient data. Stat Med. 1995, 14 (19): 2057-2079. 10.1002/sim.4780141902.CrossRefPubMed Stewart LA: Practical methodology of meta-analyses (overviews) using updated individual patient data. Stat Med. 1995, 14 (19): 2057-2079. 10.1002/sim.4780141902.CrossRefPubMed
7.
go back to reference Riley RD, Lambert PC, Abo-Zaid G: Meta-analysis of individual participant data: rationale, conduct, and reporting. Br Med J. 2010, 340: c221-10.1136/bmj.c221.CrossRef Riley RD, Lambert PC, Abo-Zaid G: Meta-analysis of individual participant data: rationale, conduct, and reporting. Br Med J. 2010, 340: c221-10.1136/bmj.c221.CrossRef
8.
go back to reference Stewart LA, Tierney JF: To IPD or not to IPD? Advantages and disadvantages of systematic reviews using individual patient data. Eval Health Professions. 2002, 25: 76-97. 10.1177/0163278702025001006.CrossRef Stewart LA, Tierney JF: To IPD or not to IPD? Advantages and disadvantages of systematic reviews using individual patient data. Eval Health Professions. 2002, 25: 76-97. 10.1177/0163278702025001006.CrossRef
9.
go back to reference Ioannidis JPA, Rosenberg PS, Goedert JJ, O’Brien TR: Commentary: meta-analysis of individual participants’ data in genetic epidemiology. A J Epidemiol. 2002, 156 (3): 204-210. 10.1093/aje/kwf031.CrossRef Ioannidis JPA, Rosenberg PS, Goedert JJ, O’Brien TR: Commentary: meta-analysis of individual participants’ data in genetic epidemiology. A J Epidemiol. 2002, 156 (3): 204-210. 10.1093/aje/kwf031.CrossRef
10.
go back to reference Hlatky MA, Greenland P, Arnett DK, Ballantyne CM, Criqui MH, Elkind MSV, Go AS, Harrell FEJ, Hong Y, Howard BV, Howard VJ, Hsue PY, Kramer CM, McConnell JP, Normand SLT, O’Donnell CJ, Smith SCJ, Wilson PWF: Criteria for evaluation of novel markers of cardiovascular risk: a scientific statement from the American Heart Association. Circulation. 2009, 119 (17): 2408-2416. 10.1161/CIRCULATIONAHA.109.192278.CrossRefPubMedPubMedCentral Hlatky MA, Greenland P, Arnett DK, Ballantyne CM, Criqui MH, Elkind MSV, Go AS, Harrell FEJ, Hong Y, Howard BV, Howard VJ, Hsue PY, Kramer CM, McConnell JP, Normand SLT, O’Donnell CJ, Smith SCJ, Wilson PWF: Criteria for evaluation of novel markers of cardiovascular risk: a scientific statement from the American Heart Association. Circulation. 2009, 119 (17): 2408-2416. 10.1161/CIRCULATIONAHA.109.192278.CrossRefPubMedPubMedCentral
11.
go back to reference Moons KGM: Criteria for scientific evaluation of novel markers: a perspective. Clin Chem. 2010, 56 (4): 537-541. 10.1373/clinchem.2009.134155.CrossRefPubMed Moons KGM: Criteria for scientific evaluation of novel markers: a perspective. Clin Chem. 2010, 56 (4): 537-541. 10.1373/clinchem.2009.134155.CrossRefPubMed
12.
go back to reference Riley RD, Sauerbrei W, Altman DG: Prognostic markers in cancer: the evolution of evidence from single studies to meta-analysis, and beyond. Br J Cancer. 2009, 100 (8): 1219-1229. 10.1038/sj.bjc.6604999.CrossRefPubMedPubMedCentral Riley RD, Sauerbrei W, Altman DG: Prognostic markers in cancer: the evolution of evidence from single studies to meta-analysis, and beyond. Br J Cancer. 2009, 100 (8): 1219-1229. 10.1038/sj.bjc.6604999.CrossRefPubMedPubMedCentral
13.
go back to reference Bennett DA: Review of analytical methods for prospective cohort studies using time to event data: single studies and implications for meta-analysis. Stat Methods Med Res. 2003, 12 (4): 297-319. 10.1191/0962280203sm319ra.CrossRefPubMed Bennett DA: Review of analytical methods for prospective cohort studies using time to event data: single studies and implications for meta-analysis. Stat Methods Med Res. 2003, 12 (4): 297-319. 10.1191/0962280203sm319ra.CrossRefPubMed
15.
go back to reference Falagas ME: The increasing body of research data in clinical medicine has led to the need for evidence synthesis studies. Preface. Infectious Dis Clinics North Am. 2009, 23 (2): xiii-10.1016/j.idc.2009.02.002.CrossRef Falagas ME: The increasing body of research data in clinical medicine has led to the need for evidence synthesis studies. Preface. Infectious Dis Clinics North Am. 2009, 23 (2): xiii-10.1016/j.idc.2009.02.002.CrossRef
16.
go back to reference Riley R, Abrams K, Lambert P, Sutton A, Altman D: Where Next for Evidence Synthesis of Prognostic Marker Studies? Improving the Quality and Reporting of Primary Studies to Facilitate Clinically Relevant Evidence-Based Results. Advances in Statistical Methods for the Health Sciences. Edited by: Auget J, Balakrishnan N, Mesbah M, Molenberghs G. 2007, 39-58. [Statistics for Industry and Technology]CrossRef Riley R, Abrams K, Lambert P, Sutton A, Altman D: Where Next for Evidence Synthesis of Prognostic Marker Studies? Improving the Quality and Reporting of Primary Studies to Facilitate Clinically Relevant Evidence-Based Results. Advances in Statistical Methods for the Health Sciences. Edited by: Auget J, Balakrishnan N, Mesbah M, Molenberghs G. 2007, 39-58. [Statistics for Industry and Technology]CrossRef
17.
go back to reference Sutton AJ, Cooper NJ, Jones DR: Evidence synthesis as the key to more coherent and efficient research. BMC Med Res Methodology. 2009, 9: 29-10.1186/1471-2288-9-29.CrossRef Sutton AJ, Cooper NJ, Jones DR: Evidence synthesis as the key to more coherent and efficient research. BMC Med Res Methodology. 2009, 9: 29-10.1186/1471-2288-9-29.CrossRef
18.
go back to reference Greenland S: Quantitative methods in the review of epidemiologic literature. Epidemiologic Rev. 1987, 9: 1-30. Greenland S: Quantitative methods in the review of epidemiologic literature. Epidemiologic Rev. 1987, 9: 1-30.
19.
go back to reference Steyerberg EW, Eijkemans MJ, Van Houwelingen JC, Lee KL, Habbema JD: Prognostic models based on literature and individual patient data in logistic regression analysis. Stat Med. 2000, 19 (2): 141-160. 10.1002/(SICI)1097-0258(20000130)19:2<141::AID-SIM334>3.0.CO;2-O.CrossRefPubMed Steyerberg EW, Eijkemans MJ, Van Houwelingen JC, Lee KL, Habbema JD: Prognostic models based on literature and individual patient data in logistic regression analysis. Stat Med. 2000, 19 (2): 141-160. 10.1002/(SICI)1097-0258(20000130)19:2<141::AID-SIM334>3.0.CO;2-O.CrossRefPubMed
20.
go back to reference Riley RD, Simmonds MC, Look MP: Evidence synthesis combining individual patient data and aggregate data: a systematic review identified current practice and possible methods. J Clin Epidemiol. 2007, 60 (5): 431-439.CrossRefPubMed Riley RD, Simmonds MC, Look MP: Evidence synthesis combining individual patient data and aggregate data: a systematic review identified current practice and possible methods. J Clin Epidemiol. 2007, 60 (5): 431-439.CrossRefPubMed
21.
go back to reference Sauerbrei W, Holländer N, Riley R, Altman D: Evidence-Based Assessment and Application of Prognostic Markers: The Long Way from Single Studies to Meta-Analysis. Commun Stat Theory Methods. 2006, 35 (7): 1333-1342. 10.1080/03610920600629666.CrossRef Sauerbrei W, Holländer N, Riley R, Altman D: Evidence-Based Assessment and Application of Prognostic Markers: The Long Way from Single Studies to Meta-Analysis. Commun Stat Theory Methods. 2006, 35 (7): 1333-1342. 10.1080/03610920600629666.CrossRef
22.
go back to reference Steyerberg EW, Kievit J, de Mol Van Otterloo JC, van Bockel JH, Eijkemans MJ, Habbema JD: Perioperative mortality of elective abdominal aortic aneurysm surgery. A clinical prediction rule based on literature and individual patient data. Arch Internal Med. 1995, 155 (18): 1998-2004. 10.1001/archinte.1995.00430180108012.CrossRef Steyerberg EW, Kievit J, de Mol Van Otterloo JC, van Bockel JH, Eijkemans MJ, Habbema JD: Perioperative mortality of elective abdominal aortic aneurysm surgery. A clinical prediction rule based on literature and individual patient data. Arch Internal Med. 1995, 155 (18): 1998-2004. 10.1001/archinte.1995.00430180108012.CrossRef
23.
go back to reference Greenland S, Mickey RM: Closed Form and Dually Consistent Methods for Inference on Strict Collapsibility in 2 x 2 x K and 2 x J x K Tables. J R Stat Soc Ser C (Appl Stat). 1988, 37 (3): 335-343. Greenland S, Mickey RM: Closed Form and Dually Consistent Methods for Inference on Strict Collapsibility in 2 x 2 x K and 2 x J x K Tables. J R Stat Soc Ser C (Appl Stat). 1988, 37 (3): 335-343.
24.
go back to reference Robinson LD, Jewell NP: Some Surprising Results about Covariate Adjustment in Logistic Regression Models. Int Stat Rev / Revue Internationale de Statistique. 1991, 59 (2): 227-240. 10.2307/1403444. Robinson LD, Jewell NP: Some Surprising Results about Covariate Adjustment in Logistic Regression Models. Int Stat Rev / Revue Internationale de Statistique. 1991, 59 (2): 227-240. 10.2307/1403444.
25.
go back to reference Davison A, Hinkley D: Bootstrap Methods App. No. 1 in Cambridge Series in Statistical and Probabilistic Mathematics. 1997, Cambridge: CambridgeUniversity Press, Davison A, Hinkley D: Bootstrap Methods App. No. 1 in Cambridge Series in Statistical and Probabilistic Mathematics. 1997, Cambridge: CambridgeUniversity Press,
26.
go back to reference Albert A, Anderson J: On the existence of maximum likelihood estimates in logistic regression models. Biometrika. 1984, 71: 1-10. 10.1093/biomet/71.1.1.CrossRef Albert A, Anderson J: On the existence of maximum likelihood estimates in logistic regression models. Biometrika. 1984, 71: 1-10. 10.1093/biomet/71.1.1.CrossRef
27.
go back to reference Lesaffre E, Albert A: Partial separation in Logistic Discrimination. J R Stat Soc Ser B (Methodological). 1989, 51: 109-116. Lesaffre E, Albert A: Partial separation in Logistic Discrimination. J R Stat Soc Ser B (Methodological). 1989, 51: 109-116.
28.
go back to reference Peduzzi P, Concato J, Kemper E, Holford TR, Feinstein AR: A simulation study of the number of events per variable in logistic regression analysis. J Clin Epidemioly. 1996, 49 (12): 1373-1379. 10.1016/S0895-4356(96)00236-3.CrossRef Peduzzi P, Concato J, Kemper E, Holford TR, Feinstein AR: A simulation study of the number of events per variable in logistic regression analysis. J Clin Epidemioly. 1996, 49 (12): 1373-1379. 10.1016/S0895-4356(96)00236-3.CrossRef
29.
go back to reference Gelman A, Jakulin A, Pittau MG, Su YS: A weakly informative default prior distribution for logistic and other regression models. Ann Appl Stat. 2008, 2 (4): 1360-1383. 10.1214/08-AOAS191.CrossRef Gelman A, Jakulin A, Pittau MG, Su YS: A weakly informative default prior distribution for logistic and other regression models. Ann Appl Stat. 2008, 2 (4): 1360-1383. 10.1214/08-AOAS191.CrossRef
30.
go back to reference Normand SL: Meta-analysis: formulating, evaluating, combining, and reporting. Stat Med. 1999, 18 (3): 321-359. 10.1002/(SICI)1097-0258(19990215)18:3<321::AID-SIM28>3.0.CO;2-P.CrossRefPubMed Normand SL: Meta-analysis: formulating, evaluating, combining, and reporting. Stat Med. 1999, 18 (3): 321-359. 10.1002/(SICI)1097-0258(19990215)18:3<321::AID-SIM28>3.0.CO;2-P.CrossRefPubMed
31.
go back to reference Hedges LV, Vevea JL: Fixed- and Random-Effects Models in Meta-Analysis. Psychological Methods. 1998, 3 (4): 486-504.CrossRef Hedges LV, Vevea JL: Fixed- and Random-Effects Models in Meta-Analysis. Psychological Methods. 1998, 3 (4): 486-504.CrossRef
32.
go back to reference Burton A, Altman DG, Royston P, Holder RL: The design of simulation studies in medical statistics. Stat Med. 2006, 25 (24): 4279-4292. 10.1002/sim.2673.CrossRefPubMed Burton A, Altman DG, Royston P, Holder RL: The design of simulation studies in medical statistics. Stat Med. 2006, 25 (24): 4279-4292. 10.1002/sim.2673.CrossRefPubMed
33.
go back to reference Mason CH, Perreault WDJ: Collinearity, Power, and Interpretation of Multiple Regression Analysis. J Marketing Res. 1991, 28: 268-280. 10.2307/3172863.CrossRef Mason CH, Perreault WDJ: Collinearity, Power, and Interpretation of Multiple Regression Analysis. J Marketing Res. 1991, 28: 268-280. 10.2307/3172863.CrossRef
34.
go back to reference Greenland S: Invited commentary: a critical look at some popular meta-analytic methods. Am J Epidemiol. 1994, 140 (3): 290-296.PubMed Greenland S: Invited commentary: a critical look at some popular meta-analytic methods. Am J Epidemiol. 1994, 140 (3): 290-296.PubMed
35.
go back to reference Higgins JPT, Thompson SG, Deeks JJ, Altman DG: Measuring inconsistency in meta-analyses. Br Med J. 2003, 327 (7414): 557-560. 10.1136/bmj.327.7414.557.CrossRef Higgins JPT, Thompson SG, Deeks JJ, Altman DG: Measuring inconsistency in meta-analyses. Br Med J. 2003, 327 (7414): 557-560. 10.1136/bmj.327.7414.557.CrossRef
36.
go back to reference Debray TPA, Koffijberg H, Vergouwe Y, Moons KGM, Steyerberg EW: Aggregating published prediction models with individual participant data: a comparison of different approaches. Stat Med. 2012, 31 (23): Accepted for publication [doi:10.1002/sim.5412] Debray TPA, Koffijberg H, Vergouwe Y, Moons KGM, Steyerberg EW: Aggregating published prediction models with individual participant data: a comparison of different approaches. Stat Med. 2012, 31 (23): Accepted for publication [doi:10.1002/sim.5412]
Metadata
Title
Incorporating published univariable associations in diagnostic and prognostic modeling
Authors
Thomas P A Debray
Hendrik Koffijberg
Difei Lu
Yvonne Vergouwe
Ewout W Steyerberg
Karel G M Moons
Publication date
01-12-2012
Publisher
BioMed Central
Published in
BMC Medical Research Methodology / Issue 1/2012
Electronic ISSN: 1471-2288
DOI
https://doi.org/10.1186/1471-2288-12-121

Other articles of this Issue 1/2012

BMC Medical Research Methodology 1/2012 Go to the issue