Skip to main content
Top
Published in: Systematic Reviews 1/2019

Open Access 01-12-2019 | Methodology

An algorithm for the classification of study designs to assess diagnostic, prognostic and predictive test accuracy in systematic reviews

Authors: Tim Mathes, Dawid Pieper

Published in: Systematic Reviews | Issue 1/2019

Login to get access

Abstract

Results of medical tests are the main source to inform clinical decision making. The main information to assess the usefulness of medical tests for correct discrimination of patients are accuracy measures. For the estimation of test accuracy measures, many different study designs can be used. The study design is related to the clinical question to be answered (diagnosis, prognosis, prediction), determines the accuracy measures that can be calculated and it might have an influence on risk of bias. Therefore, a clear and consistent distinction of the different study designs in systematic reviews on test accuracy studies is very important. In this paper, we propose an algorithm for the classification of study designs of test accuracy, that compare the results of an index test (the test to be evaluated) with the results of a reference test (the test whose results are considered as correct/the gold standard) studies in systematic reviews.
Literature
1.
go back to reference Schünemann HJ, Mustafa R, Brozek J, Santesso N, Alonso-Coello P, Guyatt G, Scholten R, Langendam M, Leeflang MM, Akl EA, et al. GRADE Guidelines: 16. GRADE evidence to decision frameworks for tests in clinical practice and public health. J Clin Epidemiol. 2016;76(Supplement C):89–98.CrossRef Schünemann HJ, Mustafa R, Brozek J, Santesso N, Alonso-Coello P, Guyatt G, Scholten R, Langendam M, Leeflang MM, Akl EA, et al. GRADE Guidelines: 16. GRADE evidence to decision frameworks for tests in clinical practice and public health. J Clin Epidemiol. 2016;76(Supplement C):89–98.CrossRef
2.
go back to reference Bae JH, Park SH, Ye BD, Kim SO, Cho YK, Youn EJ, Lee HS, Hwang SW, Yang DH, Kim KJ, et al. Development and validation of a novel prediction model for differential diagnosis between Crohn's disease and intestinal tuberculosis. Inflamm Bowel Dis. 2017;23(9):1614–23.CrossRef Bae JH, Park SH, Ye BD, Kim SO, Cho YK, Youn EJ, Lee HS, Hwang SW, Yang DH, Kim KJ, et al. Development and validation of a novel prediction model for differential diagnosis between Crohn's disease and intestinal tuberculosis. Inflamm Bowel Dis. 2017;23(9):1614–23.CrossRef
3.
go back to reference Hilvering B, Vijverberg SJH, Jansen J, Houben L, Schweizer RC, Go S, Xue L, Pavord ID, Lammers JJ, Koenderman L. Diagnosing eosinophilic asthma using a multivariate prediction model based on blood granulocyte responsiveness. Allergy. 2017;72(8):1202–11.CrossRef Hilvering B, Vijverberg SJH, Jansen J, Houben L, Schweizer RC, Go S, Xue L, Pavord ID, Lammers JJ, Koenderman L. Diagnosing eosinophilic asthma using a multivariate prediction model based on blood granulocyte responsiveness. Allergy. 2017;72(8):1202–11.CrossRef
4.
go back to reference Giannini V, Mazzetti S, Marmo A, Montemurro F, Regge D, Martincich L. A computer-aided diagnosis (CAD) scheme for pretreatment prediction of pathological response to neoadjuvant therapy using dynamic contrast-enhanced MRI texture features. Br J Radiol. 2017;90(1077):20170269.CrossRef Giannini V, Mazzetti S, Marmo A, Montemurro F, Regge D, Martincich L. A computer-aided diagnosis (CAD) scheme for pretreatment prediction of pathological response to neoadjuvant therapy using dynamic contrast-enhanced MRI texture features. Br J Radiol. 2017;90(1077):20170269.CrossRef
5.
go back to reference Dubey D, Singh J, Britton JW, Pittock SJ, Flanagan EP, Lennon VA, Tillema JM, Wirrell E, Shin C, So E, et al. Predictive models in the diagnosis and treatment of autoimmune epilepsy. Epilepsia. 2017;58(7):1181–9.CrossRef Dubey D, Singh J, Britton JW, Pittock SJ, Flanagan EP, Lennon VA, Tillema JM, Wirrell E, Shin C, So E, et al. Predictive models in the diagnosis and treatment of autoimmune epilepsy. Epilepsia. 2017;58(7):1181–9.CrossRef
6.
go back to reference Chitty LS, Finning K, Wade A, Soothill P, Martin B, Oxenford K, Daniels G, Massey E. Diagnostic accuracy of routine antenatal determination of fetal RHD status across gestation: population based cohort study. BMJ (Clin Res Ed). 2014;349:g5243. Chitty LS, Finning K, Wade A, Soothill P, Martin B, Oxenford K, Daniels G, Massey E. Diagnostic accuracy of routine antenatal determination of fetal RHD status across gestation: population based cohort study. BMJ (Clin Res Ed). 2014;349:g5243.
7.
go back to reference Nwachuku EL, Balzer JR, Yabes JG, Habeych ME, Crammond DJ, Thirumala PD. Diagnostic value of somatosensory evoked potential changes during carotid endarterectomy: a systematic review and meta-analysis. JAMA Neurol. 2015;72(1):73–80.CrossRef Nwachuku EL, Balzer JR, Yabes JG, Habeych ME, Crammond DJ, Thirumala PD. Diagnostic value of somatosensory evoked potential changes during carotid endarterectomy: a systematic review and meta-analysis. JAMA Neurol. 2015;72(1):73–80.CrossRef
8.
go back to reference van den Bosch WB, Mangnus L, Reijnierse M, Huizinga TW, van der Helm-van Mil AH. The diagnostic accuracy of the squeeze test to identify arthritis: a cross-sectional cohort study. Ann Rheum Dis. 2015;74(10):1886–9.CrossRef van den Bosch WB, Mangnus L, Reijnierse M, Huizinga TW, van der Helm-van Mil AH. The diagnostic accuracy of the squeeze test to identify arthritis: a cross-sectional cohort study. Ann Rheum Dis. 2015;74(10):1886–9.CrossRef
9.
go back to reference Andreeva E, Pokhaznikova M, Lebedev A, Moiseeva I, Kozlov A, Kuznetsova O, Degryse JM. The RESPECT study: RESearch on the PrEvalence and the diagnosis of COPD and its tobacco-related etiology: a study protocol. BMC Public Health. 2015;15:831.CrossRef Andreeva E, Pokhaznikova M, Lebedev A, Moiseeva I, Kozlov A, Kuznetsova O, Degryse JM. The RESPECT study: RESearch on the PrEvalence and the diagnosis of COPD and its tobacco-related etiology: a study protocol. BMC Public Health. 2015;15:831.CrossRef
10.
go back to reference Perry JJ, Stiell IG, Sivilotti ML, Bullard MJ, Emond M, Symington C, Sutherland J, Worster A, Hohl C, Lee JS, et al. Sensitivity of computed tomography performed within six hours of onset of headache for diagnosis of subarachnoid haemorrhage: prospective cohort study. BMJ (Clin Res Ed). 2011;343:d4277.CrossRef Perry JJ, Stiell IG, Sivilotti ML, Bullard MJ, Emond M, Symington C, Sutherland J, Worster A, Hohl C, Lee JS, et al. Sensitivity of computed tomography performed within six hours of onset of headache for diagnosis of subarachnoid haemorrhage: prospective cohort study. BMJ (Clin Res Ed). 2011;343:d4277.CrossRef
11.
go back to reference Nickolas TL, O'Rourke MJ, Yang J, Sise ME, Canetta PA, Barasch N, Buchen C, Khan F, Mori K, Giglio J, et al. Sensitivity and specificity of a single emergency department measurement of urinary neutrophil gelatinase-associated lipocalin for diagnosing acute kidney injury. Ann Intern Med. 2008;148(11):810–9.CrossRef Nickolas TL, O'Rourke MJ, Yang J, Sise ME, Canetta PA, Barasch N, Buchen C, Khan F, Mori K, Giglio J, et al. Sensitivity and specificity of a single emergency department measurement of urinary neutrophil gelatinase-associated lipocalin for diagnosing acute kidney injury. Ann Intern Med. 2008;148(11):810–9.CrossRef
12.
go back to reference Craig JC, Williams GJ, Jones M, Codarini M, Macaskill P, Hayen A, Irwig L, Fitzgerald DA, Isaacs D, McCaskill M. The accuracy of clinical symptoms and signs for the diagnosis of serious bacterial infection in young febrile children: prospective cohort study of 15 781 febrile illnesses. BMJ (Clin Res Ed). 2010;340:c1594.CrossRef Craig JC, Williams GJ, Jones M, Codarini M, Macaskill P, Hayen A, Irwig L, Fitzgerald DA, Isaacs D, McCaskill M. The accuracy of clinical symptoms and signs for the diagnosis of serious bacterial infection in young febrile children: prospective cohort study of 15 781 febrile illnesses. BMJ (Clin Res Ed). 2010;340:c1594.CrossRef
13.
go back to reference Luqmani R, Lee E, Singh S, Gillett M, Schmidt WA, Bradburn M, Dasgupta B, Diamantopoulos AP, Forrester-Barker W, Hamilton W, et al. The role of ultrasound compared to biopsy of temporal arteries in the diagnosis and treatment of giant cell arteritis (TABUL): a diagnostic accuracy and cost-effectiveness study. Health Technol Assess (Winchester, England). 2016;20(90):1–238.CrossRef Luqmani R, Lee E, Singh S, Gillett M, Schmidt WA, Bradburn M, Dasgupta B, Diamantopoulos AP, Forrester-Barker W, Hamilton W, et al. The role of ultrasound compared to biopsy of temporal arteries in the diagnosis and treatment of giant cell arteritis (TABUL): a diagnostic accuracy and cost-effectiveness study. Health Technol Assess (Winchester, England). 2016;20(90):1–238.CrossRef
14.
go back to reference Acute Abdominal Pain (AAP) Study group. Diagnostic accuracy of surgeons and trainees in assessment of patients with acute abdominal pain. Br J Surg. 2016;103(10):1343–9.CrossRef Acute Abdominal Pain (AAP) Study group. Diagnostic accuracy of surgeons and trainees in assessment of patients with acute abdominal pain. Br J Surg. 2016;103(10):1343–9.CrossRef
15.
go back to reference Allen VB, Gurusamy KS, Takwoingi Y, Kalia A, Davidson BR. Diagnostic accuracy of laparoscopy following computed tomography (CT) scanning for assessing the resectability with curative intent in pancreatic and periampullary cancer. Cochrane Database Syst Rev. 2016;7:CD009323. John Wiley & Sons, LtdPubMed Allen VB, Gurusamy KS, Takwoingi Y, Kalia A, Davidson BR. Diagnostic accuracy of laparoscopy following computed tomography (CT) scanning for assessing the resectability with curative intent in pancreatic and periampullary cancer. Cochrane Database Syst Rev. 2016;7:CD009323. John Wiley & Sons, LtdPubMed
16.
go back to reference Hartling L, Bond K, Santaguida PL, Viswanathan M, Dryden DM. Testing a tool for the classification of study designs in systematic reviews of interventions and exposures showed moderate reliability and low accuracy. J Clin Epidemiol. 2011;64(8):861–71.CrossRef Hartling L, Bond K, Santaguida PL, Viswanathan M, Dryden DM. Testing a tool for the classification of study designs in systematic reviews of interventions and exposures showed moderate reliability and low accuracy. J Clin Epidemiol. 2011;64(8):861–71.CrossRef
17.
go back to reference Steyerberg EW, Moons KGM, van der Windt DA, Hayden JA, Perel P, Schroter S, Riley RD, Hemingway H. Altman DG, for the PG: prognosis research strategy (PROGRESS) 3: prognostic model research. PLoS Med. 2013;10(2):e1001381.CrossRef Steyerberg EW, Moons KGM, van der Windt DA, Hayden JA, Perel P, Schroter S, Riley RD, Hemingway H. Altman DG, for the PG: prognosis research strategy (PROGRESS) 3: prognostic model research. PLoS Med. 2013;10(2):e1001381.CrossRef
18.
go back to reference van Stralen KJ, Stel VS, Reitsma JB, Dekker FW, Zoccali C, Jager KJ. Diagnostic methods I: sensitivity, specificity, and other measures of accuracy. Kidney Int. 2009;75(12):1257–63.CrossRef van Stralen KJ, Stel VS, Reitsma JB, Dekker FW, Zoccali C, Jager KJ. Diagnostic methods I: sensitivity, specificity, and other measures of accuracy. Kidney Int. 2009;75(12):1257–63.CrossRef
19.
go back to reference Mustafa RA, Wiercioch W, Cheung A, Prediger B, Brozek J, Bossuyt P, Garg AX, Lelgemann M, Büehler D, Schünemann HJ. Decision-making about healthcare related tests and diagnostic strategies: a review of methodological and practical challenges. J Clin Epidemiol. 2017;92:18–28.CrossRef Mustafa RA, Wiercioch W, Cheung A, Prediger B, Brozek J, Bossuyt P, Garg AX, Lelgemann M, Büehler D, Schünemann HJ. Decision-making about healthcare related tests and diagnostic strategies: a review of methodological and practical challenges. J Clin Epidemiol. 2017;92:18–28.CrossRef
21.
go back to reference Riley RD, Hayden JA, Steyerberg EW, Moons KGM, Abrams K, Kyzas PA, Malats N, Briggs A, Schroter S, Altman DG, et al. Prognosis research strategy (PROGRESS) 2: prognostic factor research. PLoS Med. 2013;10(2):e1001380.CrossRef Riley RD, Hayden JA, Steyerberg EW, Moons KGM, Abrams K, Kyzas PA, Malats N, Briggs A, Schroter S, Altman DG, et al. Prognosis research strategy (PROGRESS) 2: prognostic factor research. PLoS Med. 2013;10(2):e1001380.CrossRef
22.
go back to reference Mathes T, Pieper D. Study design classification of registry-based studies in systematic reviews. J Clin Epidemiol. 2017;93:84–7.CrossRef Mathes T, Pieper D. Study design classification of registry-based studies in systematic reviews. J Clin Epidemiol. 2017;93:84–7.CrossRef
23.
go back to reference Ferrante di Ruffano L, Hyde CJ, McCaffery KJ, Bossuyt PMM, Deeks JJ. Assessing the value of diagnostic tests: a framework for designing and evaluating trials. BMJ (Clin Res Ed). 2012;344:e686.CrossRef Ferrante di Ruffano L, Hyde CJ, McCaffery KJ, Bossuyt PMM, Deeks JJ. Assessing the value of diagnostic tests: a framework for designing and evaluating trials. BMJ (Clin Res Ed). 2012;344:e686.CrossRef
24.
go back to reference Collins GS, Reitsma JB, Altman DG, Moons KG. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. BMC Med. 2015;13(1):1.CrossRef Collins GS, Reitsma JB, Altman DG, Moons KG. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. BMC Med. 2015;13(1):1.CrossRef
25.
go back to reference Hemingway H, Croft P, Perel P, Hayden JA, Abrams K, Timmis A, Briggs A, Udumyan R, Moons KGM, Steyerberg EW, et al. Prognosis research strategy (PROGRESS) 1: a framework for researching clinical outcomes. BMJ. 2013;346:e5595.CrossRef Hemingway H, Croft P, Perel P, Hayden JA, Abrams K, Timmis A, Briggs A, Udumyan R, Moons KGM, Steyerberg EW, et al. Prognosis research strategy (PROGRESS) 1: a framework for researching clinical outcomes. BMJ. 2013;346:e5595.CrossRef
26.
go back to reference Knottnerus JA, Muris JW. Assessment of the accuracy of diagnostic tests: the cross-sectional study. J Clin Epidemiol. 2003;56(11):1118–28.CrossRef Knottnerus JA, Muris JW. Assessment of the accuracy of diagnostic tests: the cross-sectional study. J Clin Epidemiol. 2003;56(11):1118–28.CrossRef
27.
go back to reference Bossuyt PM LMCDCfISIC, September HfSRoDTAVu, 2008. The Cochrane collaboration. Bossuyt PM LMCDCfISIC, September HfSRoDTAVu, 2008. The Cochrane collaboration.
28.
go back to reference Kamarudin AN, Cox T, Kolamunnage-Dona R. Time-dependent ROC curve analysis in medical research: current methods and applications. BMC Med Res Methodol. 2017;17(1):53.CrossRef Kamarudin AN, Cox T, Kolamunnage-Dona R. Time-dependent ROC curve analysis in medical research: current methods and applications. BMC Med Res Methodol. 2017;17(1):53.CrossRef
29.
go back to reference Whiting PF, Rutjes AW, Westwood ME, Mallett S, Deeks JJ, Reitsma JB, Leeflang MM, Sterne JA, Bossuyt PM. QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Ann Intern Med. 2011;155(8):529–36.CrossRef Whiting PF, Rutjes AW, Westwood ME, Mallett S, Deeks JJ, Reitsma JB, Leeflang MM, Sterne JA, Bossuyt PM. QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Ann Intern Med. 2011;155(8):529–36.CrossRef
30.
go back to reference Hayden JA, van der Windt DA, Cartwright JL, Cote P, Bombardier C. Assessing bias in studies of prognostic factors. Ann Intern Med. 2013;158(4):280–6.CrossRef Hayden JA, van der Windt DA, Cartwright JL, Cote P, Bombardier C. Assessing bias in studies of prognostic factors. Ann Intern Med. 2013;158(4):280–6.CrossRef
31.
go back to reference Creavin ST, Wisniewski S, Noel-Storr AH, Trevelyan CM, Hampton T, Rayment D, Thom VM, Nash KJ, Elhamoui H, Milligan R, et al. Mini-mental state examination (MMSE) for the detection of dementia in clinically unevaluated people aged 65 and over in community and primary care populations. Cochrane Database Syst Rev. 2016;(1):CD011145. Creavin ST, Wisniewski S, Noel-Storr AH, Trevelyan CM, Hampton T, Rayment D, Thom VM, Nash KJ, Elhamoui H, Milligan R, et al. Mini-mental state examination (MMSE) for the detection of dementia in clinically unevaluated people aged 65 and over in community and primary care populations. Cochrane Database Syst Rev. 2016;(1):CD011145.
32.
go back to reference Arevalo-Rodriguez I, Smailagic N, IFM R, Ciapponi A, Sanchez-Perez E, Giannakou A, Pedraza OL, Bonfill Cosp X, Cullum S. Mini-mental state examination (MMSE) for the detection of Alzheimer’s disease and other dementias in people with mild cognitive impairment (MCI). Cochrane Database Syst Rev. 2015;(3):CD010783. Arevalo-Rodriguez I, Smailagic N, IFM R, Ciapponi A, Sanchez-Perez E, Giannakou A, Pedraza OL, Bonfill Cosp X, Cullum S. Mini-mental state examination (MMSE) for the detection of Alzheimer’s disease and other dementias in people with mild cognitive impairment (MCI). Cochrane Database Syst Rev. 2015;(3):CD010783.
33.
go back to reference Rutjes AWS, Reitsma JB, Vandenbroucke JP, Glas AS, Bossuyt PMM. Case–control and two-gate designs in diagnostic accuracy studies. Clin Chem. 2005;51(8):1335–41.CrossRef Rutjes AWS, Reitsma JB, Vandenbroucke JP, Glas AS, Bossuyt PMM. Case–control and two-gate designs in diagnostic accuracy studies. Clin Chem. 2005;51(8):1335–41.CrossRef
34.
go back to reference Mathes T, Pieper D. Clarifying the distinction between case series and cohort studies in systematic reviews of comparative studies: potential impact on body of evidence and workload. BMC Med Res Methodol. 2017;17(1):107.CrossRef Mathes T, Pieper D. Clarifying the distinction between case series and cohort studies in systematic reviews of comparative studies: potential impact on body of evidence and workload. BMC Med Res Methodol. 2017;17(1):107.CrossRef
35.
go back to reference Higgins JP, Ramsay C, Reeves BC, Deeks JJ, Shea B, Valentine JC, Tugwell P, Wells G. Issues relating to study design and risk of bias when including non-randomized studies in systematic reviews on the effects of interventions. Res Synth Methods. 2013;4(1):12–25.CrossRef Higgins JP, Ramsay C, Reeves BC, Deeks JJ, Shea B, Valentine JC, Tugwell P, Wells G. Issues relating to study design and risk of bias when including non-randomized studies in systematic reviews on the effects of interventions. Res Synth Methods. 2013;4(1):12–25.CrossRef
36.
go back to reference Collins MG, Teo E, Cole SR, Chan C-Y, McDonald SP, Russ GR, Young GP, Bampton PA, Coates PT. Screening for colorectal cancer and advanced colorectal neoplasia in kidney transplant recipients: cross sectional prevalence and diagnostic accuracy study of faecal immunochemical testing for haemoglobin and colonoscopy. BMJ. 2012;345:e4657.CrossRef Collins MG, Teo E, Cole SR, Chan C-Y, McDonald SP, Russ GR, Young GP, Bampton PA, Coates PT. Screening for colorectal cancer and advanced colorectal neoplasia in kidney transplant recipients: cross sectional prevalence and diagnostic accuracy study of faecal immunochemical testing for haemoglobin and colonoscopy. BMJ. 2012;345:e4657.CrossRef
37.
go back to reference Brown J, Pengas G, Dawson K, Brown LA, Clatworthy P. Self administered cognitive screening test (TYM) for detection of Alzheimer’s disease: cross sectional study. BMJ (Clin Res Ed). 2009;338:b2030.CrossRef Brown J, Pengas G, Dawson K, Brown LA, Clatworthy P. Self administered cognitive screening test (TYM) for detection of Alzheimer’s disease: cross sectional study. BMJ (Clin Res Ed). 2009;338:b2030.CrossRef
38.
go back to reference Dagan N, Cohen-Stavi C, Leventer-Roberts M, Balicer RD. External validation and comparison of three prediction tools for risk of osteoporotic fractures using data from population based electronic health records: retrospective cohort study. BMJ (Clin Res Ed). 2017;356:i6755.CrossRef Dagan N, Cohen-Stavi C, Leventer-Roberts M, Balicer RD. External validation and comparison of three prediction tools for risk of osteoporotic fractures using data from population based electronic health records: retrospective cohort study. BMJ (Clin Res Ed). 2017;356:i6755.CrossRef
39.
go back to reference Holmstrom B, Johansson M, Bergh A, Stenman UH, Hallmans G, Stattin P. Prostate specific antigen for early detection of prostate cancer: longitudinal study. BMJ (Clin Res Ed). 2009;339:b3537.CrossRef Holmstrom B, Johansson M, Bergh A, Stenman UH, Hallmans G, Stattin P. Prostate specific antigen for early detection of prostate cancer: longitudinal study. BMJ (Clin Res Ed). 2009;339:b3537.CrossRef
40.
go back to reference Schünemann HJ, Oxman AD, Brozek J, Glasziou P, Jaeschke R, Vist GE, Williams JW, Kunz R, Craig J, Montori VM, et al. Grading quality of evidence and strength of recommendations for diagnostic tests and strategies. BMJ (Clin Res Ed). 2008;336(7653):1106–10.CrossRef Schünemann HJ, Oxman AD, Brozek J, Glasziou P, Jaeschke R, Vist GE, Williams JW, Kunz R, Craig J, Montori VM, et al. Grading quality of evidence and strength of recommendations for diagnostic tests and strategies. BMJ (Clin Res Ed). 2008;336(7653):1106–10.CrossRef
Metadata
Title
An algorithm for the classification of study designs to assess diagnostic, prognostic and predictive test accuracy in systematic reviews
Authors
Tim Mathes
Dawid Pieper
Publication date
01-12-2019
Publisher
BioMed Central
Published in
Systematic Reviews / Issue 1/2019
Electronic ISSN: 2046-4053
DOI
https://doi.org/10.1186/s13643-019-1131-4

Other articles of this Issue 1/2019

Systematic Reviews 1/2019 Go to the issue