Skip to main content
Key Specialties
Cardiology Diabetology Endocrinology Gastroenterology Geriatrics and Gerontology Gynecology and Obstetrics Hematology Infectious Disease Internal Medicine Nephrology Neurology Oncology Pediatrics Respiratory Medicine Rheumatology Surgery
Congress Coverage
2024 ACC Congress 2023 ESMO Congress 2023 EASD Congress 2023 ERS Congress 2023 ESC Congress
Clinical Collections
Advances in Alzheimer’s

At a glance: The ONWARDS insulin icodec trials

A round-up of the ONWARDS phase 3 clinical trials evaluating the novel weekly insulin icodec in people with diabetes.
ADVANCED SEARCH
Log in
Top
BMC Medical Research Methodology
Published in: BMC Medical Research Methodology 1/2019

Open Access 01-12-2019 | Debate

DEBATE-statistical analysis plans for observational studies

Authors: Bart Hiemstra, Frederik Keus, Jørn Wetterslev, Christian Gluud, Iwan C. C. van der Horst

Published in: BMC Medical Research Methodology | Issue 1/2019

Login to get access

Abstract

Background

All clinical research benefits from transparency and validity. Transparency and validity of studies may increase by prospective registration of protocols and by publication of statistical analysis plans (SAPs) before data have been accessed to discern data-driven analyses from pre-planned analyses.

Main message

Like clinical trials, recommendations for SAPs for observational studies increase the transparency and validity of findings. We appraised the applicability of recently developed guidelines for the content of SAPs for clinical trials to SAPs for observational studies. Of the 32 items recommended for a SAP for a clinical trial, 30 items (94%) were identically applicable to a SAP for our observational study. Power estimations and adjustments for multiplicity are equally important in observational studies and clinical trials as both types of studies usually address multiple hypotheses. Only two clinical trial items (6%) regarding issues of randomisation and definition of adherence to the intervention did not seem applicable to observational studies. We suggest to include one new item specifically applicable to observational studies to be addressed in a SAP, describing how adjustment for possible confounders will be handled in the analyses.

Conclusion

With only few amendments, the guidelines for SAP of a clinical trial can be applied to a SAP for an observational study. We suggest SAPs should be equally required for observational studies and clinical trials to increase their transparency and validity.
Appendix
Available only for authorised users
Literature
1.
Skoog M, Saarimäki J, Gluud C, Scheinin M, Erlendsson K, Aamdal S. Transparency and registration in clinical research in the Nordic countries (Report). NordForsk: Nordic Trial Alliance. 2015:1–108.
2.
Schulz KF, Altman DG, Moher D. CONSORT group: CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. Int J Surg. 2011;9(8):672–7.PubMedCrossRef
3.
Moher D, Liberati A, Tetzlaff J, Altman DG. PRISMA group: preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. J Clin Epidemiol. 2009;62(10):1006–12.PubMedCrossRef
4.
Chan AW, Tetzlaff JM, Gotzsche PC, Altman DG, Mann H, Berlin JA, Dickersin K, Hrobjartsson A, Schulz KF, Parulekar WR, Krleza-Jeric K, Laupacis A, Moher D. SPIRIT 2013 explanation and elaboration: guidance for protocols of clinical trials. BMJ. 2013;346:e7586.PubMedPubMedCentralCrossRef
5.
Chien PF, Khan KS, Siassakos D. Registration of systematic reviews: PROSPERO. BJOG. 2012;119(8):903–5.PubMedCrossRef
6.
Ebrahim S, Sohani ZN, Montoya L, Agarwal A, Thorlund K, Mills EJ, Ioannidis JP. Reanalyses of randomized clinical trial data. JAMA. 2014;312(10):1024–32.PubMedCrossRef
7.
Gamble C, Krishan A, Stocken D, Lewis S, Juszczak E, Dore C, Williamson PR, Altman DG, Montgomery A, Lim P, Berlin J, Senn S, Day S, Barbachano Y, Loder E. Guidelines for the content of statistical analysis plans in clinical trials. JAMA. 2017;318(23):2337–43.PubMedCrossRef
8.
von Elm E, Altman DG, Egger M, Pocock SJ, Gotzsche PC, Vandenbroucke JP. STROBE initiative: strengthening the reporting of observational studies in epidemiology (STROBE) statement: guidelines for reporting observational studies. BMJ. 2007;335(7624):806–8.CrossRef
9.
Collins GS, Reitsma JB, Altman DG, Moons KG. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. J Clin Epidemiol. 2015;68(2):134–43.PubMedCrossRef
10.
Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP, Irwig L, Lijmer JG, Moher D, Rennie D, de Vet HC, Kressel HY, Rifai N, Golub RM, Altman DG, Hooft L, Korevaar DA, Cohen JF. STARD group: STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies. BMJ. 2015;351:h5527.PubMedPubMedCentralCrossRef
11.
PLOS. Medicine editors: observational studies: getting clear about transparency. PLoS Med. 2014;11(8):e1001711.CrossRef
12.
13.
The Lancet. Should protocols for observational research be registered? Lancet. 2010;375(9712):1.
14.
15.
Eisenach JC, Kheterpal S, Houle TT. Reporting of observational research in ANESTHESIOLOGY: the importance of the analysis plan. Anesthesiology. 2016;124(5):998–1000.PubMedCrossRef
16.
Hiemstra B, Eck RJ, Koster G, Wetterslev J, Perner A, Pettilä V, Snieder H, Hummel YM, Wiersema R, de Smet AM, Keus F, van der Horst I. Clinical examination, critical care ultrasonography and outcomes in the critically ill: cohort profile of the simple intensive care studies-I. BMJ Open. 2017;7(9):e017170.PubMedPubMedCentralCrossRef
17.
Krleza-Jeric K, Chan AW, Dickersin K, Sim I, Grimshaw J, Gluud C. Principles for international registration of protocol information and results from human trials of health related interventions: Ottawa statement (part 1). BMJ. 2005;330(7497):956–8.PubMedPubMedCentralCrossRef
18.
Garattini S, Jakobsen JC, Wetterslev J, Bertele V, Banzi R, Rath A, Neugebauer EA, Laville M, Masson Y, Hivert V, Eikermann M, Aydin B, Ngwabyt S, Martinho C, Gerardi C, Szmigielski CA, Demotes-Mainard J, Gluud C. Evidence-based clinical practice: overview of threats to the validity of evidence and how to minimise them. Eur J Intern Med. 2016;32:13–21.PubMedCrossRef
19.
Chu C, Umanski G, Blank A, Grossberg R, Selwyn PA. HIV-infected patients and treatment outcomes: an equivalence study of community-located, primary care-based HIV treatment vs. hospital-based specialty care in the Bronx, New York. AIDS Care. 2010;22(12):1522–9.PubMedPubMedCentralCrossRef
20.
Pyo JH, Lee H, Min BH, Lee JH, Choi MG, Lee JH, Sohn TS, Bae JM, Kim KM, Ahn JH, Carriere KC, Kim JJ, Kim S. Long-term outcome of endoscopic resection vs. surgery for early gastric cancer: a non-inferiority-matched cohort study. Am J Gastroenterol. 2016;111(2):240–9.PubMedCrossRef
21.
Parker K, Perikala V, Aminazad A, Deng Z, Borg B, Buchan C, Toghill J, Irving LB, Goldin J, Charlesworth D, Mahal A, Illesinghe S, Naughton MT, Young A. Models of care for non-invasive ventilation in the acute COPD comparison of three tertiary hospitals (ACT3) study. Respirology. 2018;23(5):492–7.PubMedCrossRef
22.
Austevoll IM, Gjestad R, Brox JI, Solberg TK, Storheim K, Rekeland F, Hermansen E, Indrekvam K, Hellum C. The effectiveness of decompression alone compared with additional fusion for lumbar spinal stenosis with degenerative spondylolisthesis: a pragmatic comparative non-inferiority observational study from the Norwegian registry for spine surgery. Eur Spine J. 2017;26(2):404–13.PubMedCrossRef
23.
O'Brien PC, Fleming TR. A multiple testing procedure for clinical trials. Biometrics. 1979;35(3):549–56.PubMedCrossRef
24.
25.
Šidák Z. Rectangular confidence regions for the means of multivariate normal distributions. J Am Stat Assoc. 1967;62(318):626–33.
26.
Anonymous International Conference on Harmonisation of Technical Requirements for Registration of Pharmaceuticals for Human Use (ICH) adopts Consolidated Guideline on Good Clinical Practice in the Conduct of Clinical Trials on Medicinal Products for Human Use. Int Dig Health Legis. 1997;48(2):231–4.
27.
28.
Jakobsen JC, Wetterslev J, Winkel P, Lange T, Gluud C. Thresholds for statistical and clinical significance in systematic reviews with meta-analytic methods. BMC Med Res Methodol. 2014;14:120.PubMedPubMedCentralCrossRef
29.
Vandenbroucke JP, von Elm E, Altman DG, Gotzsche PC, Mulrow CD, Pocock SJ, Poole C, Schlesselman JJ, Egger M. STROBE initiative: strengthening the reporting of observational studies in epidemiology (STROBE): explanation and elaboration. Int J Surg. 2014;12(12):1500–24.CrossRefPubMed
30.
Nguyen TL, Collins GS, Lamy A, Devereaux PJ, Daures JP, Landais P, Le Manach Y. Simple randomization did not protect against bias in smaller trials. J Clin Epidemiol. 2017;84:105–13.PubMedCrossRef
31.
Pouwels KB, Widyakusuma NN, Groenwold RH, Hak E. Quality of reporting of confounding remained suboptimal after the STROBE guideline. J Clin Epidemiol. 2016;69:217–24.PubMedCrossRef
32.
Greenberg L, Jairath V, Pearse R, Kahan BC. Pre-specification of statistical analysis approaches in published clinical trial protocols was inadequate. J Clin Epidemiol. 2018;101:53–60.PubMedCrossRef
33.
Perkins NJ, Cole SR, Harel O, Tchetgen Tchetgen EJ, Sun B, Mitchell EM, Schisterman EF. Principled approaches to missing data in epidemiologic studies. Am J Epidemiol. 2018;187(3):568–75.PubMedCrossRef
34.
Jakobsen JC, Gluud C, Wetterslev J, Winkel P. When and how should multiple imputation be used for handling missing data in randomised clinical trials - a practical guide with flowcharts. BMC Med Res Methodol. 2017;17(1):162.PubMedPubMedCentralCrossRef
35.
McCulloch P, Altman DG, Campbell WB, Flum DR, Glasziou P, Marshall JC, Nicholl J, Balliol Collaboration, Aronson JK, Barkun JS, Blazeby JM, Boutron IC, Campbell WB, Clavien PA, Cook JA, Ergina PL, Feldman LS, Flum DR, Maddern GJ, Nicholl J, Reeves BC, Seiler CM, Strasberg SM, Meakins JL, Ashby D, Black N, Bunker J, Burton M, Campbell M, Chalkidou K, Chalmers I, de Leval M, Deeks J, Ergina PL, Grant A, Gray M, Greenhalgh R, Jenicek M, Kehoe S, Lilford R, Littlejohns P, Loke Y, Madhock R, McPherson K, Meakins J, Rothwell P, Summerskill B, Taggart D, Tekkis P, Thompson M, Treasure T, Trohler U, Vandenbroucke J. No surgical innovation without evaluation: the IDEAL recommendations. Lancet. 2009;374(9695):1105–12.CrossRefPubMed
36.
van Staa TP, Smeeth L, Persson I, Parkinson J, Leufkens HG. What is the harm-benefit ratio of cox-2 inhibitors? Int J Epidemiol. 2008;37(2):405–13.PubMedCrossRef
37.
Jespersen CM, Als-Nielsen B, Damgaard M, Hansen JF, Hansen S, Helo OH, Hildebrandt P, Hilden J, Jensen GB, Kastrup J, Kolmos HJ, Kjoller E, Lind I, Nielsen H, Petersen L, Gluud C. CLARICOR trial group: randomised placebo controlled multicentre trial to assess short term clarithromycin for patients with stable coronary heart disease: CLARICOR trial. BMJ. 2006;332(7532):22–7.PubMedPubMedCentralCrossRef
38.
Editors. The registration of observational studies--when metaphors go bad. Epidemiology. 2010;21(5):607–9.
39.
Savitz DA. Registration of observational studies does not enhance validity. Clin Pharmacol Ther. 2011;90(5):646–8.PubMedCrossRef
40.
Lash TL. Preregistration of study protocols is unlikely to improve the yield from our science, but other strategies might. Epidemiology. 2010;21(5):612–3.PubMedCrossRef
41.
Vandenbroucke JP. Preregistration of epidemiologic studies: an ill-founded mix of ideas. Epidemiology. 2010;21(5):619–20.PubMedCrossRef
42.
Vandenbroucke JP. Registering observational research: second thoughts. Lancet. 2010;375(9719):982–3.PubMedCrossRef
43.
Pearce N. Registration of protocols for observational research is unnecessary and would do more harm than good. Occup Environ Med. 2011;68(2):86–8.PubMedCrossRef
44.
Lash TL, Vandenbroucke JP. Should preregistration of epidemiologic study protocols become compulsory? Reflections and a counterproposal. Epidemiology. 2012;23(2):184–8.PubMedCrossRef
45.
Dal-Re R, Ioannidis JP, Bracken MB, Buffler PA, Chan AW, Franco EL, La Vecchia C, Weiderpass E. Making prospective registration of observational research a reality. Sci Transl Med. 2014;6(224):224cm1.PubMedCrossRef
46.
Bracken MB. Preregistration of epidemiology protocols: a commentary in support. Epidemiology. 2011;22(2):135–7.PubMedCrossRef
47.
Thomas L, Peterson ED. The value of statistical analysis plans in observational research: defining high-quality research from the start. JAMA. 2012;308(8):773–4.PubMedCrossRef
48.
Onukwugha E. Improving confidence in observational studies : should statistical analysis plans be made publicly available? Pharmacoeconomics. 2013;31(3):177–9.PubMedCrossRef
49.
Ioannidis JP. The importance of potential studies that have not existed and registration of observational data sets. JAMA. 2012;308(6):575–6.PubMedCrossRef
50.
Deeks JJ, Dinnes J, D'Amico R, Sowden AJ, Sakarovitch C, Song F, Petticrew M, Altman DG. International stroke trial collaborative group, European carotid surgery trial collaborative group: evaluating non-randomised intervention studies. Health Technol Assess. 2003;7(27):173.CrossRef
51.
Ioannidis JP. Why most discovered true associations are inflated. Epidemiology. 2008;19(5):640–8.PubMedCrossRef
52.
Schoenfeld JD, Ioannidis JP. Is everything we eat associated with cancer? A systematic cookbook review. Am J Clin Nutr. 2013;97(1):127–34.PubMedCrossRef
Metadata
Title
DEBATE-statistical analysis plans for observational studies
Authors
Bart Hiemstra
Frederik Keus
Jørn Wetterslev
Christian Gluud
Iwan C. C. van der Horst
Publication date
01-12-2019
Publisher
BioMed Central
Published in
BMC Medical Research Methodology / Issue 1/2019
Electronic ISSN: 1471-2288
DOI
https://doi.org/10.1186/s12874-019-0879-5

Other articles of this Issue 1/2019

BMC Medical Research Methodology 1/2019 Go to the issue

Research article

Does the provision of a DVD-based audio-visual presentation improve recruitment in a clinical trial? A randomised trial of DVD trial invitations

Research article

Understanding the predictive value of continuous markers for censored survival data using a likelihood ratio approach

Research article

Providing quality data in health care - almost perfect inter-rater agreement in the Norwegian tonsil surgery register

Technical advance

Bayes factors for superiority, non-inferiority, and equivalence designs

Research article

The agreement between chronic diseases reported by patients and derived from administrative data in patients undergoing joint arthroplasty

Research article

The key therapeutic factors needed to deliver behavioural change interventions to decrease risky substance use (drug and alcohol) for looked after children and care leavers: a qualitative exploration with young people, carers and front line workers