Top

Critical Care

Published in:

Open Access 01-04-2010 | Research

Delta inflation: a bias in the design of randomized controlled trials in critical care medicine

Authors: Scott K Aberegg, D Roxanne Richards, James M O'Brien

Published in: Critical Care | Issue 2/2010

Abstract

Introduction

Mortality is the most widely accepted outcome measure in randomized controlled trials of therapies for critically ill adults, but most of these trials fail to show a statistically significant mortality benefit. The reasons for this are unknown.

Methods

We searched five high impact journals (Annals of Internal Medicine, British Medical Journal, JAMA, The Lancet, New England Journal of Medicine) for randomized controlled trials comparing mortality of therapies for critically ill adults over a ten year period. We abstracted data on the statistical design and results of these trials to compare the predicted delta (delta; the effect size of the therapy compared to control expressed as an absolute mortality reduction) to the observed delta to determine if there is a systematic overestimation of predicted delta that might explain the high prevalence of negative results in these trials.

Results

We found 38 trials meeting our inclusion criteria. Only 5/38 (13.2%) of the trials provided justification for the predicted delta. The mean predicted delta among the 38 trials was 10.1% and the mean observed delta was 1.4% (P < 0.0001), resulting in a delta-gap of 8.7%. In only 2/38 (5.3%) of the trials did the observed delta exceed the predicted delta and only 7/38 (18.4%) of the trials demonstrated statistically significant results in the hypothesized direction; these trials had smaller delta-gaps than the remainder of the trials (delta-gap 0.9% versus 10.5%; P < 0.0001). For trials showing non-significant trends toward benefit greater than 3%, large increases in sample size (380% - 1100%) would be required if repeat trials use the observed delta from the index trial as the predicted delta for a follow-up study.

Conclusions

Investigators of therapies for critical illness systematically overestimate treatment effect size (delta) during the design of randomized controlled trials. This bias, which we refer to as "delta inflation", is a potential reason that these trials have a high rate of negative results.

"Absence of evidence is not evidence of absence."

Available only for authorised users

Ospina-Tascon GA, Buchele GL, Vincent JL: Multicenter, randomized, controlled trials evaluating mortality in intensive care: doomed to fail? Crit Care Med 2008, 36: 1311-1322. 10.1097/CCM.0b013e318168ea3eCrossRefPubMed

Berghe G, Wouters P, Weekers F, Verwaest C, Bruyninckx F, Schetz M, Vlasselaers D, Ferdinande P, Lauwers P, Bouillon R: Intensive insulin therapy in critically ill patients. N Engl J Med 2001, 345: 1359-1367. 10.1056/NEJMoa011300CrossRefPubMed

Berghe G, Wilmer A, Hermans G, Meersseman W, Wouters PJ, Milants I, Van Wijngaerden E, Bobbaers H, Bouillon R: Intensive insulin therapy in the medical ICU. N Engl J Med 2006, 354: 449-461. 10.1056/NEJMoa052521CrossRefPubMed

The NICE-SUGAR Study Investigators: Intensive versus conventional glucose control in critically ill patients. N Engl J Med 2009, 360: 1283-1297. 10.1056/NEJMoa0810625CrossRef

Annane D, Sébille V, Charpentier C, Bollaert PE, François B, Korach JM, Capellier G, Cohen Y, Azoulay E, Troché G, Chaumet-Riffaud P, Bellissant E: Effect of treatment with low doses of hydrocortisone and fludrocortisone on mortality in patients with septic shock. JAMA 2002, 288: 862-871. 10.1001/jama.288.7.862CrossRefPubMed

Sprung CL, Annane D, Keh D, Moreno R, Singer M, Freivogel K, Weiss YG, Benbenishty J, Kalenka A, Forst H, Laterre PF, Reinhart K, Cuthbertson BH, Payen D, Briegel J, CORTICUS Study Group: Hydrocortisone therapy for patients with septic shock. N Engl J Med 2008, 358: 111-124. 10.1056/NEJMoa071366CrossRefPubMed

Brunkhorst FM, Engel C, Bloos F, Meier-Hellmann A, Ragaller M, Weiler N, Moerer O, Gruendling M, Oppert M, Grond S, Olthoff D, Jaschinski U, John S, Rossaint R, Welte T, Schaefer M, Kern P, Kuhnt E, Kiehntopf M, Hartog C, Natanson C, Loeffler M, Reinhart K, German Competence Network Sepsis (SepNet): Intensive insulin therapy and pentastarch resuscitation in severe sepsis. N Engl J Med 2008,358(2):125-139. 10.1056/NEJMoa070716CrossRefPubMed

Charles P, Giraudeau B, Dechartres A, Baron G, Ravaud P: Reporting of sample size calculation in randomised controlled trials: review. BMJ 2009, 338: b1732. 10.1136/bmj.b1732PubMedCentralCrossRefPubMed

Moher D, Schulz KF, Altman D, for the CONSORT Group: The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomized trials. JAMA 2001, 285: 1987-1991. 10.1001/jama.285.15.1987CrossRefPubMed

10.

Chan KB, Man-Son-Hing M, Molnar FJ, Laupacis A: How well is the clinical importance of study results reported? An assessment of randomized controlled trials. CMAJ 2001, 165: 1197-1202.PubMedCentralPubMed

11.

Schulz KF, Grimes DA: Sample size calculations in randomised trials: mandatory and mystical. Lancet 2005, 365: 1348-1353. 10.1016/S0140-6736(05)61034-3CrossRefPubMed

12.

Matthews JN: Small clinical trials: are they all bad? Stat Med 1995, 14: 115-126. 10.1002/sim.4780140204CrossRefPubMed

13.

Goodman SN, Berlin JA: The use of predicted confidence intervals when planning experiments and the misuse of power when interpreting results. Ann Intern Med 1994, 121: 200-206.CrossRefPubMed

14.

Guyatt GH, Mills EJ, Elbourne D: In the era of systematic reviews, does the size of an individual trial still matter. PLoS Med 2008, 5: e4. 10.1371/journal.pmed.0050004PubMedCentralCrossRefPubMed

15.

Harvey S, Harrison DA, Singer M, Ashcroft J, Jones CM, Elbourne D, Brampton W, Williams D, Young D, Rowan K, PAC-Man study collaboration: Assessment of the clinical effectiveness of pulmonary artery catheters in management of patients in intensive care (PAC-Man): a randomised controlled trial. Lancet 2005, 366: 472-477. 10.1016/S0140-6736(05)67061-4CrossRefPubMed

16.

The National Heart LaBIARDSACTN: Efficacy and safety of corticosteroids for persistent acute respiratory distress syndrome. N Engl J Med 2006, 354: 1671-1684. 10.1056/NEJMoa051693CrossRef

17.

Abraham E, Reinhart K, Opal S, Demeyer I, Doig C, Rodriguez AL, Beale R, Svoboda P, Laterre PF, Simon S, Light B, Spapen H, Stone J, Seibert A, Peckelsen C, De Deyne C, Postier R, Pettilä V, Artigas A, Percell SR, Shu V, Zwingelstein C, Tobias J, Poole L, Stolzenbach JC, Creasey AA, OPTIMIST Trial Study Group: Efficacy and safety of tifacogin (recombinant tissue factor pathway inhibitor) in severe sepsis: a randomized controlled trial. JAMA 2003, 290: 238-247. 10.1001/jama.290.2.238CrossRefPubMed

18.

Finfer S, Bellomo R, Boyce N, French J, Myburgh J, Norton R, SAFE Study Investigators: A comparison of albumin and saline for fluid resuscitation in the intensive care unit. N Engl J Med 2004, 350: 2247-2256. 10.1056/NEJMoa040232CrossRefPubMed

19.

Bernard GR, Vincent JL, Laterre PF, LaRosa SP, Dhainaut JF, Lopez-Rodriguez A, Steingrub JS, Garber GE, Helterbrand JD, Ely EW, Fisher CJ Jr, Recombinant human protein C Worldwide Evaluation in Severe Sepsis (PROWESS) study group: Efficacy and safety of recombinant human activated protein C for severe sepsis. N Engl J Med 2001, 344: 699-709. 10.1056/NEJM200103083441001CrossRefPubMed

20.

Rivers E, Nguyen B, Havstad S, Ressler J, Muzzin A, Knoblich B, Peterson E, Tomlanovich M, Early Goal-Directed Therapy Collaborative Group: Early goal-directed therapy in the treatment of severe sepsis and septic shock. N Engl J Med 2001, 345: 1368-1377. 10.1056/NEJMoa010307CrossRefPubMed

21.

Esteban A, Anzueto A, Frutos F, Alía I, Brochard L, Stewart TE, Benito S, Epstein SK, Apezteguía C, Nightingale P, Arroliga AC, Tobin MJ, Mechanical Ventilation International Study Group: Characteristics and outcomes in adult patients receiving mechanical ventilation: a 28-day international study. JAMA 2002, 287: 345-355. 10.1001/jama.287.3.345CrossRefPubMed

22.

Sloan EP, Koenigsberg M, Gens D, Cipolle M, Runge J, Mallory MN, Rodman G Jr: Diaspirin cross-linked hemoglobin (DCLHb) in the treatment of severe traumatic hemorrhagic shock: a randomized controlled efficacy trial. JAMA 1999, 282: 1857-1864. 10.1001/jama.282.19.1857CrossRefPubMed

23.

Weaver CS, Leonardi-Bee J, Bath-Hextall FJ, Bath PM: Sample size calculations in acute stroke trials: a systematic review of their reporting, characteristics, and relationship with outcome. Stroke 2004, 35: 1216-1224. 10.1161/01.STR.0000125010.70652.93CrossRefPubMed

24.

Raju TN, Langenberg P, Sen A, Aldana O: How much 'better' is good enough? The magnitude of treatment effect in clinical trials. Am J Dis Child 1992, 146: 407-411.CrossRefPubMed

25.

Moher D, Dulberg CS, Wells GA: Statistical power, sample size, and their reporting in randomized controlled trials. JAMA 1994, 272: 122-124. 10.1001/jama.272.2.122CrossRefPubMed

26.

Chan AW, Hrobjartsson A, Jorgensen KJ, Gotzsche PC, Altman DG: Discrepancies in sample size calculations and data analyses reported in randomised trials: comparison of publications with protocols. BMJ 2008, 337: a2299. 10.1136/bmj.a2299PubMedCentralCrossRefPubMed

27.

Chalmers I, Matthews R: What are the implications of optimism bias in clinical research? Lancet 2006, 367: 449-450. 10.1016/S0140-6736(06)68153-1CrossRefPubMed

28.

Aberegg SK, O'Brien J Jr, Khoury P, Patel R, Arkes HR: The influence of treatment effect size on willingness to adopt a therapy. Med Decis Making 2009, 29: 599-605. 10.1177/0272989X09336078CrossRefPubMed

29.

Gould ALL: Planning and revising the sample size for a trial. Stat Med 1995, 14: 1039-1051. 10.1002/sim.4780140922CrossRefPubMed

30.

Naylor CD, Llewellyn-Thomas HA: Can there be a more patient-centred approach to determining clinically important effect sizes for randomized treatment trials? J Clin Epidemiol 1994, 47: 787-795. 10.1016/0895-4356(94)90176-7CrossRefPubMed

31.

32.

Decullier E, Chan AW, Chapuis F: Inadequate dissemination of phase I trials: a retrospective cohort study. PLoS Med 2009, 6: e1000034. 10.1371/journal.pmed.1000034PubMedCentralCrossRefPubMed

33.

Bedard PL, Krzyzanowska MK, Pintilie M, Tannock IF: Statistical power of negative randomized controlled trials presented at American Society for Clinical Oncology annual meetings. J Clin Oncol 2007, 25: 3482-3487. 10.1200/JCO.2007.11.3670CrossRefPubMed

34.

Hebert RS, Wright SM, Dittus RS, Elasy TA: Prominent medical journals often provide insufficient information to assess the validity of studies with negative results. J Negat Results Biomed 2002, 1: 1. 10.1186/1477-5751-1-1PubMedCentralCrossRefPubMed

35.

National Heart, Lung, and Blood Institute Acute Respiratory Distress Syndrome (ARDS) Clinical Trials Network, Wiedemann HP, Wheeler AP, Bernard GR, Thompson BT, Hayden D, deBoisblanc B, Connors AF Jr, Hite RD, Harabin AL: Comparison of two fluid-management strategies in acute lung injury. N Engl J Med 2006, 354: 2564-2575. 10.1056/NEJMoa062200CrossRef

36.

Rivers EP: Fluid-management strategies in acute lung injury -- liberal, conservative, or both? N Engl J Med 2006, 354: 2598-2600. 10.1056/NEJMe068105CrossRefPubMed

37.

Feinstein AR, Concato J: The quest for "power": contradictory hypotheses and inflated sample sizes. J Clin Epidemiol 1998, 51: 537-545. 10.1016/S0895-4356(98)00029-8CrossRefPubMed

38.

Halpern SD, Karlawish JH, Berlin JA: The continuing unethical conduct of underpowered clinical trials. JAMA 2002, 288: 358-362. 10.1001/jama.288.3.358CrossRefPubMed

39.

Altman DG, Bland JM: Absence of evidence is not evidence of absence. Aust Vet J 1996, 74: 311. 10.1111/j.1751-0813.1996.tb13786.xCrossRefPubMed

40.

Ioannidis JP, Evans SJ, Gøtzsche PC, O'Neill RT, Altman DG, Schulz K, Moher D, CONSORT Group: Better reporting of harms in randomized trials: an extension of the CONSORT statement. Ann Intern Med 2004, 141: 781-788.CrossRefPubMed

41.

Freemantle N, Calvert M, Wood J, Eastaugh J, Griffin C: Composite outcomes in randomized trials: greater precision but with greater uncertainty? JAMA 2003, 289: 2554-2559. 10.1001/jama.289.19.2554CrossRefPubMed

42.

Lim E, Brown A, Helmy A, Mussa S, Altman DG: Composite outcomes in cardiovascular research: a survey of randomized trials. Ann Intern Med 2008, 149: 612-617.CrossRefPubMed

43.

Ferreira-González I, Busse JW, Heels-Ansdell D, Montori VM, Akl EA, Bryant DM, Alonso-Coello P, Alonso J, Worster A, Upadhye S, Jaeschke R, Schünemann HJ, Permanyer-Miralda G, Pacheco-Huergo V, Domingo-Salvany A, Wu P, Mills EJ, Guyatt GH: Problems with use of composite end points in cardiovascular trials: systematic review of randomised controlled trials. BMJ 2007, 334: 786. 10.1136/bmj.39136.682083.AEPubMedCentralCrossRefPubMed

44.

Dowdy DW, Eid MP, Sedrakyan A, Mendez-Tellez PA, Pronovost PJ, Herridge MS, Needham DM: Quality of life in adult survivors of critical illness: a systematic review of the literature. Intensive Care Med 2005, 31: 611-620. 10.1007/s00134-005-2592-6CrossRefPubMed

Title: Delta inflation: a bias in the design of randomized controlled trials in critical care medicine
Authors: Scott K Aberegg
D Roxanne Richards
James M O'Brien
Publication date: 01-04-2010
Publisher: BioMed Central
Published in: Critical Care / Issue 2/2010
Electronic ISSN: 1364-8535
DOI: https://doi.org/10.1186/cc8990

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Delta inflation: a bias in the design of randomized controlled trials in critical care medicine

Abstract

Introduction

Methods

Results

Conclusions

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Abstract

Introduction

Methods

Results

Conclusions

Please log in to get access to this content

Other articles of this Issue 2/2010

Effect of an analgo-sedation protocol for neurointensive patients: a two-phase interventional non-randomized pilot study

Near-infrared spectroscopy during stagnant ischemia estimates central venous oxygen saturation and mixed venous oxygen saturation discrepancy in patients with severe left heart failure and additional sepsis/septic shock

Non-invasive ventilation for critically ill patients with pandemic H1N1 2009 influenza A virus infection

Influence of Factor V Leiden on susceptibility to and outcome from critical illness: a genetic association study

Prognostic value of cell-free plasma DNA in patients with cardiac arrest outside the hospital: an observational cohort study

Vacuum-assisted closure device in intensive care unit patients and dissemination of Gram-negative bacteria