Skip to main content
Key Specialties
Cardiology Diabetology Endocrinology Gastroenterology Geriatrics and Gerontology Gynecology and Obstetrics Hematology Infectious Disease Internal Medicine Nephrology Neurology Oncology Pediatrics Respiratory Medicine Rheumatology Surgery
Congress Coverage
2024 ACC Congress 2023 ESMO Congress 2023 EASD Congress 2023 ERS Congress 2023 ESC Congress
Clinical Collections
Advances in Alzheimer’s

At a glance: The STEP trials

A round-up of the STEP phase 3 clinical trials evaluating semaglutide for weight loss in people with overweight or obesity.
ADVANCED SEARCH
Log in
Top
Trials
Published in: Trials 1/2015

Open Access 01-12-2015 | Review

A literature review on the representativeness of randomized controlled trial samples and implications for the external validity of trial results

Authors: Tessa Kennedy-Martin, Sarah Curtis, Douglas Faries, Susan Robinson, Joseph Johnston

Published in: Trials | Issue 1/2015

Login to get access

Abstract

Randomized controlled trials (RCTs) are conducted under idealized and rigorously controlled conditions that may compromise their external validity. A literature review was conducted of published English language articles that reported the findings of studies assessing external validity by a comparison of the patient sample included in RCTs reporting on pharmaceutical interventions with patients from everyday clinical practice. The review focused on publications in the fields of cardiology, mental health, and oncology. A range of databases were interrogated (MEDLINE; EMBASE; Science Citation Index; Cochrane Methodology Register). Double-abstract review and data extraction were performed as per protocol specifications. Out of 5,456 de-duplicated abstracts, 52 studies met the inclusion criteria (cardiology, n = 20; mental health, n = 17; oncology, n = 15). Studies either performed an analysis of the baseline characteristics (demographic, socioeconomic, and clinical parameters) of RCT-enrolled patients compared with a real-world population, or assessed the proportion of real-world patients who would have been eligible for RCT inclusion following the application of RCT inclusion/exclusion criteria. Many of the included studies concluded that RCT samples are highly selected and have a lower risk profile than real-world populations, with the frequent exclusion of elderly patients and patients with co-morbidities. Calculation of ineligibility rates in individual studies showed that a high proportion of the general disease population was often excluded from trials. The majority of studies (n = 37 [71.2 %]) explicitly concluded that RCT samples were not broadly representative of real-world patients and that this may limit the external validity of the RCT. Authors made a number of recommendations to improve external validity. Findings from this review indicate that there is a need to improve the external validity of RCTs such that physicians treating patients in real-world settings have the appropriate evidence on which to base their clinical decisions. This goal could be achieved by trial design modification to include a more representative patient sample and by supplementing RCT evidence with data generated from observational studies. In general, a thoughtful approach to clinical evidence generation is required in which the trade-offs between internal and external validity are considered in a holistic and balanced manner.
Appendix
Available only for authorised users
Literature
1.
Schulz KF, Altman DG, Moher D, for the CONSORT Group. CONSORT 2010 Statement: Updated guidelines for reporting parallel group randomized trials. Ann Intern Med. 2010;152:726–32.CrossRefPubMed
2.
Van Spall HG, Toren A, Kiss A, Fowler RA, et al. Eligibility criteria of randomized controlled trials published in high-impact general medical journals: a systematic sampling review. JAMA. 2007;297:1233–40.CrossRefPubMed
3.
Rothwell PM. External validity of randomised controlled trials: “to whom do the results of this trial apply?”. Lancet. 2005;365:82–93.CrossRefPubMed
4.
Singal AG, Higgins PDR, Waljee AK. A primer on effectiveness and efficacy trials. Clin Trans Gastroenterol. 2014;5:e45.CrossRef
5.
6.
Nallamothu BK, Hayward RA, Bates ER. Beyond the randomized clinical trial: the role of effectiveness studies in evaluating cardiovascular therapies. Circulation. 2008;118:1294–303.CrossRefPubMed
7.
Sniderman AD, LaChapelle KJ, Rachon NA, Furberg CD, et al. The necessity for clinical reasoning in the era of evidence-based medicine. Mayo Clin Proc. 2013;88:1108–14.CrossRefPubMed
8.
Franciosa JA. The potential role of community-based registries to complement the limited applicability of clinical trial results to the community setting: heart failure as an example. Am J Manag Care. 2004;10:487–92.PubMed
9.
Saunders C, Byrne CD, Guthrie B, Lindsay RS, McKnight JA, Sattar N, et al. External validity of randomized controlled trials of glycaemic control and vascular disease: how representative are participants? Diabet Med. 2013;30:300–8.CrossRefPubMed
10.
Hordijk-Trion M, Lenzen M, Wijns W, De Jagere P, Simmons ML, Scholte Op Reimer WJM, et al. Patients enrolled in coronary intervention trials are not representative of patients in clinical practice: results from the Euro Heart Survey on Coronary Revascularization. Eur Heart J. 2006;27:671–8.CrossRefPubMed
11.
Maasland L, Van Oostenbrugge RJ, Franke CF, Scholte Op Reimer WJM, Koudstaal PJ, Dippel DWJ, et al. Patients enrolled in large randomized clinical trials of antiplatelet treatment for prevention after transient ischemic attack or ischemic stroke are not representative of patients in clinical practice: the Netherlands Stroke Survey. Stroke. 2009;40:2662–8.CrossRefPubMed
12.
Travers J, Marsh S, Williams M, Weatherall M, Caldwekk B, Shirtcliffe P, et al. External validity of randomised controlled trials in asthma: to whom do the results of the trials apply? Thorax. 2007;62:219–23.CrossRefPubMed
13.
Villela R, Yuen SY, Pope JE, Baron M. Assessment of unmet needs and the lack of generalizability in the design of randomized controlled trials for scleroderma treatment. Arthritis Rheum. 2008;59:706–13.CrossRefPubMed
14.
Arksey H, O’Malley L. Scoping studies: towards a methodological framework. Int J Soc Res Methodol. 2005;8:19–32.CrossRef
15.
Badano LP, Di Lenarda A, Bellotti P, Albanese MC, Sinagra G, Fioretti PM. Patients with chronic heart failure encountered in daily clinical practice are different from the “typical” patient enrolled in therapeutic trials. Ital Heart J. 2003;4:84–91.PubMed
16.
Bahit MC, Cannon CP, Antman EM, Murphy SA, Gibson MC, McCabe CH, et al. Thrombolysis in myocardial infarction. Direct comparison of characteristics, treatment, and outcomes of patients enrolled versus patients not enrolled in a clinical trial at centers participating in the TIMI 9 Trial and TIMI 9 registry. Am Heart J. 2003;145:109–17.CrossRefPubMed
17.
Björklund E, Lindahl B, Stenestrand U, Swahn E, Delborg M, Pehrsson K, et al. Outcome of ST-elevation myocardial infarction treated with thrombolysis in the unselected population is vastly different from samples of eligible patients in a large-scale clinical trial. Am Heart J. 2004;148:566–73.CrossRefPubMed
18.
Blanco C, Olfson M, Goodwin RD, Ogburn E, Liebowitz MR, Nunes EV, et al. Generalizability of clinical trial results for major depression to community samples: results from the National Epidemiologic Survey on Alcohol and Related Conditions. J Clin Psychiatry. 2008;69:1276–80.CrossRefPubMed
19.
Bosch X, Delgado V, Verbal F, Bórquez E, Loma-Osorio P, Díez-Aja S, et al. Causes of ineligibility in randomized controlled trials and long-term mortality in patients with non-ST-segment elevation acute coronary syndromes. Int J Cardiol. 2008;124:86–91.CrossRefPubMed
20.
Clarey J, Kao SC, Clarke SJ, Vardy J. The eligibility of advanced non-small-cell lung cancer patients for targeted therapy clinical trials. Ann Oncol. 2012;23:1229–33.CrossRefPubMed
21.
Costantino G, Rusconi AM, Duca PG, Giorgia Duca P, Guzzetti S, Bossi I, et al. Eligibility criteria in heart failure randomized controlled trials: a gap between evidence and clinical practice. Intern Emerg Med. 2009;4:117–22.CrossRefPubMed
22.
Dhruva SS, Redberg RF. Variations between clinical trial participants and Medicare beneficiaries in evidence used for Medicare national coverage decisions. Arch Intern Med. 2008;168:136–40.CrossRefPubMed
23.
Elting LS, Cooksley C, Bekele BN, Frumovitz M, Avritscher EBC, Sun C, et al. Generalizability of cancer clinical trial results: prognostic differences between participants and nonparticipants. Cancer. 2006;106:2452–8.CrossRefPubMed
24.
Ezekowitz JA, Hu J, Delgado D, Hernandez AF, Kaul P, Leader R, et al. Acute heart failure: perspectives from a randomized trial and a simultaneous registry. Circ Heart Fail. 2012;5:735–41.CrossRefPubMed
25.
Fraser J, Steele N, Al Zaman A, Yule A. Are patients in clinical trials representative of the general population? Dose intensity and toxicities associated with FE100C-D chemotherapy in a non-trial population of node positive breast cancer patients compared with PACS-01 trial group. Eur J Cancer. 2011;47:215–20.CrossRefPubMed
26.
Goedhard LE, Stolker JJ, Nijman HL, Egberts TCG, Heerdink ER. Trials assessing pharmacotherapeutic management of aggression in psychiatric patients: comparability with clinical practice. Pharmacopsychiatry. 2010;43:205–9.CrossRefPubMed
27.
Golomb BA, Chan VT, Evans MA, Koperski S, White HL, Criqui MH. The older the better: are elderly study participants more non-representative? A cross-sectional analysis of clinical trial and observational study samples. BMJ Open. 2012;2:e000833.CrossRefPubMedPubMedCentral
28.
Hoertel N, Le Strat Y, Lavaud P, Dubertret C, Limosin F. Generalizability of clinical trial results for bipolar disorder to community samples: findings from the National Epidemiologic Survey on Alcohol and Related Conditions. J Clin Psychiatry. 2013;74:265–70.CrossRefPubMed
29.
Hutchinson-Jaffe AB, Goodman SG, Yan RT, Wald R, Elbarouni B, Rose B, et al. Comparison of baseline characteristics, management and outcome of patients with non-ST-segment elevation acute coronary syndrome in versus not in clinical trials. Am J Cardiol. 2010;106:1389–96.CrossRefPubMed
30.
Jennens RR, Giles GG, Fox RM. Increasing underrepresentation of elderly patients with advanced colorectal or non-small-cell lung cancer in chemotherapy trials. Intern Med J. 2006;36:216–20.CrossRefPubMed
31.
Kalata P, Martus P, Zettl H, Rödel C, Hohenberger W, Raab R, et al. Differences between clinical trial participants and patients in a population-based registry: the German Rectal Cancer Study vs the Rostock Cancer Registry. Dis Colon Rectum. 2009;52:425–37.CrossRefPubMed
32.
Keitner GI, Posternak MA, Ryan CE. How many subjects with major depressive disorder meet eligibility requirements of an antidepressant efficacy trial? J Clin Psychiatry. 2003;64:1091–3.CrossRefPubMed
33.
Khan AY, Preskorn SH, Baker B. Effect of study criteria on recruitment and generalizability of the results. J Clin Psychopharmacol. 2005;25:271–5.CrossRefPubMed
34.
Koeth O, Zahn R, Gitt AK, Bauer T, Juenger C, Senges J, et al. Clinical benefit of early reperfusion therapy in patients with ST-elevation myocardial infarction usually excluded from randomized clinical trials (results from the Maximal Individual Therapy in Acute Myocardial Infarction Plus [MITRA Plus] registry). Am J Cardiol. 2009;104:1074–7.CrossRefPubMed
35.
Lenzen MJ, Boersma E, Scholte Op Reimer WJM, Balk AHMM, Komajda M, Swedberg K, et al. Under-utilization of evidence-based drug treatment in patients with heart failure is only partially explained by dissimilarity to patients enrolled in landmark trials: a report from the Euro Heart Survey on Heart Failure. Eur Heart J. 2005;26:2706–13.CrossRefPubMed
36.
Masoudi FA, Havranek EP, Wolfe P, Gross CP, Rathore SS, Steiner JF, et al. Most hospitalized older persons do not meet the enrollment criteria for clinical trials in heart failure. Am Heart J. 2003;146:250–7.CrossRefPubMed
37.
Melloni C, Berger JS, Wang TY, Gunes F, Stebbins A, Pieper KS, et al. Representation of women in randomized clinical trials of cardiovascular disease prevention. Circ Cardiovasc Qual Outcomes. 2010;3:135–42.CrossRefPubMed
38.
Mengis C, Aebi S, Tobler A, Dähler W, Fey MF. Assessment of differences in patient populations selected for excluded from participation in clinical phase III acute myelogenous leukemia trials. J Clin Oncol. 2003;21:3933–9.CrossRefPubMed
39.
Somer RA, Sherman E, Langer CJ. Restrictive eligibility limits access to newer therapies in non-small-cell lung cancer: the implications of Eastern Cooperative Oncology Group 4599. Clin Lung Cancer. 2008;9:102–5.CrossRefPubMed
40.
Steg PG, López-Sendón J, Lopez De Sa E, Goodman SG, Gore JM, Anderson FA, et al. External validity of clinical trials in acute myocardial infarction. Arch Intern Med. 2007;167:68–73.CrossRefPubMed
41.
Storosum JG, Fouwels A, Gispen-de Wied CC, Wohlfarth T, Van Zwieten BJ, van den Brink W. How real are patients in placebo-controlled studies of acute manic episode? Eur Neuropsychopharmacol. 2004;14:319–23.CrossRefPubMed
42.
Surman CB, Monuteaux MC, Petty CR, Faraone SV, Spencer TJ, Chu NF, et al. Representativeness of participants in a clinical trial for attention-deficit/hyperactivity disorder? Comparison with adults from a large observational study. J Clin Psychiatry. 2010;71:1612–6.CrossRefPubMedPubMedCentral
43.
Terschüren C, Gierer S, Brillant C, Paulus U, Löffler M, Hoffmann W. Are patients with Hodgkin lymphoma and high-grade non-Hodgkin lymphoma in clinical therapy optimization protocols representative of these groups of patients in Germany? Ann Oncol. 2010;21:2045–51.CrossRefPubMed
44.
Uijen AA, Bakx JC, Mokkink HG, Van Weel C. Hypertension patients participating in trials differ in many aspects from patients treated in general practices. J Clin Epidemiol. 2007;60:330–5.CrossRefPubMed
45.
van der Linden N, Van Gils CW, Pescott CP, Buter J, Uyl-de Groot CA. Cetuximab in locally advanced squamous cell carcinoma of the head and neck: generalizability of EMR 062202–006 trial results. Eur Arch Otorhinolaryngol. 2014;271:1673–8.PubMed
46.
Vardy J, Dadasovich R, Beale P, Boyer M, Clarke SJ. Eligibility of patients with advanced non-small cell lung cancer for phase III chemotherapy trials. BMC Cancer. 2009;9:130.CrossRefPubMedPubMedCentral
47.
Wisniewski SR, Rush AJ, Nierenberg AA, Gaynes BN, Warden D, Luther JF, et al. Can phase III trial results of antidepressant medications be generalized to clinical practice? A STAR*D report. Am J Psychiatry. 2009;166:599–607.CrossRefPubMed
48.
Yennurajalingam S, Kang JH, Cheng HY, Chisholm GB, Kwon JH, Palla SL, et al. Characteristics of advanced cancer patients with cancer-related fatigue enrolled in clinical trials and patients referred to outpatient palliative care clinics. J Pain Symptom Manage. 2013;45:534–41.CrossRefPubMed
49.
Zarin DA, Young JL, West JC. Challenges to evidence-based medicine: a comparison of patients and treatments in randomized controlled trials with patients and treatments in a practice research network. Soc Psychiatry Psychiatr Epidemiol. 2005;40:27–35.CrossRefPubMed
50.
Zetin M, Hoepner CT. Relevance of exclusion criteria in antidepressant clinical trials: a replication study. J Clin Psychopharmacol. 2007;27:295–301.CrossRefPubMed
51.
Zimmerman M, Chelminski I, Posternak MA. Exclusion criteria used in antidepressant efficacy trials: consistency across studies and representativeness of samples included. J Nerv Ment Dis. 2004;192:87–94.CrossRefPubMed
52.
Baquet CR, Ellison GL, Mishra SI. Analysis of Maryland cancer patient participation in National Cancer Institute-supported cancer treatment clinical trials. J Health Care Poor Underserved. 2009;20(2 Suppl):120–34.CrossRefPubMedPubMedCentral
53.
Collet JP, Montalescot G, Fine E, Golmard J-L, Dalby M, Choussat R, et al. Enoxaparin in unstable angina patients who would have been excluded from randomized pivotal trials. J Am Coll Cardiol. 2003;41:8–14.CrossRefPubMed
54.
Filion M, Forget G, Brochu O, Provencher L, Desbien SC, Doyle C, et al. Eligibility criteria in randomized phase II and III adjuvant and neoadjuvant breast cancer trials: not a significant barrier to enrollment. Clin Trials. 2012;9:652–9.CrossRefPubMed
55.
Fortin M, Dionne J, Pinho G, Gignac J, Almirall J, Lapointe L. Randomized controlled trials: do they have external validity for patients with multiple comorbidities? Ann Fam Med. 2006;4:104–8.CrossRefPubMedPubMedCentral
56.
Krumholz HM, Gross CP, Peterson ED, Barron HV, Radford MJ, Parsons LS, et al. Is there evidence of implicit exclusion criteria for elderly subjects in randomized trials? Evidence from the GUSTO-1 study. Am Heart J. 2003;146:839–47.CrossRefPubMed
57.
Kushner SC, Quilty LC, McBride C, Bagby RM. A comparison of depressed patients in randomized vs nonrandomized trials of antidepressant medication and psychotherapy. Depress Anxiety. 2009;26:666–73.CrossRefPubMed
58.
Mol L, Koopman M, Van Gils CW, Ottevanger PB, Punt CJA. Comparison of treatment outcome in metastatic colorectal cancer patients included in a clinical trial versus daily practice in The Netherlands. Acta Oncol. 2013;52:950–5.CrossRefPubMed
59.
Rabinowitz J, Bromet EJ, Davidson M. Are patients enrolled in first episode psychosis drug trials representative of patients treated in routine clinical practice? Schizophr Res. 2003;61:149–55.CrossRefPubMed
60.
Riedel M, Strassnig M, Müller N, Zwack P, Möller H-J. How representative of everyday clinical populations are schizophrenia patients enrolled in clinical trials? Eur Arch Psychiatry Clin Neurosci. 2005;255:143–8.CrossRefPubMed
61.
Seemüller F, Möller HJ, Obermeier M, Adli M, Bauer M, Kronmüller K, et al. Do efficacy and effectiveness samples differ in antidepressant treatment outcome? An analysis of eligibility criteria in randomized controlled trials. J Clin Psychiatry. 2010;71:1425–33.CrossRefPubMed
62.
Steinberg BA, Moghbeli N, Buros J, Ruda M, Parkhomenko A, Raju BS, et al. Global outcomes of ST-elevation myocardial infarction: comparisons of the Enoxaparin and Thrombolysis Reperfusion for Acute Myocardial Infarction Treatment-Thrombolysis In Myocardial Infarction study 25 (ExTRACT-TIMI 25) registry and trial. Am Heart J. 2007;154:54–61.CrossRefPubMed
63.
Talamo A, Baldessarini RJ, Centorrino F. Comparison of mania patients suitable for treatment trials vs clinical treatment. Hum Psychopharmacol. 2008;23:447–54.CrossRefPubMedPubMedCentral
64.
van der Lem R, van der Wee NJ, Van Veen T, Zitman FG. The generalizability of antidepressant efficacy trials to routine psychiatric out-patient practice. Psychol Med. 2011;41:1353–63.CrossRefPubMed
65.
Wagner TH, Holman W, Lee K, Sethi G, Ananth L, Thai H, et al. The generalizability of participants in Veterans Affairs Cooperative Studies Program 474, a multi-site randomized cardiac bypass surgery trial. Contemp Clin Trials. 2011;32:260–6.CrossRefPubMed
66.
Yessaian A, Mendivil AA, Brewster WR. Population characteristics in cervical cancer trials: search for external validity. Am J Obstet Gynecol. 2005;192:407–13.CrossRefPubMed
67.
Garfield FB, Garfield JM. Clinical judgment and clinical practice guidelines. Int J Technol Assess Health Care. 2000;16:1050–60.CrossRefPubMed
68.
Velasco E. Inclusion criteria. In: Salkind NJ, editor. Encyclopedia of research, volume 1. Thousand Oaks: SAGE Publications, Inc; 2010. p. 589–91.
69.
Fletcher R, Fletcher SW, Fletcher GS. Chapter 9, Treatment. In: Fletcher R, Fletcher SW, Fletcher GS, editors. Clinical epidemiology: the essentials. 5th ed. Baltimore: Wolters Kluwer; 2014. p. 132–52.
70.
71.
Loudon K, Treweek S, Sullivan F, Donnan P, Thorpe KE. The PRECIS-2 tool: designing tools that are fit for purpose. BMJ. 2015;350:h2147.CrossRefPubMed
72.
73.
Prevost TC, Abrams KR, Jones DR. Hierarchical models in generalized synthesis of evidence: an example based on studies of breast cancer screening. Stat Med. 2000;19:3359–76.CrossRefPubMed
74.
75.
Stuart EA, Cole SR, Bradshaw CP, Leaf PJ. The use of propensity scores to assess the generalizability of results from randomized trials. JR Statist Soc A. 2011;174:369–86.CrossRef
76.
Pressler TR, Kaizar EE. The use of propensity scores and observational data to estimate randomized controlled trial generalizability bias. Stat Med. 2013;32:3552–68.CrossRefPubMedPubMedCentral
77.
Silverman SL. From randomized controlled trials to observational studies. Am J Med. 2009;122:114–20.CrossRefPubMed
78.
Yang W, Zilov A, Soewondo P, Bech OM, Sekkal F, Home PD. Observational studies: going beyond the boundaries of randomized controlled trial. Diabetes Res Clin Pract. 2010;88 suppl 1:S3–9.CrossRefPubMed
Metadata
Title
A literature review on the representativeness of randomized controlled trial samples and implications for the external validity of trial results
Authors
Tessa Kennedy-Martin
Sarah Curtis
Douglas Faries
Susan Robinson
Joseph Johnston
Publication date
01-12-2015
Publisher
BioMed Central
Published in
Trials / Issue 1/2015
Electronic ISSN: 1745-6215
DOI
https://doi.org/10.1186/s13063-015-1023-4

Other articles of this Issue 1/2015

Trials 1/2015 Go to the issue

Erratum

Erratum to: Disappointment and adherence among parents of newborns allocated to the control group: a qualitative study of a randomized clinical trial

Study protocol

Clinical trial to evaluate the performance of a flexible self-adherent absorbent dressing coated with a soft silicone layer compared to a standard wound dressing after orthopedic or spinal surgery: study protocol for a randomized controlled trial

Study protocol

The efficacy and safety of acupuncture for cerebral vasospasm after subarachnoid hemorrhage: study protocol for a randomized controlled trial

Study protocol

Fibrinogen concentrate versus placebo for treatment of postpartum haemorrhage: study protocol for a randomised controlled trial

Study protocol

Newborn Infection Control and Care Initiative for health facilities to accelerate reduction of newborn mortality (NICCI): study protocol for a randomized controlled trial

Study protocol

A new primary dental care service compared with standard care for child and family to reduce the re-occurrence of childhood dental caries (Dental RECUR): study protocol for a randomised controlled trial