Skip to main content
Top
Published in: BMC Medical Research Methodology 1/2017

Open Access 01-12-2017 | Research article

Pragmatic clinical trials embedded in healthcare systems: generalizable lessons from the NIH Collaboratory

Authors: Kevin P. Weinfurt, Adrian F. Hernandez, Gloria D. Coronado, Lynn L. DeBar, Laura M. Dember, Beverly B. Green, Patrick J. Heagerty, Susan S. Huang, Kathryn T. James, Jeffrey G. Jarvik, Eric B. Larson, Vincent Mor, Richard Platt, Gary E. Rosenthal, Edward J. Septimus, Gregory E. Simon, Karen L. Staman, Jeremy Sugarman, Miguel Vazquez, Douglas Zatzick, Lesley H. Curtis

Published in: BMC Medical Research Methodology | Issue 1/2017

Login to get access

Abstract

Background

The clinical research enterprise is not producing the evidence decision makers arguably need in a timely and cost effective manner; research currently involves the use of labor-intensive parallel systems that are separate from clinical care. The emergence of pragmatic clinical trials (PCTs) poses a possible solution: these large-scale trials are embedded within routine clinical care and often involve cluster randomization of hospitals, clinics, primary care providers, etc. Interventions can be implemented by health system personnel through usual communication channels and quality improvement infrastructure, and data collected as part of routine clinical care. However, experience with these trials is nascent and best practices regarding design operational, analytic, and reporting methodologies are undeveloped.

Methods

To strengthen the national capacity to implement cost-effective, large-scale PCTs, the Common Fund of the National Institutes of Health created the Health Care Systems Research Collaboratory (Collaboratory) to support the design, execution, and dissemination of a series of demonstration projects using a pragmatic research design.

Results

In this article, we will describe the Collaboratory, highlight some of the challenges encountered and solutions developed thus far, and discuss remaining barriers and opportunities for large-scale evidence generation using PCTs.

Conclusion

A planning phase is critical, and even with careful planning, new challenges arise during execution; comparisons between arms can be complicated by unanticipated changes. Early and ongoing engagement with both health care system leaders and front-line clinicians is critical for success. There is also marked uncertainty when applying existing ethical and regulatory frameworks to PCTS, and using existing electronic health records for data capture adds complexity.
Literature
1.
go back to reference Koh C, Zhao X, Samala N, Sakiani S, Liang TJ, Talwalkar JA. AASLD clinical practice guidelines: a critical review of scientific evidence and evolving recommendations. Hepatology. 2013;58:2142–52.CrossRefPubMedPubMedCentral Koh C, Zhao X, Samala N, Sakiani S, Liang TJ, Talwalkar JA. AASLD clinical practice guidelines: a critical review of scientific evidence and evolving recommendations. Hepatology. 2013;58:2142–52.CrossRefPubMedPubMedCentral
2.
go back to reference Roos M, Brodbeck J, Sarkozy A, Chierchia GB, De Asmundis C, Brugada P. A critical analysis of the scientific evidence behind international guidelines related to cardiac arrhythmias. Circ Arrhythm Electrophysiol. 2011;4:202–10.CrossRefPubMed Roos M, Brodbeck J, Sarkozy A, Chierchia GB, De Asmundis C, Brugada P. A critical analysis of the scientific evidence behind international guidelines related to cardiac arrhythmias. Circ Arrhythm Electrophysiol. 2011;4:202–10.CrossRefPubMed
3.
go back to reference Neuman MD, Goldstein JN, Cirullo MA, Schwartz JS. Durability of class I American College of Cardiology/American Heart Association clinical practice guideline recommendations. JAMA. 2014;311:2092–100.CrossRefPubMedPubMedCentral Neuman MD, Goldstein JN, Cirullo MA, Schwartz JS. Durability of class I American College of Cardiology/American Heart Association clinical practice guideline recommendations. JAMA. 2014;311:2092–100.CrossRefPubMedPubMedCentral
4.
go back to reference Wright JD, Pawar N, Gonzalez JSR, Lewin SN, Burke WM, Simpson LL, et al. Scientific evidence underlying the American College of Obstetricians and Gynecologists’ practice bulletins. Obstet Gynecol. 2011;118:505–12.CrossRefPubMed Wright JD, Pawar N, Gonzalez JSR, Lewin SN, Burke WM, Simpson LL, et al. Scientific evidence underlying the American College of Obstetricians and Gynecologists’ practice bulletins. Obstet Gynecol. 2011;118:505–12.CrossRefPubMed
5.
go back to reference Feuerstein JD, Akbari M, Gifford AE, Hurley CM, Leffler DA, Sheth SG, et al. Systematic analysis underlying the quality of the scientific evidence and conflicts of interest in interventional medicine subspecialty guidelines. Mayo Clin Proc. 2014;89:16–24.CrossRefPubMed Feuerstein JD, Akbari M, Gifford AE, Hurley CM, Leffler DA, Sheth SG, et al. Systematic analysis underlying the quality of the scientific evidence and conflicts of interest in interventional medicine subspecialty guidelines. Mayo Clin Proc. 2014;89:16–24.CrossRefPubMed
6.
go back to reference Tricoci P, Allen JM, Kramer JM, Califf RM, Smith SC. Scientific evidence underlying the ACC/AHA clinical practice guidelines. JAMA. 2009;301:831–41.CrossRefPubMed Tricoci P, Allen JM, Kramer JM, Califf RM, Smith SC. Scientific evidence underlying the ACC/AHA clinical practice guidelines. JAMA. 2009;301:831–41.CrossRefPubMed
7.
go back to reference Han H, Chao H, Guerra A, Sosa A, Christopoulos G, Christakopoulos GE, et al. Evolution of the American College of Cardiology/American Heart Association clinical guidelines. J Am Coll Cardiol. 2015;65:2726–34.CrossRefPubMed Han H, Chao H, Guerra A, Sosa A, Christopoulos G, Christakopoulos GE, et al. Evolution of the American College of Cardiology/American Heart Association clinical guidelines. J Am Coll Cardiol. 2015;65:2726–34.CrossRefPubMed
8.
go back to reference Califf RM. The cycle of quality as a model for improving health outcomes in the treatment of hypertension. European Heart Journal Supplements. 2007;9:B8–12.CrossRef Califf RM. The cycle of quality as a model for improving health outcomes in the treatment of hypertension. European Heart Journal Supplements. 2007;9:B8–12.CrossRef
9.
go back to reference Crowley WF Jr, Sherwood L, Salber P, Scheinberg D, Slavkin H, Tilson H, et al. Clinical research in the United States at a crossroads: proposal for a novel public-private partnership to establish a national clinical research enterprise. JAMA. 2004;291:1120–6.CrossRefPubMed Crowley WF Jr, Sherwood L, Salber P, Scheinberg D, Slavkin H, Tilson H, et al. Clinical research in the United States at a crossroads: proposal for a novel public-private partnership to establish a national clinical research enterprise. JAMA. 2004;291:1120–6.CrossRefPubMed
10.
go back to reference Alper J, Grossmann C. Integrating research and practice: health system leaders working toward high-value care: workshop summary. Washington, D.C: The National Academies Press; 2015. Alper J, Grossmann C. Integrating research and practice: health system leaders working toward high-value care: workshop summary. Washington, D.C: The National Academies Press; 2015.
11.
go back to reference Committee on the Learning Health Care System in America, Institute of Medicine. Best Care at Lower Cost: The Path to Continuously Learning Health Care in America [Internet]. Smith M, Saunders R, Stuckhardt L, McGinnis JM, editors. Washington (DC): National Academies Press (US); 2013 [cited 2015 Jan 28] Available from: http://www.ncbi.nlm.nih.gov/books/NBK207225/ Committee on the Learning Health Care System in America, Institute of Medicine. Best Care at Lower Cost: The Path to Continuously Learning Health Care in America [Internet]. Smith M, Saunders R, Stuckhardt L, McGinnis JM, editors. Washington (DC): National Academies Press (US); 2013 [cited 2015 Jan 28] Available from: http://​www.​ncbi.​nlm.​nih.​gov/​books/​NBK207225/​
12.
go back to reference Ford I, Norrie J. Pragmatic trials. Drazen JM, Harrington DP, McMurray JJV, Ware JH, Woodcock J, editors. N Engl J Med 2016;375:454–463. Ford I, Norrie J. Pragmatic trials. Drazen JM, Harrington DP, McMurray JJV, Ware JH, Woodcock J, editors. N Engl J Med 2016;375:454–463.
13.
go back to reference Loudon K, Treweek S, Sullivan F, Donnan P, Thorpe KE, Zwarenstein M. The PRECIS-2 tool: designing trials that are fit for purpose. BMJ. 2015;350:h2147.CrossRefPubMed Loudon K, Treweek S, Sullivan F, Donnan P, Thorpe KE, Zwarenstein M. The PRECIS-2 tool: designing trials that are fit for purpose. BMJ. 2015;350:h2147.CrossRefPubMed
14.
go back to reference Thorpe KE, Zwarenstein M, Oxman AD, Treweek S, Furberg CD, Altman DG, et al. A pragmatic-explanatory continuum indicator summary (PRECIS): a tool to help trial designers. J Clin Epidemiol. 2009;62:464–75.CrossRefPubMed Thorpe KE, Zwarenstein M, Oxman AD, Treweek S, Furberg CD, Altman DG, et al. A pragmatic-explanatory continuum indicator summary (PRECIS): a tool to help trial designers. J Clin Epidemiol. 2009;62:464–75.CrossRefPubMed
15.
go back to reference O’Rourke PP, Carrithers J, Patrick-Lake B, Rice TW, Corsmo J, Hart R, et al. Harmonization and streamlining of research oversight for pragmatic clinical trials. Clin Trials. 2015;12:449–56.CrossRefPubMedPubMedCentral O’Rourke PP, Carrithers J, Patrick-Lake B, Rice TW, Corsmo J, Hart R, et al. Harmonization and streamlining of research oversight for pragmatic clinical trials. Clin Trials. 2015;12:449–56.CrossRefPubMedPubMedCentral
17.
go back to reference Sugarman J. Ethics of research in usual care settings: data on point. AJOB Empirical Bioethics. 2016;7:71–5.CrossRef Sugarman J. Ethics of research in usual care settings: data on point. AJOB Empirical Bioethics. 2016;7:71–5.CrossRef
18.
go back to reference Weinfurt KP, Bollinger JM, Brelsford KM, Crayton TJ, Topazian RJ, Kass NE, et al. Patients’ views concerning research on medical practices: implications for consent. AJOB Empirical Bioethics. 2016;7:76–91.CrossRefPubMed Weinfurt KP, Bollinger JM, Brelsford KM, Crayton TJ, Topazian RJ, Kass NE, et al. Patients’ views concerning research on medical practices: implications for consent. AJOB Empirical Bioethics. 2016;7:76–91.CrossRefPubMed
20.
go back to reference Smalley JB, Merritt MW, Al-Khatib SM, McCall D, Staman KL, Stepnowsky C. Ethical responsibilities toward indirect and collateral participants in pragmatic clinical trials. Clinical Trials. 2015;12:476–84.CrossRefPubMedPubMedCentral Smalley JB, Merritt MW, Al-Khatib SM, McCall D, Staman KL, Stepnowsky C. Ethical responsibilities toward indirect and collateral participants in pragmatic clinical trials. Clinical Trials. 2015;12:476–84.CrossRefPubMedPubMedCentral
21.
go back to reference Finkelstein JA, Brickman AL, Capron A, Ford DE, Gombosev A, Greene SM, et al. Oversight on the borderline: quality improvement and pragmatic research. Clinical Trials. 2015;12:457–66.CrossRefPubMedPubMedCentral Finkelstein JA, Brickman AL, Capron A, Ford DE, Gombosev A, Greene SM, et al. Oversight on the borderline: quality improvement and pragmatic research. Clinical Trials. 2015;12:457–66.CrossRefPubMedPubMedCentral
22.
go back to reference McGraw D, Greene SM, Miner CS, Staman KL, Welch MJ, Rubel A. Privacy and confidentiality in pragmatic clinical trials. Clinical Trials. 2015;12:520–9.CrossRefPubMedPubMedCentral McGraw D, Greene SM, Miner CS, Staman KL, Welch MJ, Rubel A. Privacy and confidentiality in pragmatic clinical trials. Clinical Trials. 2015;12:520–9.CrossRefPubMedPubMedCentral
23.
go back to reference McKinney RE, Beskow LM, Ford DE, Lantos JD, McCall J, Patrick-Lake B, et al. Use of altered informed consent in pragmatic clinical research. Clin Trials. 2015;12:494–502.CrossRefPubMedPubMedCentral McKinney RE, Beskow LM, Ford DE, Lantos JD, McCall J, Patrick-Lake B, et al. Use of altered informed consent in pragmatic clinical research. Clin Trials. 2015;12:494–502.CrossRefPubMedPubMedCentral
24.
go back to reference Ali J, Andrews JE, Somkin CP, Rabinovich CE. Harms, benefits, and the nature of interventions in pragmatic clinical trials. Clinical Trials. 2015;12:467–75.CrossRefPubMedPubMedCentral Ali J, Andrews JE, Somkin CP, Rabinovich CE. Harms, benefits, and the nature of interventions in pragmatic clinical trials. Clinical Trials. 2015;12:467–75.CrossRefPubMedPubMedCentral
25.
go back to reference Anderson ML, Griffin J, Goldkind SF, Zeitler EP, Wing L, Al-Khatib SM, et al. The Food and Drug Administration and pragmatic clinical trials of marketed medical products. Clinical Trials. 2015;12:511–9.CrossRefPubMedPubMedCentral Anderson ML, Griffin J, Goldkind SF, Zeitler EP, Wing L, Al-Khatib SM, et al. The Food and Drug Administration and pragmatic clinical trials of marketed medical products. Clinical Trials. 2015;12:511–9.CrossRefPubMedPubMedCentral
26.
go back to reference Ellenberg SS, Culbertson R, Gillen DL, Goodman S, Schrandt S, Zirkle M. Data monitoring committees for pragmatic clinical trials. Clinical Trials. 2015;12:530–6.CrossRefPubMedPubMedCentral Ellenberg SS, Culbertson R, Gillen DL, Goodman S, Schrandt S, Zirkle M. Data monitoring committees for pragmatic clinical trials. Clinical Trials. 2015;12:530–6.CrossRefPubMedPubMedCentral
27.
go back to reference Lantos JD, Wendler D, Septimus E, Wahba S, Madigan R, Bliss G. Considerations in the evaluation and determination of minimal risk in pragmatic clinical trials. Clinical Trials. 2015;12:485–93.CrossRefPubMedPubMedCentral Lantos JD, Wendler D, Septimus E, Wahba S, Madigan R, Bliss G. Considerations in the evaluation and determination of minimal risk in pragmatic clinical trials. Clinical Trials. 2015;12:485–93.CrossRefPubMedPubMedCentral
28.
go back to reference Welch MJ, Lally R, Miller JE, Pittman S, Brodsky L, Caplan AL, et al. The ethics and regulatory landscape of including vulnerable populations in pragmatic clinical trials. Clinical Trials. 2015;12:503–10.CrossRefPubMedPubMedCentral Welch MJ, Lally R, Miller JE, Pittman S, Brodsky L, Caplan AL, et al. The ethics and regulatory landscape of including vulnerable populations in pragmatic clinical trials. Clinical Trials. 2015;12:503–10.CrossRefPubMedPubMedCentral
29.
go back to reference Whicher DM, Miller JE, Dunham KM, Califf RM, Joffe S. Gatekeepers for pragmatic clinical trials. Clinical Trials. 2015;12:442–8. Whicher DM, Miller JE, Dunham KM, Califf RM, Joffe S. Gatekeepers for pragmatic clinical trials. Clinical Trials. 2015;12:442–8.
30.
go back to reference Locklear T, DeBar LL, Willig J, Rundell S, Blackhall L, Zatzick D, et al. Case Studies from the Clinic: Initiating and Implementing Patient-Reported Outcome Measures. eGEMs (Generating Evidence & Methods to improve patient outcomes). 2017;5:1. Locklear T, DeBar LL, Willig J, Rundell S, Blackhall L, Zatzick D, et al. Case Studies from the Clinic: Initiating and Implementing Patient-Reported Outcome Measures. eGEMs (Generating Evidence & Methods to improve patient outcomes). 2017;5:1.
31.
go back to reference Richesson RL, Hammond WE, Nahm M, Wixted D, Simon GE, Robinson JG, et al. Electronic health records based phenotyping in next-generation clinical trials: a perspective from the NIH health care systems Collaboratory. J Am Med Inform Assoc. 2013;20(e2):e226–31. Richesson RL, Hammond WE, Nahm M, Wixted D, Simon GE, Robinson JG, et al. Electronic health records based phenotyping in next-generation clinical trials: a perspective from the NIH health care systems Collaboratory. J Am Med Inform Assoc. 2013;20(e2):e226–31.
32.
go back to reference Rachel L. Richesson, Beverly B. Green, Reesa Laws, Jon puro, Michael G. Kahn, Alan Bauck, et al. Pragmatic (trial) informatics: a perspective from the NIH health care systems research Collaboratory. J Am Med Inform Assoc. 2017;24(5):996–1001. Rachel L. Richesson, Beverly B. Green, Reesa Laws, Jon puro, Michael G. Kahn, Alan Bauck, et al. Pragmatic (trial) informatics: a perspective from the NIH health care systems research Collaboratory. J Am Med Inform Assoc. 2017;24(5):996–1001.
33.
go back to reference Zozus MN, Richesson RL, Walden A, Tenenbaum JD, Hammond WE. Research reproducibility in longitudinal multi-center studies using data from electronic health records. AMIA Jt Summits Transl Sci Proc. 2016;2016:279–85.PubMedPubMedCentral Zozus MN, Richesson RL, Walden A, Tenenbaum JD, Hammond WE. Research reproducibility in longitudinal multi-center studies using data from electronic health records. AMIA Jt Summits Transl Sci Proc. 2016;2016:279–85.PubMedPubMedCentral
34.
go back to reference Johnson KE, Tachibana C, Coronado GD, Dember LM, Glasgow RE, Huang SS, et al. A guide to research partnerships for pragmatic clinical trials. BMJ. 2014;349:g6826.CrossRefPubMedPubMedCentral Johnson KE, Tachibana C, Coronado GD, Dember LM, Glasgow RE, Huang SS, et al. A guide to research partnerships for pragmatic clinical trials. BMJ. 2014;349:g6826.CrossRefPubMedPubMedCentral
35.
go back to reference Larson EB, Tachibana C, Thompson E, Coronado GD, DeBar L, Dember LM, et al. Trials without tribulations: Minimizing the burden of pragmatic research on healthcare systems. Healthcare. 2015;4:138–44. Larson EB, Tachibana C, Thompson E, Coronado GD, DeBar L, Dember LM, et al. Trials without tribulations: Minimizing the burden of pragmatic research on healthcare systems. Healthcare. 2015;4:138–44.
37.
go back to reference Moloney RM, Tambor ES, Tunis SR. Patient and clinician support for the learning healthcare system: recommendations for enhancing value. J Comp Eff Res. 2016;5:123–8. Moloney RM, Tambor ES, Tunis SR. Patient and clinician support for the learning healthcare system: recommendations for enhancing value. J Comp Eff Res. 2016;5:123–8.
38.
go back to reference Li F, Lokhnygina Y, Murray DM, Heagerty PJ, DeLong ER. An evaluation of constrained randomization for the design and analysis of group-randomized trials. Stat Med. 2016;35:1565–79.CrossRefPubMed Li F, Lokhnygina Y, Murray DM, Heagerty PJ, DeLong ER. An evaluation of constrained randomization for the design and analysis of group-randomized trials. Stat Med. 2016;35:1565–79.CrossRefPubMed
39.
go back to reference Cook AJ, Delong E, Murray DM, Vollmer WM, Heagerty PJ. Statistical lessons learned for designing cluster randomized pragmatic clinical trials from the NIH health care systems Collaboratory biostatistics and design Core. Clinical Trials. 2016;13:504–12.CrossRefPubMed Cook AJ, Delong E, Murray DM, Vollmer WM, Heagerty PJ. Statistical lessons learned for designing cluster randomized pragmatic clinical trials from the NIH health care systems Collaboratory biostatistics and design Core. Clinical Trials. 2016;13:504–12.CrossRefPubMed
40.
go back to reference Langley GJ, editor. The improvement guide: a practical approach to enhancing organizational performance. 2nd ed. San Francisco: Jossey-Bass; 2009. Langley GJ, editor. The improvement guide: a practical approach to enhancing organizational performance. 2nd ed. San Francisco: Jossey-Bass; 2009.
43.
go back to reference Topazian R, Bollinger J, Weinfurt KP, Dvoskin R, Mathews D, Brelsford K, et al. Physicians’ perspectives regarding pragmatic clinical trials. J Comp Eff Res. 2016;5:499–506.CrossRefPubMedPubMedCentral Topazian R, Bollinger J, Weinfurt KP, Dvoskin R, Mathews D, Brelsford K, et al. Physicians’ perspectives regarding pragmatic clinical trials. J Comp Eff Res. 2016;5:499–506.CrossRefPubMedPubMedCentral
46.
go back to reference Sugarman J, Califf RM. Ethics and regulatory complexities for pragmatic clinical trials. JAMA. 2014;311:2381–2.CrossRefPubMed Sugarman J, Califf RM. Ethics and regulatory complexities for pragmatic clinical trials. JAMA. 2014;311:2381–2.CrossRefPubMed
49.
go back to reference Neff KM, Nawarskas JJ. Hydrochlorothiazide versus chlorthalidone in the management of hypertension. Cardiol Rev. 2010;18:51–6.CrossRefPubMed Neff KM, Nawarskas JJ. Hydrochlorothiazide versus chlorthalidone in the management of hypertension. Cardiol Rev. 2010;18:51–6.CrossRefPubMed
Metadata
Title
Pragmatic clinical trials embedded in healthcare systems: generalizable lessons from the NIH Collaboratory
Authors
Kevin P. Weinfurt
Adrian F. Hernandez
Gloria D. Coronado
Lynn L. DeBar
Laura M. Dember
Beverly B. Green
Patrick J. Heagerty
Susan S. Huang
Kathryn T. James
Jeffrey G. Jarvik
Eric B. Larson
Vincent Mor
Richard Platt
Gary E. Rosenthal
Edward J. Septimus
Gregory E. Simon
Karen L. Staman
Jeremy Sugarman
Miguel Vazquez
Douglas Zatzick
Lesley H. Curtis
Publication date
01-12-2017
Publisher
BioMed Central
Published in
BMC Medical Research Methodology / Issue 1/2017
Electronic ISSN: 1471-2288
DOI
https://doi.org/10.1186/s12874-017-0420-7

Other articles of this Issue 1/2017

BMC Medical Research Methodology 1/2017 Go to the issue