Top

Published in:

01-12-2022 | Care | Research

Measuring implementation fidelity in a cluster-randomized pragmatic trial: development and use of a quantitative multi-component approach

Authors: Miranda B. Olson, Ellen M. McCreedy, Rosa R. Baier, Renée R. Shield, Esme E. Zediker, Rebecca Uth, Kali S. Thomas, Vincent Mor, Roee Gutman, James L. Rudolph

Published in: Trials | Issue 1/2022

Abstract

Background

In pragmatic trials, on-site partners, rather than researchers, lead intervention delivery, which may result in implementation variation. There is a need to quantitatively measure this variation. Applying the Framework for Implementation Fidelity (FIF), we develop an approach for measuring variability in site-level implementation fidelity. This approach is then applied to measure site-level fidelity in a cluster-randomized pragmatic trial of Music & Memory^SM (M&M), a personalized music intervention targeting agitated behaviors in residents living with dementia, in US nursing homes (NHs).

Methods

Intervention NHs (N = 27) implemented M&M using a standardized manual, utilizing provided staff trainings and iPods for participating residents. Quantitative implementation data, including iPod metadata (i.e., song title, duration, number of plays), were collected during baseline, 4-month, and 8-month site visits. Three researchers developed four FIF adherence dimension scores. For Details of Content, we independently reviewed the implementation manual and reached consensus on six core M&M components. Coverage was the total number of residents exposed to the music at each NH. Frequency was the percent of participating residents in each NH exposed to M&M at least weekly. Duration was the median minutes of music received per resident day exposed. Data elements were scaled and summed to generate dimension-level NH scores, which were then summed to create a Composite adherence score. NHs were grouped by tercile (low-, medium-, high-fidelity).

Results

The 27 NHs differed in size, resident composition, and publicly reported quality rating. The Composite score demonstrated significant variation across NHs, ranging from 4.0 to 12.0 [8.0, standard deviation (SD) 2.1]. Scaled dimension scores were significantly correlated with the Composite score. However, dimension scores were not highly correlated with each other; for example, the correlation of the Details of Content score with Coverage was τ_b = 0.11 (p = 0.59) and with Duration was τ_b = − 0.05 (p = 0.78). The Composite score correlated with CMS quality star rating and presence of an Alzheimer’s unit, suggesting face validity.

Conclusions

Guided by the FIF, we developed and used an approach to quantitatively measure overall site-level fidelity in a multi-site pragmatic trial. Future pragmatic trials, particularly in the long-term care environment, may benefit from this approach.

Trial registration

Clinicaltrials.gov NCT03821844. Registered on 30 January 2019, https://clinicaltrials.gov/ct2/show/NCT03821844.

Available only for authorised users

Weinfurt KP, Hernandez AF, Coronado GD, DeBar LL, Dember LM, Green BB, et al. Pragmatic clinical trials embedded in healthcare systems: generalizable lessons from the NIH Collaboratory. BMC Med Res Methodol. 2017;17(1):144 Available from: https://pubmed.ncbi.nlm.nih.gov/28923013.CrossRefPubMedPubMedCentral

Gitlin LN, Marx K, Stanley IH. Hodgson N. Translating evidence-based dementia caregiving interventions into practice: state-of-the-science and next steps. Gerontologist. 2015;55(2):210–26 Available from: https://pubmed.ncbi.nlm.nih.gov/26035597.CrossRefPubMedPubMedCentral

Chalkidou K, Tunis S, Whicher D, Fowler R, Zwarenstein M. The role for pragmatic randomized controlled trials (pRCTs) in comparative effectiveness research. Clin Trials J Soc Clin Trials. 2012;2, 9(4):–446. https://doi.org/10.1177/1740774512450097.

Curran GM, Bauer M, Mittman B, Pyne JM, Stetler C. Effectiveness-implementation hybrid designs: combining elements of clinical effectiveness and implementation research to enhance public health impact. Med Care. 2012;50(3):217–26. Available from: https://pubmed.ncbi.nlm.nih.gov/22310560. https://doi.org/10.1097/MLR.0b013e3182408812.CrossRefPubMedPubMedCentral

Glasgow RE, Magid DJ, Beck A, Ritzwoller D, Estabrooks PA. Practical clinical trials for translating research to practice: design and measurement recommendations. Med Care, 7. 2005;43(6):551 Available from: https://journals.lww.com/lww-medicalcare/Fulltext/2005/06000/Practical_Clinical_Trials_for_Translating_Research.5.aspx.

Dusenbury L, Brannigan R, Falco M, Hansen WB. A review of research on fidelity of implementation: implications for drug abuse prevention in school settings. Health Educ Res. 2003;18(2):–256. https://doi.org/10.1093/her/18.2.237.

Hill HC, Erickson A. Using implementation fidelity to aid in interpreting program impacts: a brief review. Educ Res. 2019;48(9):590–8. Available from. https://doi.org/10.3102/0013189X19891436.CrossRef

Elliott DS, Mihalic S. Issues in disseminating and replicating effective prevention programs. Prev Sci. 2004;5(1):47–53. Available from. https://doi.org/10.1023/B:PREV.0000013981.28071.52.CrossRefPubMed

Inouye SK, Bogardus ST Jr, Williams CS, Leo-Summers L, Agostini JV. The role of adherence on the effectiveness of nonpharmacologic interventions: evidence from the delirium prevention trial. Arch Intern Med. 2003;163(8):958–64. Available from. https://doi.org/10.1001/archinte.163.8.958.CrossRefPubMed

10.

Gesell SB, Bushnell CD, Jones SB, Coleman SW, Levy SM, Xenakis JG, et al. Implementation of a billable transitional care model for stroke patients: the COMPASS study. BMC Health Serv Res. 2019;19(1):978 Available from: https://pubmed.ncbi.nlm.nih.gov/31856808.CrossRefPubMedPubMedCentral

11.

Mitchell SL, Volandes AE, Gutman R, Gozalo PL, Ogarek JA, Loomer L, et al. Advance care planning video intervention among long-stay nursing home residents: a pragmatic cluster randomized clinical trial. JAMA Intern Med. 2020;180(8):1070–8. Available from. https://doi.org/10.1001/jamainternmed.2020.2366.CrossRefPubMedPubMedCentral

12.

Lin S-Y, Schneider CE, Bristol AA, Clancy M, Sprague SA, Aldridge M, et al. Findings of sequential pilot trials of Aliviado Dementia Care to inform an embedded pragmatic clinical trial. Gerontologist. 2020:gnaa220 Available from. https://doi.org/10.1093/geront/gnaa220.

13.

Axford N, Bjornstad G, Clarkson S, Ukoumunne OC, Wrigley Z, Matthews J, et al. The effectiveness of the KiVa bullying prevention program in Wales, UK: results from a pragmatic cluster randomized controlled trial. Prev Sci. 2020;21(5):615–26. Available from. https://doi.org/10.1007/s11121-020-01103-9.CrossRefPubMedPubMedCentral

14.

Toomey E, Matthews J, Guerin S, Hurley DA. Development of a feasible implementation fidelity protocol within a complex physical therapy–led self-management intervention. Phys Ther. 2016;96(8):1287–98. Available from. https://doi.org/10.2522/ptj.20150446.CrossRefPubMed

15.

Palmer JA, Parker VA, Barre LR, Mor V, Volandes AE, Belanger E, et al. Understanding implementation fidelity in a pragmatic randomized clinical trial in the nursing home setting:a mixed-methods examination. Trials. 2019;20(1):656. Available from. https://doi.org/10.1186/s13063-019-3725-5.CrossRefPubMedPubMedCentral

16.

Harris-Kojetin L, Sengupta M, Lendon J, Rome V, Valverde R, Caffrey C. Long-term care providers and services users in the United States, 2015-2016. Vital Heal Stat. 2019;3(43):23.

17.

Matthews KA, Xu W, Gaglioti AH, Holt JB, Croft JB, Mack D, et al. Racial and ethnic estimates of Alzheimer’s disease and related dementias in the United States (2015-2060) in adults aged ≥65 years. Alzheimers Dement. 2019;15(1):17–24 Available from: https://pubmed.ncbi.nlm.nih.gov/30243772.CrossRefPubMed

18.

Zuidema S, Koopmans R, Verhey F. Prevalence and predictors of neuropsychiatric symptoms in cognitively impaired nursing home patients. J Geriatr Psychiatry Neurol. 2007;29(1):–49. https://doi.org/10.1177/0891988706292762.

19.

Chiu Y, Bero L, Hessol NA, Lexchin J, Harrington C, et al. Health Policy (New York). 2015;119(6):802–13 Available from: https://www.sciencedirect.com/science/article/pii/S0168851015000652.CrossRef

20.

Why Get Certified: Music and Memory. https://musicandmemory.org. Accessed 1 January 2021.

21.

Jacobsen J-H, Stelzer J, Fritz TH, Chételat G, La Joie R, Turner R. Why musical memory can be preserved in advanced Alzheimer’s disease. Brain. 2015;138(8).

22.

Kwak J, Anderson K, O’Connell VK. Findings from a prospective randomized controlled trial of an individualized music listening program for persons with dementia. J Appl Gerontol. 2020;39(6):–575. https://doi.org/10.1177/0733464818778991.

23.

Thomas KS, Baier R, Kosar C, Ogarek J, Trepman A, Mor V, et al. Am J Geriatr Psychiatry. 2017;25(9):931–8 Available from: https://pubmed.ncbi.nlm.nih.gov/28483436.CrossRefPubMedPubMedCentral

24.

Bakerjian D, Bettega K, Cachu AM, Azzis L, Taylor S, et al. J Am Med Dir Assoc. 2020;21(8):1045–1050.e2 Available from: https://pubmed.ncbi.nlm.nih.gov/32146039.CrossRefPubMedPubMedCentral

25.

McCreedy EM, Yang X, Baier RR, Rudolph JL, Thomas KS, Mor V. Measuring effects of nondrug interventions on behaviors: Music & Memory pilot study. J Am Geriatr Soc. 2019;67(10):–2138. https://doi.org/10.1111/jgs.16069.

26.

McCreedy EM, Gutman R, Baier R, Rudolph JL, Thomas KS, Dvorchak F, et al. Measuring the effects of a personalized music intervention on agitated behaviors among nursing home residents with dementia: design features for cluster-randomized adaptive trial. Trials. 2021;22(1):681. Available from. https://doi.org/10.1186/s13063-021-05620-y.CrossRefPubMedPubMedCentral

27.

Landes SJ, McBain SA, Curran GM. An introduction to effectiveness-implementation hybrid designs. Psychiatry Res. 2019;280:112513 Available from: https://pubmed.ncbi.nlm.nih.gov/31434011.CrossRefPubMedPubMedCentral

28.

Carroll C, Patterson M, Wood S, Booth A, Rick J, Balain S. A conceptual framework for implementation fidelity. Implement Sci. 2007;2(1):40. Available from. https://doi.org/10.1186/1748-5908-2-40.CrossRefPubMedPubMedCentral

29.

MUSIC & MEMORY® Certification Training. https://musicandmemory.org/get-trained/certification-training/. Accessed 1 March 2021.

30.

Long-term Care: Facts on Care in the US. Brown University Center for Gerontology and Healthcare Research. 2019. https://ltcfocus.org. Accessed 1 January 2020.

31.

Five-Star Quality Rating System. Centers for Medicare and Medicaid Services. 2019. https://www.cms.gov/Medicare/Provider-Enrollment-and-Certification/CertificationandComplianc/FSQRS. Accessed 1 January 2020.

32.

Morris JN, Fries BE, Morris SA. Scaling ADLs Within the MDS. J Gerontol Ser A. 1999;54(11):M546–M553. Available from: https://doi.org/10.1093/gerona/54.11.M546

33.

Medicare and Medicaid Basics - MLN Booklet. Centers for Medicare and Medicaid Services. 2018. https://www.cms.gov/Outreach-and-Education/Medicare-Learning-Network-MLN/MLNProducts/Downloads/ProgramBasics.pdf. Accessed 11 December 2021.

34.

Hasson H. Systematic evaluation of implementation fidelity of complex interventions in health and social care. Implement Sci. 2010;5(1):67. Available from. https://doi.org/10.1186/1748-5908-5-67.CrossRefPubMedPubMedCentral

35.

Muntinga ME, Van Leeuwen KM, Schellevis FG, Nijpels G, Jansen APD. From concept to content: assessing the implementation fidelity of a chronic care model for frail, older people who live at home. BMC Health Serv Res. 2015;15(1):18. Available from. https://doi.org/10.1186/s12913-014-0662-6.CrossRefPubMedPubMedCentral

36.

Willeboordse F, Schellevis FG, Meulendijk MC, Hugtenburg JG, PJM E. Implementation fidelity of a clinical medication review intervention: process evaluation. Int J Clin Pharm. 2018;40(3):550–65 Available from: https://pubmed.ncbi.nlm.nih.gov/29556930.CrossRefPubMedPubMedCentral

37.

Berry CA, Nguyen AM, Cuthel AM, Cleland CM, Siman N, Pham-Singer H, et al. Measuring implementation strategy fidelity in HealthyHearts NYC: a complex intervention using practice facilitation in primary care. Am J Med Qual. 2020. Available from;36(4):270–6. https://doi.org/10.1177/1062860620959450.CrossRef

38.

Smith J, Green J, Siddiqi N, Inouye SK, Collinson M, Farrin A, et al. Investigation of ward fidelity to a multicomponent delirium prevention intervention during a multicentre, pragmatic, cluster randomised, controlled feasibility trial. Age Ageing. 2020;49(4):648–55. Available from. https://doi.org/10.1093/ageing/afaa042.CrossRefPubMedPubMedCentral

39.

Hasson H, Blomberg S, Dunér A. Fidelity and moderating factors in complex interventions: a case study of a continuum of care program for frail elderly people in health and social care. Implement Sci. 2012;7:23 Available from: https://pubmed.ncbi.nlm.nih.gov/22436121.CrossRefPubMedPubMedCentral

40.

Bragstad LK, Bronken BA, Sveen U, Hjelle EG, Kitzmüller G, Martinsen R, et al. Implementation fidelity in a complex intervention promoting psychosocial well-being following stroke: an explanatory sequential mixed methods study. BMC Med Res Methodol. 2019;19(1):59 Available from: https://pubmed.ncbi.nlm.nih.gov/30876403.CrossRefPubMedPubMedCentral

Title: Measuring implementation fidelity in a cluster-randomized pragmatic trial: development and use of a quantitative multi-component approach
Authors: Miranda B. Olson
Ellen M. McCreedy
Rosa R. Baier
Renée R. Shield
Esme E. Zediker
Rebecca Uth
Kali S. Thomas
Vincent Mor
Roee Gutman
James L. Rudolph
Publication date: 01-12-2022
Publisher: BioMed Central
Keywords: Care
Dementia
Dementia
Published in: Trials / Issue 1/2022
Electronic ISSN: 1745-6215
DOI: https://doi.org/10.1186/s13063-022-06002-8

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Measuring implementation fidelity in a cluster-randomized pragmatic trial: development and use of a quantitative multi-component approach

Abstract

Background

Methods

Results

Conclusions

Trial registration

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Abstract

Background

Methods

Results

Conclusions

Trial registration

Please log in to get access to this content

Other articles of this Issue 1/2022

Factors associated with reporting of the Prevention of Falls Network Europe (ProFaNE) core outcome set domains in randomized trials on falls in older people: a citation analysis and correlational study

Comparison of explanatory and pragmatic design choices in a cluster-randomized hypertension trial: effects on enrollment, participant characteristics, and adherence

Early essential newborn care for cesarean section newborns in China: study protocol for a multi-centered randomized controlled trial

Reshaping the path of mild cognitive impairment by refining exercise prescription: a study protocol of a randomized controlled trial to understand the “what,” “for whom,” and “how” of exercise to promote cognitive function

Financially incentivized knowledge assessments to improve provider compliance with treatment guidelines: a cluster-randomized controlled trial

Effectiveness of an adapted diving mask (Owner mask) for non-invasive ventilation in the COVID-19 pandemic scenario: study protocol for a randomized clinical trial