Top

Administration and Policy in Mental Health and Mental Health Services Research

Published in:

Open Access 01-03-2011 | Original Paper

Outcomes for Implementation Research: Conceptual Distinctions, Measurement Challenges, and Research Agenda

Authors: Enola Proctor, Hiie Silmere, Ramesh Raghavan, Peter Hovmand, Greg Aarons, Alicia Bunger, Richard Griffey, Melissa Hensley

Published in: Administration and Policy in Mental Health and Mental Health Services Research | Issue 2/2011

Abstract

An unresolved issue in the field of implementation research is how to conceptualize and evaluate successful implementation. This paper advances the concept of “implementation outcomes” distinct from service system and clinical treatment outcomes. This paper proposes a heuristic, working “taxonomy” of eight conceptually distinct implementation outcomes—acceptability, adoption, appropriateness, feasibility, fidelity, implementation cost, penetration, and sustainability—along with their nominal definitions. We propose a two-pronged agenda for research on implementation outcomes. Conceptualizing and measuring implementation outcomes will advance understanding of implementation processes, enhance efficiency in implementation research, and pave the way for studies of the comparative effectiveness of implementation strategies.

Aarons, G. A. (2004). Mental health provider attitudes toward adoption of evidence-based practice: The evidence-based practice attitude scale (EBPAS). Mental Health Service Research, 6(2), 61–74.CrossRef

Aarons, G. A., & Palinkas, L. A. (2007). Implementation of evidence-based practice in child welfare: Service provider perspectives. Mental Health Services Research, 34, 411–419.CrossRef

Adily, A., Westbrook, J., Coiera, E., & Ward, J. (2004). Use of on-line evidence databases by Australian public health practitioners. Medical Informatics & the Internet in Medicine, 29(2), 127–136.CrossRef

Bartholomew, N. G., Joe, G. W., Rowan-Szal, G. A., & Simpson, D. D. (2007). Counselor assessments of training and adoption barriers. Journal of Substance Abuse Treatment, 33, 193–199.PubMedCrossRef

Bates, D. M., & Sarkar, D. (2007). lme4: Linear mixed-effects models using S4 classes. R Package Version, 0, 99875–99876.

Berwick, D. M. (2003). Disseminating innovations in health care. Journal of the American Medical Association, 289(15), 1969–1975.PubMedCrossRef

Bond, G. R., McHugo, G. J., Becker, D. R., Rapp, C. A., & Whitley, R. (2008). Fidelity of supported employment: Lessons learned from the national evidence-based practice project. Psychiatric Rehabilitation Journal, 31(4), 300–305.PubMedCrossRef

Callum, P., Iger, J., Ray, M., Sims, C. A., & Falk, R. E. (2010). Outcome and experience of implementing spinal muscular atrophy carrier screening on sperm donors. Fertility and Sterility, 94(5), 1912–1914.PubMedCrossRef

Chamberlain, P., Brown, C. H., Saldana, L., Reid, J., Wang, W., Marsenich, L., et al. (2008). Engaging and recruiting counties in an experiment on implementing evidence-based practice in California. Administration and Policy In Mental Health, 35, 250–260.PubMedCrossRef

Cooke, M., Mattick, R. P., & Walsh, R. A. (2001). Implementation of the ‘fresh start’ smoking cessation programme to 23 antenatal clinics: A randomized controlled trial investigating two methods of dissemination. Drug and Alcohol Review, 20, 19–28.CrossRef

Dane, A. V., & Schneider, B. H. (1998). Program integrity in primary and early secondary prevention: Are implementation effects out of control? Clinical Psychology Review, 18, 23–45.PubMedCrossRef

Davis, F. (1993). User acceptance of information technology: System characteristics, user perceptions and behavioral impacts. International Journal of Man-Machine Studies, 38, 475–487.CrossRef

Dusenbury, L., Brannigan, R., Falco, M., & Hansen, W. B. (2003). A review of research on fidelity of implementation: Implications for drug abuse prevention in school settings. Health Education Research, 18(2), 237–256.PubMedCrossRef

Fischer, M. A., Vogeli, C., Stedman, M. R., Ferris, T. G., & Weissman, J. S. (2008). Uptake of electronic prescribing in community-based practices. Journal of General Internal Medicine, 23(4), 358–363.PubMedCrossRef

Fixsen, D. L., Naoom, S. F., Blasé, K. A., Friedman, R. M., & Wallace, F. (2005). Implementation research: A synthesis of the literature (No. FMHI Publication #231). Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute, National Implementation Research Network.

Frambach, R. T., & Schillewaert, N. (2002). Organizational innovation adoption: A multi-level framework of determinants and opportunities for future research. Journal of Business Research Special Issue: Marketing Theory in the Next Millennium, 55, 163–176.

Glasgow, R. E. (2007a). eHealth evaluation and dissemination research. American Journal of Preventive Medicine, 32(5S), S119–S126.PubMedCrossRef

Glasgow, R. E. (2007b). The RE-AIM model for planning, evaluation and reporting on implementation and dissemination research. Paper presented at the NIH Conference on Building the Science of Dissemination and Implementation in the Service of Public Health: Bethesda, MD.

Glasgow, R. E., Vogt, T. M., & Boles, S. M. (1999). Evaluating the public health impact of health promotion interventions: The RE-AIM framework. American Journal of Public Health, 89, 1322–1327.PubMedCrossRef

Goodman, R. M., McLeroy, K. R., Steckler, A. B., & Hoyle, R. H. (1993). Development of level of institutionalization scales for health promotion programs. Health Education Quarterly, 20(2), 161–179.PubMed

Graham, I. D., Brouwers, M., Davies, C., & Tetroe, J. (2007). Ontario doctors’ attitudes toward and use of clinical practice guidelines in oncology. Journal of Evaluation in Clinical Practice, 13(4), 607–615.PubMedCrossRef

Grimshaw, J., Eccles, M., Thomas, R., MacLennan, G., Ramsay, C., Fraser, C., et al. (2006). Toward evidence-based quality improvement: Evidence (and its limitations) of the effectiveness of guideline dissemination and implementation strategies 1966–1998. Journal of General Internal Medicine, 21, S14–S20.PubMed

Haug, N. A., Shopshire, M., Tajima, B., Gruber, V., & Guydish, J. (2008). Adoption of evidence-based practices among substance abuse treatment providers. Journal of Drug Education, 38(2), 181–192.PubMedCrossRef

Hayes, S. C. (1998). President’s message: Dissemination research now. The Behavior Therapist, 21(9), 166–169.

Henggeler, S. W., Lee, T., & Burns, J. A. (2002). What happens after the innovation is identified? Clinical Psychology: Science & Practice, 9(2), 191–194.CrossRef

Henggeler, S. W., Chapman, J. E., Rowland, M. D., Halliday-Boykinds, C. A., Randall, J., Shackelford, J., et al. (2008). Statewide adoption and initial implementation of contingency management for substance-abusing adolescents. Journal of Consulting and Clinical Psychology, 76(4), 556–567.PubMedCrossRef

Hides, L., Lubman, D. I., Elkins, K., Catania, L. S., & Rogers, N. (2007). Feasibility and acceptability of a mental health screening tool and training programme in the youth alcohol and other drug (AOD) sector. Drug and Alcohol Review, 26, 509–515.PubMedCrossRef

Hogue, A., Liddle, H. A., & Rowe, C. (1996). Treatment adherence process research in family therapy: A rationale and some practical guidelines. Psychotherapy, 33(2), 332–345.

Hogue, A., Dauber, S., Chincilla, P., Fried, A., Henderson, C., Inclan, J., et al. (2008). Assessing fidelity in individual and family therapy for adolescent substance abuse. Journal of Substance Abuse Treatment, 35, 137–147.PubMedCrossRef

Hovmand, P. S., & Gillespie, D. F. (2010). Implementation of evidence based practice and organizational performance. Journal of Behavioral Health Services & Research, 37(1), 79–94.CrossRef

Institute of Medicine Committee on Crossing the Quality Chasm. (2006). Adaption to mental health and addictive disorder: Improving the quality of health care for mental and substance-use conditions. Washington, DC: Institute of Medicine, National Academies Press.

Institute of Medicine Committee on Quality of Health Care in America. (2001). Crossing the quality chasm: A new health system for the 21st century. Washington, DC: Institute of Medicine, National Academy Press.

Johnson, K., Hays, C., Center, H., & Daley, C. (2004). Building capacity and sustainable prevention innovations: A sustainability planning model. Evaluation Program Planning, 27, 135–149.CrossRef

Jones, K. (1997). Mission drift in qualitative research, or moving toward a systematic review of qualitative studies. The Qualitative Report, 9(1), 95–112.

Karlsson, A., & Bendtsen, P. (2005). Acceptability of a computerized alcohol screening and advice routine in an emergency department setting―a patient perspective. Addictive Behaviors, 30, 767–776.PubMedCrossRef

Karsh, B. T. (2004). Beyond usability: Designing effective technology implementation systems to promote patient safety. Quality and Safety in Health Care, 13, 388–394.PubMedCrossRef

Kirkevoid, M. (1997). Integrative nursing research—an important strategy to further the development of nursing science and practice. Journal of Advanced Nursing, 25, 977–984.CrossRef

Klein, K. J., & Knight, A. P. (2005). Innovation implementation: Overcoming the challenge. Current Directions in Psychological Science, 14(5), 243–246.CrossRef

Klein, K. J., & Sorra, J. S. (1996). The challenge of innovation implementation. (Special topic forum on the management of innovation). Academy of Management Review, 21(4), 1055–1081.CrossRef

Luke, D. A. (2004). Multilevel modeling. Sage University Papers Series on Quantitative Applications in the Social Sciences. Thousand Oaks, CA: Sage.

McHugh, R. K., Otto, M. W., Barlow, D. H., Gorman, J. M., Shear, M. K., & Woods, S. W. (2007). Cost-efficacy of individual and combined treatments for panic disorder. The Journal of Clinical Psychiatry, 68(7), 1038–1044.PubMedCrossRef

McPheeters, M. L., Briss, P., Teutsch, S. J., & Truman, B. (2006). Systematic reviews in public health. In R. C. Brownson & D. B. Petitti (Eds.), Applied epidemiology: Theory to practice (2nd ed., pp. 99–124). New York: Oxford University Press.CrossRef

Michie, S., Johnson, M., Abraham, C., Lawton, R., Parker, D., & Walker, A. (2005). Making psychological theory useful for implementing evidence based practice: A consensus approach. Quality and Safety in Health Care, 14(1), 26–33.PubMedCrossRef

Michie, S., Pilling, S., Garety, P., Whitty, P., Eccles, M. P., Johnston, M., et al. (2007). Difficulties implementing a mental health guideline: An exploratory investigation using psychosocial theory. Implementation Science, 2(8), 1–8.

Michie, S., Fixsen, D., Grimshaw, J., & Eccles, M. (2009). Specifying and reporting complex behavior change interventions: The need for a scientific method. Implementation Science, 4(1), 40.PubMedCrossRef

Mihalic, S. (2004). The importance of implementation fidelity. Emotional and Behavioral Disorders in Youth, 4, 83–86.

Proctor, E. K., Landsverk, J., Aarons, G., Chambers, D., Glisson, C., & Mittman, B. (2009). Implementation research in mental health services: An emerging science with conceptual, methodological, and training challenges. Administration and Policy in Mental Health and Mental Health Services Research, 36(1), 24–34.PubMedCrossRef

Rabin, B. A., Brownson, R. C., Haire-Joshu, D., Kreuter, M. W., & Weaver, N. L. (2008). A glossary for dissemination and implementation research in health. Journal of Public Health Management and Practice, 14(2), 117–123.PubMed

Raghavan, R., Bright, C., & Shadoin, A. (2008). Toward a policy ecology of implementation of evidence-based practices in public mental health settings. Implementation Science, 3, 26.PubMedCrossRef

Repenning, N. P. (2002). A simulation-based approach to understanding the dynamics of innovation implementation. Organization Science, 13(2), 109–127.CrossRef

Rogers, E. M. (1995). Diffusion of innovations (4th ed.). New York: The Free Press.

Ronckers, E., Groot, W., Steenbakkers, M., Ruland, E., & Ament, A. (2006). Costs of the ‘hartslag limburg’ community heart health intervention. BMC Public Health, 6(1), 51.PubMedCrossRef

Rosen, A., & Proctor, E. K. (1981). Distinctions between treatment outcomes and their implications for treatment evaluation. Journal of Consulting and Clinical Psychology, 49(3), 418–425.PubMedCrossRef

Rye, C. B., & Kimberly, J. R. (2007). The adoption of innovations by provider organizations in health care. Medical Care Research and Review, 64(3), 235–278.PubMedCrossRef

Schoenwald, S. K., Letourneau, E. J., & Halliday-Boykins, C. A. (2005). Predicting therapist adherence to a transported family-based treatment for youth. Journal of Clinical Child & Adolescent Psychology, 34, 658–670.CrossRef

Shumway, M., Saunders, T., Shern, D., Pines, E., Downs, A., Burbine, T., et al. (2003). Preferences for schizophrenia treatment outcomes among public policy makers, consumers, families, and provides. Psychiatric Services, 54, 1124–1128.PubMedCrossRef

Steckler, A., Goodman, R., & McElory, K. (1992). Measuring the diffusion of innovative health promotion programs. American Journal of Health Promotion, 6, 214–224.PubMed

Stiles, P. G., Boothroyd, R. A., Snyder, K., & Zong, X. (2002). Service penetration by persons with severe mental illness: How should it be measured? Journal of Behavioral Health Services & Research, 29(2), 198.CrossRef

Turner, K. M. T., & Sanders, M. R. (2006). Dissemination of evidence-based parenting and family support strategies: Learning from the triple p-positive parenting program system approach. Aggression and Violent Behavior, 11(2), 176–193.CrossRef

Waldorff, F., Steenstrup, A., Nielsen, B., Rubak, J., & Bro, F. (2008). Diffusion of an e-learning programme among Danish general practitioners: A nation-wide prospective survey. BMC Family Practice, 9(1), 24.PubMedCrossRef

Wilkie, D. J., Judge, M. K. M., Berry, D. L., Dell, J., Zong, S., Gilespie, R., et al. (2003). Usability of a computerized PAINReportIt in the general public with pain and people with cancer pain. Journal of Pain and Symptom Management, 25(3), 213–224.PubMedCrossRef

Woolf, S. H. (2008). The meaning of translational research and why it matters. Journal of the American Medical Association, 299(2), 211–213.PubMedCrossRef

Title: Outcomes for Implementation Research: Conceptual Distinctions, Measurement Challenges, and Research Agenda
Authors: Enola Proctor
Hiie Silmere
Ramesh Raghavan
Peter Hovmand
Greg Aarons
Alicia Bunger
Richard Griffey
Melissa Hensley
Publication date: 01-03-2011
Publisher: Springer US
Published in: Administration and Policy in Mental Health and Mental Health Services Research / Issue 2/2011
Print ISSN: 0894-587X
Electronic ISSN: 1573-3289
DOI: https://doi.org/10.1007/s10488-010-0319-7

At a glance: The ONWARDS insulin icodec trials

Springer Medicine

Outcomes for Implementation Research: Conceptual Distinctions, Measurement Challenges, and Research Agenda

Abstract

At a glance: The ONWARDS insulin icodec trials

Springer Medicine

Abstract

Please log in to get access to this content

Other articles of this Issue 2/2011

Medication Adherence, Ethnicity, and the Influence of Multiple Psychosocial and Financial Barriers

The Role of Housing: A Comparison of Front-Line Provider Views in Housing First and Traditional Programs

The Nathan Kline Institute Cultural Competency Assessment Scale: Psychometrics and Implications for Disparity Reduction

Investigating the Relationship Between Neighborhood Experiences and Psychiatric Distress for Individuals with Serious Mental Illness

Randomized Trial of Enhanced Profiling in Substance Abuse Treatment

Demand Characteristics and Quality Improvement: Who is Fooling Whom?