Skip to main content
Key Specialties
Cardiology Diabetology Endocrinology Gastroenterology Geriatrics and Gerontology Gynecology and Obstetrics Hematology Infectious Disease Internal Medicine Nephrology Neurology Oncology Pediatrics Respiratory Medicine Rheumatology Surgery
Congress Coverage
2024 ACC Congress 2023 ESMO Congress 2023 EASD Congress 2023 ERS Congress 2023 ESC Congress 2023 EHA Congress 2023 ASCO Annual Meeting
Clinical Collections
Advances in Alzheimer’s

At a glance: The ONWARDS insulin icodec trials

A round-up of the ONWARDS phase 3 clinical trials evaluating the novel weekly insulin icodec in people with diabetes.
ADVANCED SEARCH
Log in
Top
Implementation Science
Published in: Implementation Science 1/2014

Open Access 01-12-2014 | Debate

Instrumentation issues in implementation science

Authors: Ruben G Martinez, Cara C Lewis, Bryan J Weiner

Published in: Implementation Science | Issue 1/2014

Login to get access

Abstract

Background

Like many new fields, implementation science has become vulnerable to instrumentation issues that potentially threaten the strength of the developing knowledge base. For instance, many implementation studies report findings based on instruments that do not have established psychometric properties. This article aims to review six pressing instrumentation issues, discuss the impact of these issues on the field, and provide practical recommendations.

Discussion

This debate centers on the impact of the following instrumentation issues: use of frameworks, theories, and models; role of psychometric properties; use of `home-grown’ and adapted instruments; choosing the most appropriate evaluation method and approach; practicality; and need for decision-making tools. Practical recommendations include: use of consensus definitions for key implementation constructs; reporting standards (e.g., regarding psychometrics, instrument adaptation); when to use multiple forms of observation and mixed methods; and accessing instrument repositories and decision aid tools.

Summary

This debate provides an overview of six key instrumentation issues and offers several courses of action to limit the impact of these issues on the field. With careful attention to these issues, the field of implementation science can potentially move forward at the rapid pace that is respectfully demanded by community stakeholders.
Appendix
Available only for authorised users
Literature
1.
Siegel AE: Sidney Siegel: a memoir. Decision and Choice, Contributions of Sidney Siegel. 1964, McGraw-Hill, New York, NY
2.
Proctor EK, Landsverk J, Aarons G, Chambers D, Glisson C, Mittman B: Implementation research in mental health services: an emerging science with conceptual, methodological, and training challenges. Adm Policy Ment Health. 2009, 36: 24-34. 10.1007/s10488-008-0197-4.CrossRefPubMed
3.
Cook DA, Beckman TJ: Current concepts in validity and reliability for psychometric instruments: theory and application. Am J Med. 2006, 119: 166.e17-166.e16. 10.1016/j.amjmed.2005.10.036.CrossRef
4.
Downing SM, Haladyna TM: Test item development: validity evidence from quality assurance procedures. Appl Meas Educ. 1997, 10: 61-82. 10.1207/s15324818ame1001_4.CrossRef
5.
Proctor EK, Brownson RC: Measurement Issues in Dissemination and Implementation Research. Dissemination and Implementation Research in Health: Translating Science to Practice. Edited by: Brownson R, Colditz G, Proctor E. 2012, Oxford University Press, Inc, New York, NY
6.
Grimshaw J, Eccles M, Thomas R, MacLennan G, Ramsay C, Fraser C, Vale L: Toward evidence based quality improvement. J Gen Intern Med. 2006, 21: S14-S20.PubMedPubMedCentral
7.
Achenbach TM: Advancing assessment of children and adolescents: commentary on evidence-based assessment of child and adolescent disorders. J Clin Child Adolesc. 2005, 34: 541-547. 10.1207/s15374424jccp3403_9.CrossRef
8.
Hunsley J, Mash EJ: Evidence-based assessment. The Oxford handbook of clinical psychology. 2011, 76-97.
9.
Michie S, Fixsen D, Grimshaw JM, Eccles MP: Specifying and reporting complex behaviour change interventions: the need for a scientific method. Implement Sci. 2009, 4: 40-10.1186/1748-5908-4-40.CrossRefPubMedPubMedCentral
10.
Michie S, Johnston M, Francis J, Hardeman W, Eccles M: From theory to intervention: mapping theoretically derived behavioural determinants to behaviour change techniques. Appl Psychol. 2008, 57: 660-680. 10.1111/j.1464-0597.2008.00341.x.CrossRef
11.
Rabin BA, Brownson RC, Haire-Joshu D, Kreuter MW, Weaver NL: A glossary for dissemination and implementation research in health. J Public Health Manag Pract. 2008, 14: 117-123. 10.1097/01.PHH.0000311888.06252.bb.CrossRefPubMed
12.
Tabak RG, Khoong EC, Chambers DA, Brownson RC: Bridging research and practice: models for dissemination and implementation research. Am J Prev Med. 2012, 43: 337-350. 10.1016/j.amepre.2012.05.024.CrossRefPubMedPubMedCentral
13.
Weinert CR, Mann HJ: The science of implementation: changing the practice of critical care. Curr Opin Crit Care. 2008, 14: 460-465. 10.1097/MCC.0b013e3283079eb5.CrossRefPubMed
14.
Aarons GA, Hurlburt M, Horwitz SM: Advancing a conceptual model of evidence-based practice implementation in public service sectors. Adm Policy Ment Health. 2011, 38: 4-23. 10.1007/s10488-010-0327-7.CrossRefPubMed
15.
Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC: Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009, 4: 50-10.1186/1748-5908-4-50.CrossRefPubMedPubMedCentral
16.
Messick S: Validity of psychological assessment: validation of inferences from persons' responses and performances as scientific inquiry into score meaning. Am Psychol. 1995, 50: 741-749. 10.1037/0003-066X.50.9.741.CrossRef
17.
Proctor EK, Silmere H, Raghavan R, Hovmand P, Aarons GA, Bunger A, Griffey R, Hensley M: Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health. 2011, 38: 65-76. 10.1007/s10488-010-0319-7.CrossRefPubMed
18.
Chaudoir SR, Dugan AG, Barr CHI: Measuring factors affecting implementation of health innovations: a systematic review of structural, organizational, provider, patient, and innovation level measures. Implement Sci. 2013, 8: 22-10.1186/1748-5908-8-22.CrossRefPubMedPubMedCentral
19.
20.
21.
22.
Stichter J, Conroy MA: A critical analysis of the role of measurement on the validity of emotional and behavioral disorders (EBD) research. Behav Disorders. 2004, 30: 7-18.
23.
24.
Chor KHB, Wisdom JP, Olin S-CS, Hoagwood KE, Horwitz SM: Measures for predictors of innovation adoption.Adm Policy Ment Health 2014, 1-29.,
25.
Association AER: American Psychological Association, National Council on Measurement in Education: The standards for educational and psychological testing. 1999, American Psychological Association, Washington, DC
26.
Streiner DL: Starting at the beginning: an introduction to coefficient alpha and internal consistency. J Pers Assess. 2003, 80: 99-103. 10.1207/S15327752JPA8001_18.CrossRefPubMed
27.
Walsh WB, Betz NE: Tests and assessment. 1995, Prentice-Hall Inc, Englewood Cliffs, NJ
28.
Eyde LD, Robertson GJ, Krug SE: Responsible Test Use: Case Studies for Assessing Human Behavior. 1995, American Psychological Association, Washington, DC
29.
Osipow SH: Developing instruments for use in counseling. J Couns Dev. 1991, 70: 322-326. 10.1002/j.1556-6676.1991.tb01605.x.CrossRef
30.
Kendler KS: The dappled nature of causes of psychiatric illness: replacing the organic-functional/hardware-software dichotomy with empirically based pluralism. Mol Psychiatry. 2012, 17: 377-388. 10.1038/mp.2011.182.CrossRefPubMedPubMedCentral
31.
Glisson C, Landsverk J, Schoenwald S, Kelleher K, Hoagwood KE, Mayberg S, Green P: Assessing the organizational social context (OSC) of mental health services: Implications for research and practice. Adm Policy Ment Health. 2008, 35: 98-113. 10.1007/s10488-007-0148-5.CrossRefPubMed
32.
Martinez RG, Lewis CC: Instrumentation Issues in Implementation Science. In Seattle, WA: Presented at Seattle Implementation Research Conference; 2013.
33.
Hinkin TR: A brief tutorial on the development of measures for use in survey questionnaires. Organ Res Methods. 1998, 1: 104-121. 10.1177/109442819800100106.CrossRef
34.
DeVellis RF: Scale development: Theory and applications. 2011, Sage Publications, Thousand Oaks, CA
35.
Collins D: Pretesting survey instruments: an overview of cognitive methods. Qual Life Res. 2003, 12: 229-238. 10.1023/A:1023254226592.CrossRefPubMed
36.
Palinkas LA, Aarons GA, Horwitz S, Chamberlain P, Hurlburt M, Landsverk J: Mixed method designs in implementation research. Adm Policy Ment Health. 2011, 38: 44-53. 10.1007/s10488-010-0314-z.CrossRefPubMed
37.
Squires JE, Hutchinson AM, Boström A-M, O'Rourke HM, Cobban SJ, Estabrooks CA: To what extent do nurses use research in clinical practice? A systematic review. Implement Sci. 2011, 6: 21-10.1186/1748-5908-6-21.CrossRefPubMedPubMedCentral
38.
Podsakoff PM, MacKenzie SB, Lee J-Y, Podsakoff NP: Common method biases in behavioral research: a critical review of the literature and recommended remedies. J Appl Psychol. 2003, 88: 879-903. 10.1037/0021-9010.88.5.879.CrossRefPubMed
39.
Hofmann W, Gawronski B, Gschwendner T, Le H, Schmitt M: A meta-analysis on the correlation between the Implicit Association Test and explicit self-report measures. Pers Soc Psychol Bull. 2005, 31: 1369-1385. 10.1177/0146167205275613.CrossRefPubMed
40.
Greenwald AG, Poehlman TA, Uhlmann EL, Banaji MR: Understanding and using the Implicit Association Test: III. Meta-analysis of predictive validity. J Pers Soc Psychol. 2009, 97: 17-10.1037/a0015575.CrossRefPubMed
41.
Martino S, Ball S, Nich C, Frankforter TL, Carroll KM: Correspondence of motivational enhancement treatment integrity ratings among therapists, supervisors, and observers. Psychother Res. 2009, 19: 181-193. 10.1080/10503300802688460.CrossRefPubMedPubMedCentral
42.
Kimberly J, Cook JM: Organizational measurement and the implementation of innovations in mental health services. Adm Policy Ment Health. 2008, 35: 11-20. 10.1007/s10488-007-0143-x.CrossRefPubMed
43.
Dattilio FM, Edwards DJA, Fishman DB: Case studies within a mixed methods paradigm: toward a resolution of the alienation between researcher and practitioner in psychotherapy research. Psychother Theor Res Pract Train. 2010, 47: 427-441. 10.1037/a0021181.CrossRef
44.
Gaglio B, Glasgow RE: Evaluation approaches for dissemination and implementation research. Dissemination and implementation research in health: translating science to practice. Edited by: Brownsone RC, Colditz GA, Proctor EK. 2012, Oxford University Press, New York, NY, 327-356.CrossRef
45.
McLeod BD, Jensen-Doss A, Ollendick TH: Diagnostic and behavioral assessment in children and adolescents: a clinical guide. 2013, The Guilford Press, New York, NY
46.
Stange KC, Zyzanski SJ, Smith TF, Kelly R, Langa DM, Flocke SA, Jaén CR: How valid are medical records and patient questionnaires for physician profiling and health services research?: A comparison with direct observation of patient visits. Med Care. 1998, 36: 851-867. 10.1097/00005650-199806000-00009.CrossRefPubMed
47.
English DJ, Brandford CC, Coghlan L: Data-based organizational change: the use of administrative data to improve child welfare programs and policy. Child Welfare. 2000, 79: 499-515.PubMed
48.
Powell BJ, McMillen JC, Proctor EK, Carpenter CR, Griffey RT, Bunger AC, Glass JE, York JL: A compilation of strategies for implementing clinical innovations in health and mental health. Med Care Res Rev. 2012, 69: 123-157. 10.1177/1077558711430690.CrossRefPubMed
49.
Glasgow RE, Riley WT: Pragmatic measures: what they are and why we need them. Am J Prev Med. 2013, 45: 237-243. 10.1016/j.amepre.2013.03.010.CrossRefPubMed
50.
51.
52.
Aarons GA: Mental health provider attitudes toward adoption of evidence-based practice: the Evidence-Based Practice Attitude Scale (EBPAS). Ment Health Serv Res. 2004, 6: 61-74. 10.1023/B:MHSR.0000024351.12294.65.CrossRefPubMedPubMedCentral
53.
Rogers EM: Diffusion of innovations. 2003, Simon and Schuster, New York, NY
54.
Kramer DM, Cole DC: Sustained, intensive engagement to promote health and safety knowledge transfer to and utilization by workplaces. Sci Commun. 2003, 25: 56-82. 10.1177/1075547003252334.CrossRef
55.
Jacobson N, Butterill D, Goering P: Development of a framework for knowledge translation: understanding user context. J Health Serv Res Pol. 2003, 8: 94-99. 10.1258/135581903321466067.CrossRef
56.
Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O: Diffusion of innovations in service organizations: Systematic review and recommendations. Milbank Q. 2004, 82: 581-629. 10.1111/j.0887-378X.2004.00325.x.CrossRefPubMedPubMedCentral
57.
Dodson EA, Brownson RC, Weiss SW: Policy dissemination research. Dissemination and implementation research in health: translating science to practice. Edited by: Brownson RC, Colditz GA, Proctor EK. 2012, Oxford University Press, New York, NY, 437-458.CrossRef
58.
Atun R, de Jongh T, Secci F, Ohiri K, Adeyi O: Integration of targeted health interventions into health systems: a conceptual framework for analysis. Health Policy Plan. 2010, 25: 104-111. 10.1093/heapol/czp055.CrossRefPubMed
59.
Glasgow RE, Vogt TM, Boles SM: Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health. 1999, 89: 1322-1327. 10.2105/AJPH.89.9.1322.CrossRefPubMedPubMedCentral
60.
Wandersman A, Duffy J, Flaspohler P, Noonan R, Lubell K, Stillman L, Blachman M, Dunville R, Saul J: Bridging the gap between prevention research and practice: the interactive systems framework for dissemination and implementation. Am J Community Psychol. 2008, 41: 171-181. 10.1007/s10464-008-9174-z.CrossRefPubMed
61.
Farkas M, Anthony WA: Bridging science to service: using Rehabilitation Research and Training Center program to ensure that research-based knowledge makes a difference. J Rehabil Res Dev. 2007, 44: 879-892. 10.1682/JRRD.2006.08.0101.CrossRefPubMed
62.
Damush T, Bravata D, Plue L, Woodward-Hagg H, William L: Facilitation of Best Practices (FAB) Framework: Stroke QUERI Center Annual Report. Am J Prev Med. 2008, 43: 337-350.
63.
Feldstein AC, Glasgow RE: A practical, robust implementation and sustainability model (PRISM) for integrating research findings into practice. Jt Comm J Qual Patient Saf. 2008, 34: 228-243.PubMed
64.
65.
Fixsen DL, Naoom SF, Blase KA, Friedman RM, Wallace F: Implementation research: A synthesis of the literature. 2005, University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network, Tampa, FL
66.
Glisson C, Schoenwald SK: The ARC organizational and community intervention strategy for implementing evidence-based children's mental health treatments. Ment Health Serv Res. 2005, 7: 243-259. 10.1007/s11020-005-7456-1.CrossRefPubMed
67.
Kitson A, Harvey G, McCormack B: Enabling the implementation of evidence based practice: a conceptual framework. Qual Health Care. 1998, 7: 149-158. 10.1136/qshc.7.3.149.CrossRefPubMedPubMedCentral
68.
Torrey WC, Lynde DW, Gorman P: Promoting the implementation of practices that are supported by research: The National Implementing Evidence-Based Practice Project. Child Adolesc Psychiatr Clin N Am. 2005, 14: 297-306. 10.1016/j.chc.2004.05.004.CrossRefPubMed
Metadata
Title
Instrumentation issues in implementation science
Authors
Ruben G Martinez
Cara C Lewis
Bryan J Weiner
Publication date
01-12-2014
Publisher
BioMed Central
Published in
Implementation Science / Issue 1/2014
Electronic ISSN: 1748-5908
DOI
https://doi.org/10.1186/s13012-014-0118-8

Other articles of this Issue 1/2014

Implementation Science 1/2014 Go to the issue

Study protocol

Examining the ethical and social issues of health technology design through the public appraisal of prospective scenarios: a study protocol describing a multimedia-based deliberative method

Study protocol

Transforming mental health services: a participatory mixed methods study to promote and evaluate the implementation of recovery-oriented services

Study protocol

Increasing the quantity and quality of searching for current best evidence to answer clinical questions: protocol and intervention design of the MacPLUS FS Factorial Randomized Controlled Trials

Research

Theory of planned behaviour can help understand processes underlying the use of two emergency medicine diagnostic imaging rules

Study protocol

Designing and evaluating an interprofessional shared decision-making and goal-setting decision aid for patients with diabetes in clinical care - systematic decision aid development and study protocol

Study protocol

A cluster randomized trial to evaluate external support for the implementation of positive behavioral interventions and supports by school personnel