Skip to main content
Key Specialties
Cardiology Diabetology Endocrinology Gastroenterology Geriatrics and Gerontology Gynecology and Obstetrics Hematology Infectious Disease Internal Medicine Nephrology Neurology Oncology Pediatrics Respiratory Medicine Rheumatology Surgery
Congress Coverage
2024 ACC Congress 2023 ESMO Congress 2023 EASD Congress 2023 ERS Congress 2023 ESC Congress
Clinical Collections
Advances in Alzheimer’s

At a glance: The STEP trials

A round-up of the STEP phase 3 clinical trials evaluating semaglutide for weight loss in people with overweight or obesity.
ADVANCED SEARCH
Log in
Top
Implementation Science
Published in: Implementation Science 1/2015

Open Access 01-12-2015 | Methodology

Health Information Technologies—Academic and Commercial Evaluation (HIT-ACE) methodology: description and application to clinical feedback systems

Authors: Aaron R. Lyon, Cara C. Lewis, Abigail Melvin, Meredith Boyd, Semret Nicodimos, Freda F. Liu, Nathaniel Jungbluth

Published in: Implementation Science | Issue 1/2015

Login to get access

Abstract

Background

Health information technologies (HIT) have become nearly ubiquitous in the contemporary healthcare landscape, but information about HIT development, functionality, and implementation readiness is frequently siloed. Theory-driven methods of compiling, evaluating, and integrating information from the academic and commercial sectors are necessary to guide stakeholder decision-making surrounding HIT adoption and to develop pragmatic HIT research agendas. This article presents the Health Information Technologies—Academic and Commercial Evaluation (HIT-ACE) methodology, a structured, theory-driven method for compiling and evaluating information from multiple sectors. As an example demonstration of the methodology, we apply HIT-ACE to mental and behavioral health measurement feedback systems (MFS). MFS are a specific class of HIT that support the implementation of routine outcome monitoring, an evidence-based practice.

Results

HIT-ACE is guided by theories and frameworks related to user-centered design and implementation science. The methodology involves four phases: (1) coding academic and commercial materials, (2) developer/purveyor interviews, (3) linking putative implementation mechanisms to hit capabilities, and (4) experimental testing of capabilities and mechanisms. In the current demonstration, phase 1 included a systematic process to identify MFS in mental and behavioral health using academic literature and commercial websites. Using user-centered design, implementation science, and feedback frameworks, the HIT-ACE coding system was developed, piloted, and used to review each identified system for the presence of 38 capabilities and 18 additional characteristics via a consensus coding process. Bibliometic data were also collected to examine the representation of the systems in the scientific literature. As an example, results are presented for the application of HIT-ACE phase 1 to MFS wherein 49 separate MFS were identified, reflecting a diverse array of characteristics and capabilities.

Conclusions

Preliminary findings demonstrate the utility of HIT-ACE to represent the scope and diversity of a given class of HIT beyond what can be identified in the academic literature. Phase 2 data collection is expected to confirm and expand the information presented and phases 3 and 4 will provide more nuanced information about the impact of specific HIT capabilities. In all, HIT-ACE is expected to support adoption decisions and additional HIT development and implementation research.
Literature
1.
2.
Patient Protection and Affordable Care Act. 2010
3.
Health information technology for economic and clinical health act. 2009
4.
Buntin MB, Burke MF, Hoaglin MC, Blumenthal D. The benefits of health information technology: a review of the recent literature shows predominantly positive results. Health Aff (Millwood). 2011;30:464–71.CrossRef
5.
Calisir F, Calisir F. The relation of interface usability characteristics, perceived usefulness, and perceived ease of use to end-user satisfaction with enterprise resource planning (ERP) systems. Comput Hum Behav. 2004;20:505–15.CrossRef
6.
Tao D, Or CK. Effects of self-management health information technology on glycaemic control for patients with diabetes: a meta-analysis of randomized controlled trials. J Telemed Telecare. 2013;19:133–43.CrossRefPubMed
7.
Singh K, Drouin K, Newmark LP, Rozenblum R, Lee J, Landman A, Pabo E, Klinger EV, Bates DW. Developing a framework for evaluating the patient engagement, quality, and safety of mobile health applications. Issue Brief (Commonw Fund). 2016;5:1–11.
8.
Baxter SK, Blank L, Woods HB, Payne N, Rimmer M, Goyder E. Using logic model methods in systematic review synthesis: describing complex pathways in referral management interventions. BMC Med Res Methodol. 2014;14:62.CrossRefPubMedPubMedCentral
9.
Sutcliffe K, Thomas J, Stokes G, Hinds K, Bangpan M. Intervention component analysis (ICA): a pragmatic approach for identifying the critical features of complex interventions. Syst Rev. 2015;4:140.CrossRefPubMedPubMedCentral
10.
11.
Collins LM, Murphy SA, Strecher V. The multiphase optimization strategy (MOST) and the sequential multiple assignment randomized trial (SMART): new methods for more potent eHealth interventions. Am J Prev Med. 2007;32:S112–8 [Critical Issues in eHealth ResearchCritical Issues in eHealth Research 2005].CrossRefPubMedPubMedCentral
12.
Shibata N, Kajikawa Y, Sakata I. Extracting the commercialization gap between science and technology—case study of a solar cell. Technol Forecast Soc Change. 2010;77:1147–55.CrossRef
13.
Yen P-Y, Bakken S. Review of health information technology usability study methodologies. J Am Med Inform Assoc. 2012;19:413–22.CrossRefPubMed
14.
Powell BJ, McMillen JC, Proctor EK, Carpenter CR, Griffey RT, Bunger AC, Glass JE, York JL. A compilation of strategies for implementing clinical innovations in health and mental health. Med Care Res Rev. 2012;69:123–57.CrossRefPubMed
15.
Powell BJ, Waltz TJ, Chinman MJ, Damschroder LJ, Smith JL, Matthieu MM, Proctor EK, Kirchner JE. A refined compilation of implementation strategies: results from the expert recommendations for implementing change (ERIC) project. Implement Sci. 2015;10:21.CrossRefPubMedPubMedCentral
16.
Bickman L. A measurement feedback system (MFS) is necessary to improve mental health outcomes. J Am Acad Child Adolesc Psychiatry. 2008;47:1114–9.CrossRefPubMed
17.
Bickman L, Lyon AR, Wolpert M. Achieving precision mental health through effective assessment, monitoring, and feedback processes. Adm Policy Ment Health. 2016;43:271–6.CrossRefPubMedPubMedCentral
18.
Roe D, Drake RE, Slade M. Routine outcome monitoring: an international endeavour. Int Rev Psychiatry. 2015;27:257–60.CrossRefPubMed
19.
Bickman L, Kelley SD, Breda C, de Andrade AR, Riemer M. Effects of routine feedback to clinicians on mental health outcomes of youths: results of a randomized trial. Psychiatr Serv. 2011;62:1423–9.CrossRefPubMed
20.
Lambert MJ, Whipple JL, Hawkins EJ, Vermeersch DA, Nielsen SL, Smart DW. Is it time for clinicians to routinely track patient outcome? A meta-analysis. Clin Psychol Sci Pract. 2003;10:288–301.CrossRef
21.
22.
23.
Kellermann AL, Jones SS. What it will take to achieve the as-yet-unfulfilled promises of health information technology. Health Aff (Millwood). 2013;32:63–8.CrossRef
24.
Bergen M, Peteraf MA. Competitor identification and competitor analysis: a broad-based managerial approach. Manag Decis Econ. 2002;23:157–69.CrossRef
25.
Blount A, Schoenbaum M, Kathol R, Rollman BL, Thomas M, O’Donohue W, Peek CJ. The economics of behavioral health services in medical settings: a summary of the evidence. Prof Psychol Res Pract. 2007;38:290–7.CrossRef
26.
Kazdin AE. Mediators and mechanisms of change in psychotherapy research. Annu Rev Clin Psychol. 2007;3:1–27.CrossRefPubMed
27.
Lewis CC, Boyd M, Beidas RS, Lyon AR, Chambers D, Aarons GA, Mittman B. A research agenda for mechanistic dissemination and implementation. 2015.
28.
Waltz TJ, Powell BJ, Matthieu MM, Damschroder LJ, Chinman MJ, Smith JL, Proctor EK, Kirchner JE. Use of concept mapping to characterize relationships among implementation strategies and assess their feasibility and importance: results from the expert recommendations for implementing change (ERIC) study. Implement Sci. 2015;10:109.CrossRefPubMedPubMedCentral
29.
30.
Aarons GA, Hurlburt M, Horwitz SM. Advancing a conceptual model of evidence-based practice implementation in public service sectors. Adm Policy Ment Health Ment Health Serv Res. 2010;38:4–23.CrossRef
31.
Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC, et al. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4:50.CrossRefPubMedPubMedCentral
32.
Insel T, Cuthbert B, Garvey M, Heinssen R, Pine DS, Quinn K, Sanislow C, Wang P. Research domain criteria (rdoc): toward a new classification framework for research on mental disorders. Am J Psychiatry. 2010;167:748–51.CrossRefPubMed
33.
Hsu C-C, Sandford BA. The Delphi technique: making sense of consensus. Pract Assess Res Eval. 2007;12:1–8.
34.
35.
Leijten P, Dishion TJ, Thomaes S, Raaijmakers MAJ, Orobio de Castro B, Matthys W. Bringing parenting interventions back to the future: how randomized microtrials may benefit parenting intervention efficacy. Clin Psychol Sci Pract. 2015;22:47–57.CrossRef
36.
Lyon AR, Koerner K. User-centered design for psychosocial intervention development and implementation. Clin Psychol Sci Pract. 2016;23:180–200.CrossRef
37.
Kluger AN, DeNisi A. The effects of feedback interventions on performance: a historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychol Bull. 1996;119:254–84.CrossRef
38.
Riemer M, Rosof-Williams J, Bickman L. Theories related to changing clinician practice. Child Adolesc Psychiatr Clin N Am. 2005;14:241–54 [Evidence-Based Practice, Part II: Effecting Change].CrossRefPubMed
39.
Courage C, Baxter K. Understanding your users: a practical guide to user requirements: methods, tools, and techniques. San Francisco, CA :Gulf Professional Publishing; 2005.
40.
Pea RD. User centered system design: new perspectives on human-computer interaction. J Educ Comput Res. 1987;3(1):129–34.
41.
Eden J, Maslow K, Le M, Blazer D. et al. The mental health and substance use workforce for older adults: in whose hands? San Francisco, CA: National Academies Press; 2012
42.
DIS I: 9241–210: 2010. Ergonomics of human system interaction-Part 210: human-centred design for interactive systems. Int Stand Organ ISO Switz 2009.
43.
Rubin J, Chisnell D. Handbook of usability testing: how to plan, design and conduct effective tests. Indianapolis: John Wiley & Sons; 2008.
44.
Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, Griffey R, Hensley M. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health Ment Health Serv Res. 2010;38:65–76.CrossRef
45.
Rogers EM. Diffusion of innovations. 4th ed. New York: The Free Press; 2010.
46.
Tabak RG, Khoong EC, Chambers DA, Brownson RC. Bridging research and practice: models for dissemination and implementation research. Am J Prev Med. 2012;43:337–50.CrossRefPubMedPubMedCentral
47.
Douglas S, Button S, Casey SE. Implementing for sustainability: promoting use of a measurement feedback system for innovation and quality improvement. Adm Policy Ment Health Ment Health Serv Res. 2014;43:286–91.CrossRef
48.
Edbrooke-Childs J, Wolpert M, Deighton J. Introduction to the special section on implementing feedback from outcome measures in child and adult mental health services. Adm Policy Ment Health Ment Health Serv Res. 2016;43(3):277–278.CrossRef
49.
Lyon AR, Lewis CC. Designing health information technologies for uptake: development and implementation of measurement feedback systems in mental health service delivery. Adm Policy Ment Health Ment Health Serv Res. 2015;43:344–9.CrossRef
50.
Landes SJ, Carlson EB, Ruzek JI, Wang D, Hugo E, DeGaetano N, Chambers JG, Lindley SE. Provider-driven development of a measurement feedback system to enhance measurement-based care in VA mental health. Cogn Behav Pract. 2015;22:87–100.CrossRef
51.
Lyon AR, Wasse JK, Ludwig K, Zachry M, Bruns EJ, Unützer J, McCauley E. The contextualized technology adaptation process (CTAP): Optimizing health information technology to improve mental health systems. Administration and Policy in Mental Health and Mental Health Services Research. 2016;43:394-409.
52.
Lyon AR, Lewis CC, Boyd MR, Hendrix E, Liu F. Capabilities and characteristics of digital measurement feedback systems: results from a comprehensive review. Adm Policy Ment Health Ment Health Serv Res. 2016;43:441–66.CrossRef
53.
Fernández-Alemán JL, Señor IC, Lozoya PÁO, Toval A. Security and privacy in electronic health records: a systematic literature review. J Biomed Inform. 2013;46:541–62.CrossRefPubMed
54.
Grant RW, Wald JS, Poon EG, Schnipper JL, Gandhi TK, Volk LA, Middleton B. Design and implementation of a web-based patient portal linked to an ambulatory care electronic health record: Patient gateway for diabetes collaborative care. Diabetes Technol Ther. 2006;8:576–86.CrossRefPubMed
55.
Staroselsky M, Volk LA, Tsurikova R, Pizziferri L, Lippincott M, Wald J, Bates DW. Improving electronic health record (EHR) accuracy and increasing compliance with health maintenance clinical guidelines through patient access and input. Int J Med Inf. 2006;75:693–700.CrossRef
56.
Hill CE, Thompson BJ, Williams EN. A guide to conducting consensual qualitative research. Couns Psychol. 1997;25:517–72.CrossRef
57.
Hill CE, Knox S, Thompson BJ, Williams EN, Hess SA, Ladany N. Consensual qualitative research: an update. J Couns Psychol. 2005;52:196–205.CrossRef
58.
59.
Brin S, Page L. Reprint of: the anatomy of a large-scale hypertextual web search engine. Comput Netw. 2012;56:3825–33 [The WEB We Live in].CrossRef
60.
Why do Google results vary in different locations? - LCN.com. Grow a successful website with the LCN.com Business Hub. 2014.
Metadata
Title
Health Information Technologies—Academic and Commercial Evaluation (HIT-ACE) methodology: description and application to clinical feedback systems
Authors
Aaron R. Lyon
Cara C. Lewis
Abigail Melvin
Meredith Boyd
Semret Nicodimos
Freda F. Liu
Nathaniel Jungbluth
Publication date
01-12-2015
Publisher
BioMed Central
Published in
Implementation Science / Issue 1/2015
Electronic ISSN: 1748-5908
DOI
https://doi.org/10.1186/s13012-016-0495-2

Other articles of this Issue 1/2015

Implementation Science 1/2015 Go to the issue

Research

Understanding the dynamics of the Seguro Popular de Salud policy implementation in Mexico from a complex adaptive systems perspective

Research

How organizational escalation prevention potential affects success of implementation of innovations: electronic medical records in hospitals

Research

A complex endeavour: an ethnographic study of the implementation of the Sepsis Six clinical care bundle

Research

Evaluation of a large healthy lifestyle program: informing program implementation and scale-up in the prevention of obesity

Erratum

Erratum to: ‘Toward the sustainability of health interventions implemented in sub-Saharan Africa: a systematic review and conceptual framework’

Debate

Developing the ethics of implementation research in health