Skip to main content
Key Specialties
Cardiology Diabetology Endocrinology Gastroenterology Geriatrics and Gerontology Gynecology and Obstetrics Hematology Infectious Disease Internal Medicine Nephrology Neurology Oncology Pediatrics Respiratory Medicine Rheumatology Surgery
Congress Coverage
2024 ACC Congress 2023 ESMO Congress 2023 EASD Congress 2023 ERS Congress 2023 ESC Congress
Clinical Collections
Advances in Alzheimer’s

At a glance: The STEP trials

A round-up of the STEP phase 3 clinical trials evaluating semaglutide for weight loss in people with overweight or obesity.
ADVANCED SEARCH
Log in
Top
Implementation Science
Published in: Implementation Science 1/2015

Open Access 01-12-2015 | Research

Health system guidance appraisal—concept evaluation and usability testing

Authors: Denis E. Ako-Arrey, Melissa C. Brouwers, John N. Lavis, Mita K. Giacomini, on behalf of the AGREE HS Team

Published in: Implementation Science | Issue 1/2015

Login to get access

Abstract

Background

Health system guidance (HSG) provides recommendations aimed to address health system challenges. However, there is a paucity of methods to direct, appraise, and report HSG. Earlier research identified 30 candidate criteria (concepts) that can be used to evaluate the quality of HSG and guide development and reporting requirements. The objective of this paper was to describe two studies aimed at evaluating the importance of these 30 criteria, design a draft HSG appraisal tool, and test its usability.

Methods

This study involved a two-step survey process. In step 1, respondents rated the 30 concepts for appropriateness to, relevance to, and priority for health system decisions and HSG. This led to a draft tool. In step 2, respondents reviewed HSG documents, appraised them using the tool, and answered a series of questions. Descriptive analyses were computed.

Results

Fifty participants were invited in step 1, and we had a response rate of 82 %. The mean response rates for each concept within each survey question were universally favorable. There was also an overall agreement about the need for a high-quality tool to systematically direct the development, appraisal, and reporting of HSG. Qualitative feedback and a consensus process by the team led to refinements to some of the concepts and the creation of a beta (draft) version of the HSG tool. In step 2, 35 participants were invited and we had a response rate of 74 %. Exploratory analyses showed that the quality of the HSGs reviewed varied as a function of the HSG item and the specific document assessed. A favorable consensus was reached with participants agreeing that the HSG items were easy to understand and easy to apply. Moreover, the overall agreement was high for the usability of the tool to systematically direct the development (85 %), appraisal (92 %), and reporting (81 %) of HSG. From this process, version 1.0 of the HSG appraisal tool was generated complete with 32 items (and their descriptions) and 4 domains.

Conclusions

The final tool, named the Appraisal of Guidelines for Research and Evaluation for Health Systems (AGREE-HS) (version 1), defines expectations of HSG and facilitates informed decisions among policymakers on health system delivery, financial, and governance arrangements.
Appendix
Available only for authorised users
Literature
1.
2.
Lavis JN, Ross SE, Hurley JE, Hohenadel JM, Stoddart GL, Woodward CA, et al. Examining the role of health services research in public policymaking. Milbank Q. 2002;80:125–54.PubMedCentralCrossRefPubMed
3.
4.
Lavis JN, Wilson MG, Moat KA, Hammill AC, Boyko JA, Grimshaw JM, et al. Developing and refining the methods for a ‘one-stop shop’for research evidence about health systems. Health Res Policy and Systems. 2015;13:10.CrossRef
5.
Coker RJ, Atun R, McKee M. Health care system frailties and public health control of communicable diseases on the European Union’s new eastern border. Lancet. 2004;363:1389–92.CrossRefPubMed
6.
Lessof S, Figueras J, Duran A, McKee M, Suhrcke M, Nolte E, et al. Health systems, health, and wealth: a European perspective. Lancet. 2009;373:349–51.CrossRefPubMed
7.
Mendis S, Bekedam H, Wright A, Samb B, Desai N, Nishtar S, et al. Prevention and management of chronic disease: a litmus test for health-systems strengthening in low-income and middle-income countries. Lancet. 2010;376:1785–97.CrossRefPubMed
8.
Atun R. Health systems, systems thinking and innovation. Health Policy Plan. 2012;27 suppl 4:iv4–8.CrossRefPubMed
9.
Travis P, Bennett S, Haines A, Pang T, Bhutta Z, Hyder AA, et al. Overcoming health-systems constraints to achieve the millennium development goals. Lancet. 2004;364(9437):900–6.CrossRefPubMed
10.
Pierson P. When effect becomes cause: policy feedback and political change. World Politics. 1993;45(July):595–628.CrossRef
11.
Szreter S, Woolcock M. Health by association? Social capital, social theory, and the political economy of public health. Int J Epidemiol. 2004;33(4):650–67.CrossRefPubMed
12.
13.
Bosch-Capblanch X, Lavis JN, Lewin S, Atun R, Røttingen JA, Dröschel D, et al. Guidance for evidence-informed decisions about health systems: rationale for and challenges of guidance development. PLoS Med. 2012;9:e1001185.
14.
15.
Degrasse C, Green E, Mackay JA, Vandenberg T, Coakley N, Nayler J, et al. A framework for the organization and delivery of systemic treatment. Curr Oncol. 2009;16(1):4–15.PubMedCentralPubMed
16.
17.
Evans D, Murray CJ. Health systems performance assessment. Office of health economics. 2006.
18.
Frenk J, Murray CJ. A WHO framework for health system performance assessment. Evidence and information for policy, world health organization. 1999.
19.
20.
Bosch-Capblanch X. Handbook for supporting the development of health system guidance. Basel: Swiss Centre for International Health; 2011.
21.
Gilson L, Lavis JN, Røttingen JA, Bosch-Capblanch X, Atun R, El-Jardali F, et al. Guidance for evidence-informed policies about health systems: linking guidance development to policy development. PLoS Med. 2012;9:e1001186.PubMedCentralCrossRefPubMed
22.
23.
Brouwers M, Kho ME, Browman GP, Burgers JS, Cluzeau F, Feder G, et al. for the AGREE Next Steps Consortium. AGREE II: advancing guideline development, reporting and evaluation in healthcare. Can Med Assoc J. 2010 Dec; 2010c; 182:18. E839-E842
24.
Brouwers, M. C., Kho, M. E., Browman, G. P., Burgers, J. S., Cluzeau, F., Feder, G., Makarski, J. Development of the AGREE II, part 1: performance, usefulness and areas for improvement. Canadian Medical Association Journal, 2010a; 182(10), 1045–1052
25.
Brouwers, M. C., Kho, M. E., Browman, G. P., Burgers, J. S., Cluzeau, F., Feder, G., Makarski, J. Development of the AGREE II, part 2: assessment of validity of items and tools to support application. Canadian Medical Association Journal, 2010b; 182(10), E472-E478
26.
Birtwhistle R, Pottie K, Shaw E, Dickinson JA, Brauer P, Fortin M, et al. Canadian task force on preventive health care: we’re back! Can Fam Physician. 2012;58:13–5.PubMedCentralPubMed
27.
on behalf of the AGREE-HS Team, Ako-Arrey ED, Brouwers MC, Lavis JN, Giacomini M. Health systems guidance appraisal concepts: a critical interpretive synthesis. Manuscript submitted for publication. 2015.
28.
Streiner DL, Norman GR. Health measurement scales: a practical guide to their development and use. 4th ed. Oxford, UK: Oxford University Press; 2003.
Metadata
Title
Health system guidance appraisal—concept evaluation and usability testing
Authors
Denis E. Ako-Arrey
Melissa C. Brouwers
John N. Lavis
Mita K. Giacomini
on behalf of the AGREE HS Team
Publication date
01-12-2015
Publisher
BioMed Central
Published in
Implementation Science / Issue 1/2015
Electronic ISSN: 1748-5908
DOI
https://doi.org/10.1186/s13012-015-0365-3

Other articles of this Issue 1/2015

Implementation Science 1/2015 Go to the issue

Study protocol

Implementing an evidence-based computerized decision support system to improve patient care in a general hospital: the CODES study protocol for a randomized controlled trial

Research

The impact of leadership hubs on the uptake of evidence-informed nursing practices and workplace policies for HIV care: a quasi-experimental study in Jamaica, Kenya, Uganda and South Africa

Debate

A modified theoretical framework to assess implementation fidelity of adaptive public health interventions

Study protocol

Tailoring implementation strategies for evidence-based recommendations using computerised clinical decision support systems: protocol for the development of the GUIDES tools

Research

Structuring successful collaboration: a longitudinal social network analysis of a translational research network

Methodology

Figuring out fidelity: a worked example of the methods used to identify, critique and revise the essential elements of a contextualised intervention in health policy agencies