Skip to main content
Top
Published in: BMC Public Health 1/2009

Open Access 01-12-2009 | Research article

The development of an evaluation framework for injury surveillance systems

Authors: Rebecca J Mitchell, Ann M Williamson, Rod O'Connor

Published in: BMC Public Health | Issue 1/2009

Login to get access

Abstract

Background

Access to good quality information from injury surveillance is essential to develop and monitor injury prevention activities. To determine if information obtained from surveillance is of high quality, the limitations and strengths of a surveillance system are often examined. Guidelines have been developed to assist in evaluating certain types of surveillance systems. However, to date, no standard guidelines have been developed to specifically evaluate an injury surveillance system. The aim of this research is to develop a framework to guide the evaluation of injury surveillance systems.

Methods

The development of an Evaluation Framework for Injury Surveillance Systems (EFISS) involved a four stage process. First, a literature review was conducted to identify an initial set of characteristics that were recognised as important and/or had been recommended to be assessed in an evaluation of a surveillance system. Second, this set of characteristics was assessed using SMART criteria. Third, those surviving were presented to an expert panel using a two round modified-Delphi study to gain an alternative perspective on characteristic definitions, practicality of assessment, and characteristic importance. Finally, a rating system was created for the EFISS characteristics.

Results

The resulting EFISS consisted of 18 characteristics that assess three areas of an injury surveillance system – five characteristics assess data quality, nine characteristics assess the system's operation, and four characteristics assess the practical capability of an injury surveillance system. A rating system assesses the performance of each characteristic.

Conclusion

The development of the EFISS builds upon existing evaluation guidelines for surveillance systems and provides a framework tailored to evaluate an injury surveillance system. Ultimately, information obtained through an evaluation of an injury data collection using the EFISS would be useful for agencies to recommend how a collection could be improved to increase its usefulness for injury surveillance and in the long-term injury prevention.
Literature
1.
go back to reference Holder Y, Peden M, Krug E, Lund J, Gururaj G, Kobusingye OC: Injury Surveillance Guidelines. 2001, Geneva: World Health Organization Holder Y, Peden M, Krug E, Lund J, Gururaj G, Kobusingye OC: Injury Surveillance Guidelines. 2001, Geneva: World Health Organization
2.
go back to reference Thacker S, Berkelman R: Public health surveillance in the United States. Epidemiologic Reviews. 1988, 10: 164-190.PubMed Thacker S, Berkelman R: Public health surveillance in the United States. Epidemiologic Reviews. 1988, 10: 164-190.PubMed
3.
go back to reference Ing R: Surveillance in injury prevention. Public Health Reports. 1985, 100 (6): 586-588. Ing R: Surveillance in injury prevention. Public Health Reports. 1985, 100 (6): 586-588.
4.
go back to reference Harrison J: The context of new developments in injury surveillance. Working as a Nation to Prevent Injury. 1995, Darling Harbour, Sydney: Commonwealth Department of Human Services and Health Harrison J: The context of new developments in injury surveillance. Working as a Nation to Prevent Injury. 1995, Darling Harbour, Sydney: Commonwealth Department of Human Services and Health
5.
go back to reference Thacker S, Parrish R, Trowbridge F: A method for evaluate systems of epidemiologic surveillance. World Health Statistics Quarterly. 1988, 41: 11-18. Thacker S, Parrish R, Trowbridge F: A method for evaluate systems of epidemiologic surveillance. World Health Statistics Quarterly. 1988, 41: 11-18.
7.
go back to reference Macarthur C, Pless IB: Evaluation of the quality of an injury surveillance system. American Journal of Epidemiology. 1999, 149 (6): 586-92.CrossRefPubMed Macarthur C, Pless IB: Evaluation of the quality of an injury surveillance system. American Journal of Epidemiology. 1999, 149 (6): 586-92.CrossRefPubMed
8.
go back to reference Centers for Disease Control and Prevention: Updated guidelines for evaluating public health surveillance systems: recommendations from the Guidelines Working Group. Morbidity & Mortality Weekly Report CDC Surveillance Summaries. 2001, 50: 1-51. Centers for Disease Control and Prevention: Updated guidelines for evaluating public health surveillance systems: recommendations from the Guidelines Working Group. Morbidity & Mortality Weekly Report CDC Surveillance Summaries. 2001, 50: 1-51.
9.
go back to reference Health Surveillance Coordinating Committee Health Canada: Framework and Tools for Evaluating Health Surveillance Systems. 2004, Ottawa: Health Canada Health Surveillance Coordinating Committee Health Canada: Framework and Tools for Evaluating Health Surveillance Systems. 2004, Ottawa: Health Canada
10.
go back to reference Centers for Disease Control and Prevention: Framework for Evaluating Public Health Surveillance Systems for Early Detection of Outbreaks. 2004, US Department of Health and Human Services: Atlanta Centers for Disease Control and Prevention: Framework for Evaluating Public Health Surveillance Systems for Early Detection of Outbreaks. 2004, US Department of Health and Human Services: Atlanta
11.
go back to reference World Health Organization: Protocol for the Assessment of National Communicable Disease Surveillance and Response Systems: Guidelines for Assessment Teams. 2001, World Health Organization: Geneva World Health Organization: Protocol for the Assessment of National Communicable Disease Surveillance and Response Systems: Guidelines for Assessment Teams. 2001, World Health Organization: Geneva
12.
go back to reference Locke E, Latham G: Theory of goal setting and task performance. 1990, Englewood Cliffs, N.J: Prentice Hall Locke E, Latham G: Theory of goal setting and task performance. 1990, Englewood Cliffs, N.J: Prentice Hall
13.
go back to reference Shahin A, Mahbod M: Prioritization of key performance indicators. An integration of analytical hierarchy process and goal setting. International Journal of Productivity and Performance Management. 2007, 56 (3): 226-240.CrossRef Shahin A, Mahbod M: Prioritization of key performance indicators. An integration of analytical hierarchy process and goal setting. International Journal of Productivity and Performance Management. 2007, 56 (3): 226-240.CrossRef
14.
go back to reference Holliman R, Johnson J, Adjei O: The objective assessment of international collaboration between pathology laboratories. Journal of Evaluation in Clinical Practice. 2006, 12 (1): 1-7.CrossRefPubMed Holliman R, Johnson J, Adjei O: The objective assessment of international collaboration between pathology laboratories. Journal of Evaluation in Clinical Practice. 2006, 12 (1): 1-7.CrossRefPubMed
15.
go back to reference Bowles S, Cunningham C, De La Rosa G, Picano J: Coaching leaders in middle and executive management: goals, performance, buy-in. Leadership and Organization Development Journal. 2007, 28 (5): 388-408.CrossRef Bowles S, Cunningham C, De La Rosa G, Picano J: Coaching leaders in middle and executive management: goals, performance, buy-in. Leadership and Organization Development Journal. 2007, 28 (5): 388-408.CrossRef
16.
go back to reference Custer R, Scarcella J, Stewart B: The modified delphi technique – a rotational modification. Journal of Vocational and Technical Education. 1999, 15 (2): Custer R, Scarcella J, Stewart B: The modified delphi technique – a rotational modification. Journal of Vocational and Technical Education. 1999, 15 (2):
17.
go back to reference Stewart B, Bristow D: Tech prep programs: the role of essential elements. Journal of Vocational and Technical Education. 1997, 13 (2): Stewart B, Bristow D: Tech prep programs: the role of essential elements. Journal of Vocational and Technical Education. 1997, 13 (2):
18.
go back to reference Burns SP, Rivara FP, Johansen JM, Thompson DC: Rehabilitation of traumatic injuries: use of the delphi method to identify topics for evidence-based review. American Journal of Physical Medicine & Rehabilitation. 2003, 82 (5): 410-4. Burns SP, Rivara FP, Johansen JM, Thompson DC: Rehabilitation of traumatic injuries: use of the delphi method to identify topics for evidence-based review. American Journal of Physical Medicine & Rehabilitation. 2003, 82 (5): 410-4.
19.
go back to reference Nathens AB, Rivara FP, Jurkovich GJ, Maier RV, Johansen JM, Thompson DC: Management of the injured patient: identification of research topics for systematic review using the delphi technique. Journal of Trauma-Injury Infection & Critical Care. 2003, 54 (3): 595-601.CrossRef Nathens AB, Rivara FP, Jurkovich GJ, Maier RV, Johansen JM, Thompson DC: Management of the injured patient: identification of research topics for systematic review using the delphi technique. Journal of Trauma-Injury Infection & Critical Care. 2003, 54 (3): 595-601.CrossRef
20.
go back to reference Riggs W: The Delphi technique: an experimental evaluation. Technological forecasting and social change. 1983, 23: 89-94.CrossRef Riggs W: The Delphi technique: an experimental evaluation. Technological forecasting and social change. 1983, 23: 89-94.CrossRef
21.
go back to reference Jurkovich GJ, Rivara FP, Johansen JM, Maier RV: Centers for Disease Control and Prevention injury research agenda: identification of acute care research topics of interest to the Centers for disease Control and Prevention – National Center for Injury Prevention and Control. Journal of Trauma-Injury Infection & Critical Care. 2004, 56 (5): 1166-70.CrossRef Jurkovich GJ, Rivara FP, Johansen JM, Maier RV: Centers for Disease Control and Prevention injury research agenda: identification of acute care research topics of interest to the Centers for disease Control and Prevention – National Center for Injury Prevention and Control. Journal of Trauma-Injury Infection & Critical Care. 2004, 56 (5): 1166-70.CrossRef
22.
go back to reference Rowe G, Wright G: The Delphi technique as a forecasting tool: issues and analysis. International Journal of Forecasting. 1999, 15: 353-375.CrossRef Rowe G, Wright G: The Delphi technique as a forecasting tool: issues and analysis. International Journal of Forecasting. 1999, 15: 353-375.CrossRef
23.
go back to reference Snyder-Halpern R, Thompson C, Schaffer J: Comparison of mailed vs. internet applications of the Delphi technique in clinical informatics research. Proceedings – AMIA Symposium. 2000, 809-813. Snyder-Halpern R, Thompson C, Schaffer J: Comparison of mailed vs. internet applications of the Delphi technique in clinical informatics research. Proceedings – AMIA Symposium. 2000, 809-813.
24.
go back to reference Microsoft: Microsoft Excel. 2003, Microsoft Corporation: Redmond Microsoft: Microsoft Excel. 2003, Microsoft Corporation: Redmond
25.
go back to reference SPSS: SPSS: Statistical Program for the Social Sciences version 14.0. 2005, SPSS: Chicago SPSS: SPSS: Statistical Program for the Social Sciences version 14.0. 2005, SPSS: Chicago
26.
go back to reference Sharkey S, Sharples A: An approach to consensus building using the Delphi technique: developing a learning resource in mental health. Nurse Education Today. 2001, 21: 398-408.CrossRefPubMed Sharkey S, Sharples A: An approach to consensus building using the Delphi technique: developing a learning resource in mental health. Nurse Education Today. 2001, 21: 398-408.CrossRefPubMed
27.
go back to reference Evidence-based medicine working group: Evidence-based medicine. A new approach to teaching the practice of medicine. JAMA. 1992, 268 (17): 2420-2425.CrossRef Evidence-based medicine working group: Evidence-based medicine. A new approach to teaching the practice of medicine. JAMA. 1992, 268 (17): 2420-2425.CrossRef
28.
29.
go back to reference Treacy W, Carey M: Credit risk rating systems at large US banks. Journal of Banking and Finance. 2000, 24: 167-201.CrossRef Treacy W, Carey M: Credit risk rating systems at large US banks. Journal of Banking and Finance. 2000, 24: 167-201.CrossRef
30.
go back to reference Krahnen J, Weber M: Generally accepted rating principles: a primer. Journal of Banking and Finance. 2001, 25 (1): 3-23.CrossRef Krahnen J, Weber M: Generally accepted rating principles: a primer. Journal of Banking and Finance. 2001, 25 (1): 3-23.CrossRef
31.
go back to reference Day L, Scott M, Williams R, Rechnitzer G, Walsh P, Boyle S: Development of the safe tractor assessment rating system. Journal of Agricultural Safety and Health. 2005, 11 (3): 353-364.CrossRefPubMed Day L, Scott M, Williams R, Rechnitzer G, Walsh P, Boyle S: Development of the safe tractor assessment rating system. Journal of Agricultural Safety and Health. 2005, 11 (3): 353-364.CrossRefPubMed
32.
go back to reference Stefani R: Survey of the major world sports rating systems. Journal of Applied Statistics. 1997, 24 (6): 635-646.CrossRef Stefani R: Survey of the major world sports rating systems. Journal of Applied Statistics. 1997, 24 (6): 635-646.CrossRef
35.
go back to reference Klaucke D: Evaluating public health surveillance systems. Public Health Surveillance. Edited by: Halperin W, Baker E. 1992, John Wiley & Sons Inc: New York, 26-41. Klaucke D: Evaluating public health surveillance systems. Public Health Surveillance. Edited by: Halperin W, Baker E. 1992, John Wiley & Sons Inc: New York, 26-41.
36.
go back to reference Romaguera R, German R, Klaucke D: Evaluating public health surveillance. Principles and Practice of Public Health Surveillance. Edited by: Teutsch S, Churchill R. 2000, Oxford University Press: Oxford, 176-193. Second Romaguera R, German R, Klaucke D: Evaluating public health surveillance. Principles and Practice of Public Health Surveillance. Edited by: Teutsch S, Churchill R. 2000, Oxford University Press: Oxford, 176-193. Second
37.
go back to reference Klaucke D: Evaluating public health surveillance. Principles and Practice of Public Health Surveillance. Edited by: Teutsch S, Churchill R. 2000, Oxford University Press: Oxford, 158-174. Second Klaucke D: Evaluating public health surveillance. Principles and Practice of Public Health Surveillance. Edited by: Teutsch S, Churchill R. 2000, Oxford University Press: Oxford, 158-174. Second
38.
go back to reference Rahman F, Andersson R, Svanstrom L: Potential of using existing injury information for injury surveillance at the local level in developing countries: experiences from Bangladesh. Public Health. 2000, 114 (2): 133-6.PubMed Rahman F, Andersson R, Svanstrom L: Potential of using existing injury information for injury surveillance at the local level in developing countries: experiences from Bangladesh. Public Health. 2000, 114 (2): 133-6.PubMed
39.
go back to reference Laflamme L, Svanstrom L, Schelp L: Safety Promotion Research. A Public Health Approach to Accident and Injury Prevention. 1999, Stockholm: Karolinska Institute Laflamme L, Svanstrom L, Schelp L: Safety Promotion Research. A Public Health Approach to Accident and Injury Prevention. 1999, Stockholm: Karolinska Institute
40.
go back to reference Declich S, Carter A: Public health surveillance: historical origins, methods and evaluation. Bulletin of the World Health Organization. 1994, 72 (2): 285-304.PubMedPubMedCentral Declich S, Carter A: Public health surveillance: historical origins, methods and evaluation. Bulletin of the World Health Organization. 1994, 72 (2): 285-304.PubMedPubMedCentral
41.
go back to reference World Health Organization: Overview of the WHO framework for monitoring and evaluating surveillance and response systems for communicable diseases. Weekly Epidemiological Record. 2004, 36 (79): 322-326. World Health Organization: Overview of the WHO framework for monitoring and evaluating surveillance and response systems for communicable diseases. Weekly Epidemiological Record. 2004, 36 (79): 322-326.
42.
go back to reference Graitcer P: The development of state and local injury surveillance systems. Journal of Safety Research. 1987, 18: 191-198.CrossRef Graitcer P: The development of state and local injury surveillance systems. Journal of Safety Research. 1987, 18: 191-198.CrossRef
43.
go back to reference Thacker S, Parrish R, Trowbridge F: A method for evaluating systems of epidemiologic surveillance. World Health Statistics Quarterly. 1988, 41: 11-18. Thacker S, Parrish R, Trowbridge F: A method for evaluating systems of epidemiologic surveillance. World Health Statistics Quarterly. 1988, 41: 11-18.
44.
go back to reference Thacker S, Berkelman R, Stroup D: The science of public health surveillance. Journal of Public Health Policy. 1989, Summer: 187-203.CrossRef Thacker S, Berkelman R, Stroup D: The science of public health surveillance. Journal of Public Health Policy. 1989, Summer: 187-203.CrossRef
45.
go back to reference Teutsch S: Considerations in planning a surveillance system. Principles and Practice of Public Health Surveillance. Edited by: Teutsch S, Churchill R. 2000, Oxford University Press: Oxford, 17-29. Second Teutsch S: Considerations in planning a surveillance system. Principles and Practice of Public Health Surveillance. Edited by: Teutsch S, Churchill R. 2000, Oxford University Press: Oxford, 17-29. Second
46.
go back to reference Teutsch S, Thacker S: Planning a public health surveillance system. Epidemiological Bulletin. 1995, 16 (1): 1-6.PubMed Teutsch S, Thacker S: Planning a public health surveillance system. Epidemiological Bulletin. 1995, 16 (1): 1-6.PubMed
47.
48.
go back to reference Gilbert R, Logan S: Assessing diagnostic and screening tests. Evidence-based Pediatrics and Child Health. Edited by: Moyer V. 2004, BMJ: Cornwall, 31-43.CrossRef Gilbert R, Logan S: Assessing diagnostic and screening tests. Evidence-based Pediatrics and Child Health. Edited by: Moyer V. 2004, BMJ: Cornwall, 31-43.CrossRef
49.
go back to reference Eylenbosch W, Noah D: Surveillance in Health and Disease. 1988, Oxford: Oxford University Press Eylenbosch W, Noah D: Surveillance in Health and Disease. 1988, Oxford: Oxford University Press
50.
go back to reference Langley J: The role of surveillance in reducing morbidity and mortality from injuries. Morbidity & Mortality Weekly Report. 1992, 41 (Suppl): 181-191. Langley J: The role of surveillance in reducing morbidity and mortality from injuries. Morbidity & Mortality Weekly Report. 1992, 41 (Suppl): 181-191.
51.
go back to reference Ing R, Baker S, Frankowski R, Guyer B, Hollinshead W, Pine J, Rockett I: Injury surveillance systems – strengths, weaknesses, and issues workshop. Public Health Reports. 1985, 100 (6): 582-586. Ing R, Baker S, Frankowski R, Guyer B, Hollinshead W, Pine J, Rockett I: Injury surveillance systems – strengths, weaknesses, and issues workshop. Public Health Reports. 1985, 100 (6): 582-586.
52.
go back to reference Thacker S, Stroup DF: Future directions for comprehensive public health surveillance and health information systems in the United States. American Journal of Epidemiology. 1994, 140 (5): 383-397.PubMed Thacker S, Stroup DF: Future directions for comprehensive public health surveillance and health information systems in the United States. American Journal of Epidemiology. 1994, 140 (5): 383-397.PubMed
53.
go back to reference Rychetnik L, Frommer M, Hawe P, Shiell A: Criteria for evaluating evidence on public health interventions. Journal of Epidemiology & Community Health. 2002, 56: 119-127.CrossRef Rychetnik L, Frommer M, Hawe P, Shiell A: Criteria for evaluating evidence on public health interventions. Journal of Epidemiology & Community Health. 2002, 56: 119-127.CrossRef
54.
go back to reference Hasbrouck LM, Durant T, Ward E, Gordon G: Surveillance of interpersonal violence in Kingston, Jamaica: an evaluation. Injury Control & Safety Promotion. 2002, 9 (4): 249-53.CrossRef Hasbrouck LM, Durant T, Ward E, Gordon G: Surveillance of interpersonal violence in Kingston, Jamaica: an evaluation. Injury Control & Safety Promotion. 2002, 9 (4): 249-53.CrossRef
55.
go back to reference Hedegaard H, Wake M, Hoffman R: Firearm-related injury surveillance in Colorado. American Journal of Preventive Medicine. 1998, 15 (3 Suppl): 38-45.CrossRefPubMed Hedegaard H, Wake M, Hoffman R: Firearm-related injury surveillance in Colorado. American Journal of Preventive Medicine. 1998, 15 (3 Suppl): 38-45.CrossRefPubMed
56.
go back to reference Wiersema B, Loftin C, Mullen RC, Daub EM, Sheppard MA, Smialek JE, McDowall D: Fatal firearm-related injury surveillance in Maryland. American Journal of Preventive Medicine. 1998, 15 (3 Suppl): 46-56.CrossRefPubMed Wiersema B, Loftin C, Mullen RC, Daub EM, Sheppard MA, Smialek JE, McDowall D: Fatal firearm-related injury surveillance in Maryland. American Journal of Preventive Medicine. 1998, 15 (3 Suppl): 46-56.CrossRefPubMed
57.
go back to reference McClure RJ, Burnside J: The Australian Capital Territory Injury Surveillance and Prevention Project. Academic Emergency Medicine. 1995, 2 (6): 529-34.CrossRefPubMed McClure RJ, Burnside J: The Australian Capital Territory Injury Surveillance and Prevention Project. Academic Emergency Medicine. 1995, 2 (6): 529-34.CrossRefPubMed
58.
go back to reference Klaucke D, Buehler J, Thacker S, Parrish R, Trowbridge F, Berkelman R: Guidelines for evaluating surveillance systems. Morbidity & Mortality Weekly Report CDC Surveillance Summaries. 1988, 37 (S-5): 1-18. Klaucke D, Buehler J, Thacker S, Parrish R, Trowbridge F, Berkelman R: Guidelines for evaluating surveillance systems. Morbidity & Mortality Weekly Report CDC Surveillance Summaries. 1988, 37 (S-5): 1-18.
59.
go back to reference Williams P, Webb C: The Delphi technique: a methodological discussion. Journal of Advanced Nursing. 1994, 19: 180-186.CrossRefPubMed Williams P, Webb C: The Delphi technique: a methodological discussion. Journal of Advanced Nursing. 1994, 19: 180-186.CrossRefPubMed
60.
go back to reference van Zoligen S, Klaassen C: Selection processes in a Delphi study about key qualifications in Senior Secondary Vocational Education. Technological forecasting and social change. 2003, 70: 317-340.CrossRef van Zoligen S, Klaassen C: Selection processes in a Delphi study about key qualifications in Senior Secondary Vocational Education. Technological forecasting and social change. 2003, 70: 317-340.CrossRef
61.
go back to reference Sumsion T: The Delphi technique: an adaptive research tool. British Journal of Occupational Therapy. 1998, 61 (4): 153-156.CrossRef Sumsion T: The Delphi technique: an adaptive research tool. British Journal of Occupational Therapy. 1998, 61 (4): 153-156.CrossRef
62.
go back to reference Mitchell V: The Delphi technique: an exposition and application. Technology analysis and strategic management. 1991, 3 (4): 333-358.CrossRef Mitchell V: The Delphi technique: an exposition and application. Technology analysis and strategic management. 1991, 3 (4): 333-358.CrossRef
63.
go back to reference Brockhoff K: The performance of forecasting groups in computer dialogue and face-to-face discussion. The Delphi Method: Technique and Applications. Edited by: Linstone H, Turoff M. 1975, Addison-Wesley: London, 285-311. Brockhoff K: The performance of forecasting groups in computer dialogue and face-to-face discussion. The Delphi Method: Technique and Applications. Edited by: Linstone H, Turoff M. 1975, Addison-Wesley: London, 285-311.
64.
go back to reference Hasson F, Keeney S, McKenna H: Research guidelines for the Delphi survey technique. Journal of Advanced Nursing. 2000, 32 (4): 1008-1015.PubMed Hasson F, Keeney S, McKenna H: Research guidelines for the Delphi survey technique. Journal of Advanced Nursing. 2000, 32 (4): 1008-1015.PubMed
65.
go back to reference Fink A, Kosecoff J, Chassin M, Brook R: Consensus methods: characteristics and guidelines for use. American Journal of Public Health. 1984, 74 (9): 979-983.CrossRefPubMedPubMedCentral Fink A, Kosecoff J, Chassin M, Brook R: Consensus methods: characteristics and guidelines for use. American Journal of Public Health. 1984, 74 (9): 979-983.CrossRefPubMedPubMedCentral
Metadata
Title
The development of an evaluation framework for injury surveillance systems
Authors
Rebecca J Mitchell
Ann M Williamson
Rod O'Connor
Publication date
01-12-2009
Publisher
BioMed Central
Published in
BMC Public Health / Issue 1/2009
Electronic ISSN: 1471-2458
DOI
https://doi.org/10.1186/1471-2458-9-260

Other articles of this Issue 1/2009

BMC Public Health 1/2009 Go to the issue