Skip to main content
Top
Published in: BMC Public Health 1/2010

Open Access 01-12-2010 | Correspondence

How to develop a theory-driven evaluation design? Lessons learned from an adolescent sexual and reproductive health programme in West Africa

Authors: Sara B Van Belle, Bruno Marchal, Dominique Dubourg, Guy Kegels

Published in: BMC Public Health | Issue 1/2010

Login to get access

Abstract

Background

This paper presents the development of a study design built on the principles of theory-driven evaluation. The theory-driven evaluation approach was used to evaluate an adolescent sexual and reproductive health intervention in Mali, Burkina Faso and Cameroon to improve continuity of care through the creation of networks of social and health care providers.

Methods/design

Based on our experience and the existing literature, we developed a six-step framework for the design of theory-driven evaluations, which we applied in the ex-post evaluation of the networking component of the intervention. The protocol was drafted with the input of the intervention designer. The programme theory, the central element of theory-driven evaluation, was constructed on the basis of semi-structured interviews with designers, implementers and beneficiaries and an analysis of the intervention's logical framework.

Discussion

The six-step framework proved useful as it allowed for a systematic development of the protocol. We describe the challenges at each step. We found that there is little practical guidance in the existing literature, and also a mix up of terminology of theory-driven evaluation approaches. There is a need for empirical methodological development in order to refine the tools to be used in theory driven evaluation. We conclude that ex-post evaluations of programmes can be based on such an approach if the required information on context and mechanisms is collected during the programme.
Appendix
Available only for authorised users
Literature
1.
go back to reference Chen H-T: The conceptual framework of the theory-driven perspective. Evaluation and Program Planning. 1989, 12: 391-396. 10.1016/0149-7189(89)90057-8.CrossRef Chen H-T: The conceptual framework of the theory-driven perspective. Evaluation and Program Planning. 1989, 12: 391-396. 10.1016/0149-7189(89)90057-8.CrossRef
2.
go back to reference Chen H-T, Rossi P: Issues in the theory-driven perspective. Evaluation and Program Planning. 1989, 12 (4): 299-306. 10.1016/0149-7189(89)90046-3.CrossRef Chen H-T, Rossi P: Issues in the theory-driven perspective. Evaluation and Program Planning. 1989, 12 (4): 299-306. 10.1016/0149-7189(89)90046-3.CrossRef
3.
go back to reference Campbell D, Stanley J: Experimental And Quasi-experimental Designs For Research. 1963, Skokie, Illinois: Rand-McNally Campbell D, Stanley J: Experimental And Quasi-experimental Designs For Research. 1963, Skokie, Illinois: Rand-McNally
4.
go back to reference Cronbach L: Associates Toward reform or program evaluation: aims, methods and institutional arrangements. 1980, San Francisco: Jossey-Bass Cronbach L: Associates Toward reform or program evaluation: aims, methods and institutional arrangements. 1980, San Francisco: Jossey-Bass
5.
go back to reference Chen H-T: Practical program evaluation. 2005, Thousand Oaks: SAGE PublicationsCrossRef Chen H-T: Practical program evaluation. 2005, Thousand Oaks: SAGE PublicationsCrossRef
6.
go back to reference Fulbright-Anderson A, Kubisch A, Connell J: New approaches to evaluating community initiatives. 1998, Washington, DC: Aspen Institute Fulbright-Anderson A, Kubisch A, Connell J: New approaches to evaluating community initiatives. 1998, Washington, DC: Aspen Institute
7.
go back to reference Barnes M, Matka E, Sullivan H: Evidence, understanding and complexity. Evaluation in non-linear systems. Evaluation. 2003, 9 (3): 265-284. 10.1177/13563890030093003.CrossRef Barnes M, Matka E, Sullivan H: Evidence, understanding and complexity. Evaluation in non-linear systems. Evaluation. 2003, 9 (3): 265-284. 10.1177/13563890030093003.CrossRef
8.
go back to reference Mason P, Barnes M: Constructing theories of change. Evaluation. 2007, 13 (2): 151-170. 10.1177/1356389007075221.CrossRef Mason P, Barnes M: Constructing theories of change. Evaluation. 2007, 13 (2): 151-170. 10.1177/1356389007075221.CrossRef
9.
go back to reference Weiss C: Nothing as practical as good theory: exploring theory-based evaluation for comprehensive community initiatives. New approaches to evaluating comprehensive community initiatives: concepts, methods and contexts. Edited by: Connell J, Kubisch A, Schorr L, Weiss C. 1995, Washington, DC: The Aspen Institute Weiss C: Nothing as practical as good theory: exploring theory-based evaluation for comprehensive community initiatives. New approaches to evaluating comprehensive community initiatives: concepts, methods and contexts. Edited by: Connell J, Kubisch A, Schorr L, Weiss C. 1995, Washington, DC: The Aspen Institute
10.
go back to reference Douglas FC, Gray DA, van Teijlingen ER: Using a realist approach to evaluate smoking cessation interventions targeting pregnant women and young people. BMC Health Serv Res. 2010, 10: 49-10.1186/1472-6963-10-49.CrossRefPubMedPubMedCentral Douglas FC, Gray DA, van Teijlingen ER: Using a realist approach to evaluate smoking cessation interventions targeting pregnant women and young people. BMC Health Serv Res. 2010, 10: 49-10.1186/1472-6963-10-49.CrossRefPubMedPubMedCentral
11.
go back to reference Pawson R, Tilley N: Realistic Evaluation. 1997, London: Sage Pawson R, Tilley N: Realistic Evaluation. 1997, London: Sage
12.
go back to reference Ogrinc G, Batalden P: Realist evaluation as a framework for the assessment of teaching about the improvement of care. J Nurs Educ. 2009, 48 (12): 661-667. 10.3928/01484834-20091113-08.CrossRefPubMed Ogrinc G, Batalden P: Realist evaluation as a framework for the assessment of teaching about the improvement of care. J Nurs Educ. 2009, 48 (12): 661-667. 10.3928/01484834-20091113-08.CrossRefPubMed
13.
go back to reference Connelly J: A realistic theory of health sector management. The case for critical realism. Journal of Management in Medicine. 2000, 14 (5/6): 262-271. 10.1108/02689230010362882.CrossRefPubMed Connelly J: A realistic theory of health sector management. The case for critical realism. Journal of Management in Medicine. 2000, 14 (5/6): 262-271. 10.1108/02689230010362882.CrossRefPubMed
14.
go back to reference Koenig G: Realistic evaluation and case studies: stretching the potential. Evaluation. 2009, 15 (1): 9-30. 10.1177/1356389008097869.CrossRef Koenig G: Realistic evaluation and case studies: stretching the potential. Evaluation. 2009, 15 (1): 9-30. 10.1177/1356389008097869.CrossRef
15.
go back to reference Chen H-T: Theory-driven evaluations. 1990, Newbury Park, California: Sage Publications, 1 Chen H-T: Theory-driven evaluations. 1990, Newbury Park, California: Sage Publications, 1
16.
go back to reference Van Der Knaap P: Theory-based evaluation and learning: possibilities and challenges. Evaluation. 2004, 10 (1): 16-34. 10.1177/1356389004042328.CrossRef Van Der Knaap P: Theory-based evaluation and learning: possibilities and challenges. Evaluation. 2004, 10 (1): 16-34. 10.1177/1356389004042328.CrossRef
17.
go back to reference Donaldson S: Program theory-driven evaluation science. 2007, Strategies and applications New York: Lawrence Erlbaum Associates Donaldson S: Program theory-driven evaluation science. 2007, Strategies and applications New York: Lawrence Erlbaum Associates
18.
go back to reference Stame N: Theory-based evaluation and types of complexity. Evaluation. 2004, 10 (1): 58-76. 10.1177/1356389004043135.CrossRef Stame N: Theory-based evaluation and types of complexity. Evaluation. 2004, 10 (1): 58-76. 10.1177/1356389004043135.CrossRef
19.
go back to reference Dickinson H: The evaluation of health and social care partnerships: an analysis of approaches and synthesis for the future. Health Soc Care Community. 2006, 14 (5): 375-383. 10.1111/j.1365-2524.2006.00650.x.CrossRefPubMed Dickinson H: The evaluation of health and social care partnerships: an analysis of approaches and synthesis for the future. Health Soc Care Community. 2006, 14 (5): 375-383. 10.1111/j.1365-2524.2006.00650.x.CrossRefPubMed
20.
go back to reference Kazi M, Rostila I: The practice of realist evaluation in two countries. European Evaluation Society Conference. 2002, Sevilla, Spain Kazi M, Rostila I: The practice of realist evaluation in two countries. European Evaluation Society Conference. 2002, Sevilla, Spain
21.
go back to reference MacKenzie M, Blamey A: The practice and theory. Lessons from the application of a Theories of Change approach. Evaluation. 2005, 11 (2): 151-168. 10.1177/1356389005055538.CrossRef MacKenzie M, Blamey A: The practice and theory. Lessons from the application of a Theories of Change approach. Evaluation. 2005, 11 (2): 151-168. 10.1177/1356389005055538.CrossRef
22.
go back to reference Leone L: Realistic evaluation of an illicit drug deterrence programme. Evaluation. 2008, 14 (1): 19-28. 10.1177/1356389007084673.CrossRef Leone L: Realistic evaluation of an illicit drug deterrence programme. Evaluation. 2008, 14 (1): 19-28. 10.1177/1356389007084673.CrossRef
23.
go back to reference Ying Ho S: Evaluating urban regeneration programmes in Britain. Exploring the potential of the realist approach. Evaluation. 1999, 5 (4): 422-438. 10.1177/135638999400830084.CrossRef Ying Ho S: Evaluating urban regeneration programmes in Britain. Exploring the potential of the realist approach. Evaluation. 1999, 5 (4): 422-438. 10.1177/135638999400830084.CrossRef
24.
go back to reference Pommier J, Guével M-R, Jourdan D: Evaluation of health promotion in schools: a realistic evaluation approach using mixed methods. BMC Public Health. 2010, 10 (43): Pommier J, Guével M-R, Jourdan D: Evaluation of health promotion in schools: a realistic evaluation approach using mixed methods. BMC Public Health. 2010, 10 (43):
25.
go back to reference Guichard A, Ridde V: Etude exploratoire des mécanismes de l'efficacité des interventions visant à réduire les inégalités sociales de santé. 2009, Etude pilote dans trois régions françaises: Institut National de Prévention et d'Education pour la Santé (INPES), France Guichard A, Ridde V: Etude exploratoire des mécanismes de l'efficacité des interventions visant à réduire les inégalités sociales de santé. 2009, Etude pilote dans trois régions françaises: Institut National de Prévention et d'Education pour la Santé (INPES), France
26.
go back to reference Mays N, Wyke S, D E: The evaluation of complex health policy. Evaluation. 2001, 7 (4): 405-426.CrossRef Mays N, Wyke S, D E: The evaluation of complex health policy. Evaluation. 2001, 7 (4): 405-426.CrossRef
27.
go back to reference Sullivan H, Barnes M, Matka E: Building collaborative capacity through 'Theories of change'. Early lessons from the evaluation of Health Action Zones in England. Evaluation. 2002, 8 (2): 205-226. 10.1177/1358902002008002514.CrossRef Sullivan H, Barnes M, Matka E: Building collaborative capacity through 'Theories of change'. Early lessons from the evaluation of Health Action Zones in England. Evaluation. 2002, 8 (2): 205-226. 10.1177/1358902002008002514.CrossRef
28.
go back to reference Byng R: Using the 'Realistic evaluation' framework to make a retrospective qualitative evaluation of a practice level intervention to improve primary care for patients with long-term mental illness. The 2002 EES Conference Three movements in Contemporary Evaluation: Learning, Theory and Evidence October 10-12: 2002. 2002, European Evaluation Society Byng R: Using the 'Realistic evaluation' framework to make a retrospective qualitative evaluation of a practice level intervention to improve primary care for patients with long-term mental illness. The 2002 EES Conference Three movements in Contemporary Evaluation: Learning, Theory and Evidence October 10-12: 2002. 2002, European Evaluation Society
29.
go back to reference Wong G, Greenhalgh T, Pawson R: Internet-based medical education: a realist review of what works, for whom and in what circumstances. BMC Med Educ. 2010, 10: 12-10.1186/1472-6920-10-12.CrossRefPubMedPubMedCentral Wong G, Greenhalgh T, Pawson R: Internet-based medical education: a realist review of what works, for whom and in what circumstances. BMC Med Educ. 2010, 10: 12-10.1186/1472-6920-10-12.CrossRefPubMedPubMedCentral
30.
go back to reference Blaise P, Kegels G: A realistic approach to the evaluation of the quality management movement in health care systems: a comparison between European and African contexts based on Mintzberg's organizational models. Int J Health Plann Manage. 2004, 19 (4): 337-364. 10.1002/hpm.769.CrossRefPubMed Blaise P, Kegels G: A realistic approach to the evaluation of the quality management movement in health care systems: a comparison between European and African contexts based on Mintzberg's organizational models. Int J Health Plann Manage. 2004, 19 (4): 337-364. 10.1002/hpm.769.CrossRefPubMed
31.
go back to reference Marchal B, Dedzo M, Kegels G: A realist evaluation of the management of a well-performing regional hospital in Ghana. BMC Health Services Research. 2010, 10 (24): Marchal B, Dedzo M, Kegels G: A realist evaluation of the management of a well-performing regional hospital in Ghana. BMC Health Services Research. 2010, 10 (24):
32.
go back to reference Blamey A, Mackenzie M: Theories of change and realistic evaluation. Peas in a pod or apples and oranges?. Evaluation. 2007, 13 (439-455): Blamey A, Mackenzie M: Theories of change and realistic evaluation. Peas in a pod or apples and oranges?. Evaluation. 2007, 13 (439-455):
33.
go back to reference Pedersen L, Rieper O: Is realist evaluation a realistic approach for complex reforms?. Evaluation. 2008, 14 (3): 271-293. 10.1177/1356389008090856.CrossRef Pedersen L, Rieper O: Is realist evaluation a realistic approach for complex reforms?. Evaluation. 2008, 14 (3): 271-293. 10.1177/1356389008090856.CrossRef
34.
go back to reference Davies R: Scale, complexity and the representation of theories of change. Evaluation. 2004, 10 (1): 101-121. 10.1177/1356389004043124.CrossRef Davies R: Scale, complexity and the representation of theories of change. Evaluation. 2004, 10 (1): 101-121. 10.1177/1356389004043124.CrossRef
35.
go back to reference Rogers P: Using programme theory to evaluate complicated and complex aspects of interventions. Evaluation. 2008, 14 (1): 29-48. 10.1177/1356389007084674.CrossRef Rogers P: Using programme theory to evaluate complicated and complex aspects of interventions. Evaluation. 2008, 14 (1): 29-48. 10.1177/1356389007084674.CrossRef
36.
go back to reference Lipsey M, Pollard J: Driving toward theory in program evaluation: more models to choose from. Evaluation and Program Planning. 1989, 12: 317-328. 10.1016/0149-7189(89)90048-7.CrossRef Lipsey M, Pollard J: Driving toward theory in program evaluation: more models to choose from. Evaluation and Program Planning. 1989, 12: 317-328. 10.1016/0149-7189(89)90048-7.CrossRef
37.
go back to reference Cole G: Advancing the development and application of theory-based evaluation in the practice of public health. American Journal of Evaluation. 1999, 20: 453-470.CrossRef Cole G: Advancing the development and application of theory-based evaluation in the practice of public health. American Journal of Evaluation. 1999, 20: 453-470.CrossRef
38.
go back to reference Yin R: Case study research. Design and methods. 2003, London: Sage Publications, Third Yin R: Case study research. Design and methods. 2003, London: Sage Publications, Third
39.
go back to reference Weick KE: Theory construction as disciplined imagination. Academy of Management Review. 1989, 14 (4): 516-531. 10.2307/258556. Weick KE: Theory construction as disciplined imagination. Academy of Management Review. 1989, 14 (4): 516-531. 10.2307/258556.
40.
go back to reference Bourgeois LJ: Toward a method of middle-range theorizing. Academy of Management Review. 1979, 4 (3): 443-447. 10.2307/257201. Bourgeois LJ: Toward a method of middle-range theorizing. Academy of Management Review. 1979, 4 (3): 443-447. 10.2307/257201.
41.
go back to reference Campbell C, MacPhail C: Peer education, gender and the development of critical consciousness: participatory HIV prevention by South African youth. Soc Sci Med. 2002, 55 (2): 331-345. 10.1016/S0277-9536(01)00289-1.CrossRefPubMed Campbell C, MacPhail C: Peer education, gender and the development of critical consciousness: participatory HIV prevention by South African youth. Soc Sci Med. 2002, 55 (2): 331-345. 10.1016/S0277-9536(01)00289-1.CrossRefPubMed
42.
go back to reference Määtä M, Rantala K: The evaluator as a critical interpreter. Evaluation. 2007, 13 (4): 457-476. 10.1177/1356389007082131.CrossRef Määtä M, Rantala K: The evaluator as a critical interpreter. Evaluation. 2007, 13 (4): 457-476. 10.1177/1356389007082131.CrossRef
Metadata
Title
How to develop a theory-driven evaluation design? Lessons learned from an adolescent sexual and reproductive health programme in West Africa
Authors
Sara B Van Belle
Bruno Marchal
Dominique Dubourg
Guy Kegels
Publication date
01-12-2010
Publisher
BioMed Central
Published in
BMC Public Health / Issue 1/2010
Electronic ISSN: 1471-2458
DOI
https://doi.org/10.1186/1471-2458-10-741

Other articles of this Issue 1/2010

BMC Public Health 1/2010 Go to the issue