Skip to main content
Top
Published in: BMC Health Services Research 1/2017

Open Access 01-12-2017 | Debate

The challenge of complexity in evaluating health policies and programs: the case of women’s participatory groups to improve antenatal outcomes

Authors: Sara Van Belle, Susan Rifkin, Bruno Marchal

Published in: BMC Health Services Research | Issue 1/2017

Login to get access

Abstract

Background

During the last years, randomized designs have been promoted as the cornerstone of evidence-based policymaking. Also in the field of community participation, Random Control Trials (RCTs) have been the dominant design, used for instance to examine the contribution of community participation to health improvement. We aim at clarifying why RCTs and related (quasi-) experimental designs may not be the most appropriate approach to evaluate such complex programmes.

Results

We argue that the current methodological debate could be more fruitful if it would start from the position that the choice of designs should fit the nature of the program and research questions rather than be driven by methodological preferences. We present how realist evaluation, a theory-driven approach to research and evaluation, is a relevant methodology that could be used to assess whether and how community participation works. Using the realist evaluation approach to examine the relationship between participation and action of women groups and antenatal outcomes would enable evaluators to examine in detail the underlying mechanisms which influence actual practices and outcomes, as well as the context conditions required to make it work.

Conclusions

Realist research in fact allows opening the black boxes of “community” and “participation” in order to examine the role they play in ensuring cost-effective, sustainable interventions. This approach yields important information for policy makers and programme managers considering how such programs could be implemented in their own setting.
Literature
1.
go back to reference MRC. A framework for the development and evaluation of RCTs for complex interventions to improve health. London: Medical Research Council; 2000. MRC. A framework for the development and evaluation of RCTs for complex interventions to improve health. London: Medical Research Council; 2000.
2.
go back to reference Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M. Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ. 2008;337:a1655.CrossRefPubMedPubMedCentral Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M. Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ. 2008;337:a1655.CrossRefPubMedPubMedCentral
3.
go back to reference Moore G, Audrey S, Barker M, Bond L, Bonell C, Hardeman W, Moore L, O'Cathain A, Tinati T, Wight D, et al. Process evaluation of complex interventions. UK Medical research Council (MRC) guidance. London: Medical Research Council; 2015. Moore G, Audrey S, Barker M, Bond L, Bonell C, Hardeman W, Moore L, O'Cathain A, Tinati T, Wight D, et al. Process evaluation of complex interventions. UK Medical research Council (MRC) guidance. London: Medical Research Council; 2015.
4.
go back to reference Rifkin SB. Examining the links between community participation and health outcomes: a review of the literature. Health Policy Plan. 2014;29(6):ii98–ii106.CrossRefPubMedPubMedCentral Rifkin SB. Examining the links between community participation and health outcomes: a review of the literature. Health Policy Plan. 2014;29(6):ii98–ii106.CrossRefPubMedPubMedCentral
5.
go back to reference Prost A, Colbourn T, Seward N, Azad K, Coomarasamy A, Copas A, Houweling T, Fottrell E, Kuddus A, Lewycka S, et al. Women’s groups practising participatory learning and action to improve maternal and newborn health in low-resource settings: a systematic review and meta-analysis. Lancet. 2013;381:1736–46.CrossRefPubMedPubMedCentral Prost A, Colbourn T, Seward N, Azad K, Coomarasamy A, Copas A, Houweling T, Fottrell E, Kuddus A, Lewycka S, et al. Women’s groups practising participatory learning and action to improve maternal and newborn health in low-resource settings: a systematic review and meta-analysis. Lancet. 2013;381:1736–46.CrossRefPubMedPubMedCentral
6.
go back to reference Rifkin S. Trials of participation to improve maternal and newborn health. Lancet. 2013;382(9893):681.CrossRefPubMed Rifkin S. Trials of participation to improve maternal and newborn health. Lancet. 2013;382(9893):681.CrossRefPubMed
7.
go back to reference Rifkin SB. Paradigms Lost: Toward a new understanding of community participation in health programmes. Acta Trop. 1996;61:79–92.CrossRefPubMed Rifkin SB. Paradigms Lost: Toward a new understanding of community participation in health programmes. Acta Trop. 1996;61:79–92.CrossRefPubMed
8.
go back to reference Prost A, Rosato M, Tripathy P, Costello A. Trials of participation to improve maternal and newborn health – Authors' reply. Lancet. 2013;382(9893):681–2.CrossRefPubMed Prost A, Rosato M, Tripathy P, Costello A. Trials of participation to improve maternal and newborn health – Authors' reply. Lancet. 2013;382(9893):681–2.CrossRefPubMed
9.
go back to reference Gilson L. Health Policy and Systems Research. A methodology reader. Geneva: Alliance for Health Policy and Systems Research & World Health Organisation; 2012. Gilson L. Health Policy and Systems Research. A methodology reader. Geneva: Alliance for Health Policy and Systems Research & World Health Organisation; 2012.
10.
go back to reference Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Pettigrew M. Developing and evaluating complex interventions: new guidance. London: Medical Research Council; 2008. Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Pettigrew M. Developing and evaluating complex interventions: new guidance. London: Medical Research Council; 2008.
11.
go back to reference Cartwright N. RCTs, evidence and predicting policy effectiveness. In: Kincaid H, editor. The Oxford Handbook of Philosophy of Social Science. edn ed. Oxford: Oxford University Press; 2012. Cartwright N. RCTs, evidence and predicting policy effectiveness. In: Kincaid H, editor. The Oxford Handbook of Philosophy of Social Science. edn ed. Oxford: Oxford University Press; 2012.
12.
go back to reference Hedström P, Ylikoski P. Causal mechanisms in the social sciences. Annu Rev Sociol. 2010;36:49–67.CrossRef Hedström P, Ylikoski P. Causal mechanisms in the social sciences. Annu Rev Sociol. 2010;36:49–67.CrossRef
13.
go back to reference Udehn L. Methodological Individualism: Background, History and Meaning. London: Routledge; 2001.CrossRef Udehn L. Methodological Individualism: Background, History and Meaning. London: Routledge; 2001.CrossRef
14.
go back to reference Byrne D, Uprichard E. Useful complex causality. In: Kincaid H, editor. The Oxford Handbook of Philosophy of Social Science. edn ed. Oxford: Oxford University Press; 2012. Byrne D, Uprichard E. Useful complex causality. In: Kincaid H, editor. The Oxford Handbook of Philosophy of Social Science. edn ed. Oxford: Oxford University Press; 2012.
15.
go back to reference Bhaskar R, Lawson T. Introduction. Basic texts and developments. In: Archer M, Bhaskar R, Collier A, Lawson T, Norrie A, editors. Critical realism Essential readings. edn ed. London & New York: Routledge; 1998. p. 3–15. Bhaskar R, Lawson T. Introduction. Basic texts and developments. In: Archer M, Bhaskar R, Collier A, Lawson T, Norrie A, editors. Critical realism Essential readings. edn ed. London & New York: Routledge; 1998. p. 3–15.
16.
go back to reference Pawson R. The science of evaluation: a realist manifesto. London: SAGE Publications; 2013.CrossRef Pawson R. The science of evaluation: a realist manifesto. London: SAGE Publications; 2013.CrossRef
17.
go back to reference Pawson R, Tilley N. Realistic Evaluation. London: Sage; 1997. Pawson R, Tilley N. Realistic Evaluation. London: Sage; 1997.
18.
go back to reference Merton RK. Social theory and social structure. New York: The Free Press; 1968. Merton RK. Social theory and social structure. New York: The Free Press; 1968.
19.
go back to reference Pawson R, Tilley N. Realistic Evaluation Bloodlines. Am J Eval. 2001;22(3):317–24.CrossRef Pawson R, Tilley N. Realistic Evaluation Bloodlines. Am J Eval. 2001;22(3):317–24.CrossRef
20.
go back to reference Mansuri G, Rao V. Localizing Development. Does participation work? edn ed. Washington, DC: The World Bank; 2013. Mansuri G, Rao V. Localizing Development. Does participation work? edn ed. Washington, DC: The World Bank; 2013.
21.
go back to reference WHO. WHO recommendation on community mobilization through facilitated participatory learning and action cycles with women’s groups for maternal and newborn health. Geneva: World Health Organisation; 2014. WHO. WHO recommendation on community mobilization through facilitated participatory learning and action cycles with women’s groups for maternal and newborn health. Geneva: World Health Organisation; 2014.
22.
go back to reference Goicolea I, Vives-Cases C, San Sebastian M, Marchal B, Kegels G, Hurtig AK. How do primary health care teams learn to integrate intimate partner violence (IPV) management? A realist evaluation protocol. Implement Sci. 2013;8:36.CrossRefPubMedPubMedCentral Goicolea I, Vives-Cases C, San Sebastian M, Marchal B, Kegels G, Hurtig AK. How do primary health care teams learn to integrate intimate partner violence (IPV) management? A realist evaluation protocol. Implement Sci. 2013;8:36.CrossRefPubMedPubMedCentral
23.
go back to reference Prashanth N, Marchal B, Hoerée T, Devadasan N, Macq J, Kegels G, Criel B. How does capacity building of health managers work? A realist evaluation study protocol. BMJ Open. 2012;2:e000882.CrossRefPubMedPubMedCentral Prashanth N, Marchal B, Hoerée T, Devadasan N, Macq J, Kegels G, Criel B. How does capacity building of health managers work? A realist evaluation study protocol. BMJ Open. 2012;2:e000882.CrossRefPubMedPubMedCentral
24.
go back to reference Vareilles G, Pommier J, Kane S, Pictet G, Marchal B. Understanding the motivation and performance of community health volunteers involved in the delivery of health programs: a realist evaluation protocol. BMJ Open. 2015;5:e006752.CrossRefPubMedPubMedCentral Vareilles G, Pommier J, Kane S, Pictet G, Marchal B. Understanding the motivation and performance of community health volunteers involved in the delivery of health programs: a realist evaluation protocol. BMJ Open. 2015;5:e006752.CrossRefPubMedPubMedCentral
25.
go back to reference Randell R, Greenhalgh J, Hindmarsh J, Dowding D, Jayne D, Pearman A, Gardner P, Croft J, Kotze A. Integration of robotic surgery into routine practice and impacts on communication, collaboration, and decision making: a realist process evaluation protocol. Implem Sci. 2014;9(52). Randell R, Greenhalgh J, Hindmarsh J, Dowding D, Jayne D, Pearman A, Gardner P, Croft J, Kotze A. Integration of robotic surgery into routine practice and impacts on communication, collaboration, and decision making: a realist process evaluation protocol. Implem Sci. 2014;9(52).
26.
go back to reference Mukumbang F, Van Belle S, Marchal B, Van Wyck B. A realist evaluation of the antiretroviral treatment adherence club programme in selected primary health care facilities in the metropolitan area of Western Cape Province, South Africa: A study protocol. BMJ Open. 2016;6:e009977.CrossRefPubMedPubMedCentral Mukumbang F, Van Belle S, Marchal B, Van Wyck B. A realist evaluation of the antiretroviral treatment adherence club programme in selected primary health care facilities in the metropolitan area of Western Cape Province, South Africa: A study protocol. BMJ Open. 2016;6:e009977.CrossRefPubMedPubMedCentral
27.
go back to reference Goicolea I, Hurtig AK, San Sebastian M, Vives-Cases C, Marchal B. Developing a programme theory to explain how primary health care teams learn to respond to intimate partner violence: A realist case-study. BMC Health Serv Res. 2015;15:228.CrossRefPubMedPubMedCentral Goicolea I, Hurtig AK, San Sebastian M, Vives-Cases C, Marchal B. Developing a programme theory to explain how primary health care teams learn to respond to intimate partner violence: A realist case-study. BMC Health Serv Res. 2015;15:228.CrossRefPubMedPubMedCentral
28.
go back to reference Prashanth N, Marchal B, Devadasan N, Kegels G, Criel B. Advancing the application of systems thinking in health: A realist evaluation of a capacity building programme for district managers in Tumkur, India. Health Res Policy Syst. 2014;12:42.CrossRefPubMedPubMedCentral Prashanth N, Marchal B, Devadasan N, Kegels G, Criel B. Advancing the application of systems thinking in health: A realist evaluation of a capacity building programme for district managers in Tumkur, India. Health Res Policy Syst. 2014;12:42.CrossRefPubMedPubMedCentral
29.
go back to reference Vareilles G, Marchal B, Kane S, Petric T, Pictet G, Pommier J. Understanding the motivation and performance of community health volunteers involved in the delivery of health programmes in Kampala, Uganda: a realist evaluation. BMJ Open. 2015;5:e006752.CrossRefPubMedPubMedCentral Vareilles G, Marchal B, Kane S, Petric T, Pictet G, Pommier J. Understanding the motivation and performance of community health volunteers involved in the delivery of health programmes in Kampala, Uganda: a realist evaluation. BMJ Open. 2015;5:e006752.CrossRefPubMedPubMedCentral
30.
go back to reference Hernandez A, Hurtig A, San Sebastian M. More than a checklist: a realist evaluation of supervision of mid-level health workers in rural Guatemala. BMC Health Serv Res. 2014;14:112.CrossRefPubMedPubMedCentral Hernandez A, Hurtig A, San Sebastian M. More than a checklist: a realist evaluation of supervision of mid-level health workers in rural Guatemala. BMC Health Serv Res. 2014;14:112.CrossRefPubMedPubMedCentral
31.
go back to reference Kwamie A, van Dijk H, Agyepong IA. Advancing the application of systems thinking in health: realist evaluation of the Leadership Development Programme for district manager decision-making in Ghana. Health Res Policy Syst. 2014;12:29.CrossRefPubMedPubMedCentral Kwamie A, van Dijk H, Agyepong IA. Advancing the application of systems thinking in health: realist evaluation of the Leadership Development Programme for district manager decision-making in Ghana. Health Res Policy Syst. 2014;12:29.CrossRefPubMedPubMedCentral
32.
go back to reference Adams A, Sedalia S, McNab S, Sarker M. Lessons learned in using realist evaluation to assess maternal and newborn health programming in rural Bangladesh. Health Policy Plan. 2015;31(2):267–75.CrossRefPubMed Adams A, Sedalia S, McNab S, Sarker M. Lessons learned in using realist evaluation to assess maternal and newborn health programming in rural Bangladesh. Health Policy Plan. 2015;31(2):267–75.CrossRefPubMed
33.
go back to reference Greenhalgh T, Wong G, Jagosh J, Greenhalgh J, Manzano A, Westhorp G, Pawson R. Protocol—the RAMESES II study: developing guidance and reporting standards for realist evaluation. BMJ Open. 2015;5:e008567.CrossRefPubMedPubMedCentral Greenhalgh T, Wong G, Jagosh J, Greenhalgh J, Manzano A, Westhorp G, Pawson R. Protocol—the RAMESES II study: developing guidance and reporting standards for realist evaluation. BMJ Open. 2015;5:e008567.CrossRefPubMedPubMedCentral
34.
go back to reference Wong G, Westhorp G, Manzano A, Greenhalgh J, Jagosh J, Greenhalgh T. RAMESES II reporting standards for realist evaluations. BMC Med. 2016;14:–96. Wong G, Westhorp G, Manzano A, Greenhalgh J, Jagosh J, Greenhalgh T. RAMESES II reporting standards for realist evaluations. BMC Med. 2016;14:–96.
35.
go back to reference Bonell C, Fletcher A, Morton M, Lorenc T, Moore L. Realist randomised controlled trials: a new approach to evaluating complex public health interventions. Soc Sci Med. 2012;75(12):2299–306.CrossRefPubMed Bonell C, Fletcher A, Morton M, Lorenc T, Moore L. Realist randomised controlled trials: a new approach to evaluating complex public health interventions. Soc Sci Med. 2012;75(12):2299–306.CrossRefPubMed
36.
go back to reference Marchal B, Westhorp G, Wong G, Van Belle S, Greenhalgh T, Kegels G, Pawson G. Realist RCTs of complex interventions - an oxymoron. Soc Sci Med. 2013;94:124–8.CrossRefPubMed Marchal B, Westhorp G, Wong G, Van Belle S, Greenhalgh T, Kegels G, Pawson G. Realist RCTs of complex interventions - an oxymoron. Soc Sci Med. 2013;94:124–8.CrossRefPubMed
37.
go back to reference Jamal F, Fletcher A, Shackleton N, Elbourne D, Viner R, Bonell C. The three stages of building and testing mid- level theories in a realist RCT: a theoretical and methodological case-example. Trials. 2015;16:466.CrossRefPubMedPubMedCentral Jamal F, Fletcher A, Shackleton N, Elbourne D, Viner R, Bonell C. The three stages of building and testing mid- level theories in a realist RCT: a theoretical and methodological case-example. Trials. 2015;16:466.CrossRefPubMedPubMedCentral
38.
go back to reference Van Belle S, Wong G, Westhorp G, Pearson M, Emmel N, Manzano A, Marchal B. Can "realist" randomised controlled trials be genuinely realist? Trials. 2016;17:313.CrossRefPubMedPubMedCentral Van Belle S, Wong G, Westhorp G, Pearson M, Emmel N, Manzano A, Marchal B. Can "realist" randomised controlled trials be genuinely realist? Trials. 2016;17:313.CrossRefPubMedPubMedCentral
39.
go back to reference Marchal B, Van Belle S, van Olmen J, Hoerée T, Kegels G. Is realist evaluation keeping its promise? A literature review of methodological practice in health systems research. Evaluation. 2012;18(2):192–212.CrossRef Marchal B, Van Belle S, van Olmen J, Hoerée T, Kegels G. Is realist evaluation keeping its promise? A literature review of methodological practice in health systems research. Evaluation. 2012;18(2):192–212.CrossRef
Metadata
Title
The challenge of complexity in evaluating health policies and programs: the case of women’s participatory groups to improve antenatal outcomes
Authors
Sara Van Belle
Susan Rifkin
Bruno Marchal
Publication date
01-12-2017
Publisher
BioMed Central
Published in
BMC Health Services Research / Issue 1/2017
Electronic ISSN: 1472-6963
DOI
https://doi.org/10.1186/s12913-017-2627-z

Other articles of this Issue 1/2017

BMC Health Services Research 1/2017 Go to the issue