Top

Published in:

Open Access 01-12-2021 | Research article

Using path analysis to test theory of change: a quantitative process evaluation of the MapSan trial

Authors: Sarah Bick, Helen Buxton, Rachel P. Chase, Ian Ross, Zaida Adriano, Drew Capone, Jackie Knee, Joe Brown, Rassul Nalá, Oliver Cumming, Robert Dreibelbis

Published in: BMC Public Health | Issue 1/2021

Abstract

Background

Although theory-driven evaluations should have empirical components, few evaluations of public health interventions quantitatively test the causal model made explicit in the theory of change (ToC). In the context of a shared sanitation trial (MapSan) in Maputo, Mozambique, we report findings of a quantitative process evaluation assessing intervention implementation, participant response and impacts on hypothesised intermediary outcomes on the pathway to trial health outcomes. We examine the utility of path analysis in testing intervention theory using process indicators from the intervention’s ToC.

Methods

Process data were collected through a cross-sectional survey of intervention and control compounds of the MapSan trial > 24-months post-intervention, sampling adult residents and compound leaders. Indicators of implementation fidelity (dose received, reach) and participant response (participant behaviours, intermediary outcomes) were compared between trial arms. The intervention’s ToC (formalised post-intervention) was converted to an initial structural model with multiple alternative pathways. Path analysis was conducted through linear structural equation modelling (SEM) and generalised SEM (probit model), using a model trimming process and grouped analysis to identify parsimonious models that explained variation in outcomes, incorporating demographics of respondents and compounds.

Results

Among study compounds, the MapSan intervention was implemented with high fidelity, with a strong participant response in intervention compounds: improvements were made to intermediary outcomes related to sanitation ‘quality’ – latrine cleanliness, maintenance and privacy – but not to handwashing (presence of soap / soap residue). These outcomes varied by intervention type: single-cabin latrines or multiple-cabin blocks (designed for > 20 users). Path analysis suggested that changes in intermediary outcomes were likely driven by direct effects of intervention facilities, with little contribution from hygiene promotion activities nor core elements expected to mediate change: a compound sanitation committee and maintenance fund. A distinct structural model for two compound size subgroups (≤ 20 members vs. > 20 members) explained differences by intervention type, and other contextual factors influenced specific model parameters.

Conclusions

While process evaluation found that the MapSan intervention achieved sufficient fidelity and participant response, the path analysis approach applied to test the ToC added to understanding of possible ‘mechanisms of change’, and has value in disentangling complex intervention pathways.

Trial registration

MapSan trial registration: NCT02362932 Feb-13-2015.

Available only for authorised users

Oakley A, Strange V, Bonell C, Allen E, Stephenson J, Team RS. Process evaluation in randomised controlled trials of complex interventions. BMJ. 2006;332(7538):413–6. https://doi.org/10.1136/bmj.332.7538.413.CrossRefPubMedPubMedCentral

Moore GF, Audrey S, Barker M, Bond L, Bonell C, Hardeman W, et al. Process evaluation of complex interventions: Medical Research Council guidance. BMJ. 2015;350:h1258. https://doi.org/10.1136/bmj.h1258.CrossRefPubMedPubMedCentral

Coryn CLS, Noakes LA, Westine CD, Schröter DC. A systematic review of theory-driven evaluation practice from 1990 to 2009. Am J Eval. 2010;32(2):199–226. https://doi.org/10.1177/1098214010389321.

Astbury B, Leeuw FL. Unpacking black boxes: mechanisms and theory building in evaluation. Am J Eval. 2010;31(3):363–81. https://doi.org/10.1177/1098214010371972.CrossRef

Bonell C, Fletcher A, Morton M, Lorenc T, Moore L. Realist randomised controlled trials: a new approach to evaluating complex public health interventions. Soc Sci Med. 2012;75(12):2299–306. https://doi.org/10.1016/j.socscimed.2012.08.032.CrossRefPubMed

Pawson R, Tilley N. Realistic evaluation. London: Sage; 1997.

Weiss C. Nothing as practical as a good theory: exploring theory-based evaluation in complex community initiatives for children and families. In: Connell J, Kubisch A, Schorr L, C W, editors. New approaches to evaluating community initiatives: volume 1, concepts, methods, and contexts. Washington, DC: The Aspen Institute; 1995.

De Silva MJ, Breuer E, Lee LA, Asher L, Chowdhary N, Lund C, et al. Theory of change: a theory-driven approach to enhance the Medicial research Council's framework for complex interventions. Trials. 2014;15(1):267. https://doi.org/10.1186/1745-6215-15-267.CrossRefPubMedPubMedCentral

Breuer E, Lee L, De Silva M, Lund C. Using theory of change to design and evaluate public health interventions: a systematic review. Implement Sci. 2016;11(1):63. https://doi.org/10.1186/s13012-016-0422-6.

10.

Chen H-T. Practical program evaluation: assessing and improving planning, implementation, and effectiveness. Thousand Oaks: Sage; 2005.

11.

Fletcher A, Jamal F, Moore G, Evans RE, Murphy S, Bonell C. Realist complex intervention science: applying realist principles across all phases of the Medical Research Council framework for developing and evaluating complex interventions. Evaluation (Lond). 2016;22(3):286–303. https://doi.org/10.1177/1356389016652743.CrossRef

12.

Lipsey MW. Theory as method: small theories of treatments. New Direct Prog Eval. 1993;1993(57):5–38. https://doi.org/10.1002/ev.1637.CrossRef

13.

National Institute for Health and Care Excellence. Behaviour change: general approaches. London: NICE; 2007.

14.

Smith NL. Using path analysis to develop and evaluate program theory and impact. New Direct Prog Eval. 1990;1990(47):53–7. https://doi.org/10.1002/ev.1554.CrossRef

15.

Pedhazur EJ. Multiple Regression in Behavioral Research: Explanation and Prediction. 3rd ed. Fort Worth: Harcourt Brace College Publishers; 1997.

16.

Wright S. The method of path coefficients. Ann Math Stat. 1934;5(3):161–215. https://doi.org/10.1214/aoms/1177732676.CrossRef

17.

Hennessy M, Greenberg J. Bringing it all together: modeling intervention processes using structural equation modeling. Am J Eval. 1999;20(3):471–80. https://doi.org/10.1177/109821409902000306.CrossRef

18.

Pruss-Ustun A, Wolf J, Bartram J, Clasen T, Cumming O, Freeman MC, et al. Burden of disease from inadequate water, sanitation and hygiene for selected adverse health outcomes: an updated analysis with a focus on low- and middle-income countries. Int J Hyg Environ Health. 2019;222(5):765–77. https://doi.org/10.1016/j.ijheh.2019.05.004.CrossRefPubMedPubMedCentral

19.

UNICEF, World Health Organization. Progress on household drinking water, sanitation and hygiene 2000–2017: special focus on inequalities. New York: United Nations Children’s Fund (UNICEF) and World Health Organization; 2019.

20.

UN-HABITAT. The challenge of slums: global report on human settlements 2003. London: Earthscan Publications; 2003.

21.

UN-HABITAT. Mozambique Cities Profile: Maputo, Nacala and Manica. Nairobi: United Nations Human Settlements Programme (UN-HABITAT); 2010. Report No. 9789211322675.

22.

Peal A, Evans B, Blackett I, Hawkins P, Heymans C. Fecal sludge management: a comparative analysis of 12 cities. J Water Sanit Hyg Dev. 2014;4(4):563–75. https://doi.org/10.2166/washdev.2014.026.

23.

Simiyu S, Swilling M, Cairncross S, Rheingans R. Determinants of quality of shared sanitation facilities in informal settlements: case study of Kisumu, Kenya. BMC Public Health. 2017;17(1):68. https://doi.org/10.1186/s12889-016-4009-6.CrossRefPubMedPubMedCentral

24.

Kwiringira J, Atekyereza P, Niwagaba C, Gunther I. Descending the sanitation ladder in urban Uganda: evidence from Kampala slums. BMC Public Health. 2014;14(1):624. https://doi.org/10.1186/1471-2458-14-624.CrossRefPubMedPubMedCentral

25.

Tumwebaze IK, Mosler HJ. Shared toilet users' collective cleaning and determinant factors in Kampala slums, Uganda. BMC Public Health. 2014;14(1):1260. https://doi.org/10.1186/1471-2458-14-1260.CrossRefPubMedPubMedCentral

26.

Brown J, Cumming O, Bartram J, Cairncross S, Ensink J, Holcomb D, et al. A controlled, before-and-after trial of an urban sanitation intervention to reduce enteric infections in children: research protocol for the Maputo sanitation (MapSan) study, Mozambique. BMJ Open. 2015;5(6):e008215. https://doi.org/10.1136/bmjopen-2015-008215.CrossRefPubMedPubMedCentral

27.

Water & Sanitation for the Urban Poor (WSUP). An integrated approach to peri-urban sanitation and hygiene in Maputo. Working with city authorities to improve services and practices. Policy Brief. Maputo: WSUP; 2018. https://www.wsup.com/insights/an-integrated-approach-to-peri-urban-sanitation-and-hygiene-in-maputo/.

28.

Knee J, Sumner T, Adriano Z, Anderson C, Bush F, Capone D, et al. Effects of an urban sanitation intervention on childhood enteric infection and diarrhea in Maputo, Mozambique: a controlled before-and-after trial. eLife. 2021;10. https://doi.org/10.7554/eLife.62278.

29.

Knee J, Sumner T, Adriano Z, Berendes D, de Bruijn E, Schmidt WP, et al. Risk factors for childhood enteric infection in urban Maputo, Mozambique: a cross-sectional study. Plos Neglect Trop D. 2018;12(11):e0006956. https://doi.org/10.1371/journal.pntd.0006956.CrossRef

30.

Mattson K. Final evaluation of the Water & Sanitation for the urban poor JSDF funded Maputo Peri-urban sanitation project. Report prepared for WSUP; 2016.

31.

Shiras T, Cumming O, Brown J, Muneme B, Nala R, Dreibelbis R. Shared Sanitation Management and the Role of Social Capital: Findings from an Urban Sanitation Intervention in Maputo, Mozambique. Int J Environ Res Public Health. 2018;15(10):2222. http://dx.doi.org/10.3390/ijerph15102222.

32.

Carroll C, Patterson M, Wood S, Booth A, Rick J, Balain S. A conceptual framework for implementation fidelity. Implement Sci. 2007;2(1):40. https://doi.org/10.1186/1748-5908-2-40.CrossRefPubMedPubMedCentral

33.

Linnan L, Steckler A. Process evaluation for public health interventions and research. San Francisco: Jossey-Boss: A Wiley Imprint; 2002.

34.

Capone D, Buxton H, Cumming O, Dreibelbis R, Knee J, Nala R, et al. Impact of an intervention to improve pit latrine emptying practices in low income urban neighborhoods of Maputo, Mozambique. Int J Hyg Environ Health. 2020;226:113480. https://doi.org/10.1016/j.ijheh.2020.113480.CrossRefPubMedPubMedCentral

35.

Cicchetti DV. Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychol Assess. 1994;6(4):284–90. https://doi.org/10.1037/1040-3590.6.4.284.CrossRef

36.

Schreiner M. Simple poverty scorecard-assessment tool Mozambique; 2013.

37.

Rheingans R, Anderson JD, Luyendijk R, Cumming O. Measuring disparities in sanitation access: does the measure matter? Trop Med Int Health. 2014;19(1):2–13. https://doi.org/10.1111/tmi.12220.

38.

Evans B, Hueso A, Johnston R, Norman G, Pérez E, Slaymaker T, et al. Limited services? The role of shared sanitation in the 2030 agenda for sustainable development. J Water Sanit Hyg Dev. 2017;7(3):349–51. https://doi.org/10.2166/washdev.2017.023.CrossRef

39.

Sinharoy SS, Pittluck R, Clasen T. Review of drivers and barriers of water and sanitation policies for urban informal settlements in low-income and middle-income countries. Util Policy. 2019;60:100957. https://doi.org/10.1016/j.jup.2019.100957.CrossRefPubMedPubMedCentral

40.

Tidwell JB, Chipungu J, Bosomprah S, Aunger R, Curtis V, Chilengi R. Effect of a behaviour change intervention on the quality of peri-urban sanitation in Lusaka, Zambia: a randomised controlled trial. Lancet Planet Health. 2019;3(4):e187–e96. https://doi.org/10.1016/S2542-5196(19)30036-1.CrossRefPubMed

41.

Tumwebaze IK, Mosler HJ. Effectiveness of group discussions and commitment in improving cleaning behaviour of shared sanitation users in Kampala, Uganda slums. Soc Sci Med. 2015;147:72–9. https://doi.org/10.1016/j.socscimed.2015.10.059.CrossRefPubMed

42.

Alam MU, Winch PJ, Saxton RE, Nizame FA, Yeasmin F, Norman G, et al. Behaviour change intervention to improve shared toilet maintenance and cleanliness in urban slums of Dhaka: a cluster-randomised controlled trial. Trop Med Int Health. 2017;22(8):1000–11. https://doi.org/10.1111/tmi.12902.

43.

Isunju JB, Schwartz K, Schouten MA, Johnson WP, van Dijk MP. Socio-economic aspects of improved sanitation in slums: a review. Public Health. 2011;125(6):368–76. https://doi.org/10.1016/j.puhe.2011.03.008.CrossRefPubMed

44.

Tumwebaze IK. Prevalence and determinants of the cleanliness of shared toilets in Kampala slums, Uganda. J Public Health. 2013;22(1):33–9. https://doi.org/10.1007/s10389-013-0590-7.

45.

Hanssen CE. Structural equation modeling as a tool for multisite evaluation. 2004.

46.

Bagozzi RP, Yi Y. Specification, evaluation, and interpretation of structural equation models. J Acad Market Sci. 2012;40(1):8–34. https://doi.org/10.1007/s11747-011-0278-x.CrossRef

47.

MacCallum RC, Austin JT. Applications of structural equation modeling in psychological research. Annu Rev Psychol. 2000;51(1):201–26. https://doi.org/10.1146/annurev.psych.51.1.201.CrossRefPubMed

48.

Bollen KA, Noble MD. Structural equation models and the quantification of behavior. Proc Natl Acad Sci U S A. 2011;108(Suppl 3):15639–46. https://doi.org/10.1073/pnas.1010661108.

49.

VanderWeele TJ. Mediation analysis: a Practitioner's guide. Annu Rev Public Health. 2016;37(1):17–32. https://doi.org/10.1146/annurev-publhealth-032315-021402.CrossRefPubMed

50.

Aunger R, Curtis V. Behaviour Centred design: towards an applied science of behaviour change. Health Psychol Rev. 2016;10(4):425–46. https://doi.org/10.1080/17437199.2016.1219673.CrossRefPubMedPubMedCentral

51.

Petraitis PS, Dunham AE, Niewiarowski PH. Inferring multiple causality: the limitations of path analysis. Funct Ecol. 1996;10(4):421–31. https://doi.org/10.2307/2389934.CrossRef

52.

Bonell C, Jamal F, Melendez-Torres GJ, Cummins S. 'Dark logic': theorising the harmful consequences of public health interventions. J Epidemiol Community Health. 2015;69(1):95–8. https://doi.org/10.1136/jech-2014-204671.CrossRefPubMed

Title: Using path analysis to test theory of change: a quantitative process evaluation of the MapSan trial
Authors: Sarah Bick
Helen Buxton
Rachel P. Chase
Ian Ross
Zaida Adriano
Drew Capone
Jackie Knee
Joe Brown
Rassul Nalá
Oliver Cumming
Robert Dreibelbis
Publication date: 01-12-2021
Publisher: BioMed Central
Published in: BMC Public Health / Issue 1/2021
Electronic ISSN: 1471-2458
DOI: https://doi.org/10.1186/s12889-021-11364-w

2024 ASCO Annual Meeting

Springer Medicine

Using path analysis to test theory of change: a quantitative process evaluation of the MapSan trial

Abstract

Background

Methods

Results

Conclusions

Trial registration

2024 ASCO Annual Meeting

Springer Medicine

Abstract

Background

Methods

Results

Conclusions

Trial registration

Please log in to get access to this content

Other articles of this Issue 1/2021

Evaluating the impact of Archway: a personalized program for 1st year student success and mental health and wellbeing

Syringe service program-based telemedicine linkage to opioid use disorder treatment: protocol for the STAMINA randomized control trial

A multifaceted approach increased staff confidence to develop outside of school hours care as a health promoting setting

Working conditions and health status of 6,317 front line public health workers across five provinces in China during the COVID-19 epidemic: a cross-sectional study

A review of the inclusion of equity stratifiers for the measurement of health inequalities within health and social care data collections in Ireland

Prevalences of metabolic syndrome and its sex-specific association with socioeconomic status in rural China: a cross-sectional study