Evaluating the effectiveness of a multifaceted, multilevel continuous quality improvement program in primary health care: developing a realist theory of change

Variation in effectiveness of continuous quality improvement (CQI) interventions between services is commonly reported, but with little explanation of how contextual and other factors may interact to produce this variation. Therefore, there is scant information available on which policy makers can draw to inform effective implementation in different settings. In this paper, we explore how patterns of change in delivery of services may have been achieved in a diverse range of health centers participating in a wide-scale program to achieve improvements in quality of care for Indigenous Australians.

We elicited key informants’ interpretations of factors explaining patterns of change in delivery of guideline-scheduled services over three or more years of a wide-scale CQI project, and inductively analyzed these interpretations to propose fine-grained realist hypotheses about what works for whom and in what circumstances. Data were derived from annual clinical audits from 36 health centers operating in diverse settings, quarterly project monitoring reports, and workshops with 12 key informants who had key roles in project implementation. We abstracted potential context-mechanism-outcome configurations from the data, and based on these, identified potential program-strengthening strategies.

Several context-specific, mechanism-based explanations for effectiveness of this CQI project were identified. These were collective valuing of clinical data for improvement purposes; collective efficacy; and organizational change towards a population health orientation. Health centers with strong central management of CQI, and those in which CQI efforts were more dependent on local health center initiative and were adapted to resonate with local priorities were both favorable contexts for collective valuing of clinical data. Where health centers had prior positive experiences of collaboration, effects appeared to be achieved at least partly through the mechanism of collective efficacy. Strong community linkages, staff ability to identify with patients, and staff having the skills and support to take broad ranging action, were favorable contexts for the mechanism of increased population health orientation.

Our study provides evidence to support strategies for program strengthening described in the literature, and extends the understanding of mechanisms through which strategies may be effective in achieving particular outcomes in different contexts.