Paying for performance to improve the delivery of health interventions in low and middle‐income countries
Abstract
This is a protocol for a Cochrane Review (Intervention). The objectives are as follows:
The overall objective of the systematic review is to summarise the current evidence for the effects of paying for performance on the provision of health care and health outcomes in low and middle‐income settings.
The analyses will focus on the following comparisons:
-
Paying for performance targeted at health workers vs no paying for performance
-
Paying for performance targeted at facilities vs no paying for performance
-
Paying for performance targeted at areas or groups of providers vs no paying for performance
-
Paying for performance targeted at national governments (aid) vs no paying for performance
-
Paying for performance targeted at multiple levels vs no paying for performance
-
Comparisons of different types of paying for performance, including paying for performance targeted at different levels, and different magnitudes of incentives
Background
Description of the condition
Improving the performance of health care delivery systems is an important objective in high‐income settings but even more critically in low and middle‐income settings, where resources for health are much more constrained.
Paying for performance (sometimes shortened to P4P) is currently receiving increased attention as a strategy for improving the performance of healthcare providers, organisations and governments. It is also promoted as an important tool for achieving the health Millennium Development Goals, improving the effectiveness of development aid, and motivating patients to improve their attendance at health facilities and compliance with recommended health interventions. However, there is currently a lack of rigorous evidence on the effectiveness of these strategies in improving health care and health, particularly in lower income countries (Oxman 2008; Eldridge 2009).
Description of the intervention
Paying for performance refers to the transfer of money or material goods conditional on taking a measurable action or achieving a predetermined performance target (CGD working group on performance‐based incentives (Eichler 2006). Paying for performance is also referred to as results‐based financing, performance‐based funding, and output based aid. While paying for performance is relatively a simple concept, it includes a wide range of interventions that vary with respect to the level at which the incentives are targeted (recipients of healthcare, individual providers of healthcare, health care facilities, private sector organizations, public sector organizations and national or sub‐national levels). Paying for performance interventions also measure specific targeted results such as health outcomes, delivery of effective interventions (for instance immunization), utilization of services (such as prenatal visits or births at an accredited facility), quality of care etc. Also central to paying for performance are interventions on ancillary components schemes such as increasing the availability of resources to health care, education, supplies, technical support or training, monitoring and feedback, increasing salaries, construction of new facilities, improvements in planning and management or information systems (Oxman 2008).
While it is conceivable that pay increases designed to increase motivation and retention of staff might fall within this definition (coming to work is, after all, a measurable action), in this review we focus on reforms which are explicitly linked to changing patterns of activity, output or outcome indicators (thus excluding routine changes to pay or public funding flows or user fee regimes).
Another systematic review has recently addressed the use of conditional cash transfers for service users (demand side paying for performance) for improving the uptake of health interventions in low and middle‐income countries (LMIC) (Lagarde 2007). This review will therefore focus on evidence of the impacts of supply side paying for performance aimed at improving the delivery of health interventions. We will include all impact evaluations of paying for performance interventions which have a supply side component.
In this review, paying for performance includes both paying for performance schemes (including ancillary components) and paying for performance per se (where any ancillary components are controlled for).
How the intervention might work
On one level, paying for performance by individuals is nothing new. It has taken the form of user fees, and in many low and middle income countries it remains one of the main forms of health financing. However, public funding for health (including aid funding, where this is channeled through governments) has traditionally not been linked to specific activities, but has taken the form of budget flows, which are linked to indicators such as staffing levels or bed numbers (for facilities), inputs (such as estimated drug needs), population numbers (for regions and districts, in some cases) and also historical trends in expenditure (all modified by overall budget constraints).
These bureaucratic mechanisms offer the advantage of stability, and predictability, and rely on local clinical judgement as to how and what services to offer. The disadvantage, however, is that health systems based on budget funding and salaried staff can lack incentives to improve quality, to increase outputs and to improve outcomes. Paying for performance aims to reintroduce those incentives by linking pay (at individual or facility level) to desired activities and/or outcome indicators. It may in addition increase resources (by providing supplementary funding) or may be an alternative mechanism for channeling existing funding resources (substituting for existing funds).
In OECD countries, paying for performance is generally described as a tool for improving quality (Christianson 2007). In LMICs, however, it generally has wider objectives (Eldridge 2009), including:
-
to increase the allocative efficiency of health services (by encouraging the provision of high priority and cost effective services)
-
to increase the technical efficiency (by making better use of existing resources such as health staff)
-
to improve equity of outcomes (for example, by encouraging expansion of services to hard‐to‐reach groups)
Paying for performance for providers (P4P4P) is clearly premised on the assumption that for these three dimensions to shift, a change in behaviour on the provider side is required. If, however, the barriers are more connected with demand side factors (such as low affordability of services), then paying for performance for providers alone will not be effective.
Paying for performance for providers in LMICs can operate at a number of levels. It can be offered directly to health workers (in public, private or private not‐for‐profit sectors). It can be linked to facility budgets. It can be used to set budget or supplement budgets at higher organisational units, such as health districts or regions. It can also be used at national level, in particular by donor organisations negotiating aid to a national health sector. Clearly, incentives would be expected to operate differently at these different levels: incentives to individuals are likely to be more directly motivating (incentives to organisations only affect behaviour indirectly, if passed on in some way to individuals), but may undermine cooperation (unlike organisational incentives, which might be expected to reinforce cooperation).
It seems intuitive that paying more money for the delivery of effective services will improve health care, but health care does not operate like a classic free market. Human behaviour is complex and there are many theories that attempt to explain both health behaviour and professional behaviour. Principalagent theory addresses relationships where one individual (the patient) cannot directly observe or know the level of skill or effort expended by the other individual (the professional) doing the contracted work. Because patients do not have perfect knowledge of their medical condition, their need for care, or the expected outcome of health care services, they are willing to have health care professionals act as their agents in providing information and services. Because patients have asymmetric information about the need for and outcomes of health care, patient demand for health care may be unresponsive to technical quality. Therefore, one theoretical advantage of performance pay is that explicit financial incentives are provided even when patient demand for health care is unresponsive to quality. Professional effort in providing high quality is rewarded, regardless of whether patients recognise it. This theoretical advantage relies however on a whole host of assumptions, including the ability to assess quality, the linkage of paying for performance systems with quality measures, the absence of adverse consequences etc.
Moreover, as indicated above, in LMICs in particular, paying for performance is being deployed for a wide range of reasons other than quality. It is envisaged more ambitiously as a tool to increase the responsiveness of staff and the health system generally to priority areas.
It is also important to note that although financial incentives and health care payment systems are likely to have an important influence on professional behaviour, this influence is far from exclusive. In economic terms, professionals are viewed as maximizing their utility function. (Utility can be defined as well‐being.) Important factors included in the utility function, besides income, are professional and social status, altruistic concerns, the cost of the effort to provide care, and the uncertainty of the clinical effectiveness of treatment. Moreover there may be other barriers to changing professional behaviour, even when professionals are motivated, including patient factors, lack of time, lack of technical skills, lack of resources, and organisational constraints.
It is generally accepted that professionals are motivated by the satisfaction of doing their jobs well (intrinsic motivation). Indeed, it is doubtful whether some valued‐but‐difficult‐ to‐observe dimensions of quality (such as empathy or listening in the medical encounter) would be provided at all if physicians were solely interested in income. Thus, professionals have both non‐monetary (that is, personal ethics, professional norms, regulatory control, clinical uncertainty) and monetary (from the payment system) incentives, all of which affect effort. It is possible that financial incentives may dilute professionals’ intrinsic motivation. On the other hand, where health workers pay is low in absolute terms, incentives may be an important channel to improve motivation through increasing their income levels.
The timescale of evaluation is another important consideration. Financial incentives might be effective in the short run for simple and distinct, well‐defined behavioural goals, but these are not necessarily sustained in the longer run.
Paying for performance schemes are often accompanied by ancillary features, such as an increase in resources. For paying for performance schemes it may be impossible to disentangle the impact of paying for performance per se from the impact of increased resources.
Why it is important to do this review
Although both demand and supply side paying for performance are widely advocated in LMICs, there are currently no systematic reviews of the impacts of supply‐side paying for performance in LMICs (Oxman 2008). Several non‐Cochrane systematic reviews have addressed the impacts of paying for performance or financial incentives in high‐income countries. Another Cochrane review (Giuffrida 1999) assessed evidence on target payments for primary care. However, this had a more limited scope (focusing on target payments alone) and only found OECD‐country studies. With the growth in interest in paying for performance in developing countries, it is believed that more rigorous studies may have been produced in the last few years which can now warrant a review focused on the LMICs.
While reviews of schemes in high‐income countries can help to inform decisions in LMICs, there are several reasons for undertaking a review of the impacts of paying for performance in LMICs specifically. The potential benefits, harms and costs of paying for performance may be greater in LMICs, where there are fewer resources and financial incentives than in high‐income countries, weak health systems, inadequate supplies, facilities and human resources and greater inequities; and where paying for performance schemes are often introduced by donors and include ancillary components, such as increased resources and technical support.
Paying for performance is a complex intervention with uncertain benefits and potential harms. It may, just to give one example, lead to the concentration of resources in areas where targets are easier to meet (which typically are better served areas), thus increasing inequity of provision. The extent to which benefits attributed to paying for performance in LMICs are attributable to conditionality (versus ancillary components of paying for performance schemes in LMICs such as increased resources and technical support) is also uncertain. Paying for performance may not be a good use of resources, even when it is effective, due to potentially small effects and high costs. For these reasons a systematic review of evaluations of the impacts of paying for performance is needed to inform decisions about whether and when to use paying for performance, how to design these schemes, and how to monitor and evaluate them in LMICs.
Finally, this is an area of growing interest for funders and for LMIC governments, so a review of evidence on effectiveness would be timely.
Objectives
The overall objective of the systematic review is to summarise the current evidence for the effects of paying for performance on the provision of health care and health outcomes in low and middle‐income settings.
The analyses will focus on the following comparisons:
-
Paying for performance targeted at health workers vs no paying for performance
-
Paying for performance targeted at facilities vs no paying for performance
-
Paying for performance targeted at areas or groups of providers vs no paying for performance
-
Paying for performance targeted at national governments (aid) vs no paying for performance
-
Paying for performance targeted at multiple levels vs no paying for performance
-
Comparisons of different types of paying for performance, including paying for performance targeted at different levels, and different magnitudes of incentives
Methods
Criteria for considering studies for this review
Types of studies
The review will include:
-
Randomized Controlled Trials (RCTs)
-
Controlled Clinical Trials (CCTs)
-
Controlled Before‐and‐After (CBA) studies with:
-
-
at least two clusters in each comparison
-
pre and post intervention periods for study and control groups are the same
-
the choice of the control site is appropriate, e.g. similar socio‐economic characteristics and/or and no major differences in the baseline group
-
-
Interrupted Time Series (ITS) studies with at least three measurements before and after introducing the intervention.
Well‐designed cluster randomised trials protect against selection bias and are likely to provide the most rigorous estimates of the impacts of paying for performance schemes. Although we are aware of some trials that are underway, few if any have been reported and advocates of paying for performance frequently cite the results of CBA studies. Although these studies have a high risk of bias, we believe it is important, at least at this time, to include these studies. ITS studies may be problematic due to changes in information systems and the reliability of information systems used in paying for performance schemes in LMICs. However, they may potentially have less of a risk of bias than CBA studies and cluster randomised trials may not be practical for evaluating some paying for performance schemes; e.g. when there is simultaneous system‐wide implementation. ITS studies, particularly evaluations of the impact of GAVI, are also frequently cited (and contested) as evidence of the benefits of paying for performance (Lu 2006; Chee 2007).
Other study designs may provide useful information about acceptability, potential effects, or explanations for observed effects of paying for performance, but are unlikely to provide useful estimates of the impacts of paying for performance on the main outcomes of this review.
Types of participants
The participants are providers of health care services in low and middle income countries (as defined by the World Bank). These include health workers, facilities, sub‐national organisations (health administrations, NGOs or local governments), national governments and combinations of the above. All sectors (public, private, and private not‐for‐profit) will be included in the review.
Types of interventions
Paying for performance takes three main forms:
-
Conditional cash payment
-
Conditional provision of material goods
-
Target payments (payments for reaching a certain level of coverage, which can be defined in absolute terms or relative to a starting point)
We will include impact evaluations of paying for performance schemes (including ancillary components), compared to any alternative (including non‐conditional financial incentives and different levels of conditional financial incentives). We will include comparisons with alternatives where there may be differences in ancillary components, such as increased resources, as well as differences in paying for performance.
We will exclude studies which focus on:
-
The demand side of health care only (i.e. payments to consumers, not producers)
-
Payment to health workers or facilities which are not explicitly linked to changing patterns of performance (e.g. for coming to work; salary increases; routine increases in activity‐based payments such as DRGs or fees for service; or changes to budget flows which are routine or intended to motivate, but without being conditional on specific activity or output measures)
Types of outcome measures
Primary outcomes
To be included, a study must report at least one of the following outcomes:
-
Changes in targeted measures of provider performance, such as the delivery or utilization of healthcare services, or patient outcomes
-
Unintended effects, including motivating unintended behaviors, distortions (ignoring important tasks that are not rewarded with incentives), cherry‐picking/cream‐skimming (prioritising patients that are most profitable over those who release fewer financial rewards), gaming (improving or cheating on reporting rather than improving performance), increased inequities, and dependency on financial incentives
-
Changes in resource use, including for incentives, administration and services
Secondary outcomes
The following other outcomes of interest will be included if reported in included studies:
-
Acceptability
-
Patient or provider satisfaction
-
Impacts on management or information systems (if not a targeted measure of performance)
-
Impacts on overall financing or resource allocation
The results of process evaluations or qualitative studies conducted alongside impact evaluations will be included.
Search methods for identification of studies
Electronic searches
We will search the Database of Abstracts of Reviews of Effectiveness (DARE) for related reviews.
We will search the following electronic databases for primary studies
-
Cochrane Effective Practice and Organisation of Care Group Specialised Register (and the database of studies awaiting assessment)
-
Cochrane Central Register of Controlled Trials (CENTRAL)
-
MEDLINE
-
MEDLINE In‐Process & Other Non‐Indexed Citations
-
EMBASE
-
PsycINFO
-
EconLit
-
Sociological Abstract
-
LILACS
-
WHOLIS
-
World Bank
-
Social Science Citation Index
-
Science Citation Index
In addition we will select relevant databases from the LMIC database list at: http://epocoslo.cochrane.org
We will develop strategies that incorporate the methodological component of the EPOC search strategy combined with selected index terms and free text terms. No language or date restrictions will be placed on the search strategy. The MEDLINE search strategy will be translated into the other databases using the appropriate controlled vocabulary as applicable.
An appendix containing the search strategy for MEDLINE is attached (Appendix 1).
Searching other resources
International experts in the field will be contacted, including the authors of relevant articles that are retrieved. We will ask them to identify additional web sites, academic (or other) institutions active in this field, and other experts in the field, as well as additional relevant studies.
In addition, we will search the web sites of organisations likely to be active in the field, including: the World Bank, U.S. Agency for International Development (USAID)., Management Sciences for Health (MSH), Centre for Global Development, World Health Organization (WHO), Swiss Tropical Institute, Deutsche Gesellschaft für Technische Zusammenarbeit (GTZ), KfW Entwicklungsbank, Department for International Development (DFID), The Global Alliance for Vaccines and Immunization (GAVI), The Global Fund to Fight AIDS, Tuberculosis and Malaria, Asian Development Bank, and Pan American Health Organization (PAHO).
We will also search the web sites of academic institutions active in this field, such as the London School of Hygiene and Tropical Medicine, the Harvard School of Public Health, University of Cape Town, Institute of Policy Studies of Sri Lanka (IPS), the Kenya Institute of Policy Analysis and Research (IPAR), and Institute of Tropical Medicine, Belgium.
We will check references from included studies and other relevant articles to identify other relevant studies that meet the inclusion criteria.
We will search ISI Web of Science for papers which cite studies included in the review.
Data collection and analysis
Selection of studies
Two authors will independently review abstracts to identify all studies that potentially meet the inclusion criteria and should be retrieved.
The same two authors will independently assess each full text article that is retrieved to determine whether it met all of the selection criteria. Any disagreements and uncertainties will be resolved by discussion, and/or the involvement of a third author.
Data extraction and management
Two authors will independently extract the following information from the included studies using a modified version of the EPOC data collection checklist, including the following items:
Study design (RCT, CCT, CBA, ITS)
Type of targeted behaviour (clinical prevention services, diagnosis, test ordering etc.)
Study setting (country, urban, rural)
Participants
-
Targets for paying for performance scheme
-
Description of patient group(s) affected by intervention
Setting
Methods
-
Unit of allocation
-
Unit of analysis
-
Power calculation
-
Quality criteria
Intervention
-
Magnitude of incentives
-
Incentives relative to appropriate measure (e.g. percentage of health workers' wage)
-
Are incentives additional to ordinary wage/funding?
-
Ancillary components
-
Source of funding
Outcomes
-
Main outcome measures
-
Economic variables (e.g. change in resource use)
-
Length of time during which outcomes were measured after initiation of intervention
-
Length of post‐ intervention follow‐up period
-
Measurement of outcome indicator ‐ by who?
Results
-
Changes in targeted measures of provider performance
-
Changes in the delivery of health care services
-
Changes in the utilisation of health care services
-
Changes in patient outcomes
-
Unintended effects
-
Changes in resource use
-
Acceptability
-
Patient or provider satisfaction
-
Impacts on management or information systems (if not a targeted measure of performance)
-
Impacts on overall financing or resource allocation
-
Equity‐consideration: Evidence of differential impact on different parts of the population?
Data will be entered and managed using Excel.
Assessment of risk of bias in included studies
Criteria recommended by EPOC will be used to assess the risk of bias for each main outcome in all studies included in the review (EPOC Review Group Checklist, 2008).
An overall assessment of the risk of bias (high, moderate or low risk of bias) will be assigned to each main outcome in all included studies using the approach suggested in Chapter 12 of the Cochrane Handbook for Systematic Reviews of Interventions (Higgins 2008).
The quality of evidence for each main outcome that is the extent of confidence in the estimate of effect across studies ‐ will be assessed using GRADEpro and the GRADE approach (Guyatt 2008).
Measures of treatment effect
For RCTs, CCTs and CBA studies we will record outcomes in each comparison group. Where possible we will record risk ratios (RRs) (for undesirable outcomes) for dichotomous outcomes and weighted mean differences (WMDs) for continuous outcomes. If adjusted analyses are reported (adjusting for potential confounders in non‐randomised studies), we will record the estimates of effect together with the standard error. For a plain fixed effects meta‐analysis, we will record the number of events and total number in each group (for risk ratio), or mean and standard deviation in each group (for weighted mean difference).
For ITS studies we will record changes in level and in slope and their standard errors. Where analysis of ITS data is inappropriate, we will try to re‐analyse if possible.
Unit of analysis issues
For cluster randomised trials and CBA studies we will control that an appropriate analysis has been done that adjusts for clustering in calculating confidence intervals or P‐values. If an analysis has not done this, we will attempt to extract the necessary data (intracluster correlation coefficients ‐ ICCs) or obtain these data from the investigators and re analyse the results. If this is not possible, we will report point estimates, but not the reported confidence intervals or P‐values. If there are sufficiently similar studies to conduct meta‐analyses, we will undertake sensitivity analyses using imputed ICCs based on data from the other included studies and other studies with comparable outcome measures (Campbell 2000).
Dealing with missing data
We will contact the authors of studies to obtain missing data, including details of the intervention, the context, overall resource inputs, ancillary components and the results.
Assessment of heterogeneity
If we decide to conduct meta‐analyses, we will assess the extent of heterogeneity in results across comparable studies using forest plots, the I2 statistic and the Chi2 test.
Assessment of reporting biases
Selective outcome reporting will be assessed using the approach described in Chapter 8 of the Cochrane Handbook for Systematic Reviews of Interventions (Higgins 2008). Publication bias will be assessed qualitatively based on the results and characteristics of the included studies, including the extent to which only small effects in favour of the intervention are reported, the extent to which funders or investigators are advocates of paying for performance or have a vested interest in the results, and the extent to which the authors’ interpretations of the results are supported by the actual results.
Data synthesis
The primary analysis will only include estimates of the impact of paying for performance where there are not important differences in ancillary components.
For each of the comparisons, the results of each relevant included study will be summarised in a table that includes the main characteristics of the study, the results in natural units as reported by the investigators, and standardised results.
Studies of paying for performance are likely to be heterogeneous, in relation to context, study design, characteristics of the participants and the interventions, and the outcome measures. It is therefore unlikely to be informative to calculate average effects across studies. If relevant, we will report median effects and the range of effects for each group of studies and across groups for each main outcome. If there are studies that are similar enough that it would be meaningful to combine their results, we will perform statistical analysis using the Review Manager software (RevMan 5). We will use a fixed effect model to combine data in the absence of important heterogeneity, using risk ratios (for undesirable outcomes) for dichotomous outcomes and weighted mean differences for continuous outcomes, if possible. If possible, changes in level and changes in slopes, from ITS studies, will be combined using the Generic Inverse Variance method. This method will also be used to combine results from reported adjusted analyses in primary studies.
Comparisons where there are important differences in ancillary components, such as increased resources, will be reported separately, and will not be included in the primary analysis. For these comparisons, we will describe the key ancillary components and explore the extent to which reported impacts are likely to be attributable to paying for performance per se (conditionality) versus the other components of paying for performance schemes.
Subgroup analysis and investigation of heterogeneity
If we find more than one study for any of the comparisons listed above we will explore the extent to which the following factors might explain differences in the impacts of paying for performance (see Table 1).
| Indicator | Categories | Expected relationship with paying for performance outcomes |
| Magnitude of the incentive | Continuous variable; if possible, we will characterise this as the proportion of funds provided by paying for performance relative to the total amount of funds provided at the level of the paying for performance scheme | We would anticipate that larger incentives would have larger effects on targeted outcomes
|
| Ancillary components | Description: increased funding, training, supplies, technical support; management support; other quality improvement strategies; increasing salaries; construction of new facilities; improvements in information systems; changes in governance, priority setting or rationing; or processes to involve stakeholders. | Ancillary components might be expected to be very powerful in improving outcomes if they address real barriers in that context (resources, training, organisational constraints, staffing etc.). |
Of presumed greatest significance will be the magnitude and proportion of other financing sources that the incentives constitute (clearly, the higher the incentive, the more we would expect a response from providers).
Heterogeneity will be explored visually by preparing tables, bubble plots and box plots to explore the size of the observed effects in relation to each of these variables. We will consider each potential explanatory factor one at a time by looking for patterns in the distribution of the effects of paying for performance.
However, because we anticipate finding a small number of studies relative to the number of potential explanatory factors, we will investigate and report potential explanations of heterogeneity cautiously, using the criteria for interpreting subgroup analyses and meta‐regressions in Section 9.6.6 of the Cochrane Handbook for Systematic Reviews of Interventions (Higgins 2008).
To highlight the evidence and evidence gaps, we will prepare a table indicating the number and type of studies identified, by level of targeting, size of incentives, and presence or absence of ancillary measures (see Table 2).
| Target | Number of studies of paying for performance schemes including ancillary components | Number of studies of paying for performance without ancillary components | ||
| Substantial incentives (>20% of total funding) | Small incentives (20% of total funding or less) | Substantial incentives (>20% of total funding) | Small incentives (20% of total funding or less) | |
| Individuals | ||||
| Facilities | ||||
| Areas/sub‐national organisations | ||||
| National gov | ||||
| Multiple | ||||
Sensitivity analysis
We will perform sensitivity analyses by excluding studies with a high risk of bias for any outcome for which we find more than one comparable study with studies with a low or moderate risk of bias and studies with a high risk of bias.
| Indicator | Categories | Expected relationship with paying for performance outcomes |
| Magnitude of the incentive | Continuous variable; if possible, we will characterise this as the proportion of funds provided by paying for performance relative to the total amount of funds provided at the level of the paying for performance scheme | We would anticipate that larger incentives would have larger effects on targeted outcomes
|
| Ancillary components | Description: increased funding, training, supplies, technical support; management support; other quality improvement strategies; increasing salaries; construction of new facilities; improvements in information systems; changes in governance, priority setting or rationing; or processes to involve stakeholders. | Ancillary components might be expected to be very powerful in improving outcomes if they address real barriers in that context (resources, training, organisational constraints, staffing etc.). |
| Target | Number of studies of paying for performance schemes including ancillary components | Number of studies of paying for performance without ancillary components | ||
| Substantial incentives (>20% of total funding) | Small incentives (20% of total funding or less) | Substantial incentives (>20% of total funding) | Small incentives (20% of total funding or less) | |
| Individuals | ||||
| Facilities | ||||
| Areas/sub‐national organisations | ||||
| National gov | ||||
| Multiple | ||||
