Types of studies

It may be difficult to evaluate most interventions to improve health systems functioning using only randomised approaches.Therefore we will include both randomised and non‐randomised controlled studies in this Cochrane review (EPOC 2015a). We will include the following study types:

• randomised controlled trials (RCTs);

• non‐randomised controlled trials (NRCTs);

• controlled before‐and‐after studies (CBAs);

• interrupted time series (ITS): these need to have a clearly defined point in time when the intervention occurred and at least three data points before and three after the intervention;

• repeated measures studies;

• qualitative and process evaluation studies associated with included effectiveness studies.

We shall only include cluster‐RCTs, non‐randomised cluster‐RCTs and CBA studies with at least two intervention sites and two control sites. We will not restrict the inclusion of studies by geographic region, publication status, date of publication or language.

Types of participants

We will include both the institutional and staff level outcome measures because interventions may be implemented at an institutional level but operated by staff within the institution. For example, health information officers and district managers may implement a new data flow guideline within one or across several health facilities and district offices. Thus both the institutional performance and the individual performance are indicators of the success/failure of the intervention. We will include the following types of participants:

Institutional levels:

• public sector health institutions;

• private sector (for profit and not‐for‐profit health institutions);

• private‐public partnerships for delivery of health services;

• hospital‐based, primary health care and community‐based health care settings;

• sub‐district, district, regional, provincial or national levels.

Types/levels of staff:

• health service managers (from health facility level to sub‐district, district, provincial and national levels);

• professional and para‐professional health staff (from health facility level to sub‐district, district, provincial and national levels);

• lay health workers. We will use the Lewin 2005 definition.

Types of interventions

Any intervention aimed at improving the RHIS as a standard part of the health system. In Table 1 we categorise the possible areas for RHIS interventions by drawing from the PRISM framework (Aqil 2009). Intervention comparison groups could include the following comparisons:

• no RHIS intervention;

• no RHIS intervention for health systems management;

• different RHIS interventions compared to each other;

• pre‐post implementation.

We will exclude the following comparisons:

• interventions already sufficiently covered by other Cochrane Effective Practice and Organisation of Care (EPOC) reviews (such as on audit and feedback);

• complex health systems strengthening and quality improvement interventions where the study authors or implementers have not named RHISs improvement as the primary focus of the health systems strengthening and quality improvement interventions;

• interventions aimed only at clinical decision‐making for individual patient management;

• ad hoc and temporary interventions not aimed at long‐term improvement of the RHIS.

Types of outcome measures

Recognising the complexity of RHISs, we understand that there may not always be a direct causal pathway between the RHIS intervention and the more distal impact measures of health systems functioning and population health outcomes. RHIS strengthening interventions may be aimed at intermediate outcomes (for example, improved data quality), whilst others may aim to impact on more distal outcomes and impacts (such as population health). We will focus on both in this Cochrane review.

Drawing on the PRISM framework (Aqil 2009) and EPOC recommendations for categorising outcomes (EPOC 2015b), we will categorise the outcomes into four main areas. We will extract data on primary and secondary outcomes listed below.

Electronic searches

We will search for eligible studies in the following electronic databases:

• The Cochrane Central Register of Controlled Trials (CENTRAL), part of the Cochrane Library. www.cochranelibrary.com, (including the Cochrane Effective Practice and Organisation of Care (EPOC) Group Specialised Register)

• MEDLINE, OvidSP

• Embase, OvidSP

• Global Health, OvidSP

• Science Citation Index, Social Sciences Citation Index, and Emerging Sources Citation Index, ISI Web of Science

We will incorporate methodological components of the Cochrane Highly Sensitive Search Strategy and the Cochrane EPOC Group search strategy, combined with selected index terms and free text terms relating to HIS in our search strategies. We will tailor strategies to other databases and will report them in the review. See Appendix 2 for the MEDLINE search strategy.

To identify qualitative studies and process evaluations associated with included effectiveness studies, we will write to the authors of the included studies and check the reference list of each included study. In addition, we will supplement our search strategy with searches for ‘related studies’ and ‘same author’ searches in PubMed (or similar functions in other data bases). Where we identify such studies we will use them in the Discussion section of the review, to inform our understanding of the effectiveness findings of the included studies. This will not require a separate review of qualitative studies.

Searching other resources

Selection of studies

A core team comprising of at least two review authors (NL, KD) will be responsible for study selection. We will combine search results in a reference management database and remove duplicate records. The review authors, working in pairs, will each independently screen titles and abstracts of studies for potential inclusion, meaning that two different review authors will screen each abstract twice. Thereafter, we will retrieve full‐text copies of potentially eligible articles. Again working in pairs, the review authors will independently evaluate each retrieved full‐text article for inclusion. We will resolve any disagreements regarding inclusion of the full‐text articles through discussion between whichever two review authors the article was assigned to and, where necessary, by consulting a third review author from the core team for an independent assessment. Where articles do not provide sufficient information to determine eligibility, we will contact the study authors for further details.

We will report the screening process and results in a Preferred Reporting Items for Systematic Reviews and Meta‐Analyses (PRISMA) flow chart (Liberati 2009). We will list studies that appear to meet the inclusion criteria but that we excluded in the 'Characteristics of excluded studies' table.

Data extraction and management

The review authors will independently and in pairs extract data from each included study. We will develop a standardised data extraction form to extract descriptive and outcome data. Where necessary the assigned review authors will discuss any disagreements related to the extraction process until they reach consensus. We will resolve any disagreement through discussion with a third review author. We will contact study authors in the case of missing data. Two review authors, NL and KD, will perform double‐entry of the checked data into Review Manager (RevMan) (Review Manager 2014).

We will extract the following information from all included studies:

• study design (RCTs, NRCTs, CBAs, ITS and repeated measures studies);

• country, geographical location (rural, urban, peri‐urban);

• setting;

• participant characteristics (e.g. type and level of staff) and their function in the intervention (e.g. implementer or recipient);

• institutional characteristics: level of institution and nature of its function;

• intervention:

  • intervention purpose;

  • the nature of the intervention (focus, content and scope e.g. technical, behavioural, organisational, multi‐dimensional, data quality or data use or both);

  • parties involved (delivery mode and those involved in the delivery of the intervention);

  • duration and intensity of the intervention, and the length of period of follow‐up measurement;

  • resources used to enable the intervention including economic and human resources, if described;

  • fidelity and reach;

  • the theoretical basis for the intervention if described (i.e. the study authors' description of how the intervention was hypothesised to work);

  • details of the control or standard care as well as any co‐interventions delivered alongside the health information intervention;

• outcomes assessed:

  • types of outcome indicators (process, outcome, impact, technical, behavioural, organisational);

  • outcomes that can be categorised as RHIS performance, health systems performance or health status outcomes;

  • other issues of interest for outcome, specifically equity and adverse results;

  • timing of outcome assessments;

  • method(s) of assessing these outcomes;

  • results for each outcome.

Assessment of risk of bias in included studies

Two review authors, NL and KD, will independently assess the risk of bias for each included study. We will follow the guidelines from both Cochrane's 'Risk of bias' assessment tool and the Cochrane EPOC Group, which include criteria for assessing each of the included study designs (EPOC 2015c; Higgins 2011). We will summarise the risk of bias at two levels: within studies (across domains) and across studies (for each primary outcome). Judgement on the overall risk of bias will take into account the likely magnitude and direction of the bias and whether we consider that the bias will impact on the findings. We will assess studies to be at the highest risk of bias if they score high risk in one or more of the following domains: sequence generation; allocation concealment; or selective outcome reporting (based on growing empirical evidence that these three factors are the most important in influencing risk of bias) (Higgins 2011). We will judge the overall risk of bias as low if we assess these key domains as at low risk of bias; unclear if one or more key domains are assessed as at unclear risk of bias; and high if we assess one or more key domains as at high risk of bias.

We will perform further assessment of the quality of evidence related to each of the key outcomes across studies using the Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach (Guyatt 2008; Higgins 2011). We will present the main findings of the review in 'Summary of findings' tables, which we will prepare using GRADEpro GDT software (GRADEpro GDT). We will list the primary review outcomes for each comparison with estimates of relative effects along with the number of participants and studies contributing data for those outcomes. For each individual outcome, we will assess the quality of the evidence using the GRADE approach (Balshem 2011), which involves consideration of limitations in design, inconsistency, indirectness, imprecision, publication bias, magnitude of the effect, dose‐response effect and other plausible confounders. We will express the results as one of four levels of quality (high, moderate, low or very low).

Measures of treatment effect

Unit of analysis issues

For cluster RCTs that do not adequately account for clustering in their analysis, we will adjust the analysis for clustering if we can extract the following information:

• the number of clusters (or groups) randomised to each intervention group or the average (mean) size of each cluster;

• the outcome data ignoring the cluster design for the total number of individuals included in the study (for example, number or proportion of individuals with events, or means and SDs); and

• an estimate of the intra‐cluster (or intra‐class) correlation coefficient (ICC). Where no information on the ICC is reported, we will extrapolate the ICC from other included cluster RCTs, if available. If this is not possible, we will not combine the findings of these studies in a meta‐analysis but will present the results in an additional table.

We will use inflated variances to adjust appropriately for clustering (Higgins 2011). For cluster RCTs where study authors do not take clustering into account in the original analysis and where re‐analysis is not possible, we will only report the estimate of effect (and not the P value or CIs as the P value may be too small and the CIs too narrow).

Dealing with missing data

We will attempt to obtain missing data from the study authors. If this is not possible we will report the data as missing and report this in the 'Risk of bias' tables and will not attempt to impute values.

For all outcomes we will carry out analysis, as far as possible, on an intention‐to‐treat (ITT) basis based on available cases. We will attempt to include all participants randomised to each group in the analyses, and analyse data according to initial group allocation irrespective of whether or not participants received, or complied with, the planned intervention.

When assessing adverse events, adhering to the principle of ITT may be misleading and we will therefore relate the results to the treatment received. This means that for adverse effects we will base the analyses on the participants who actually received the intervention and the number of adverse events that are reported in the studies.

Assessment of heterogeneity

We will first make a qualitative assessment of the extent to which the included studies are similar to each other or not. This will include an assessment of the settings, the interventions, the participants and outcomes. We will also examine the forest plots from the meta‐analyses; visually assessing the levels of heterogeneity (in terms of the size or direction of treatment effect and by looking at the overlap between CIs around the treatment effect estimate for each included study). We will employ the Chi² test to assess whether observed differences in results across studies are compatible with chance alone. When the observed intervention effects are more different from each other than one would expect due to chance alone, we will assume that the studies have 'clinical' or statistical heterogeneity or both.

We will use the I² statistic to quantify the level of statistical heterogeneity among the trials in each analysis. If we identify substantial or considerable heterogeneity (approximately I² statistic value of 50% to 100%) we will note this in the text and explore this heterogeneity through the prespecified subgroup analyses (Subgroup analysis and investigation of heterogeneity). We will interpret results from meta‐analyses with high levels of unexplained heterogeneity with caution.

Assessment of reporting biases

We will attempt to be as comprehensive as possible in our search strategy so as to find and include all relevant studies and to reduce any possible publication bias. This will include a search of published studies, grey literature, registers of prospective trials and discussions with colleagues (Higgins 2011). We will use funnel plots to make a visual assessment of whether there is asymmetry, which may signal the presence of reporting bias, even if it is not a definitive indicator of such bias. If we find more than 10 studies in this Cochrane review that report similar outcomes, we will consider statistical testing for funnel plot asymmetry. For continuous outcomes with intervention effects measured as mean differences, we will use the test proposed by in Egger 1997 to test for funnel plot asymmetry. For dichotomous outcomes with intervention effects measured as RRs, and continuous outcomes with intervention effects measured as SMDs, we will not consider funnel plot calculations because funnel plots using risk differences are seldom of interest. We will interpret the results of tests for funnel plot asymmetry in the light of visual inspection of the funnel plot, as the statistical results may not be representative if there are small‐study effects.

Data synthesis

Assuming the breadth of the data is not too wide, we will conduct a meta‐analysis of the pooled outcome data (Review Manager 2014). We will report the results of the meta‐analysis as part of a structured synthesis and will include forest plots where appropriate (EPOC 2015d).

We will carry out a meta‐analysis to provide an overall estimate of treatment effect when more than one study examines similar interventions provided that: studies use similar methods; studies are similar regarding setting; and studies measure the same outcome in similar ways in comparable populations. We will carry out the statistical analysis using RevMan (Review Manager 2014). We will not combine results from RCTs and NRCTs together in meta‐analysis, nor will we present pooled estimates for NRCTs with different types of study designs. Evidence on different interventions may be available from different types of studies (for example, it is likely that data from interventions implemented at the national level will be reported in NRCTs). Where there is evidence on a particular outcome from both RCTs and NRCTs, we will use the evidence from trials that are at lower risk of bias to estimate treatment effect.

We will use a random‐effects meta‐analysis for combining data, as we anticipate that there may be natural heterogeneity between studies attributable to the different interventions, populations and implementation strategies. For continuous variables we will use the inverse‐variance method. For dichotomous variables we will use the method proposed by Mantel‐Haenszel (Mantel 1959). If cluster RCTs meet the inclusion criteria, we will use the generic inverse‐variance method in RevMan for meta‐analysis (Review Manager 2014).

For both RCTs and NRCTs, where results are adjusted to take account of possible confounding factors, we will use the generic inverse‐variance method in RevMan (Review Manager 2014) to carry out any meta‐analysis. Where study authors provide adjusted and non‐adjusted figures we will carry out a sensitivity analysis using the unadjusted figures to examine any possible impact on the estimate of treatment effect.

For ITS and repeated measures studies, the preferred analysis method is either a regression analysis with time trends before and after the intervention, adjusted for autocorrelation and any periodic changes, or auto‐regressive integrated moving average (ARIMA) analysis. We will attempt to present the results for outcomes as changes along two dimensions: change in level and change in slope. Since the interpretation of change in slope can be difficult, we will present the long‐term effects similarly to the way plan calculate and present the immediate effects. We will use the generic inverse‐variance method for combining the data in a meta‐analysis for ITS and CBA studies.

If the Cochrane review includes study results that we cannot pool because the settings or interventions, or both, are too heterogeneous, we will still describe the results using a structured synthesis (EPOC 2015d). This structured synthesis may include reporting on interquartile ranges and ranges of effects for relevant outcomes and we will include a summary of the findings in the review text. Guided by the model presented in Figure 1, this structured analysis may also include a description of the intervention mechanisms described across the studies. We will include information from the structured synthesis in the 'Summary of findings' table.

We will perform further assessment of the quality of evidence related to each of the key outcomes across studies using the GRADE approach (Guyatt 2008; Higgins 2011). We will present the main findings of the Cochrane review in 'Summary of findings' tables which we will prepare using GRADEpro GDT software (GRADEpro GDT).

We will list the primary review outcomes for each comparison with estimates of relative effects along with the number of participants and studies contributing data for those outcomes. For each individual outcome, we will assess the quality of the evidence using the GRADE approach (Balshem 2011), which involves consideration of limitations in design, inconsistency, indirectness, imprecision, publication bias, magnitude of the effect, dose‐response effect and other plausible confounders. We will express the results as one of four levels of quality (high, moderate, low or very low).

Subgroup analysis and investigation of heterogeneity

Subgroup analyses will check for variation in the intervention effect across different populations, interventions or setting characteristics. Using RevMan (Review Manager 2014), we will investigate differences between two or more subgroups (Deeks 2011). This analysis will test for heterogeneity across subgroup results rather than across individual study results thus investigating genuine subgroup differences rather than sampling error. We will only conduct this analysis when the data in the subgroups are independent (i.e. a set of study participants do not form part of more than one subgroup).

These subgroup analyses will depend on having sufficient trials to perform a statistically significant comparison between groups. We will perform meta‐regression to investigate both the effect of the intervention on the estimates of effects and to investigate the effect of multiple characteristics (regarding setting and the intervention) simultaneously (Deeks 2011), only if there are 10 times or more observations (studies) available than the number of independent variables (characteristics). This would mean that if we want to perform meta‐regression simultaneously on two independent variables we would need 20 or more studies, and so on. If there are fewer than 10 studies per variable, for fixed‐effect meta‐analyses we will assess subgroup differences by interaction tests (Altman 2003). For random‐effects meta‐analyses, we will use non‐overlapping CIs to indicate a statistically significant difference in treatment effect between the subgroups.

The PRISM framework will assist in guiding the analysis (see Description of the intervention and Types of interventions) and enable a better understanding of why interactions were hypothesised, the direction of the interaction (for instance, which subgroup was expected to have a larger effect and why) and how potential interactions could be analysed and interpreted.

We will perform subgroup analyses using criteria for which we may have good reasons to suspect an interaction, including:

• country income level (high‐, middle‐ and low‐income), health system level of intervention (community, PHC or hospital);

• institutional level (community‐based, primary, secondary, tertiary level health institutions or clinics and hospitals);

• private or public sector services;

• levels/types of management decision‐making (health facility, administrative, provincial/regional or national);

• technical type of intervention (electronic or paper‐based).

We will also consider subgroup analyses for the following explanatory factors:

• interventions aimed at improving data quality;

• interventions aimed at improving data use.

If we decide not to perform a meta‐analysis, we will summarise the results of the subgroups within the review text.

Sensitivity analysis

We will perform a sensitivity analysis to examine the effects of removing studies at overall high risk of bias across domains (based on 'Risk of bias' assessment within studies) from any meta‐analyses we conduct. If we combine individually and cluster RCTs, we will also perform sensitivity analyses based on varying the ICC used to adjust the results from cluster RCTs; and based on removing data obtained from cluster RCTs. If one or more included RCT or NRCT present both results adjusted for confounding factors and results not adjusted for confounding, we will carry out sensitivity analyses based on unadjusted results.