Types of studies

We included both published and unpublished empirical studies that investigated the prevalence and factors associated with selective inclusion or reporting, or both, in systematic reviews of RCTs of healthcare interventions. If empirical studies included a mixture of systematic reviews of RCTs and non‐randomised studies (NRS) and reported data separately, we only included the findings for systematic reviews of RCTs. If it was not possible to separate the data, we contacted the study authors. If the data could not be separated, we presented the results but did not include them in the quantitative syntheses.

We included empirical studies that comprised a sample or a complete set of systematic reviews (e.g. all systematic reviews registered or published during a specific time period). The empirical studies could have assessed any type of selective inclusion or reporting, such as investigations of how frequently: (1) RCT outcome data is selectively included in systematic reviews based on the results; (2) outcomes and analyses are discrepant between protocol and published review; or (3) non‐significant outcomes are partially reported in the full text or summary within systematic reviews. We defined the full text of the systematic review as the text reported in the results section, tables, forest plots and data available via online appendices. Summaries within systematic reviews included the abstract, plain language summary or 'Summary of findings' tables. We excluded empirical studies assessing discrepancies in information other than outcomes and analyses, e.g. search strategy or inclusion criteria, as discrepancies of this nature were beyond the scope of our review.

Prior to the launch in February 2011 of PROSPERO, an international online prospective register of systematic reviews hosted by the Centre for Reviews and Dissemination (University of York, United Kingdom), access to systematic review protocols was generally limited to organisations such as The Cochrane Collaboration, the Campbell Collaboration and the Joanna Briggs Institute (Booth 2010; Booth 2011). Therefore, we anticipated that it was unlikely that empirical studies that included systematic reviews other than those coming from these organisations would exist at the time of our search. We also anticipated that empirical studies comparing systematic review registry entries to published systematic reviews would be unlikely to exist at this time. However, such studies may be included in future updates of this Cochrane Review. While these issues may limit the generalisability of some of the empirical studies identified, these issues are unlikely to apply to empirical studies investigating selective inclusion of RCT outcome data in systematic reviews or selective partial reporting in systematic reviews, because both of these types of selective inclusion and reporting can be assessed in systematic reviews without a systematic review protocol or registry entry.

Types of data

We included estimates of the prevalence of selective inclusion or reporting in systematic reviews of RCTs and estimates of the association between selective inclusion or reporting and any factors predictive of this (e.g. statistical significance or direction of the effect) as assessed by the investigators who did the original empirical studies. We anticipated that these factors may have been defined by investigators differently, but included any empirical study regardless of the definitions used. The terms "full" or "complete" versus "partial" reporting in either the full text or summary of the systematic review were also anticipated to be defined variously by investigators, but we included any empirical study regardless of the definitions used.

Types of methods

We focused on five types of selective inclusion and reporting of outcomes and analyses in this review. These included:

• selective inclusion of RCT outcome data in systematic reviews;

• discrepancies between systematic review registry entries and published systematic reviews;

• discrepancies between systematic review protocols and published systematic reviews;

• discrepancies between the full text and the summaries (i.e. abstract, plain language summary or 'Summary of findings' table) in systematic reviews;

• selective partial reporting in systematic reviews.

Types of outcome measures

For each of the five types of methods, we included three primary outcomes:

1. Overall prevalence of selective inclusion or reporting;

2. Whether the overall prevalence of selective inclusion or reporting is associated with statistical significance of the effect estimate;

3. Whether the overall prevalence of selective inclusion or reporting is associated with direction of the effect estimate.

We also included four secondary outcomes:

1. Prevalence of specific examples of selective inclusion or reporting (see Appendix 1 for examples that we anticipated might be reported in empirical studies);

2. Whether the prevalence of each specific example of selective inclusion or reporting is associated with statistical significance of the effect estimate;

3. Whether the prevalence of each specific example of selective inclusion or reporting is associated with the direction of the effect estimate;

4. Whether the overall prevalence or prevalence of specific examples of selective inclusion or reporting is associated with other factors investigated by the empirical study investigators (e.g. clinical area, funding of the systematic review).

We did not exclude studies based on the outcomes reported.

Electronic searches

We searched the following electronic databases:

• Cochrane Methodology Register (to July 2012, which was the last month in which new records were added to this database);

• Ovid MEDLINE (January 1946 to May 2013);

• Ovid EMBASE (January 1980 to May 2013);

• Ovid PsycINFO (January 1806 to May 2013);

• ISI Web of Science (January 1898 to May 2013).

The search strategies for each database are reported in Appendix 2. We also searched the US Agency for Healthcare Research and Quality (AHRQ) Effective Healthcare Program's Scientific Resource Center (SRC) Methods Library (SRC Methods Library 2013) on 5th June 2013, using the following Search Descriptors: "Bias ‐ outcome selection", "Bias ‐ Publication", "Bias ‐ Reporting", "Reporting", "Registries ‐ Systematic Reviews" and "Synthesis, Quantitative ‐ Reporting".

Searching other resources

We searched the abstract books of the 2011 and 2012 Cochrane Colloquia (which were not indexed in any electronic databases at the time of the search). We also searched the article alerts for methodological work in research synthesis published from 2009 to 2011, which are available in Research Synthesis Methods (Hafdahl 2010b; Hafdahl 2010a; Hafdahl 2011a; Hafdahl 2011b; Hafdahl 2012). One review author (MJP) screened the reference lists of all included studies and any relevant reviews identified from the search. One review author (MJP) contacted the authors of all included studies and relevant reviews, as well as individuals with content expertise, to assist with the identification of published, unpublished and ongoing studies.

Selection of studies

Two of four review authors (MJP and either JK, KD or SK) independently screened the titles and available abstracts of all studies identified by the search against the inclusion criteria (see Criteria for considering studies for this review), and excluded any clearly irrelevant studies. We did not exclude studies based on the language of the publication. Two review authors (MJP and SK) independently assessed full‐text copies of reports of potentially eligible studies. The authors resolved disagreements regarding study inclusion by discussion, involving a third review author (JEM) when required.

Data extraction and management

Two of four review authors (MJP and either JK, KD or SK) independently extracted data using a data extraction form developed for this Cochrane Review. Any discrepancies between the authors were resolved through discussion until consensus was reached or by arbitration of a third review author (JEM) when required. All the data extractors pilot tested the data extraction form and modified it accordingly before use. One review author (MJP) compiled all comparisons and entered outcome data into Review Manager 5 (RevMan 2011). One review author (JEM) cross‐checked the data entered. For studies where relevant outcome data were not reported, one review author (MJP) requested further information from the study investigators. When unsuccessful, we included the study in the review and fully described it (e.g. using tables), but did not include it in any quantitative syntheses.

We extracted the following data from each study:

• characteristics of the study, in particular the types of selective inclusion or reporting investigated, the number of included systematic reviews, methods for selecting the systematic reviews, areas of health care addressed, range of years of publication of the systematic reviews, proportion of systematic reviews that were Cochrane Reviews and methodological quality of the systematic reviews (however assessed by the study authors);

• data on estimates of prevalence and association, as specified under Types of outcome measures;

• the definition of factors investigated for their association with selective inclusion or reporting (e.g. statistical significance defined as P < 0.05);

• the definition of "full" or "partial" reporting used by study investigators who assessed this practice;

• any confounding variables assessed by the study authors (e.g. funding type).

Assessment of risk of bias in included studies

There is no standard tool available to evaluate the risk of bias of empirical studies eligible for inclusion in this review. We used the following criteria:

1. What is the risk of selection bias in the empirical study? Low risk of bias: the empirical study included all systematic reviews registered during a specified time period (where registration is defined as registration of the review in an online database or publication of a protocol for the review), or included a random sample of systematic reviews. High risk of bias: the empirical study included a non‐random sample of systematic reviews. Unclear risk of bias: the sampling frame for the sample of systematic reviews of RCTs is unclear.

2. What is the risk of selective reporting bias in the empirical study? Low risk of bias: all comparisons and outcomes reported in the protocol for the empirical study are fully reported in the results section of the publication. High risk of bias: not all comparisons and outcomes reported in the protocol for the empirical study are fully reported in the results section of the publication. Unclear risk of bias: it is unclear if all comparisons and outcomes are fully reported in the results section of the publication (e.g. because a protocol for the empirical study is not available) (Dwan 2011).

Two of four authors (MJP and either JK, KD or SK) rated each criterion independently. Any discrepancies between the authors were resolved through discussion until consensus was reached or by arbitration of a third author (JEM) when required.

Measures of the effect of the methods

The measures of prevalence and associations between a factor and selective inclusion or reporting were dependent on the summary statistics reported by the empirical study investigators, and we reported the data as available. For empirical studies that reported estimates of prevalence, we reported percentages with 95% CIs. We have presented possible prevalence estimates that we considered including in Appendix 1. For empirical studies investigating the association between a factor and selective inclusion or reporting, we reported risk ratios (RRs) with 95% CIs. We standardised outcomes so that they have a consistent meaning. For example, RRs > 1 denote a higher risk of selective inclusion or reporting bias. If the empirical study investigators reported odds ratios (ORs), we converted these to RRs using the formula provided in section 12.5.4.4 of theCochrane Handbook for Systematic Reviews of Interventions (Schünemann 2011). For studies that quantified bias in meta‐analytic estimates due to selective inclusion, we reported the estimates of bias as reported in the empirical studies.

Unit of analysis issues

Within the systematic reviews evaluated in an empirical study, there is potential for overlap of included RCTs. We contacted the authors of the empirical studies to enquire how they dealt with this issue of RCT overlap. We reported whether the issue was addressed and discussed its likely impact on the results of the empirical study. Furthermore, we anticipated that there might be some overlap in the systematic reviews included in the empirical studies in our review. Based on information regarding the types of reviews (i.e. Cochrane versus non‐Cochrane), years of publication, clinical condition and types of outcomes (e.g. dichotomous versus continuous), we reported how likely the potential for overlap is and how it may have impacted on the results of our review. If we suspected that two studies were likely to have a substantial proportion of overlapping systematic reviews, we performed sensitivity analyses by removing the study with the smaller number of included systematic reviews from the meta‐analysis.

Dealing with missing data

If the empirical studies had missing information on the characteristics of the included studies (e.g. methods for selecting the systematic reviews) or missing outcome data (e.g. measure of variation for a RR denoting the association between statistical significance and discrepant outcome reporting), we contacted the authors. We did not plan to impute any missing outcome data.

Assessment of heterogeneity

We assessed methodological heterogeneity by determining whether the characteristics of the included empirical studies were similar, particularly in terms of the areas of health care addressed by the systematic reviews and type of methodological comparisons undertaken. We assessed statistical heterogeneity by inspecting the forest plots and by calculating the I² statistic with 95% CIs (Deeks 2011; Higgins 2002). We calculated 95% CIs for the I² statistic using the non‐central Chi² approximation implemented in the Stata module heterogi (this module can only be used when there are more than two studies included in a meta‐analysis) (Orsini 2006).

Assessment of reporting biases

To assess small study effects, we planned to generate funnel plots if at least 10 empirical studies examining the same methodological comparison and outcome met our inclusion criteria (Sterne 2011). To assess selective reporting in the included empirical studies, we compared outcomes reported in the empirical study protocol (if available) with outcomes reported in the results section of the publication. We searched for empirical study protocols in the electronic databases listed above (see Electronic searches). If a published protocol could not be identified, we requested one from the authors of the empirical study. If a protocol was not available, we compared the outcomes reported in the methods section of the publication with the outcomes reported in the results section of the publication.

Data synthesis

For each of the five types of selective inclusion and reporting, we combined estimates of the primary outcome of overall prevalence of selective inclusion or reporting (primary outcome #1) in a meta‐analysis using a random‐effects model. We used DerSimonian and Laird's method of moments estimator to estimate the between trial variance (DerSimonian 1986). When it was not possible to meta‐analyse (because similar measures of overall prevalence had not been measured in more than one empirical study or were incompletely reported), we reported percentages and 95% CIs for each empirical study in tables. For each of the five types of selective inclusion and reporting, we combined RR estimates of associations between overall prevalence of selective inclusion or reporting and statistical significance or direction of effect (primary outcomes #2 and #3) in a meta‐analysis using a random‐effects model. Where data could not be combined in a meta‐analysis, we reported RR estimates, 95% CIs and statistical significance for each empirical study in tables. We did not plan to meta‐analyse estimates of bias due to selective inclusion. Instead, we have reported estimates of bias as reported by the empirical study investigators in tables.

Subgroup analysis and investigation of heterogeneity

We planned to undertake subgroup analyses to investigate whether the primary prevalence and association outcomes were modified by whether the systematic reviews included in the empirical studies had a review protocol or not; and whether or not the clinical topic of the systematic reviews had an established core outcome set (COMET Initiative).

Sensitivity analysis

We planned to assess the robustness of meta‐analytic effect estimates based on the risk of bias of the included empirical studies. Specifically, we planned to compare meta‐analytic effect estimates that included all eligible empirical studies to meta‐analytic effect estimates that included empirical studies only if they were rated at low risk of selection bias, and low or unclear risk of selective reporting bias. We used the criterion of low or unclear risk of selective reporting bias because we anticipated that most empirical studies would not have a study protocol and hence would be rated as being at unclear risk of bias on this domain. We also planned to perform sensitivity analyses to determine the robustness of meta‐analysed effect estimates based on the different definitions of selective inclusion, discrepant outcome reporting or selective partial reporting used by the empirical study investigators. In cases where ORs were converted to RR, and we were unable to determine the baseline risk in the empirical study, we planned to undertake sensitivity analyses assuming a range of baseline risks from the other included studies.

We undertook a post‐hoc sensitivity analysis to investigate whether the meta‐analytic proportion was robust to the method of analysis. In our primary analysis, we meta‐analysed raw proportions. However, other methods have been shown to be preferential because they remove bias that can arise from assuming the normal approximation to the binomial and because of the correlation between the proportion and its variance (Trikalinos 2013). We undertook meta‐analyses of logit transformed proportions, arcsine transformed proportions, Freeman‐Tukey double arcsine transformed proportions (Freeman 1950) and fitted a random‐effects logistic regression (binomial‐normal model). These sensitivity analyses were performed using the 'metafor' package in the statistical package 'R' (Viechtbauer 2010).