http://orcid.org/0000-0002-2336-5323
Figures
Abstract
Citations for randomized controlled trials (RCT) are important for the dissemination of study results. However, predictors of citations for RCTs have not been investigated. The study aimed to investigate the predictors of citations for RCTs in sepsis literature. RCTs that investigated the efficacy of treatment strategies on clinical outcomes in sepsis patients were included, and publication dates were restricted to the period from 2000 to 2016. Risk of bias was assessed using the Cochrane handbook for systematic reviews and interventions. A multivariable linear regression model was built to investigate the independent variables associated with total citations. In total, 160 RCTs met our inclusion criteria and were included for analysis. The median of total citations was 28.5 (IQR: 6–76). The journal impact factor (IF) for articles was 6.312 (IQR: 3.143–7.214). The dependent variable was transformed by the square root to improve normality and meet the assumption of homoscedasticity. The journal IF (coefficient: 0.2; 95% CI: 0.16, 0.25) was independently associated with total citations. Large samples were associated with more total citations (coefficient: 0.0026; 95% CI: 0.0013, 0.0039). The study demonstrated that the journal IF was a major determinant of the RCT’s total citation number.
Citation: Zhang Z, Poucke SV (2017) Citations for Randomized Controlled Trials in Sepsis Literature: The Halo Effect Caused by Journal Impact Factor. PLoS ONE 12(1): e0169398. https://doi.org/10.1371/journal.pone.0169398
Editor: Neil R. Smalheiser, University of Illinois-Chicago, UNITED STATES
Received: July 6, 2016; Accepted: November 8, 2016; Published: January 3, 2017
Copyright: © 2017 Zhang, Poucke. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the paper and its Supporting Information files.
Funding: The authors received no specific funding for this work.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Randomized controlled trials (RCT) are fundamental to provide high-quality evidence for clinical practice. Therefore, the dissemination of RCT results is of crucial interest for authors, editors and readers. Publication in peer-reviewed journals is a major approach to disseminate the result of RCT findings. Metrics to quantify dissemination of an RCT include the number of reads and downloads from the website, which might be inaccurate. The number of citations for an article provides a much more accurate quantification of dissemination, considering that authors carefully select their reference lists and only the most important work is cited (e.g., some journals limit the number of references).
The publication of RCT should not be considered the end of the entire project. The dissemination of the knowledge is equally important, as is the verification of whether the RCT was up-to-date based on the current state of the art. Therefore, predicting the number of citations for an article is a potential interest to both authors and editors. The journal impact factor (IF) is an important bibliometric variable reflecting the impact of a journal [1–3]. Some studies suggest that IF is a strong predictor of citation, while others report that the quality of study design is equally important [4–6]. However, these studies included all types of original articles and systematic reviews. To the best of our knowledge, no study has investigated the predictors of citations in RCTs. In this study, we focused on RCTs because validated tools were available to assess the risk of bias. In addition, we narrowed our topic to the treatment of sepsis, making the study more homogeneous.
Methods
Study identification
RCTs comparing the effectiveness of different treatments on clinical outcomes were included. The subjects were patients who had sepsis at randomization. The exclusion criteria included 1) animal experiments; 2) septic arthritis; 3) duplicates or a secondary analysis of original trials; 4) non-randomized trials; 5) studies involving non-sepsis or prevention of sepsis; 6) study protocols; 7) systematic reviews; and 8) educational training about sepsis management.
The ISI Web of Science was searched for relevant articles. The searching strategy consisted of key terms of sepsis and randomized controlled trials. We restricted the publication dates from 2000 to May 2016 (the time when the study was conducted). The study was performed according to the PRISMA 2009 checklist (S1 File)
Data extraction
Data were extracted at the journal and article levels. The journal information included IF and journal title. The journal IF is calculated by dividing the number of current year citations to the source items published in that journal during the previous two years. In this study, we used the 2015 IF reported by the journal citation report (JCR). The article information included the article title, the number of participating centers, sample size, the sample size calculation, the results (negative, positive or neutral), whether an academic study group was involved in the study, publication year, doi, whether the study was presented at conferences, the total citations, the average citations per year, and the individual year citation numbers from 2000 to 2016. Only the primary outcome was used to assess whether the study result was positive, negative or neutral. If the primary outcome was not explicitly specified, the conclusion in the abstract section was assessed to determine the result.
Risk of bias
The risk of bias was assessed according to the Cochrane handbook for systematic reviews and interventions [7]. The sequence generation was judged as low risk when the authors described a random component of sequence generation such as random number table, computer random number generator, coin tossing, and throwing dice. Allocation concealment was adequate when the authors described central allocation, sequentially numbered drug containers of identical appearance, and sequentially numbered, opaque, sealed envelopes. Blinding was at low risk of bias when the description was clear or the outcome measurement was unlikely to be influenced by non-blinding. Incomplete outcome data was determined as low risk when there was no missing outcome or when the reason for missing outcome was unlikely to relate to the true outcome. Selective reporting was considered to be high risk when pre-specified outcomes were not reported.
Statistical analysis
Data were expressed as medians (interquartile range) or means (standard deviation) for continuous variables as appropriate [8]. Included RCTs were separated into highly cited and poorly cited groups using the median total citation number as the cutoff. Student’s t-test was used to compare normally distributed data, and Wilcoxon rank sum test was used for skewed data. Categorical variables were expressed as numbers and percentages, and they were compared using Chi-square tests. We assumed that total citations were associated with journal IF and publication year. Total citations were plotted against IF, stratified by publication year. Included RCTs were divided into two groups by the median of total citations. Characteristics were compared between the two groups. Multivariable linear regression model was built to explore explanatory variables that were associated with total citations. Before model fitting, associations among continuous variables were explored using means in a scatter matrix. Because the number of participating centers and the sample sizes were closely associated with each other, only sample size was incorporated into the model to avoid collinearity. As such, the full model included the following variables: sequence generation, allocation concealment, blinding of participants, blinding of outcome assessors, incomplete outcome data, selective reporting, sample size calculation, sample size, results, impact factor, cooperation group, conference, and publication year. After fitting the model, homoscedasticity was examined using a scale-location graph. There was a correlation between fitted values and square roots of standardized residuals. Furthermore, the Q-Q plot demonstrated that the normality assumption was not satisfied. The total citations were transformed by square root, and the model was refit. Stepwise backward elimination and forward selection using the Akaike information criterion was employed to select important variables [9]. All statistical analyses were performed using R (version 3.2.3), and a two-tailed p-value less than 0.05 was considered to be statistical significance.
Results
The initial search identified 244 articles. Full texts of these articles were screened manually for potential eligible studies. Eighty-four articles were excluded because there were 8 animal experiments, 4 septic arthritis studies, 2 duplicates, 10 secondary analyses of original trials, 1 non-randomized trial, 2 studies involving non-sepsis, 13 investigations of the prevention of sepsis, 19 study protocols, 23 systematic reviews, and 2 educational training documents on sepsis management. In total, 160 RCTs met our inclusion criteria and were included for analysis (Fig 1).
Fig 2 displays the risk of bias for each item. Sequence generation was performed well in most RCTs. Allocation concealment was performed less well, and 58 RCTs did not explicitly describe the method for allocation concealment. Blinding of participants was denoted as “high risk” in 32 RCTs. Because most RCTs employed a solid outcome variable such as mortality and some laboratory measurements, the assessment of outcome was less likely to be influenced by blinding. Thus, 145 RCTs (91%) were considered to have a low risk of bias. Twenty-six RCTs were considered to have a high risk of attrition bias because they reported a significant number of losses of follow-up. Selective reporting was at low risk of bias for most trials.
The characteristics of included RCTs are summarized in Table 1. Sample size calculations were performed in 105 (65.5%) RCTs. The median sample size was 90 (interquartile range [IQR]: 48–256). The median number of participating centers was 1 (IQR: 1–11). Most studies reported neutral results (97, 60.6%), followed by positive results (56, 35%) and negative results (7, 4.4%). Forty RCTs were performed by academic study groups. The median publication year was 2010 (IQR: 2006–2014). Five trials were explicitly reported as having been presented during a conference. The median of total citations was 28.5 (IQR: 6–76). Average citations per year for an RCT was 4.36 (IQR: 1.67–7.9). The journal IF for articles was 6.312 (IQR: 3.143–7.214). RCT characteristics were compared between highly and poorly cited groups. These findings showed that there were significantly more participating centers in the highly cited group compared with the poorly cited group (3 [1, 22.3] vs. 1 (1, 6); p = 0.011). RCTs in the highly cited group were published earlier than those in the poorly cited group (2007 (2003, 2010) vs. 2014 (2012, 2015); p<0.001). Journal IFs of highly cited RCTs were significantly greater than those in poorly cited RCTs (6.312 (5.473, 13) vs. 4.442 (1.856, 6.312); p<0.001).
In Fig 3, publication year was divided into three categories, with each category containing an equal number of RCTs. The total number of citations clearly increased with journal IF and publication year. However, there are large residuals that cannot be fully explained by the two variables. To avoid collinearity among independent variables, all continuous variables were plotted against each other (Fig 4). Participating centers and sample sizes appeared to be closely correlated to each other, thus we only retained sample size in the model. Initially, we fit the model without transforming the dependent variable (total citation). The model assumptions were examined with plots (Fig 5). The scale-location plot shows a correlation trend between fitted and square root of standardized residual, indicating a violation of homoscedasticity. Additionally, the Q-Q plot shows the violation of normality. Therefore, square root transformation was applied for the dependent variable [10].
Participating centers and sample sizes appeared to be closely correlated with each other, thus we only retained sample size in the model.
The dependent variable, total citation, was not transformed. The scale-location plot shows a correlation trend between fitted values and square root of the standardized residual, indicating the violation of homoscedasticity. The Q-Q plot also shows the violation of normality.
The final model was built using a stepwise approach (Table 2). Coefficients reported here were adjusted for other factors (e.g., holding all other factors constant). The journal IF (coefficient: 0.2; 95% CI: 0.16, 0.25) and publication year (coefficient: -0.48; 95% CI: -0.57, -0.39) were independently associated with the total citation. Because earlier papers had more time to accumulate citations, it was not surprising that publication year was a significant contributing factor. The risk of bias in the original RCT was not significantly associated with total number of citations, except for selective reporting. RCTs with a low risk of bias had fewer total citations than those with a high risk of bias (coefficient: -4.46; 95% CI: -7.65, -1.27). Sample size was positively correlated with the number of total citations (coefficient: 0.0026; 95% CI: 0.0013, 0.0039). RCTs with neutral results were less likely to be cited than those with negative results (coefficient: -2.73; 95% CI: -4.84, -0.26). Conference presentations tended to improve the number of total citations (coefficient: 1.97; 95% CI: -0.54, 4.49). The model showed a moderate fit to the data, accounting for 70% of total variance. The diagnostic plots showed that model assumptions were satisfied in the model with transformed dependent variables. The Q-Q plot showed that the normal assumption of the dependent variable is well satisfied (Fig 6).
The relationship between the fitted values and square root of standardized residual was weakened. The Q-Q plot showed that the normal assumption of the dependent variable was well satisfied.
Sensitivity analyses were performed by excluding RCTs with citations greater than 500. Multivariable linear regression model was fit (Table 3), which showed similar results. Journal IF was significantly associated with total citations (coefficient: 3.8, CI: 3, 4.5; p<0.001). In this model, the outcome variable was not transformed by the square root, thus the coefficient of journal IF could be interpreted such that each one point increase in journal IF resulted in an increase of 3.8 citation counts.
Discussion
The study demonstrated that journal IF was the major determinant of an RCT’s total citation count. As expected, the total number of citations was dependent on the publication year. Early publications showed significantly more total citations. Sample size, which was closely correlated with the number of participating centers, was also independently associated with total number of citations. RCTs with negative results were more likely to be cited than neutral results, but the magnitude was not large. In addition, conference presentations tended to promote dissemination of RCT results and increase its total number of citations.
Although predictors of citations to RCT have never previously been investigated, several studies have explored predictors of citations to journal articles. Consistent with our findings, journal IF of an article was found to be the most important determinant of the number of citations to that article [11]. These articles included retrospective and prospective trials. If the contribution of IF to a citation was rated as 100, the contributions of sample size, Delphi score, presentation at meeting and study design (retrospective vs. prospective) were 26.5, 26, 5.5 and 2.7, respectively. Positive results had no contribution to the citations [12]. By using multiple regression analysis, Pamela Royle and colleagues found that journal IF explained over half of the variation in citations to systematic reviews [4]. Correlation analysis showed that the 2-year IF on Web of Science was significantly associated with number of citations (Rho = 0.259), but the IF was not incorporated into subsequent multivariable analysis [13]. This study involved systematic reviews in skin diseases. Journal IF was associated with total citations to an RCT in the area of sepsis, independently from risk of bias, sample size, participating centers and other relevant trial-level characteristics. Journal IF may have a halo effect on subsequent citations [14,15]. If there are several RCTs of comparable study design and risk of bias, investigators tend to cite RCTs that are published in high IF journals. The halo effect is defined as a cognitive bias in which an observer's overall impression of an object influences the observer's feelings and thoughts about that entity's character or properties [16]. The impact of an IF on RCT’s total citations can be regarded as a halo effect in bibliometrics.
It is surprising that the risk of bias of RCT had no impact on total citations. In other words, the authors generally do not consider the quality of evidence when citing an article. This finding is inconsistent with other findings. In orthopedic literature, the study design was found to be the only variable associated with the subsequent citation rate. RCTs, meta-analyses and basic science papers had significantly more citations (mean 15.5, 9.3 and 7.6, respectively) than observational studies (mean retrospective 5.3, prospective 4.2) and case reports (mean 1.5) (p = 0.01). However, this study investigated articles published in the journal with the highest IF in orthopedics, and journal IF was not incorporated into the analysis [5]. Furthermore, this study included all types of study designs such as RCTs, retrospective designs, animal studies and systematic reviews. Researchers may have more awareness for the impact of study designs on final conclusions than the impact of risk of bias of RCTs. Furthermore, the risk of bias is usually not explicitly highlighted in an RCT article, and quality of reporting is always very poor [17–19].
Conference presentations of scientific work have been noted to help disseminate its findings [20]. Our study suggested a trend towards more total citations for articles that are also presented to a conference. The sample size of our study was probably not large enough to have the statistical power to detect this difference. Previous studies have also found that the group authorship was associated with more citations (coefficients: 11.1; 95% CI: 2.7, 19.5) [21]. This association was not identified in our multivariable analysis. Of note, data on conference presentations were extracted from the ISI web of science, which may not be exhaustive. Only five of the 160 RCTs were identified as having been presented at a conference.
The strength of the study is the robust methodology in building the regression model. There is evidence that count variables, such as the number of citations, were usually not normally distributed and the assumption of homoscedasticity is usually violated. Previous studies in this field have failed to address this problem, and the results may be biased. This problem was well addressed in the present study by transforming the dependent variable with square root. The study restricted to RCTs, whose results can be more homogenous. Previous studies incorporated all types of articles, including animal studies, retrospective studies and prospective trials [5,13,22]. In this situation, the within-study type random effects were not considered, leaving a large portion of unexplained residuals. However, several limitations of the study should be acknowledged. The study involved articles indexed in ISI web of science. Whether our results can be extrapolated to other databases that provide a citation service, such as Scopus and Google scholar, is largely unknown [23]. Our model is only moderately fitted, accounting for approximately 70% of the variance of the total citations. Some factors, such as the number of authors, article length, and open access status, may be potential predictors of citations [6,24,25].
Author Contributions
- Conceptualization: ZZ.
- Data curation: ZZ.
- Formal analysis: ZZ.
- Funding acquisition: ZZ.
- Investigation: ZZ.
- Methodology: SP.
- Project administration: ZZ.
- Resources: ZZ.
- Software: ZZ SP.
- Supervision: ZZ.
- Validation: SP.
- Visualization: SP.
- Writing – original draft: ZZ.
- Writing – review & editing: SP.
References
- 1. Puche RC. [Impact factor, its variants and its influence in academic promotion]. Medicina (B Aires). 2011;71: 484–489.
- 2. Kumar V, Upadhyay S, Medhi B. Impact of the impact factor in biomedical research: its use and misuse. Singapore Med J. 2009;50: 752–755. pmid:19710969
- 3. Weale AR, Bailey M, Lear PA. The level of non-citation of articles within a journal as a measure of quality: a comparison to the impact factor. BMC Med Res Methodol. BioMed Central; 2004;4: 14.
- 4. Royle P, Kandala N-B, Barnard K, Waugh N. Bibliometrics of systematic reviews: analysis of citation rates and journal impact factors. Syst Rev. 2013;2: 74. pmid:24028376
- 5. Bhandari M, Busse J, Devereaux PJ, Montori VM, Swiontkowski M, Tornetta P Iii, et al. Factors associated with citation rates in the orthopedic literature. Can J Surg. Canadian Medical Association; 2007;50: 119–123.
- 6. Antoniou GA, Antoniou SA, Georgakarakos EI, Sfyroeras GS, Georgiadis GS. Bibliometric analysis of factors predicting increased citations in the vascular and endovascular literature. Ann Vasc Surg. Elsevier; 2015;29: 286–292.
- 7.
Higgins JP, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions. Chichester, UK: John Wiley & Sons, Ltd; 2008. pp. 1–649.
- 8. Zhang Z. Univariate description and bivariate statistical inference: the first step delving into data. Ann Transl Med. 2016;4: 91–91. pmid:27047950
- 9. Zhang Z. Variable selection with stepwise and best subset approaches. Ann Transl Med. 2016;4: 136–136. pmid:27162786
- 10. Rosopa PJ, Schaffer MM, Schroeder AN. Managing heteroscedasticity in general linear models. Psychological Methods. 2013;18: 335–351. pmid:24015776
- 11. Falagas ME, Zarkali A, Karageorgopoulos DE, Bardakas V, Mavros MN. The impact of article length on the number of future citations: a bibliometric analysis of general medicine journals. Fortunato S, editor. PLoS ONE. Public Library of Science; 2013;8: e49476.
- 12. Callaham M, Wears RL, Weber E. Journal prestige, publication bias, and other characteristics associated with citation of published studies in peer-reviewed journals. JAMA. 2002;287: 2847–2850. pmid:12038930
- 13. Manriquez J, Cataldo K, Harz I. Factors influencing citations to systematic reviews in skin diseases: a cross-sectional study through Web of Sciences and Scopus. An Bras Dermatol. Sociedade Brasileira de Dermatologia; 2015;90: 646–652.
- 14. Golder W. [The impact factor: a critical analysis]. Rofo. 1998;169: 220–226. pmid:9779060
- 15. Ha TC, Tan SB, Soo KC. The journal impact factor: too much of an impact? Ann Acad Med Singap. 2006;35: 911–916. pmid:17219007
- 16. Nisbett RE, Wilson TD. The halo effect: Evidence for unconscious alteration of judgments. Journal of Personality and Social Psychology. 1977;35: 250–256.
- 17. Richardson M, Garner P, Donegan S. Cluster Randomised Trials in Cochrane Reviews: Evaluation of Methodological and Reporting Practice. Bruns H, editor. PLoS ONE. Public Library of Science; 2016;11: e0151818.
- 18. Lucena C, Souza EM, Voinea GC, Pulgar R, Valderrama MJ, De-Deus G. A quality assessment of randomized controlled trials reports in endodontics. Int Endod J. 2016;: n/a–n/a.
- 19. Sanclemente G, Pardo H, Sánchez S, Bonfill X. Analysis of the Quality of Clinical Trials Published in Spanish-Language Dermatology Journals Between 1997 and 2012. Actas Dermosifiliogr. 2016;107: 44–54. pmid:26546026
- 20. Sugimoto CR, Thelwall M, Larivière V, Tsou A, Mongeon P, Macaluso B. Scientists popularizing science: characteristics and impact of TED talk presenters. Bar-Ilan J, editor. PLoS ONE. Public Library of Science; 2013;8: e62403.
- 21. Kulkarni AV, Busse JW, Shams I. Characteristics associated with citation rate of the medical literature. Bacchetti P, editor. PLoS ONE. Public Library of Science; 2007;2: e403.
- 22. Willis DL, Bahler CD, Neuberger MM, Dahm P. Predictors of citations in the urological literature. BJU Int. 2011;107: 1876–1880. pmid:21332629
- 23. Kulkarni AV, Aziz B, Shams I, Busse JW. Comparisons of citations in Web of Science, Scopus, and Google Scholar for articles published in general medical journals. JAMA. American Medical Association; 2009;302: 1092–1096.
- 24. Wrafter PF, Connelly TM, Khan J, Devane L, Kelly J, Joyce WP. The 100 most influential manuscripts in colorectal cancer: A bibliometric analysis. Surgeon. Elsevier; 2016;0.
- 25. Pastorino R, Milovanovic S, Stojanovic J, Efremov L, Amore R, Boccia S. Quality Assessment of Studies Published in Open Access and Subscription Journals: Results of a Systematic Evaluation. Zaykin D, editor. PLoS ONE. Public Library of Science; 2016;11: e0154217.