Skip to main content
Key Specialties
Cardiology Diabetology Endocrinology Gastroenterology Geriatrics and Gerontology Gynecology and Obstetrics Hematology Infectious Disease Internal Medicine Nephrology Neurology Oncology Pediatrics Respiratory Medicine Rheumatology Surgery
Congress Coverage
2024 ASCO Annual Meeting 2024 ACC Congress 2023 ESMO Congress 2023 EASD Congress 2023 ERS Congress 2023 ESC Congress
Clinical Collections
Advances in Alzheimer’s

2024 ASCO Annual Meeting

Keep up to date with all the top stories and latest evidence with our hub page, including official congress videos and expert takeaways.
ADVANCED SEARCH
Log in
Top
Systematic Reviews
Published in: Systematic Reviews 1/2013

Open Access 01-12-2013 | Research

Bibliometrics of systematic reviews: analysis of citation rates and journal impact factors

Authors: Pamela Royle, Ngianga-Bakwin Kandala, Katharine Barnard, Norman Waugh

Published in: Systematic Reviews | Issue 1/2013

Login to get access

Abstract

Background

Systematic reviews are important for informing clinical practice and health policy. The aim of this study was to examine the bibliometrics of systematic reviews and to determine the amount of variance in citations predicted by the journal impact factor (JIF) alone and combined with several other characteristics.

Methods

We conducted a bibliometric analysis of 1,261 systematic reviews published in 2008 and the citations to them in the Scopus database from 2008 to June 2012. Potential predictors of the citation impact of the reviews were examined using descriptive, univariate and multiple regression analysis.

Results

The mean number of citations per review over four years was 26.5 (SD ±29.9) or 6.6 citations per review per year. The mean JIF of the journals in which the reviews were published was 4.3 (SD ±4.2). We found that 17% of the reviews accounted for 50% of the total citations and 1.6% of the reviews were not cited. The number of authors was correlated with the number of citations (r = 0.215, P < 0.001). Higher numbers of citations were associated with the following characteristics: first author from the United States (36.5 citations), an ICD-10 chapter heading of Neoplasms (31.8 citations), type of intervention classified as Investigation, Diagnostics or Screening (34.7 citations) and having an international collaboration (32.1 citations). The JIF alone explained more than half of the variation in citations (R 2 = 0.59) in univariate analysis. Adjusting for both JIF and type of intervention increased the R 2 value to 0.81. Fourteen percent of reviews published in the top quartile of JIFs (≥ 5.16) received citations in the bottom quartile (eight or fewer), whereas 9% of reviews published in the lowest JIF quartile (≤ 2.06) received citations in the top quartile (34 or more). Six percent of reviews in journals with no JIF were also in the first quartile of citations.

Conclusions

The JIF predicted over half of the variation in citations to the systematic reviews. However, the distribution of citations was markedly skewed. Some reviews in journals with low JIFs were well-cited and others in higher JIF journals received relatively few citations; hence the JIF did not accurately represent the number of citations to individual systematic reviews.
Appendix
Available only for authorised users
Literature
1.
Bastian H, Glasziou P, Chalmers I: Seventy-five trials and eleven systematic reviews a day: How will we ever keep up?. PLoS Med. 2010, 7: e1000326-10.1371/journal.pmed.1000326.CrossRefPubMedPubMedCentral
2.
3.
Colledge L, de Moya-Anegón F, Guerrero-Bote V, López-Illescas C, El Aisati M, Moed HF: SJR and SNIP: two new journal metrics in Elsevier’s Scopus. Serials. 2010, 23: 215-221. 10.1629/23215.CrossRef
4.
5.
Moed HF: Measuring contextual citation impact of scientific journals. J Informetr. 2010, 4: 265-277. 10.1016/j.joi.2010.01.002.CrossRef
6.
Irwin RS, Augustyn N, French CT, Rice J, Tedeschi V, Welch SJ, Editorial Leadership Team: Spread the word about the journal in 2013: from citation manipulation to invalidation of patient-reported outcomes measures to renaming the Clara cell to new journal features. Chest. 2013, 143: 1-4. 10.1378/chest.12-2762.
7.
Sonuga-Barke EJS: “Holy Grail” or “Siren's song”? The dangers for the field of child psychology and psychiatry of over-focusing on the journal impact factor. J Child Psychol Psychiatry. 2012, 53: 915-917. 10.1111/j.1469-7610.2012.02612.x.CrossRefPubMed
8.
9.
Johnson MH, Cohen J, Grudzinskas G: The uses and abuses of bibliometrics. Reprod Biomed Online. 2012, 24: 485-486. 10.1016/j.rbmo.2012.03.007.CrossRefPubMed
10.
Pierce GN: Too much impact for the impact factor: Are a new generation of scientists in peril? [in English and French]. Can J Physiol Pharmacol. 2012, 90: iii-iv.CrossRef
11.
Waheed U, Satti HS: The impact factor: a bad impact on individual research. Health Educ J. 2012, 71: 253-254. 10.1177/0017896912444179.CrossRef
12.
13.
Opthof T, Coronel R, Piper HM: Impact factors: no totum pro parte by skewness of citation. Cardiovasc Res. 2004, 61: 201-203. 10.1016/j.cardiores.2003.11.023.CrossRefPubMed
14.
Falagas ME, Zarkali A, Karageorgopoulos DE, Bardakas V, Mavros MN: The impact of article length on the number of future citations: a bibliometric analysis of general medicine journals. PLoS One. 2013, 8: e49476-10.1371/journal.pone.0049476.CrossRefPubMedPubMedCentral
15.
16.
Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gøtzsche PC, Ioannidis JP, Clarke M, Devereaux PJ, Kleijnen J, Moher D: The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. PLoS Med. 2009, 6: e1000100-10.1371/journal.pmed.1000100.CrossRefPubMedPubMedCentral
17.
Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group: Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. J Clin Epidemiol. 2009, 62: 1006-1012. 10.1016/j.jclinepi.2009.06.005.CrossRefPubMed
18.
Petticrew M, Song F, Wilson P, Wright K: Quality-assessed reviews of health care interventions and the database of abstracts of reviews of effectiveness (DARE): NHS CRD review, dissemination, and information teams. Int J Technol Assess Health Care. 1999, 15: 671-678.PubMed
19.
20.
21.
Lokker C, McKibbon KA, McKinlay RJ, Wilczynski NL, Haynes RB: Prediction of citation counts for clinical articles at two years using data available within three weeks of publication: retrospective cohort study. BMJ. 2008, 336: 655-657. 10.1136/bmj.39482.526713.BE.CrossRefPubMedPubMedCentral
22.
Kulkarni AV, Busse JW, Shams I: Characteristics associated with citation rate of the medical literature. PLoS One. 2007, 2: e403-10.1371/journal.pone.0000403.CrossRefPubMedPubMedCentral
23.
Callaham M, Wears RL, Weber E: Journal prestige, publication bias, and other characteristics associated with citation of published studies in peer-reviewed journals. JAMA. 2002, 287: 2847-2850. 10.1001/jama.287.21.2847.CrossRefPubMed
24.
Weale AR, Bailey M, Lear PA: The level of non-citation of articles within a journal as a measure of quality: a comparison to the impact factor. BMC Med Res Methodol. 2004, 4: 14-10.1186/1471-2288-4-14.CrossRefPubMedPubMedCentral
25.
Allareddy V, Lee MK, Shah A, Elangovan S, Lin CY: Association between study design and citation counts of articles published in the American Journal of Orthodontics and Dentofacial Orthopedics and Angle Orthodontist. Orthodontics (Chic). 2012, 13: 184-191.
26.
Bhandari M, Busse J, Devereaux PJ, Montori VM, Swiontkowski M, Tornetta P, Einhorn TA, Khera V, Schemitsch EH: Factors associated with citation rates in the orthopedic literature. Can J Surg. 2007, 50: 119-123.PubMedPubMedCentral
27.
Montori VM, Wilczynski NL, Morgan D, Haynes RB, Hedges Team: Systematic reviews: a cross-sectional study of location and citation counts. BMC Med. 2003, 1: 2-10.1186/1741-7015-1-2.CrossRefPubMedPubMedCentral
28.
Okike K, Kocher MS, Torpey JL, Nwachukwu BU, Mehlman CT, Bhandari M: Level of evidence and conflict of interest disclosure associated with higher citation rates in orthopedics. J Clin Epidemiol. 2011, 64: 331-338. 10.1016/j.jclinepi.2010.03.019.CrossRefPubMed
29.
Patsopoulos NA, Analatos AA, Ioannidis JP: Relative citation impact of various study designs in the health sciences. JAMA. 2005, 293: 2362-2366. 10.1001/jama.293.19.2362.CrossRefPubMed
30.
Willis DL, Bahler CD, Neuberger MM, Dahm P: Predictors of citations in the urological literature. BJU Int. 2011, 107: 1876-1880. 10.1111/j.1464-410X.2010.10028.x.CrossRefPubMed
31.
Winnik S, Raptis DA, Walker JH, Hasun M, Speer T, Clavien PA, Komajda M, Bax JJ, Tendera M, Fox K, Van de Werf F, Mundow C, Lüscher TF, Ruschitzka F, Matter CM: From abstract to impact in cardiovascular research: factors predicting publication and citation. Eur Heart J. 2012, 33: 3034-3045. 10.1093/eurheartj/ehs113.CrossRefPubMedPubMedCentral
32.
Figg WD, Dunn L, Liewehr DJ, Steinberg SM, Thurman PW, Barrett JC, Birkinshaw J: Scientific collaboration results in higher citation rates of published articles. Pharmacotherapy. 2006, 26: 759-767. 10.1592/phco.26.6.759.CrossRefPubMed
33.
Evangelou E, Siontis KC, Pfeiffer T, Ioannidis JP: Perceived information gain from randomized trials correlates with publication in high-impact factor journals. J Clin Epidemiol. 2012, 65: 1274-1281. 10.1016/j.jclinepi.2012.06.009.CrossRefPubMed
34.
Siontis KC, Hernandez-Boussard T, Ioannidis JP: Overlapping meta-analyses on the same topic: survey of published studies. BMJ. 2013, 347: f4501-10.1136/bmj.f4501.CrossRefPubMedPubMedCentral
35.
Badgett R: Why would physicians undervalue reviews by the Cochrane Collaboration?. J Clin Epidemiol. 2008, 61: 419-421. 10.1016/j.jclinepi.2007.11.022.CrossRefPubMed
36.
Delaney A, Bagshaw SM, Ferland A, Laupland K, Manns B, Doig C: The quality of reports of critical care meta-analyses in the Cochrane Database of Systematic Reviews: an independent appraisal. Crit Care Med. 2007, 35: 589-594. 10.1097/01.CCM.0000253394.15628.FD.CrossRefPubMed
37.
McKinlay RJ, Cotoi C, Wilczynski NL, Haynes RB, for the McMaster Premium Literature Service (PLUS) Project: Systematic reviews and original articles differ in relevance, novelty, and use in an evidence-based service for physicians: PLUS Project. J Clin Epidemiol. 2008, 61: 449-454. 10.1016/j.jclinepi.2007.10.016.CrossRefPubMed
38.
39.
Laupacis A, Straus S: Systematic reviews: time to address clinical and policy relevance as well as methodological rigor. Ann Intern Med. 2007, 147: 273-274. 10.7326/0003-4819-147-4-200708210-00180.CrossRefPubMed
40.
Pagliaro L, Bruzzi P, Bobbio M: Why are Cochrane hepato-biliary reviews undervalued by physicians as an aid for clinical decision-making?. Dig Liver Dis. 2010, 42: 1-5. 10.1016/j.dld.2009.07.003.CrossRefPubMed
41.
Yaffe J, Montgomery P, Hopewell S, Shepard LD: Empty reviews: a description and consideration of Cochrane systematic reviews with no included studies. PLoS One. 2012, 7: e36626-10.1371/journal.pone.0036626.CrossRefPubMedPubMedCentral
Metadata
Title
Bibliometrics of systematic reviews: analysis of citation rates and journal impact factors
Authors
Pamela Royle
Ngianga-Bakwin Kandala
Katharine Barnard
Norman Waugh
Publication date
01-12-2013
Publisher
BioMed Central
Published in
Systematic Reviews / Issue 1/2013
Electronic ISSN: 2046-4053
DOI
https://doi.org/10.1186/2046-4053-2-74

Other articles of this Issue 1/2013

Systematic Reviews 1/2013 Go to the issue

Protocol

Association between caregiver quality of life and the care provided to persons with Alzheimer’s disease: protocol for a systematic review

Protocol

A systematic review investigating fatigue, psychological and cognitive impairment following TIA and minor stroke: protocol paper

Research

Interventions provided in the acute phase for mild traumatic brain injury: a systematic review

Research

Completeness of reporting of setting and health worker cadre among trials on antenatal iron and folic acid supplementation in pregnancy: an assessment based on two Cochrane reviews

Research

Insufficient evidence for the use of a physical examination to detect maltreatment in children without prior suspicion: a systematic review

Protocol

Self-collected versus clinician-collected sampling for sexually transmitted infections: a systematic review and meta-analysis protocol