Skip to main content
Key Specialties
Cardiology Diabetology Endocrinology Gastroenterology Geriatrics and Gerontology Gynecology and Obstetrics Hematology Infectious Disease Internal Medicine Nephrology Neurology Oncology Pediatrics Respiratory Medicine Rheumatology Surgery
Congress Coverage
2024 ACC Congress 2023 ESMO Congress 2023 EASD Congress 2023 ERS Congress 2023 ESC Congress 2023 EHA Congress 2023 ASCO Annual Meeting
Clinical Collections
Advances in Alzheimer’s

At a glance: The ONWARDS insulin icodec trials

A round-up of the ONWARDS phase 3 clinical trials evaluating the novel weekly insulin icodec in people with diabetes.
ADVANCED SEARCH
Log in
Top
BMC Medical Research Methodology
Published in: BMC Medical Research Methodology 1/2007

Open Access 01-12-2007 | Research article

Reliability of journal impact factor rankings

Author: Darren C Greenwood

Published in: BMC Medical Research Methodology | Issue 1/2007

Login to get access

Abstract

Background

Journal impact factors and their ranks are used widely by journals, researchers, and research assessment exercises.

Methods

Based on citations to journals in research and experimental medicine in 2005, Bayesian Markov chain Monte Carlo methods were used to estimate the uncertainty associated with these journal performance indicators.

Results

Intervals representing plausible ranges of values for journal impact factor ranks indicated that most journals cannot be ranked with great precision. Only the top and bottom few journals could place any confidence in their rank position. Intervals were wider and overlapping for most journals.

Conclusion

Decisions placed on journal impact factors are potentially misleading where the uncertainty associated with the measure is ignored. This article proposes that caution should be exercised in the interpretation of journal impact factors and their ranks, and specifically that a measure of uncertainty should be routinely presented alongside the point estimate.
Appendix
Available only for authorised users
Literature
1.
Smith R: Commentary: The power of the unrelenting impact factor – Is it a force for good or harm?. Int J Epidemiol. 2006, 35: 1129-1130. 10.1093/ije/dyl191.CrossRefPubMed
2.
3.
4.
Garfield E: Citation Indexes for Science – New Dimension in Documentation Through Association of Ideas. Science. 1955, 122: 108-111. 10.1126/science.122.3159.108.CrossRefPubMed
5.
Garfield E: Citation Analysis As A Tool in Journal Evaluation – Journals Can be Ranked by Frequency and Impact of Citations for Science Policy Studies. Science. 1972, 178: 471-479. 10.1126/science.178.4060.471.CrossRefPubMed
6.
Janke NC, Bide RW, Garfield E: Journal Evaluation. Science. 1973, 182: 1196-1198. 10.1126/science.182.4118.1196.CrossRefPubMed
7.
Wade N: A New Tool for Science Administrators. Science. 1975, 188: 429-432. 10.1126/science.188.4187.429.CrossRefPubMed
8.
Klerer M, Hanley HJM, Arditti J, Machol RE: Citation Analysis. Science. 1975, 188: 1064-10.1126/science.188.4193.1064.CrossRefPubMed
9.
Garfield E, Cawkell AE: Citation Analysis Studies. Science. 1975, 189: 397-10.1126/science.189.4200.397.CrossRefPubMed
10.
Tainer JA, Abt HA, Hargens LL, Bott DM, Lancaster FW, Pannell JH, et al: Science, Citation, and Funding. Science. 1991, 251: 1408-1411. 10.1126/science.2006414.CrossRef
11.
Warner J: A critical review of the application of citation studies to the Research Assessment Exercises. Journal of Information Science. 2000, 26: 453-459. 10.1177/016555150002600607.CrossRef
12.
13.
Seglen PO: Citation Frequency and Journal Impact – Valid Indicators of Scientific Quality?. J Intern Med. 1991, 229: 109-111.CrossRefPubMed
14.
15.
Weale A, Bailey M, Lear P: The level of non-citation of articles within a journal as a measure of quality: a comparison to the impact factor. BMC Med Res Methodol. 2004, 4: 14-10.1186/1471-2288-4-14.CrossRefPubMedPubMedCentral
16.
Ebrahim S: Entelechy, citation indexes, and the association of ideas. Int J Epidemiol. 2006, 35: 1117-1118. 10.1093/ije/dyl201.CrossRef
17.
Porta M, Fernandez E, Bolumar F: Commentary: The 'bibliographic impact factor' and the still uncharted sociology of epidemiology. Int J Epidemiol. 2006, 35: 1130-1135. 10.1093/ije/dyl196.CrossRefPubMed
18.
19.
Seglen PO: The Skewness of Science. J Am Soc Inform Science. 1992, 43: 628-638. 10.1002/(SICI)1097-4571(199210)43:9<628::AID-ASI5>3.0.CO;2-0.CrossRef
20.
Rostami-Hodjegan A, Tucker GT: Journal impact factors: a 'bioequivalence' issue?. British Journal of Clinical Pharmacology. 2001, 51: 111-117. 10.1111/j.1365-2125.2001.01349.x.CrossRefPubMedPubMedCentral
21.
International Committee of Medical Journal Editors: Uniform requirements for manuscripts submitted to biomedical journals. International Committee of Medical Journal Editors. Med Educ. 1999, 33: 66-78. 10.1111/j.1365-2923.1999.00339.x.CrossRef
22.
Goldstein H, Spiegelhalter DJ: League tables and their limitations: Statistical issues in comparisons of institutional performance. J Royal Stat Soc (A). 1996, 159: 385-409. 10.2307/2983325.CrossRef
23.
Marshall EC, Spiegelhalter DJ: Reliability of league tables of in vitro fertilisation clinics: retrospective analysis of live birth rates. BMJ. 1998, 316: 1701-1704.CrossRefPubMedPubMedCentral
24.
Marshall EC, Spiegelhalter DJ: Comparing institutional performance using Markov chain Monte Carlo methods. Statistical analysis of medical data: new developments. Edited by: Everitt BS, Dunn G. 1998, London: Arnold, 229-249.
25.
Marshall EC, Spiegelhalter DJ: Institutional performance. Multilevel modelling of health statistics. Edited by: Leyland AH, Goldstein H. 2001, Chichester: John Wiley & Sons, 127-142.
26.
27.
Spiegelhalter DJ, Thomas A, Best NG, Lunn D: WinBUGS user manual: version 1.4.1. 2004, Cambridge: MRC Biostatistics Unit
28.
Gilks WR, Richardson S, Spiegelhalter DJ: Markov chain monte carlo in practice. 1996, London: Chapman & Hall
29.
Brooks SP, Gelman A: Alternative methods for monitoring convergence of iterative simulations. J Comput Graph Stat. 1998, 7: 434-455. 10.2307/1390675.
30.
Bird SM, Cox D, Farewell VT, Goldstein H, Holt T, Smith PC: Performance indicators: Good, bad, and ugly. J Royal Stat Soc (A). 2005, 168: 1-27. 10.1111/j.1467-985X.2004.00333.x.CrossRef
Metadata
Title
Reliability of journal impact factor rankings
Author
Darren C Greenwood
Publication date
01-12-2007
Publisher
BioMed Central
Published in
BMC Medical Research Methodology / Issue 1/2007
Electronic ISSN: 1471-2288
DOI
https://doi.org/10.1186/1471-2288-7-48

Other articles of this Issue 1/2007

BMC Medical Research Methodology 1/2007 Go to the issue

Research article

Development, feasibility and performance of a health risk appraisal questionnaire for older persons

Research article

Use of hierarchical models to evaluate performance of cardiac surgery centres in the Italian CABG outcome study

Research article

How many mailouts? Could attempts to increase the response rate in the Iraq war cohort study be counterproductive?

Research article

Questionnaire discrimination: (re)-introducing coefficient δ

Research article

Developing search strategies for clinical practice guidelines in SUMSearch and Google Scholar and assessing their retrieval performance

Research article

Do health technology assessments comply with QUOROM diagram guidance? An empirical study