Retractions, Fake Peer Reviews, and Paper Mills

Retractions across the world are unevenly distributed and increasingly related to misconduct.1, 2 We focus here on the rise of retractions related to fake peer review (FPR) and manuscripts forged by paper mills. In addition, we advance some strategies to fight against such fakeries and foster academic integrity.

A map of retractions relative to the number of publications disclosed that, among countries with at least 100,000 papers published in the period 2003–2016, and according to the rate of retractions per 10,000 articles, the figures for the top eight countries ranged from 14.0‱ for Iran to 5.0‱ for China and Turkey; with one exception (Romania), all these are Asian countries.1 In another study that retrieved 2,087 articles retracted from Web of Science's journals during 1978–2013, the five countries with the greatest rate of retractions per 10,000 articles were Egypt (3.04‱), Iran (2.95‱), South Korea (2.19‱), China (2.16‱), and India (1.93‱); for comparison, the respective figures for the USA and UK were 0.68‱ and 0.53‱, respectively.2 In terms of individual researchers, it is remarkable that only 500 out of more than 30,000 authors of retracted papers accounted for a quarter of about 10,500 retractions analyzed in 2018.3

A PubMed search spanning the period 2013–2016 identified 1,082 retracted articles among 4,384,945 papers, i.e., there were 2.5 retractions per 10,000 publications.4 In that study, the five countries with the highest retraction rates were Iran (15.52‱), Egypt (11.75‱), China (8.26‱), India (6.67‱), and Malaysia (3.51‱); in contrast, the respective figures for USA, Canada, and the UK were 1.92‱, 0.89‱, and 0.87‱, respectively. Of significance, the former countries (except for Egypt) also led the ranking by retractions due to manipulated or FPR. A more recent inquiry (search date: August 26, 2019) in the same database documented 6,936 retractions and an overall retraction rate per 10,000 publications of 0.38 in 1985, 2.03 in 2000, and 5.95 in 2014, with Iran (up to 14.0‱), Tunisia (up to 12.0‱), Pakistan (up to 10.0‱), Bangladesh (up to 10.0‱), and India (up to 10.0‱) having the highest rates.5 This investigation did not explore the underlying reasons but highlighted the disruptive potential of retracted clinical trials and medical treatments.

A manipulated peer review or FPR implies that some authors abuse the automated manuscript submission and peer review system to suggest reviewers and supply e-mail addresses created by the submitters themselves, who then fabricate favorable reviews and thus facilitate editorial acceptance of their own papers.6

Retractions due to FPR have shown a global steady increase since they were first documented in 2012.7, 8 Yet, this misconduct may have been first identified in 2009.3 In fact, an analysis of 18,603 retractions compiled by Retraction Watch until February 2019 unveiled that FPR accounted for 676 retractions during 2012-2019 and was thus the most common reason for retractions.9 It has also been noted that retractions related to FPR often occur in journals with a low (< 5 or < 2) or null Clarivate Analytics' impact factor (IF).7, 10 Such malpractice has been documented as being widespread in some developing Asian countries.8, 10, 11, 12 For instance, of 250 papers retracted due to FPR and retrieved from the Retraction Watch database in November 2015, 187 (74.8%) came from China, 28 (11.2%) from South Korea, and 17 (6.8%) from both Iran and Pakistan.7 In contrast, FPR is either infrequent or remains undisclosed in Latin America13 and Africa.14

To expand a brief overview of retractions due to FPR15, it is worth discussing whether retraction rates differ between open access (OA) and pay-walled journals. While some analyses10, 16 have found no differences, a single study covering the 2008–2017 timeframe revealed an almost three-fold higher overall retraction rate in OA journals than in pay-walled journals included in Science Citation Index Expanded.17 FPR thus appears to be one of the main causes for retracting articles from OA journals. Actually, FPR was the leading cause among 134 retractions from BioMed Central18 and also “in the five journals with the highest rate of retractions” in another investigation that considered only journals registered in the Directory of Open Access Journals (DOAJ).10 Likewise, numerous articles have been withdrawn because of FPR from mostly domestic OA journals that are not indexed in the DOAJ.5, 10 Moreover, sizeable proportions (18/36 or 50% and 11/39 or 28%) of COVID-19 articles published in journals with an OA policy or on preprint servers were retracted before November 2, 2020.19 This fact further fuels the “publish first, judge later” controversy associated with preprints20, 21 and the quality of peer review and published articles in OA journals, including OA mega journals.22, 23, 24 In this regard, the distinction between predatory OA journals and legitimate (i.e., that are indexed and may carry an IF) OA journals is becoming increasingly obscured, leading to a gray zone of journals that can no longer be differentiated on ethical grounds.25

Some useful clues to suspect and detect FPR include generic e-mail addresses provided by the potential reviewers who in addition are difficult to find online, as well as review accounts delivered within a few days or even hours and worded quite favorably.6, 26

Tang et al.2 explored the relationship between collaborative papers and retractions by comparing 2,087 retracted papers with 3,970 “control” publications. Because the median number of authors was 4 for the former and 5 for the latter, that study did not support the conjecture that retractions are the other face of collaborations; rather, teamwork seemingly fostered a responsible behavior among coauthors. In contrast, a greater division of labor was associated with higher rates of retraction in a related study (based on the partition of tasks inherent to teamwork) that compared 195 retracted papers with 349 “control” articles published in 2004–2016 and indexed in PubMed Central.27 Since most papers currently have two or more authors and often include senior and young researchers, the consequences of retractions can differ between them. Indeed, it has been noted that the consequences of retractions are comparatively minor for eminent authors who benefit from the reverse Mathew effect2 since “they are less likely to have a paper retracted, the duration from publication to retraction is longer, and citation loss is less severe.”28

Paper mills create on-demand papers, or parts thereof, for paying clients, and have shown to be a rapidly rising menace in academic integrity that are not easy to detect during peer review since, for example, recycled figures might be widely spread among unrelated clients that publish in widely diverse journals, requiring specialized image manipulation software to detect inter-journal duplication.29, 30 Many apparent paper mills have been found to be associated with hospitals in China.31 Occasionally, paper brokers and paper mills include fake peer recommendations for the manuscripts they forge.32, 33 The production of these papers appears to be industrialized, having already churned out potentially thousands of paper-mill-derived submissions published in indexed journals that claim to follow the guidelines of the Committee on Publication Ethics (COPE), and often with an IF.33 As a result, the legitimacy, reputation, and integrity of platforms such as PubMed, which host and thus promote paper mill-derived papers, are at risk.34

Because paper mills can quickly adapt to systems aimed at detecting churned out manuscripts, Byrne and Christopher29 advised editors and reviewers to verify in all submissions reagents and details of how they were used, to look for manipulated or duplicated images, and to request and carefully scrutinize raw data before acceptance. Journals could also post submitted papers and the receipt date in a preprint server to place them in the public domain and thus avoid further deceptive manuscripts and simultaneous multiple-journal submissions. Moreover, publishers and journals must improve post-publication review processes, for instance through an editorial notice (distinct from an expression of concern) describing reported and verifiable errors in a neutral language. Finally, institutional policies should be revised and updated.29

To avoid FPR, some publishers have removed from their online submission systems the option for authors to suggest reviewers, developed strategies to identify genuine reviewers (e.g., via a universal researcher identification such as ORCID, the Open Researcher and Contributor Identifier), or looked at the reviewers' records in Publons.6, 10 Likewise, editors should verify the name and email address of every potential peer reviewer—above all when a famous scientist is presented with a noninstitutional or generic account—and check the lack of any relevant conflict of interest between referees and authors or editors.35 Moreover, some editorial associations (e.g., the International Committee of Medical Journal Editors [ICMJE] and COPE) have issued guidelines on how to avoid or retract fraudulent publications,21 although these have not been sufficient to prevent the abuse of FPR. The ICMJE and COPE ethics guidelines are also highly deficient as they fail to specifically address fake papers, authors, e-mails and affiliations associated with stings and hoaxes.36 In order to remedy such shortcomings, earlier this year the Cooperation and Liaison between Universities and Editors (CLUE) group published a set of recommendations aimed at fostering the integrity of research reports by strengthening the collaboration between research institutions and scholarly journals. Notably, one of these recommendations specifically advise journals to “retain peer review records for at least 10 years to enable the investigation of peer review manipulation or other inappropriate behaviour by authors or reviewers.”37

Countries in which researchers are financially rewarded for their academic productivity, even those for which FPR and mill-derived papers have not been recorded or detected, need to strengthen educational and awareness-raising programs aimed at students, researchers, and other research agents as well as to set up clear policies and guidelines to promote research integrity and good publication practices.12, 13, 14 Recently, a sociological perspective based on organizational theory and pathologies of science has further emphasized the relevance of training in research ethics, requiring a shared responsibility among all team members for the entire research process, and warned that even if division of labor can increase productivity, it might also be pathogenic.27

Retractions due to misconduct illustrate the sociology of deviance and constitute a system of punishment whose effectiveness is still to be proved.28 However, since retractions continue to be widely used, and perceived, as a punitive rather than a corrective mechanism, any educational program—if it plans to succeed—needs to destigmatize such withdrawals38; otherwise, retractions will continue to be combative, conflictual, and corrosive to science. To correct rewards schemes that favor poor publishing behavior, the Chinese government recently decided to stop paying cash bonuses per paper.33 It remains to be seen whether such monetary incentives will also be canceled in other countries that operate similar programs.39 In brief, the basic solution for the malpractices discussed here and related misconduct, is to abolish the current system of perverse incentives in scientific publications for both authors and editors. Otherwise, sly agents will usually circumvent control measures by introducing fresh ways of gaming the publication system.40

Whether or not retractions due to the inappropriate behaviors discussed here are more prevalent in biomedical and health sciences as compared to other fields, is still to be solved. For instance, the topic analysis of 6,936 abstracts of retracted papers retrieved from PubMed disclosed that about 5% of them were from non-biology journals.5 In contrast, a non-biomedical publication accumulated 115/834 articles by Chinese authors found to be retracted in the SCI-expanded database; actually, that journal (Acta Crystallographica Section E-Structure Reports Online with a 2015 IF of 0.0) had almost four times more retractions than the second one (Molecular Biology Reports with a 2015 IF of 1.7).12 It appears that the general relevance and sheer size of the biomedical literature, the greater attention paid to it by sleuths, and a more widespread post-publication peer scrutiny are key underlying factors.

Editors, journals, and publishers that have hosted or published FPR- and paper mill-derived papers are not totally innocent. Even if they have also been victimized, they have often promised rigorous peer review, scientific quality control, and information integrity. Moreover, they may even have benefitted from erroneous and fraudulent work, either by having the privilege of being indexed (e.g., on PubMed, Scopus, Web of Science, etc.), or by boosted journal-based metrics like IF.41 In the latter case, journals' IFs need to be corrected and adjusted downwards to accommodate for excessive (and undeserving) credit.42 Finally, we adhere to the advice offered by seasoned editors on how the trustworthiness of academic publications is threatened by blindly relying on online processing and underscore their view that even the best online systems cannot substitute for editorial intelligence.43