Skip to main content
Top
Published in: BMC Urology 1/2022

Open Access 01-12-2022 | Research article

A review of reproducible and transparent research practices in urology publications from 2014 to2018

Authors: Shelby Rauh, Bradley S. Johnson, Aaron Bowers, Daniel Tritz, Benjamin Matthew Vassar

Published in: BMC Urology | Issue 1/2022

Login to get access

Abstract

Background

Reproducibility is essential for the integrity of scientific research. Reproducibility is measured by the ability of different investigators to replicate the outcomes of an original publication using the same materials and procedures. Unfortunately, reproducibility is not currently a standard being met by most scientific research.

Methods

For this review, we sampled 300 publications in the field of urology to assess for 14 indicators of reproducibility including material availability, raw data availability, analysis script availability, pre-registration information, links to protocols, and if the publication was available free to the public. Publications were also assessed for statements about conflicts of interest and funding sources.

Results

Of the 300 sample publications, 171 contained empirical data available for analysis of reproducibility. Of the 171 articles with empirical data to analyze, 0.58% provided links to protocols, 4.09% provided access to raw data, 3.09% provided access to materials, and 4.68% were pre-registered. None of the studies provided analysis scripts. Our review is cross-sectional in nature, including only PubMed indexed journals-published in English-and within a finite time period. Thus, our results should be interpreted in light of these considerations.

Conclusion

Current urology research does not consistently provide the components needed to reproduce original studies. Collaborative efforts from investigators and journal editors are needed to improve research quality while minimizing waste and patient risk.
Appendix
Available only for authorised users
Literature
2.
go back to reference Baker M. Is there a reproducibility crisis? A Nature survey lifts the lid on how researchers view the’crisis rocking science and what they think will help. Nature. 2016;533(7604):452–5.CrossRef Baker M. Is there a reproducibility crisis? A Nature survey lifts the lid on how researchers view the’crisis rocking science and what they think will help. Nature. 2016;533(7604):452–5.CrossRef
3.
go back to reference Eisner DA. Reproducibility of science: fraud, impact factors and carelessness. J Mol Cell Cardiol. 2018;114:364–8.CrossRef Eisner DA. Reproducibility of science: fraud, impact factors and carelessness. J Mol Cell Cardiol. 2018;114:364–8.CrossRef
4.
go back to reference Loscalzo J. Irreproducible experimental results: causes, (mis)interpretations, and consequences. Circulation. 2012;125(10):1211–4.CrossRef Loscalzo J. Irreproducible experimental results: causes, (mis)interpretations, and consequences. Circulation. 2012;125(10):1211–4.CrossRef
5.
go back to reference Sfanos KS, Yegnasubramanian S, Nelson WG, Lotan TL, Kulac I, Hicks JL, et al. If this is true, what does it imply? How end-user antibody validation facilitates insights into biology and disease. Asian J Urol. 2019;6(1):10–25.CrossRef Sfanos KS, Yegnasubramanian S, Nelson WG, Lotan TL, Kulac I, Hicks JL, et al. If this is true, what does it imply? How end-user antibody validation facilitates insights into biology and disease. Asian J Urol. 2019;6(1):10–25.CrossRef
6.
go back to reference Thompson IM, Goodman PJ, Tangen CM, Lucia MS, Miller GJ, Ford LG, et al. The influence of finasteride on the development of prostate cancer. N Engl J Med. 2003;349(3):215–24.CrossRef Thompson IM, Goodman PJ, Tangen CM, Lucia MS, Miller GJ, Ford LG, et al. The influence of finasteride on the development of prostate cancer. N Engl J Med. 2003;349(3):215–24.CrossRef
7.
go back to reference Redman MW, Tangen CM, Goodman PJ, Lucia MS, Coltman CA Jr, Thompson IM. Finasteride does not increase the risk of high-grade prostate cancer: a bias-adjusted modeling approach. Cancer Prev Res. 2008;1(3):174–81.CrossRef Redman MW, Tangen CM, Goodman PJ, Lucia MS, Coltman CA Jr, Thompson IM. Finasteride does not increase the risk of high-grade prostate cancer: a bias-adjusted modeling approach. Cancer Prev Res. 2008;1(3):174–81.CrossRef
8.
go back to reference Shepherd BE, Redman MW, Ankerst DP. Does finasteride affect the severity of prostate cancer? a causal sensitivity analysis. J Am Stat Assoc. 2008;103(484):1392–404.CrossRef Shepherd BE, Redman MW, Ankerst DP. Does finasteride affect the severity of prostate cancer? a causal sensitivity analysis. J Am Stat Assoc. 2008;103(484):1392–404.CrossRef
9.
go back to reference Baker SG, Darke AK, Pinsky P, Parnes HL, Kramer BS. Transparency and reproducibility in data analysis: the prostate cancer prevention trial. Biostatistics. 2010;11(3):413–8.CrossRef Baker SG, Darke AK, Pinsky P, Parnes HL, Kramer BS. Transparency and reproducibility in data analysis: the prostate cancer prevention trial. Biostatistics. 2010;11(3):413–8.CrossRef
10.
go back to reference McIntosh LD, Juehne A, Vitale CRH, Liu X, Alcoser R, Lukas JC, et al. Repeat: a framework to assess empirical reproducibility in biomedical research. BMC Med Res Methodol. 2017;17(1):143.CrossRef McIntosh LD, Juehne A, Vitale CRH, Liu X, Alcoser R, Lukas JC, et al. Repeat: a framework to assess empirical reproducibility in biomedical research. BMC Med Res Methodol. 2017;17(1):143.CrossRef
11.
go back to reference Hardwicke TE, Wallach JD, Kidwell MC, Bendixen T, Crüwell S, Ioannidis JPA. An empirical assessment of transparency and reproducibility-related research practices in the social sciences (2014–2017). R Soc Open Sci. 2020;7(2):190806.CrossRef Hardwicke TE, Wallach JD, Kidwell MC, Bendixen T, Crüwell S, Ioannidis JPA. An empirical assessment of transparency and reproducibility-related research practices in the social sciences (2014–2017). R Soc Open Sci. 2020;7(2):190806.CrossRef
12.
go back to reference National Academies of Sciences, Engineering, and Medicine; Policy and Global Affairs; Committee on Science, Engineering, Medicine, and Public Policy; Board on Research Data and Information; Division on Engineering and Physical Sciences; Committee on Applied and Theoretical Statistics; Board on Mathematical Sciences and Analytics; Division on Earth and Life Studies; Nuclear and Radiation Studies Board; Division of Behavioral and Social Sciences and Education; Committee on National Statistics; Board on Behavioral, Cognitive, and Sensory Sciences; Committee on Reproducibility and Replicability in Science. Reproducibility and Replicability in Science. Washington (DC): National Academies Press (US); 2019. National Academies of Sciences, Engineering, and Medicine; Policy and Global Affairs; Committee on Science, Engineering, Medicine, and Public Policy; Board on Research Data and Information; Division on Engineering and Physical Sciences; Committee on Applied and Theoretical Statistics; Board on Mathematical Sciences and Analytics; Division on Earth and Life Studies; Nuclear and Radiation Studies Board; Division of Behavioral and Social Sciences and Education; Committee on National Statistics; Board on Behavioral, Cognitive, and Sensory Sciences; Committee on Reproducibility and Replicability in Science. Reproducibility and Replicability in Science. Washington (DC): National Academies Press (US); 2019.
15.
go back to reference Piccolo SR, Frampton MB. Tools and techniques for computational reproducibility. Gigascience. 2016;5(1):30.CrossRef Piccolo SR, Frampton MB. Tools and techniques for computational reproducibility. Gigascience. 2016;5(1):30.CrossRef
17.
go back to reference Ross S, Magee L, Walker M, Wood S. Protecting intellectual property associated with Canadian academic clinical trials–approaches and impact. Trials. 2012;27(13):243.CrossRef Ross S, Magee L, Walker M, Wood S. Protecting intellectual property associated with Canadian academic clinical trials–approaches and impact. Trials. 2012;27(13):243.CrossRef
18.
go back to reference Resnik DB, Shamoo AE. Reproducibility and research integrity. Account Res. 2017;24(2):116–23.CrossRef Resnik DB, Shamoo AE. Reproducibility and research integrity. Account Res. 2017;24(2):116–23.CrossRef
19.
go back to reference Chan A-W, Hróbjartsson A. Promoting public access to clinical trial protocols: challenges and recommendations. Trials. 2018;19(1):116.CrossRef Chan A-W, Hróbjartsson A. Promoting public access to clinical trial protocols: challenges and recommendations. Trials. 2018;19(1):116.CrossRef
20.
go back to reference Mistler S. Planning your analyses: advice for avoiding analysis problems in your research. Psychol Sci Agenda. 2012;26(11):550. Mistler S. Planning your analyses: advice for avoiding analysis problems in your research. Psychol Sci Agenda. 2012;26(11):550.
21.
go back to reference Simmons JP, Nelson LD, Simonsohn U. False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol Sci. 2011;22(11):1359–66.CrossRef Simmons JP, Nelson LD, Simonsohn U. False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol Sci. 2011;22(11):1359–66.CrossRef
22.
go back to reference Kerr NL. HARKing: hypothesizing after the results are known. Pers Soc Psychol Rev. 1998;2(3):196–217.CrossRef Kerr NL. HARKing: hypothesizing after the results are known. Pers Soc Psychol Rev. 1998;2(3):196–217.CrossRef
23.
go back to reference Howard B, Scott JT, Blubaugh M, Roepke B, Scheckel C, Vassar M. Systematic review: outcome reporting bias is a problem in high impact factor neurology journals. PLoS ONE. 2017;12(7):e0180986.CrossRef Howard B, Scott JT, Blubaugh M, Roepke B, Scheckel C, Vassar M. Systematic review: outcome reporting bias is a problem in high impact factor neurology journals. PLoS ONE. 2017;12(7):e0180986.CrossRef
24.
go back to reference Ghooi RB, Bhosale N, Wadhwani R, Divate P, Divate U. Assessment and classification of protocol deviations. Perspect Clin Res. 2016;7(3):132–6.CrossRef Ghooi RB, Bhosale N, Wadhwani R, Divate P, Divate U. Assessment and classification of protocol deviations. Perspect Clin Res. 2016;7(3):132–6.CrossRef
25.
go back to reference Nosek BA, Ebersole CR, DeHaven AC, Mellor DT. The preregistration revolution. Proc Natl Acad Sci U S A. 2018;115(11):2600–6.CrossRef Nosek BA, Ebersole CR, DeHaven AC, Mellor DT. The preregistration revolution. Proc Natl Acad Sci U S A. 2018;115(11):2600–6.CrossRef
26.
go back to reference Allen C, Mehler DMA. Open science challenges, benefits and tips in early career and beyond. PLoS Biol. 2019;17(5):e3000246.CrossRef Allen C, Mehler DMA. Open science challenges, benefits and tips in early career and beyond. PLoS Biol. 2019;17(5):e3000246.CrossRef
29.
go back to reference Myers MG. Diabetes Editorial Team. Diabetes takes new steps to increase transparency and reproducibility. Diabetes. 2019;68(4):681–2.CrossRef Myers MG. Diabetes Editorial Team. Diabetes takes new steps to increase transparency and reproducibility. Diabetes. 2019;68(4):681–2.CrossRef
30.
36.
go back to reference Murad MH, Wang Z. Guidelines for reporting meta-epidemiological methodology research. Evid Based Med. 2017;22(4):139–42.CrossRef Murad MH, Wang Z. Guidelines for reporting meta-epidemiological methodology research. Evid Based Med. 2017;22(4):139–42.CrossRef
37.
go back to reference Jorski A, Scott J, Heavener T, Vassar M. Reporting guideline and clinical trial registration requirements in gastroenterology and hepatology journals. Int J Evid Based Healthc. 2018;16(2):119–27.CrossRef Jorski A, Scott J, Heavener T, Vassar M. Reporting guideline and clinical trial registration requirements in gastroenterology and hepatology journals. Int J Evid Based Healthc. 2018;16(2):119–27.CrossRef
38.
go back to reference Sims MT, Bowers AM, Fernan JM, Dormire KD, Herrington JM, Vassar M. Trial registration and adherence to reporting guidelines in cardiovascular journals. Heart. 2018;104(9):753–9.CrossRef Sims MT, Bowers AM, Fernan JM, Dormire KD, Herrington JM, Vassar M. Trial registration and adherence to reporting guidelines in cardiovascular journals. Heart. 2018;104(9):753–9.CrossRef
39.
go back to reference Sims MT, Henning NM, Wayant CC, Vassar M. Do emergency medicine journals promote trial registration and adherence to reporting guidelines? A survey of “Instructions for Authors.” Scand J Trauma Resusc Emerg Med. 2016;24(1):137.CrossRef Sims MT, Henning NM, Wayant CC, Vassar M. Do emergency medicine journals promote trial registration and adherence to reporting guidelines? A survey of “Instructions for Authors.” Scand J Trauma Resusc Emerg Med. 2016;24(1):137.CrossRef
40.
go back to reference Wan J, Cain MP, Tanaka S, Nelson C. Conflict of interest, self-reporting and our profession. J Urol. 2019;201(4):678–9.CrossRef Wan J, Cain MP, Tanaka S, Nelson C. Conflict of interest, self-reporting and our profession. J Urol. 2019;201(4):678–9.CrossRef
41.
go back to reference Carlisle A, Bowers A, Wayant C, Meyer C, Vassar M. Financial conflicts of interest among authors of urology clinical practice guidelines. Eur Urol. 2018;74(3):348–54.CrossRef Carlisle A, Bowers A, Wayant C, Meyer C, Vassar M. Financial conflicts of interest among authors of urology clinical practice guidelines. Eur Urol. 2018;74(3):348–54.CrossRef
42.
go back to reference Jimbo M, Granberg CF, Osumah TS, Bandari J, Cannon GM, Routh JC, et al. Discrepancies in self-reported and actual conflicts of interest for robotic pediatric urological surgery. J Urol. 2019;201(2):393–9.CrossRef Jimbo M, Granberg CF, Osumah TS, Bandari J, Cannon GM, Routh JC, et al. Discrepancies in self-reported and actual conflicts of interest for robotic pediatric urological surgery. J Urol. 2019;201(2):393–9.CrossRef
45.
go back to reference Green S, Higgins J, Alderson P, Clarke M, Mulrow CD, Oxman AD, et al. Cochrane handbook for systematic reviews of interventions. West Sussex: Wiley; 2008. Green S, Higgins J, Alderson P, Clarke M, Mulrow CD, Oxman AD, et al. Cochrane handbook for systematic reviews of interventions. West Sussex: Wiley; 2008.
47.
go back to reference Vines TH, Albert AYK, Andrew RL, Débarre F, Bock DG, Franklin MT, et al. The availability of research data declines rapidly with article age. Curr Biol. 2014;24(1):94–7.CrossRef Vines TH, Albert AYK, Andrew RL, Débarre F, Bock DG, Franklin MT, et al. The availability of research data declines rapidly with article age. Curr Biol. 2014;24(1):94–7.CrossRef
48.
go back to reference Hardwicke TE, Ioannidis JPA. Populating the Data Ark: an attempt to retrieve, preserve, and liberate data from the most highly-cited psychology and psychiatry articles. PLoS ONE. 2018;13(8):e0201856.CrossRef Hardwicke TE, Ioannidis JPA. Populating the Data Ark: an attempt to retrieve, preserve, and liberate data from the most highly-cited psychology and psychiatry articles. PLoS ONE. 2018;13(8):e0201856.CrossRef
Metadata
Title
A review of reproducible and transparent research practices in urology publications from 2014 to2018
Authors
Shelby Rauh
Bradley S. Johnson
Aaron Bowers
Daniel Tritz
Benjamin Matthew Vassar
Publication date
01-12-2022
Publisher
BioMed Central
Published in
BMC Urology / Issue 1/2022
Electronic ISSN: 1471-2490
DOI
https://doi.org/10.1186/s12894-022-01059-8

Other articles of this Issue 1/2022

BMC Urology 1/2022 Go to the issue