Skip to main content
Key Specialties
Cardiology Diabetology Endocrinology Gastroenterology Geriatrics and Gerontology Gynecology and Obstetrics Hematology Infectious Disease Internal Medicine Nephrology Neurology Oncology Pediatrics Respiratory Medicine Rheumatology Surgery
Congress Coverage
2024 ACC Congress 2023 ESMO Congress 2023 EASD Congress 2023 ERS Congress 2023 ESC Congress
Clinical Collections
Advances in Alzheimer’s

At a glance: The STEP trials

A round-up of the STEP phase 3 clinical trials evaluating semaglutide for weight loss in people with overweight or obesity.
ADVANCED SEARCH
Log in
Top
Journal of Medical Systems
Published in: Journal of Medical Systems 1/2023

Open Access 01-12-2023 | Chickenpox | Original Paper

Analysis: Flawed Datasets of Monkeypox Skin Images

Authors: Carlos Vega, Reinhard Schneider, Venkata Satagopam

Published in: Journal of Medical Systems | Issue 1/2023

Login to get access

Abstract

The self-proclaimed first publicly available dataset of Monkeypox skin images consists of medically irrelevant images extracted from Google and photography repositories through a process denominated web-scrapping. Yet, this did not stop other researchers from employing it to build Machine Learning (ML) solutions aimed at computer-aided diagnosis of Monkeypox and other viral infections presenting skin lesions. Neither did it stop the reviewers or editors from publishing these subsequent works in peer-reviewed journals. Several of these works claimed extraordinary performance in the classification of Monkeypox, Chickenpox and Measles, employing ML and the aforementioned dataset. In this work, we analyse the initiator work that has catalysed the development of several ML solutions, and whose popularity is continuing to grow. Further, we provide a rebuttal experiment that showcases the risks of such methodologies, proving that the ML solutions do not necessarily obtain their performance from the features relevant to the diseases at issue.
Appendix
Available only for authorised users
Literature
1.
Ahsan, M.M., Uddin, M.R., Farjana, M., Sakib, A.N., Momin, K.A., Luna, S.A.: Image data collection and implementation of deep learning-based model in detecting monkeypox disease using modified vgg16. arXiv preprint arXiv:​2206.​01862 (2022)
2.
3.
Sambasivan, N., Kapania, S., Highfill, H., Akrong, D., Paritosh, P., Aroyo, L.M.: ”everyone wants to do the model work, not the data work”: Data cascades in high-stakes ai. In: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1–15 (2021)
4.
Roberts, M., Driggs, D., Thorpe, M., Gilbey, J., Yeung, M., Ursprung, S., Aviles-Rivero, A.I., Etmann, C., McCague, C., Beer, L., et al: Common pitfalls and recommendations for using machine learning to detect and prognosticate for covid-19 using chest radiographs and ct scans. Nature Machine Intelligence 3(3), 199–217 (2021)
5.
Williams, C.M., Chaturvedi, R., Urman, R.D., Waterman, R.S., Gabriel, R.A.: Artificial intelligence and a pandemic: An analysis of the potential uses and drawbacks. Springer (2021)
6.
López-Cabrera, J.D., Orozco-Morales, R., Portal-Diaz, J.A., Lovelle-Enríquez, O., Pérez-Díaz, M.: Current limitations to identify covid-19 using artificial intelligence with chest x-ray imaging. Health and Technology 11(2), 411–424 (2021)
7.
Santa Cruz, B.G., Bossa, M.N., Sölter, J., Husch, A.D.: Public covid-19 x-ray datasets and their impact on model bias–a systematic review of a significant problem. Medical image analysis 74, 102225 (2021)
8.
Bullock, J., Luccioni, A., Pham, K.H., Lam, C.S.N., Luengo-Oroz, M.: Mapping the landscape of artificial intelligence applications against covid-19. Journal of Artificial Intelligence Research 69, 807–845 (2020)
9.
Jain, N., Tanasov, A., Chodnekar, S.Y., Rakauskaitė, A., Lansiaux, E., Skuja, S., Reinis, A.: Quantitative bibliometric excellence & productivity in monkeypox (mpox) literature
10.
Maguolo, G., Nanni, L.: A critic evaluation of methods for covid-19 automatic detection from x-ray images. Information Fusion 76, 1–7 (2021)
11.
12.
13.
14.
Tartaglione, E., Barbano, C.A., Berzovini, C., Calandri, M., Grangetto, M.: Unveiling covid-19 from chest x-ray with deep learning: a hurdles race with small data. International Journal of Environmental Research and Public Health 17(18), 6933 (2020)
15.
16.
17.
18.
19.
20.
21.
22.
23.
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:​1409.​1556 (2014)
24.
25.
Islam, T., Hussain, M.A., Chowdhury, F.U.H., Islam, B.R.: Can artificial intelligence detect monkeypox from digital skin images? bioRxiv (2022)
26.
Gul, I., Liu, C., Yuan, X., Du, Z., Zhai, S., Lei, Z., Chen, Q., Raheem, M.A., He, Q., Hu, Q., et al: Current and perspective sensing methods for monkeypox virus. Bioengineering 9(10), 571 (2022)
27.
Sahin, V.H., Oztel, I., Yolcu Oztel, G.: Human monkeypox classification from skin lesion images with deep pre-trained network using mobile application. Journal of Medical Systems 46(11), 1–10 (2022)
28.
Sitaula, C., Shahi, T.B.: Monkeypox virus detection using pre-trained deep learning-based approaches. Journal of Medical Systems 46(11), 1–9 (2022)
29.
Patel, M., Surti, M., Adnan, M.: Artificial intelligence (ai) in monkeypox infection prevention. Journal of Biomolecular Structure and Dynamics, 1–5 (2022)
30.
Eid, M.M., El-Kenawy, E.-S.M., Khodadadi, N., Mirjalili, S., Khodadadi, E., Abotaleb, M., Alharbi, A.H., Abdelhamid, A.A., Ibrahim, A., Amer, G.M., et al: Meta-heuristic optimization of lstm-based deep network for boosting the prediction of monkeypox cases. Mathematics 10(20), 3845 (2022)
31.
Alcalá-Rmz, V., Villagrana-Bañuelos, K.E., Celaya-Padilla, J.M., Galván-Tejada, J.I., Gamboa-Rosales, H., Galván-Tejada, C.E.: Convolutional neural network for monkeypox detection. In: International Conference on Ubiquitous Computing and Ambient Intelligence, pp. 89–100 (2023). Springer
32.
Haque, M., Ahmed, M., Nila, R.S., Islam, S., et al.: Classification of human monkeypox disease using deep learning models and attention mechanisms. arXiv preprint arXiv:​2211.​15459 (2022)
33.
Ali, S.N., Ahmed, M.T., Paul, J., Jahan, T., Sani, S.M.S., Noor, N., Hasan, T.: Monkeypox skin lesion detection using deep learning models: A preliminary feasibility study. arXiv preprint arXiv:​2207.​03342 (2022)
34.
Islam, T., Hussain, M.A., Chowdhury, F.U.H., Islam, B.R.: A web-scrapped skin image database of monkeypox, chickenpox, smallpox, cowpox, and measles. bioRxiv (2022)
35.
36.
Obradović, S.: Publication pressures create knowledge silos. Nature Human Behaviour 3(10), 1028–1028 (2019)
37.
Forti, L.R., Solino, L.A., Szabo, J.K.: Trade-off between urgency and reduced editorial capacity affect publication speed in ecological and medical journals during 2020. Humanities and Social Sciences Communications 8(1), 1–9 (2021)
38.
Maier-Hein, L., Eisenmann, M., Reinke, A., Onogur, S., Stankovic, M., Scholz, P., Arbel, T., Bogunovic, H., Bradley, A.P., Carass, A., et al: Why rankings of biomedical image analysis competitions should be interpreted with care. Nature communications 9(1), 5217 (2018)
39.
Varoquaux, G., Cheplygina, V.: Machine learning for medical imaging: methodological failures and recommendations for the future. NPJ digital medicine 5(1), 48 (2022)
40.
Greener, J.G., Kandathil, S.M., Moffat, L., Jones, D.T.: A guide to machine learning for biologists. Nature Reviews Molecular Cell Biology 23(1), 40–55 (2022)
41.
Yang, G., Ye, Q., Xia, J.: Unbox the black-box for the medical explainable ai via multi-modal and multi-centre data fusion: A mini-review, two showcases and beyond. Information Fusion 77, 29–52 (2022)
42.
Cabitza, F., Campagner, A.: The need to separate the wheat from the chaff in medical informatics: Introducing a comprehensive checklist for the (self)-assessment of medical AI studies. Elsevier (2021)
43.
Filipp, F.V.: Opportunities for artificial intelligence in advancing precision medicine. Current genetic medicine reports 7, 208–213 (2019)
44.
Larrazabal, A.J., Nieto, N., Peterson, V., Milone, D.H., Ferrante, E.: Gender imbalance in medical imaging datasets produces biased classifiers for computer-aided diagnosis. Proceedings of the National Academy of Sciences 117(23), 12592–12594 (2020)
45.
Qayyum, A., Qadir, J., Bilal, M., Al-Fuqaha, A.: Secure and robust machine learning for healthcare: A survey. IEEE Reviews in Biomedical Engineering 14, 156–180 (2020)
46.
Vega, C., Kratochvil, M., Satagopam, V., Schneider, R.: Translational challenges of biomedical machine learning solutions in clinical and laboratory settings. In: Bioinformatics and Biomedical Engineering: 9th International Work-Conference, IWBBIO 2022, Maspalomas, Gran Canaria, Spain, June 27–30, 2022, Proceedings, Part II, pp. 353–358 (2022). Springer
47.
Vega, C.: From hume to wuhan: an epistemological journey on the problem of induction in covid-19 machine learning models and its impact upon medical research. Ieee Access 9, 97243–97250 (2021)
48.
Reyes, M., Meier, R., Pereira, S., Silva, C.A., Dahlweid, F.-M., Tengg-Kobligk, H.v., Summers, R.M., Wiest, R.: On the interpretability of artificial intelligence in radiology: challenges and opportunities. Radiology: artificial intelligence 2(3), 190043 (2020)
49.
Goyal, M., Knackstedt, T., Yan, S., Hassanpour, S.: Artificial intelligence-based image classification methods for diagnosis of skin cancer: Challenges and opportunities. Computers in Biology and Medicine 127, 104065 (2020)
50.
Hogarty, D.T., Su, J.C., Phan, K., Attia, M., Hossny, M., Nahavandi, S., Lenane, P., Moloney, F.J., Yazdabadi, A.: Artificial intelligence in dermatology-where we are and the way to the future: a review. American journal of clinical dermatology 21, 41–47 (2020)
51.
Liang, W., Tadesse, G.A., Ho, D., Fei-Fei, L., Zaharia, M., Zhang, C., Zou, J.: Advances, challenges and opportunities in creating data for trustworthy ai. Nature Machine Intelligence 4(8), 669–677 (2022)
52.
Daneshjou, R., Vodrahalli, K., Liang, W., Novoa, R.A., Jenkins, M., Rotemberg, V., Ko, J., Swetter, S.M., Bailey, E.E., Gevaert, O., et al.: Disparities in dermatology AI: Assessments using diverse clinical images. arXiv preprint arXiv:​2111.​08006 (2021)
53.
Hutchinson, B., Smart, A., Hanna, A., Denton, E., Greer, C., Kjartansson, O., Barnes, P., Mitchell, M.: Towards accountability for machine learning datasets: Practices from software engineering and infrastructure. In: Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pp. 560–575 (2021)
54.
Garcia Santa Cruz, B., Vega, C., Hertel, F.: The need of standardised metadata to encode causal relationships: Towards safer data-driven machine learning biological solutions. In: Computational Intelligence Methods for Bioinformatics and Biostatistics: 17th International Meeting, CIBB 2021, Virtual Event, November 15–17, 2021, Revised Selected Papers, pp. 200–216 (2022). Springer
55.
Holstein, K., Wortman Vaughan, J., Daumé III, H., Dudik, M., Wallach, H.: Improving fairness in machine learning systems: What do industry practitioners need? In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, pp. 1–16 (2019)
56.
Mustra, M., Delac, K., Grgic, M.: Overview of the dicom standard. In: 2008 50th International Symposium ELMAR, vol. 1, pp. 39–44 (2008). IEEE
57.
Funkhouser, W.K.: Pathology: the clinical description of human disease. In: Essential Concepts in Molecular Pathology, pp. 177–190. Elsevier, ??? (2020)
58.
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J.W., Wallach, H., Iii, H.D., Crawford, K.: Datasheets for datasets. Communications of the ACM 64(12), 86–92 (2021)
59.
Collins, G.S., de Groot, J.A., Dutton, S., Omar, O., Shanyinde, M., Tajar, A., Voysey, M., Wharton, R., Yu, L.-M., Moons, K.G., et al: External validation of multivariable prediction models: a systematic review of methodological conduct and reporting. BMC medical research methodology 14(1), 1–11 (2014)
60.
Cabitza, F., Campagner, A., Soares, F., de Guadiana-Romualdo, L.G., Challa, F., Sulejmani, A., Seghezzi, M., Carobene, A.: The importance of being external. methodological insights for the external validation of machine learning models in medicine. Computer Methods and Programs in Biomedicine 208, 106288 (2021)
61.
Rizk, B., Brat, H., Zille, P., Guillin, R., Pouchy, C., Adam, C., Ardon, R., d’Assignies, G.: Meniscal lesion detection and characterization in adult knee mri: a deep learning model approach with external validation. Physica Medica 83, 64–71 (2021)
62.
Bien, N., Rajpurkar, P., Ball, R.L., Irvin, J., Park, A., Jones, E., Bereket, M., Patel, B.N., Yeom, K.W., Shpanskaya, K., et al: Deep-learning-assisted diagnosis for knee magnetic resonance imaging: development and retrospective validation of mrnet. PLoS medicine 15(11), 1002699 (2018)
63.
Faes, L., Wagner, S.K., Fu, D.J., Liu, X., Korot, E., Ledsam, J.R., Back, T., Chopra, R., Pontikos, N., Kern, C., et al: Automated deep learning design for medical image classification by health-care professionals with no coding experience: a feasibility study. The Lancet Digital Health 1(5), 232–242 (2019)
64.
Han, S.S., Moon, I.J., Kim, S.H., Na, J.-I., Kim, M.S., Park, G.H., Park, I., Kim, K., Lim, W., Lee, J.H., et al: Assessment of deep neural networks for the diagnosis of benign and malignant skin neoplasms in comparison with dermatologists: A retrospective validation study. PLoS medicine 17(11), 1003381 (2020)
65.
Treveil, M., Team, D.: Introducing MLOps, O’Reilly Media, Inc. (2020)
66.
Webb, G.I., Hyde, R., Cao, H., Nguyen, H.L., Petitjean, F.: Characterizing concept drift. Data Mining and Knowledge Discovery 30(4), 964–994 (2016)
67.
Stirbu, V., Granlund, T., Mikkonen, T.: Continuous design control for machine learning in certified medical systems. Software Quality Journal, 1–27 (2022)
68.
Miller, T.: Explanation in artificial intelligence: Insights from the social sciences. Artificial intelligence 267, 1–38 (2019)
69.
Kim, B., Khanna, R., Koyejo, O.O.: Examples are not enough, learn to criticize! criticism for interpretability. Advances in neural information processing systems 29 (2016)
70.
Wulf, A.J., Seizov, O.: ”please understand we cannot provide further information”: evaluating content and transparency of gdpr-mandated ai disclosures. AI & SOCIETY, 1–22 (2022)
71.
Molnar, C.: Interpretable Machine Learning. Lulu. com, ??? (2020)
72.
Apley, D.W., Zhu, J.: Visualizing the effects of predictor variables in black box supervised learning models. arXiv preprint arXiv:​1612.​08468 (2016)
73.
Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. Advances in neural information processing systems 30 (2017)
74.
Simonyan, K., Vedaldi, A., Zisserman, A.: Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:​1312.​6034 (2013)
75.
Olah, C., Mordvintsev, A., Schubert, L.: Feature visualization. Distill 2(11), 7 (2017)
76.
Metadata
Title
Analysis: Flawed Datasets of Monkeypox Skin Images
Authors
Carlos Vega
Reinhard Schneider
Venkata Satagopam
Publication date
01-12-2023
Publisher
Springer US
Keyword
Chickenpox
Published in
Journal of Medical Systems / Issue 1/2023
Print ISSN: 0148-5598
Electronic ISSN: 1573-689X
DOI
https://doi.org/10.1007/s10916-023-01928-1

Other articles of this Issue 1/2023

Journal of Medical Systems 1/2023 Go to the issue

Original Paper

Detecting Patient Deterioration Early Using Continuous Heart rate and Respiratory rate Measurements in Hospitalized COVID-19 Patients

Original Paper

Virtual Reality as a Hypnotic Tool in the Management of Anxiety During the Performance of the Axillary Block

Original Paper

“ChatGPT, Can You Help Me Save My Child’s Life?” - Diagnostic Accuracy and Supportive Capabilities to Lay Rescuers by ChatGPT in Prehospital Basic Life Support and Paediatric Advanced Life Support Cases – An In-silico Analysis

Original Paper

VirtMRI: A Tool for Teaching MRI

Original Paper

Gamification to Improve Medication Adherence: A Mixed-method Usability Study for MedScrab

Original Paper

Efficacy and Patients’ Satisfaction with the ORCAM MyEye Device Among Visually Impaired People: A Multicenter Study