Skip to main content
Key Specialties
Cardiology Diabetology Endocrinology Gastroenterology Geriatrics and Gerontology Gynecology and Obstetrics Hematology Infectious Disease Internal Medicine Nephrology Neurology Oncology Pediatrics Respiratory Medicine Rheumatology Surgery
Congress Coverage
2024 ACC Congress 2023 ESMO Congress 2023 EASD Congress 2023 ERS Congress 2023 ESC Congress
Clinical Collections
Advances in Alzheimer’s

At a glance: The STEP trials

A round-up of the STEP phase 3 clinical trials evaluating semaglutide for weight loss in people with overweight or obesity.
ADVANCED SEARCH
Log in
Top
BMC Medical Imaging
Published in: BMC Medical Imaging 1/2015

Open Access 01-12-2015 | Software

Metrics for evaluating 3D medical image segmentation: analysis, selection, and tool

Authors: Abdel Aziz Taha, Allan Hanbury

Published in: BMC Medical Imaging | Issue 1/2015

Login to get access

Abstract

Background

Medical Image segmentation is an important image processing step. Comparing images to evaluate the quality of segmentation is an essential part of measuring progress in this research area. Some of the challenges in evaluating medical segmentation are: metric selection, the use in the literature of multiple definitions for certain metrics, inefficiency of the metric calculation implementations leading to difficulties with large volumes, and lack of support for fuzzy segmentation by existing metrics.

Result

First we present an overview of 20 evaluation metrics selected based on a comprehensive literature review. For fuzzy segmentation, which shows the level of membership of each voxel to multiple classes, fuzzy definitions of all metrics are provided. We present a discussion about metric properties to provide a guide for selecting evaluation metrics. Finally, we propose an efficient evaluation tool implementing the 20 selected metrics. The tool is optimized to perform efficiently in terms of speed and required memory, also if the image size is extremely large as in the case of whole body MRI or CT volume segmentation. An implementation of this tool is available as an open source project.

Conclusion

We propose an efficient evaluation tool for 3D medical image segmentation using 20 evaluation metrics and provide guidelines for selecting a subset of these metrics that is suitable for the data and the segmentation task.
Literature
1.
Zou KH, Warfield SK, Baharatha A, Tempany C, Kaus MR, Haker SJ, et al. Statistical validation of image segmentation quality based on a spatial overlap index. Academic Radiology. 2004; 11:178–89.CrossRefPubMedPubMedCentral
2.
Zou KH, Wells WM, Kikinis R, Warfield SK. Three validation metrics for automated probabilistic image segmentation of brain tumours. Stat Med. 2004; 23:1259–82.CrossRefPubMedPubMedCentral
3.
Kennedy DN, Makris N, Verne SC, Worth AJ. Neuroanatomical segmentation in MRI: Technological objectives. IJPRAI. 1997; 11(8):1161–87.
4.
Warfield SK, Westin CF, Guttmann CRG, Albert MS, Jolesz FA, Kikinis R. Fractional segmentation of white matter. In: Proceedings of Second International Conference on Medical Imaging Computing and Computer Assisted Interventions: 1999. p. 62–71. doi:10.​1007/​10704282_​7.
5.
6.
Fenster A, Chiu B. Evaluation of segmentation algorithms for medical imaging. In: Conf Proc IEEE Eng Med Biol Soc. Shanghai: 2005. p. 7186–189.
7.
Vinh NX, Epps J, Bailey J. Information theoretic measures for clusterings comparison: Variants, properties, normalization and correction for chance. J Mach Learn Res. 2010; 11:2837–854.
8.
Gerig G, Jomier M, Chakos M. Valmet: A new validation tool for assessing and improving 3D object segmentation. In: Proceedings of the 4th International Conference on Medical Image Computing and Computer-Assisted Intervention: 2001. p. 516–23. doi:10.​1007/​3-540-45468-3_​62.
9.
Jr. Maurer CR, Qi R, Raghavan V. A linear time algorithm for computing exact Euclidean distance transforms of binary images in arbitrary dimensions. IEEE Trans Pattern Anal Mach Intell. 2003; 25(2):265–70.CrossRef
10.
Udupa JK, LeBlanc VR, Zhuge Y, Imielinska C, Schmidt H, Currie LM, et al. A framework for evaluating image segmentation algorithms. Comput Med Imaging Graph. 2006; 30(2):75–87.CrossRefPubMed
11.
12.
Campello RJGB. A fuzzy extension of the Rand index and other related indexes for clustering and classification assessment. Pattern Recognit Lett. 2007; 28(7):833–41.CrossRef
13.
Dice LR. Measures of the amount of ecologic association between species. Ecology. 1945; 26(3):297–302.CrossRef
14.
Jaccard P. The distribution of the flora in the alpine zone. New Phytologist. 1912; 11(2):37–50.CrossRef
15.
Cardenes R, de Luis-Garcia R, Bach-Cuadra M. A multidimensional segmentation evaluation for medical image data. Comput Methods Prog Biomed. 2009; 96(2):108–24.CrossRef
16.
Al-Faris AQ, Ngah UK, Isa NAM, Shuaib IL. MRI breast skin-line segmentation and removal using integration method of level set active contour and morphological thinning algorithms. J Med Sci. 2013. doi:10.​3923/​jms.​2012.​286.​291.
17.
Crum WR, Camara O, Hill DLG. Generalized overlap measures for evaluation and validation in medical image analysis. IEEE Trans Med Imaging. 2006; 25(11):1451–61.CrossRefPubMed
18.
19.
Rijsbergen CJV. Information Retrieval, 2nd edn. Newton, MA, USA: Butterworth-Heinemann; 1979.
20.
Martin DR, Fowlkes C, Tal D, Malik J. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: Proc. 8th Int’l Conf. Computer Vision. Vancouver, BC: 2001. p. 416–23.
21.
Reddy AR, Prasad EV, Reddy LSS. Abnormality detection of brain mr image segmentation using iterative conditional mode algorithm. Int J Appl Inform Syst. 2013; 5(2):56–66.
22.
Vadaparthi N, Yarramalle S, Penumatsa SV, Murthy PSR. Segmentation of brain mr images based on finite skew gaussian mixture model with fuzzy c-means clustering and em algorithm. Int J Comput Appl. 2011; 28(10):18–26.
23.
Reddy AR, Prasad EV, Reddy LSS. Comparative analysis of brain tumor detection using different segmentation techniques. Int J Comput Appl. 2013; 82(14):14–28.
24.
25.
26.
Brouwer RK. Extending the Rand, adjusted Rand and Jaccard indices to fuzzy partitions. J Intell Inf Syst. 2009; 32(3):213–35.CrossRef
27.
Anderson DT, Bezdek JC, Popescu M, Keller JM. Comparing fuzzy, probabilistic, and possibilistic partitions. Trans Fuz Sys. 2010; 18(5):906–18.CrossRef
28.
Rand WM. Objective criteria for the evaluation of clustering methods. J Am Stat Assoc. 1971; 66(336):846–50.CrossRef
29.
30.
Cover TM, Thomas JA. Elements of Information Theory. New York, NY, USA: Wiley-Interscience; 1991.CrossRef
31.
Viola P, III Wells WM. Alignment by maximization of mutual information. international journal of computer. Int J Comput Vis. 1997; 24(2):137–54.CrossRef
32.
Russakoff DB, Tomasi C, Rohlfing T, Jr Maurer CR. Image similarity using mutual information of regions. In: 8th European Conference on Computer Vision, ECCV: 2004. p. 596–607. doi:10.​1007/​978-3-540-24672-5_​47.
33.
Meila M. Comparing clusterings by the variation of information. In: Learning Theory and Kernel Machines. Berlin Heidelberg: Springer: 2003. p. 173–87.
34.
Shrout P, Fleiss J. Intraclass correlations: Uses in assessing rater reliability. Psychol Bull. 1979; 86(2):420–8.CrossRefPubMed
35.
Cohen J. A coefficient of agreement for nominal scales. Educ Psychol Meas. 1960; 20:37–46.CrossRef
36.
Hanley JA, Mcneil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology. 1982; 143:29–36.CrossRefPubMed
37.
Bradley AP. The use of the area under the ROC curve in the evaluation of machine learning algorithms. Pattern Recognit. 1997; 30(7):1145–59.CrossRef
38.
Powers DMW. Evaluation: From precision, recall and F-factor to ROC, informedness, markedness correlation. J Mach Learn Technol. 2011; 2:37–63.
39.
40.
Zhang D, Lu G. Review of shape representation and discription techniques. Pattern Recognit. 2004; 37(1):1–19.CrossRef
41.
Huttenlocher DP, Klanderman GA, Rucklidge WA. Comparing images using the Hausdorff distance. IEEE Trans Pattern Anal Mach Intell. 1993; 15:850–63.CrossRef
42.
43.
Mahalanobis PC. On the generalised distance in statistics. In: Proceedings National Institute of Science, India: 1936. p. 49–55.
44.
45.
Sladoje N, Lindblad J, Nystrom I. Defuzzification of spatial fuzzy sets by feature distance minimization. Image Vis Comput. 2011; 29:127–41.CrossRef
46.
Zwick R, Karlstein E, Budiscu DV. Measures of similarity among fuzzy concepts: a comparative analysis. Int J Approx Reason. 1987; 1(2):221–42.CrossRef
47.
48.
Jain AK, Dubes RC. Algorithms for Clustering Data. Upper Saddle River, NJ, USA: Prentice-Hall, Inc.; 1988.
49.
Igual L, Soliva JC, Hernndez-Vela A, Escalera S, Vilarroya O, Radeva P. Supervised brain segmentation and classification in diagnostic of attention-deficit/hyperactivity disorder. In: HPCS. Madrid: 2012. p. 182–7.
50.
Fisher RA: Biological Monographs and Manuals. Statistical Methods for Research Workers. Edinburgh Scotland: Oliver and Boyd; 1970.
51.
52.
53.
Langs G, Mueller H, Menze BH, Hanbury A. Visceral: Towards large data in medical imaging - challenges and directions. In: MCBR-CDS MICCAI Workshop. Nice, France: 2013. p. 92–8.
54.
Jimenez del Toro OA, Goksel O, Menze B, Mueller H, Langs G, Weber MA, et al. Visceral - visual concept extraction challenge in radiology: Isbi 2014 challenge organization. In: Proceedings of the VISCERAL Challenge at ISBI. Beijing, China: 2014. p. 6–15.
55.
Fatourechi M, Ward RK, Mason SG, Huggins J, Schloegl A, Birch GE. Comparison of evaluation metrics in classification applications with imbalanced datasets. In: ICMLA. San Diego, CA: 2009. p. 777–82.
56.
Taha AA, Hanbury A, Jimenez del Toro O. A formal method for selecting evaluation metrics for image segmentation. In: 2014 IEEE International Conference on Image Processing (ICIP) (ICIP 2014). Paris, France: 2014. p. 932–6.
57.
Klein S, van der Heide UA, Raaymakers BW, Kotte ANTJ, Staring M, Pluim JPW. Segmentation of the prostate in mr images by atlas matching. In: ISBI. Arlington, VA: 2007. p. 1300–3.
58.
Cai X, Hou Y, Li C, Lee J, Wee WG. 2006. Evaluation of two segmentation methods on mri brain tissue structures.
59.
Gouttard S, Styner M, Prastawa M, Piven J, Gerig G. Assessment of reliability of multi-site neuroimaging via traveling phantom study. In: Proceedings of the 11th International Conference on Medical Image Computing and Computer-Assisted Intervention: 2008. p. 263–70. doi:10.​1007/​978-3-540-85990-1_​32.
60.
Keyvan K, Mohammad Javad D, Kamran K, Mohammad Sadegh H, Kafshgari S. Comparison evaluation of three brain mri segmentation methods in software tools. In: Biomedical Engineering (ICBME). Isfahan: 2010. p. 1–4.
61.
Babalola KO, Patenaude B, Aljabar P, Schnabel J, Kennedy D, Crum W, et al. Comparison and evaluation of segmentation techniques for subcortical structures in brain MRI. Med Image Comput Comput Assist Interv. 2008; 11(Pt 1):409–16.PubMed
62.
63.
Khotanlou H, Colliot O, Atif J, Bloch I. 3D brain tumor segmentation in MRI using fuzzy classification, symmetry analysis and spatially constrained deformable models. Fuzzy Sets Syst. 2009; 160(10):1457–73.CrossRef
64.
Pang Y, Li L, Hu W, Peng Y, Liu L, Shao Y. Computerized segmentation and characterization of breast lesions in dynamic contrast-enhanced mr images using fuzzy c-means clustering and snake algorithm. Comput Math Methods Med. 2012. doi:10.​1155/​2012/​634907.
65.
66.
Yadav S, Meshram S. Performance evaluation of basic segmented algorithms for brain tumor detection. J Electron Commun Eng IOSR. 2013; 5:08–13.CrossRef
67.
68.
69.
70.
Doring TM, Kubo TTA, Cruz LCH, Juruena MF, Fainberg J, Domingues RC, et al. Evaluation of hippocampal volume based on mr imaging in patients with bipolar affective disorder applying manual and automatic segmentation techniques. J Magn Reson Imaging. 2011; 33(3):565–72.CrossRefPubMed
71.
Morain-Nicolier F, Lebonvallet S, Baudrier E, Ruan S. Hausdorff distance based 3D quantification of brain tumor evolution from MRI images. In: 29th Annual International Conference of the IEEE Engineering in Medicine and Biology Society. Lyon, France: 2007. p. 5597–600.
72.
Narendran P, Narendira Kumar VK, Somasundaram K. 3D Brain Tumors and internal brain structures segmentation in mr images. Int J Image Graphics Signal Process. 2012; 1:ISSN: 2074-9074.
73.
Metadata
Title
Metrics for evaluating 3D medical image segmentation: analysis, selection, and tool
Authors
Abdel Aziz Taha
Allan Hanbury
Publication date
01-12-2015
Publisher
BioMed Central
Published in
BMC Medical Imaging / Issue 1/2015
Electronic ISSN: 1471-2342
DOI
https://doi.org/10.1186/s12880-015-0068-x

Other articles of this Issue 1/2015

BMC Medical Imaging 1/2015 Go to the issue

Technical advance

Cell recognition based on topological sparse coding for microscopy imaging of focused ultrasound treatment

Research article

Reproducibility of brain metabolite concentration measurements in lesion free white matter at 1.5 T

Research article

How to measure renal artery stenosis - a retrospective comparison of morphological measurement approaches in relation to hemodynamic significance

Research article

Bronchus sign on thin-section computed tomography is a powerful predictive factor for successful transbronchial biopsy using endobronchial ultrasound with a guide sheath for small peripheral lung lesions: a retrospective observational study

TECHNICAL ADVANCE

Automatic detection of abnormalities in mammograms

Research article

Determination of the human spine curve based on laser triangulation