Skip to main content
Key Specialties
Cardiology Diabetology Endocrinology Gastroenterology Geriatrics and Gerontology Gynecology and Obstetrics Hematology Infectious Disease Internal Medicine Nephrology Neurology Oncology Pediatrics Respiratory Medicine Rheumatology Surgery
Congress Coverage
2024 ACC Congress 2023 ESMO Congress 2023 EASD Congress 2023 ERS Congress 2023 ESC Congress
Clinical Collections
Advances in Alzheimer’s

At a glance: The ONWARDS insulin icodec trials

A round-up of the ONWARDS phase 3 clinical trials evaluating the novel weekly insulin icodec in people with diabetes.
ADVANCED SEARCH
Log in
Top
Radiological Physics and Technology
Published in: Radiological Physics and Technology 2/2018

01-06-2018

Overview on subjective similarity of images for content-based medical image retrieval

Author: Chisako Muramatsu

Published in: Radiological Physics and Technology | Issue 2/2018

Login to get access

Abstract

Computer-aided diagnosis systems for assisting the classification of various diseases have the potential to improve radiologists’ diagnostic accuracy and efficiency, as reported in several studies. Conventional systems generally provide the probabilities of disease types in terms of numerical values, a method that may not be efficient for radiologists who are trained by reading a large number of images. Presentation of reference images similar to those of a new case being diagnosed can supplement the probability outputs based on computerized analysis as an intuitive guide, and it can assist radiologists in their diagnosis, reporting, and treatment planning. Many studies on content-based medical image retrievals have been reported on. For retrieval of perceptually similar and diagnostically relevant images, incorporation of perceptual similarity data by radiologists has been suggested. In this paper, studies on image retrieval methods are reviewed with a special focus on quantification, utilization, and the evaluation of subjective similarities between pairs of images.
Literature
1.
Muller H, Michoux N, Bandon D, Geissbuhler A. A review of content-based image retrieval systems in medical applications—clinical benefits and future directions. Int J Med Inf. 2004;73:1–23.CrossRef
2.
Long LR, Antani S, Deserno TM, Thoma GR. Content-based image retrieval in medicine: retrospective assessment, state of the art, and future directions. Int J Health Inf Syst Inform. 2009;4(1):1–16.CrossRef
3.
Akgul CB, Rubin DL, Napel S, Beaulieu CF, Hayit G, Acar B. Content-based image retrieval in radiology: current status and future directions. J Digit Imaging. 2011;24(2):208–22.CrossRefPubMed
4.
Kumar A, Dim J, Cai W, Fulham M, Feng D. Content-based medical image retrieval: a survey of applications to multidimensional and multimodality data. J Digit Imaging. 2013;26:1025–39.CrossRefPubMedPubMedCentral
5.
Li Z, Zhang X, Muller H, Zhang S. Large-scale retrieval for medical image analytics: a comprehensive review. Med Image Anal. 2018;43:66–84.CrossRefPubMed
6.
Haralick RM, Shanmugam K, Dinstein I. Textural features for image classification. IEEE Trans Sys Manuf Cyber. 1973;SMC-3(6):610–21.CrossRef
7.
Tang X. Texture information in run-length matrices. IEEE Image Process. 1998;7(11):1602–9.CrossRef
8.
Daugman JG. Uncertainty relation for resolution in space, spatial frequency, and orientation optimized by two-dimensional visual cortical filters. J Opt Soc Am. 1985;2(7):1160–9.CrossRef
9.
Cross GR, Jain AK. Markov random field texture models. IEEE Trans Pat Anal Mach Intel. 1983; PAMI-5(1):25–39.CrossRef
10.
Ojala T, Pietikainen M, Maenpaa T. Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Trans Pat Anal Mach Intel. 2002;24(7):971–87.CrossRef
11.
Huo Z, Giger ML, Vyborny CJ, Bick U, Lu P, Wolverton DE, Schmidt RA. Analysis of spiculation in the computerized classification of mammographic masses. Med Phys. 1995;22:1569–79.CrossRefPubMed
12.
Kobatake H, Hashimoto S. Convergence index filter for vector fields. IEEE Trans Image Process. 1999;8(8):1029–38.CrossRefPubMed
13.
Bunke H, Irniger C, Neuhaus M. Graph matching – challenges and potential solutions. Int Conf Image Anal Process. 2005;LNCS3617:1–10.
14.
Sharma H, Alekseychuk A, Leskovsky P, Hellwich O, Anand RS, Zerbe N, Hufnagl P. Determining similarity in histological images using graph-teoretic description and matching methods for content-based image retrieval in medical diagnostics. Diagn Pathol. 2012;7:134.CrossRefPubMedPubMedCentral
15.
Kumar A, Kim J, Wen L, Fulham M, Feng D. A graph-based approach for the retrieval of multi-modality medical images. Med Image Anal. 2014;18:330–42.CrossRefPubMed
16.
17.
Giger ML, Huo Z, Vyborny CJ, Lan L, Bonta I, Horsch K, Nishikawa RM, Rosenbourgh I. Intelligent CAD workstation for breast imaging using similarity to known lesions and multiple visual prompt aids. Proc SPIE Med Imaging. 2002;4684:78–73.
18.
Nakagawa T, Hara T, Fujita H, Iwase T, Endo T. Image retrieval system of mammographic masses by using local pattern matching technique. In: Peitgen HO, editor. Digital Mammography. Berlin: Springer; 2003. pp. 562–5.CrossRef
19.
Muramatsu C, Li Q, Suzuki K, Schmidt RA, Shiraishi J, Newstead GM, Doi K. Investigation of psychophysical measure for evaluation of similar images for mammographic masses: preliminary results. Med Phys. 2005;32:2295–304.CrossRefPubMed
20.
Alto H, Rangayyan RM, Desautels JEL. Content-based retrieval and analysis of mammographic masses. J Electron Imaging. 2005;14(2):023016.CrossRef
21.
Zheng B, Lu A, Hardesty LA, Sumkin JH, Hakim CM, Ganott MA, Gur D. A method to improve visual similarity of breast masses for an interactive computer-aided diagnosis environment. Med Phys. 2006;33:111–7.CrossRefPubMed
22.
Kinoshita SK, Marques PMA, Pereira R, Rodrigues JAH, Rangayyan RM. Content-based retrieval of mammograms using visual features related to breast density patterns. J Digit Imaging. 2007;20:172–90.CrossRefPubMedPubMedCentral
23.
Nakayama R, Abe H, Shiraishi J, Doi K. Evaluatino of objective similarity measures for selecting similar images of mammographic lesions. J Digit Imaging. 2011;24:75–85.CrossRefPubMed
24.
Wei CH, Chen SY, Liu X. Mammogram retrieval on similar mass lesions. Comput Methods Prog Biomed. 2012;106(3):234–48.CrossRef
25.
Liu J, Zhang S, Liu W, Zhang X, Metaxas DN. Scalable mammogram retrieval using anchor graph hashing. In: IEEE International symposium on biomedical imaging, ISBI 2014; pp 898–901.
26.
Jaing M, Zhang S, Li H, Metazas DN. Computer-aided diagnosis of mammographic masses using scalable image retrieval. IEEE Trans Biomed Eng. 2015;62(2):783–92.CrossRef
27.
Bedo MVN, Pereira dos Santos D, Ponciano-Silva M, Marques PMA, Ferreira de Carvalho APL, Traina C. Endowing a content-based medical image retrieval system with perceptual similarity using ensemble strategy. J Digit Imaging. 2016;29:22–37.CrossRefPubMed
28.
Sklansky J, Tao EY, Bazargan M, Ornes CJ, Murchison RC, Teklehaimanot S. Computer-aided, case-based diagnosis of mammographic regions of interest containing microcalcifications. Acad Radiol. 2000;7:395–405.CrossRefPubMed
29.
El-Naqa I, Yang Y, Galatsanos NP, Nishikawa RM, Wernick MN. A similarity learning approach to content-based image retrieval: Application to digital mammography. IEEE Trans Med Imaging. 2004;23(10):1233–44.CrossRefPubMed
30.
Muramatsu C, Li Q, Schmidt RA, Shiraishi J, Doi K. Investigation of psyhophysical similarity measures for selection of similar images in the diagnosis of clustered microcalcifications on mammograms. Med Phys. 2008;35:5695–702.CrossRefPubMed
31.
Kuo WJ, Chang RF, Lee CC, Moon WK, Chen DR. Retrieval technique for the diagnosis of solid breast tumors on sonogram. Ultrasound Med Biol. 2002;28(7):903–9.CrossRefPubMed
32.
Cho H, Hadjiiski L, Sahiner B, Chan HP, Helvie M, Paramagul C, Nees AV. Similarity evaluation in content-based image retrieval (CBIR) CADx system for characterization of breast masses on ultrasound images. Med Phys. 2011;38(4):1820–31.CrossRefPubMedPubMedCentral
33.
Aisen AM, Broderick LS, Winer-Muram H, Brodley CE, Kak AC, Pavlopoulou C, Dy J, Shyu CR, Marchiori A. Automated storage and retrieval of thin-section CT images to assist diagnosis: System Description and preliminary assessment. Radiology. 2003;228:265–70.CrossRefPubMed
34.
Li Q, Li F, Shiraishi J, Katsuragawa S, Sone S, Doi K. Investigation of new psyhophysical measures for evaluation of similar images on thoracic CT for distinction between benign and malignant nodules. Med Phys. 2003;30:2584–93.CrossRefPubMed
35.
Kawata Y, Niki N, Ohmatsu H, Moriyama N. Example-based assisting approach for pulmonary nodule classification in three-dimensional thoracic computed tomography images. Acad Radiol. 2003;10:1402–15.CrossRefPubMed
36.
37.
Wei G, Cao H, Ma H, Qi S, Qian W, Ma Z. Content-based image retrieval for lung nodule classification using texture features and learned distance metric. J Med Syst. 2018;42:13.CrossRef
38.
Depeursinge A, Varagas A, Gaillard F, Platon A, Geissbuhler A, Poletti PA, Muller H. Case-based lung image categorization and retrieval for interstitial lung diseases clinical workflows. Int J CARS. 2012;7:97–110.CrossRef
39.
Bugatti PH, Kaster DS, Pociano-Silva M, Traina C Jr. Marques PMA, Traina AJM. PRosPer: perceptual similarity queries in medical CBIR systems through user profiles. Comput Bio Med. 2014;45:8–19.CrossRef
40.
Xu J, Faruque J, Beaulieu C, Rubin D, Napel S. A comprehensive descriptor of shape: Method and application to content-based retrieval of similar appearing lesions in medical images. J Digit Imaging. 2012;25:121–8.CrossRefPubMed
41.
Yang W, Lu Z, Yu M, Huang M, Feng Q, Chen W. Content-based retrieval of focal liver lesions using bag-of-visual-words representations of single- and multiphase contrast-enhanced CT images. J Digit Imaging. 2012;25:708–19.CrossRefPubMedPubMedCentral
42.
Dankerl P, Cavallaro A, Tsymbal A, Costa MJ, Suehling M, Janka R, Uder M, Hammon M. A retrieval-based computer-aided diagnosis system for the characterization of liver lesions in CT scans. Acad Radiol. 2013;20:1526–34.CrossRefPubMed
43.
Roy S, Chi Y, Liu J, Venkatesh SK, Brown MS. Three-dimensional spatiotemporal features for fast content-based retrieval of focal liver lesions. IEEE Trans Biomed Eng. 2014;61(11):2768–78.CrossRefPubMed
44.
Spanier AB, Caplan N, Sosna J, Acar B, Joskowicz L. A fully automatic end-to-end method for content-based image retrieval of CT scans with similar liver lesion annotation. Int J CARS. 2018;13:165–74.CrossRef
45.
Yang W, Feng Q, Yu M, Lu Z, Gao Y. Content-based retrieval of brain tumor in contrast-enhanced MRI images using tumor margin information and learned distance metric. Med Phys. 2012;39(11):6929–42.CrossRefPubMed
46.
Faria AV, Oishi K, Yoshida S, Hillis A, Miller ML, Mori S. Content-based image retrieval for brain MRI: An image-searching engine and population-based analysis to utilize past clinical data for future diagnosis. NeuroImage Clin. 2015;7:367–76.CrossRefPubMedPubMedCentral
47.
Zaki WMDW., Fauzi MFA, Besar R. Retrieval of intracranial hemorrhages in computed tomography brain images using binary coherent vector. J Electron Imaging. 2010;19(4):043021.CrossRef
48.
Chaum E, Karnowski TP, Covindasamy VP, Abdelrahman M, Tobin KW. Automated diagnosis of retinopathy by content-based image retrieval. Retina. 2008;28:1463–77.CrossRefPubMed
49.
Quellec G, Lamard M, Cazuguel G, Roux C, Cochener B. Case retrieval in medical databases by fusing heterogeneous information. IEEE Trans Med Imaging. 2011;30(1):108–18.CrossRefPubMed
50.
Kim J, Cai W, Feng D, Wu H. A new way for multidimensional medical data management: volume of interest (VOI)-based retrieval of medical images with visual and functional features. IEEE Trans Inf Technol Biomed. 2006;10(3):598–607.CrossRefPubMed
51.
Zheng X, Liu W, Dundar M, Badve S, Zhang S. Towards large-scale histopathological image analysis: Hashing-based image retrieval. IEEE Trans Med Imaging. 2015;34(2):496–506.CrossRef
52.
Zheng Y, Jiang Z, Zhang H, Xie F, Ma Y, Shi H, Zhao Y. Histopathological whole slide image analysis using context-based CBIR. IEEE Trans Med Imaging 2018 (in press).
53.
Caicedo JC, Gonzalez FA, Romero E. Content-based histopathology image retrieval using a kernel-based semantic annotation framework. J Biomed Inf. 2011;44:519–28.CrossRef
54.
Baldi A, Murace R, Dragonetti E, Manganaro M, Guerra O, Bizzi S, Galli L. Definition of an automated content-based image retrieval (CBIR) system for the comparison f dermoscopic images of pigmented skin lesions. BioMed Eng Online. 2009;8:18.CrossRefPubMedPubMedCentral
55.
Tafresh MK, Linard N, Andre B, Ayache N, Vercauteren T. Semi-automated query construction for content-based endomicroscopy video retrieval. In: Golland P, Hata N, Barillot C, Hornegger J, Howe R, editors. Medical image computing and computer-assisted intervention—MICCAI 2014 LNCS 8673, pp 89–96.
56.
Nishikawa RM, Yang Y, Huo D, Wernick M, Sennett CA, Papioannou J, Wei L. Observers’ ability to judge the similarity of clustered calcifications on mammograms. Proc SPIE Med Imaging. 2004;5372:192–8.CrossRef
57.
Wang J, Jing H, Wernick MN, Nishikawa RM, Yang Y. Analysis of perceived similarity between pairs of microcalcification clustered in mammograms. Med Phys. 2014;41(5):051904.CrossRefPubMedPubMedCentral
58.
Muramatsu C, Li Q, Schmidt R, Suzuki K, Shiraishi J, Newstead G, Doi K. Experimental determination of subjective similarity for pairs of clustered microcalcifications on mammograms: observer study results. Med Phys. 2006;33(9):3460–8.CrossRefPubMed
59.
Muramatsu C, Li Q, Schmidt RA, Shiraishi J, Suzuki K, Newstead GM, Doi K. Determination of subjective similarity for pairs of masses and pairs of clustered microcalcifications on mammograms: Comparison of similarity, ranking scores and absolute similarity ratings. Med Phys. 2007;34(7):2890–5.CrossRefPubMed
60.
Kumazawa S, Muramatsu C, Li Q, Li F, Shiraishi J, Caligiuri P, Schmidt RA, MacMahon H, Doi K. An investigation of radiologists’ perception of lesion similarity: observations with paired breast masses on mammograms and paired lung nodules on CT images. Acad Radiol. 2008;15:887–94.CrossRefPubMed
61.
Tourassi G, Yoon HJ, Xu S, Morin-Ducote G, Hudson K. Comparative analysis of data collection methods for individualized modeling of radiologists’ visual similarity judgments in mammograms. Acad Radiol. 2013;20:1371–80.CrossRefPubMed
62.
Faruque J, Rubin DL, Beaulieu CF, Napel S. Modeling perceptual similarity measures in CT images of focal liver lesions. J Digit Imaging. 2013;26:714–20.CrossRefPubMed
63.
Muramatsu C, Li Q, Schmidt RA, Shiraishi J, Doi K. Determination of similarity measures for pairs of mass lesion on mammograms by use of BI-RADS lesion descriptors and image features. Acad Radiol. 2009;16:443–9.CrossRefPubMed
64.
Nakayama R, Abe H, Shiraishi J. doi K. Evaluation of objective similarity measures for selecting similar images of mammographic lesions. J Digit Imaging. 2011;24(1):75–85.CrossRefPubMed
65.
66.
Muramatsu C, Nishimura K, Endo T, Oiwa M, Shiraiwa M, Doi K, Fujita H. Representation of lesion similarity by use of multidimensional scaling for breast masses on mammograms. J Digit Imaging. 2013;26:740–7.CrossRefPubMedPubMedCentral
67.
Nishimura K, Muramatsu C, Oiwa M, Shiraiwa M, Endo T, Doi K, Fujita H. Psychophysical similarity measure based on multi-dimensional scaling for retrieval of similar images of breast masses on mammograms. Proc SPIE Med Imaging. 2013;8670:86701R.CrossRef
68.
Muramatsu C, Takahashi T, Morita T, Endo T, Fujita H. Similar image retrieval of breast masses on ultrasonography using subjective data and multidimensional scaling. In: Tingberg A et al., editors. Proceedings of international workshop on breast imaging, IWDM 2016. Lecture notes in computer science, vol 9699. 2016; pp 43–50.
69.
Oh JH, Yang Y, El-Naqa I. Adaptive learning for relevance feedback: application to digital mammography. Med Phys 201;37(8):4432–44.
70.
Wei CH, Li Y, Huang PJ. Mammogram retrieval through machine learning within BI-RADS standard. J Biomed Inform. 2011;44:607–14.CrossRefPubMed
71.
Cho HC, Hadjiiski L, Sahiner B, Chan HP, Paramagul C, Helvie M, Nees AV, Cho HC. A similarity study of interactive content-based image retrieval scheme for classification of breast lesions. IEICE Trans Inf Syst. 2016; E99-D:1663–1670.CrossRef
72.
Liu X, Tizhoosh HR, Kofman J. Generating binary tags for fast medical image retrieval based on convolutional nets and Radon transform. In: Proceedings of the International Joint Conference on Neural Networks 2016.
73.
Anavi Y, Kogan I, Gelbart E, Geva O, Greenspan H. Visualizing and enhancing a deep learning framework using patients age and gender for chest X-ray image retrieval. Proc SPIE Med Imaging. 2016;9785:978510.CrossRef
74.
Krizhevsky A, Sutskever I, Hinton G. Imagenet classification with deep convolutional neural networks. In: Proc advances in neural information processing systems 2012; pp 1097–105.
75.
Qayyam A, Anwar SM, Awais M, Majid M. Medical image retrieval using deep convolutional neural network. Neurocomputing. 2017;266:8–20.CrossRef
76.
Khatami A, Babaie M, Tizhoosh HR, Khosravi A, Nguyen T, Nahavandi S. A wequential serach-space shrinking using CNN transfer learning and a radon projection pool for medical image retrieval. Expert Syst Appl. 2018;100:224–33.CrossRef
77.
Pang S, Orgun MA, Yu Z. A novel biomedical image indexing and retrieval system via deep preference learning. Comput Methods Programs Biomed. 2018;158:53–69.CrossRefPubMed
78.
Muramatsu C, Higuchi S, Morita T, Oiwa M, Fujita H. Similarity estimation for reference image retrieval in mammograms using convolutional neural network. Proc SPIE Med Imaging. 2018;10575:105752U.
79.
Muramatsu C, Higuchi S, Morita T, Oiwa M, Kawasaki T, Fujita H. Retrieval of reference images of breast masses on mammograms by similarity space modeling. In: Proceedings of IWBI LNCS 2018. (in press).
80.
81.
Oosawa A, Hisanaga R, Inoue T, Hoshino T, Shimura K. Development and commercialization of “SYNAPSE Case Match” content-based image retrieval system for effectively supporting the interpretation of physician. Med Imag Tech. 2014;32:26–31. (in Japanese).
82.
Kiyono M. Development of Similar case retrieval system by AI. Innervision 2017:32:46–49. (in Japanese).
83.
Korenblum D, Rubin D, Napel S, Rodriguez C, Beaulieu C. Managing biomedical image metadata for search and retrieval of similar images. J Digit Imaging. 2011;24(4):739–48.CrossRefPubMed
84.
Takahashi T, Muramatsu C, Hiramatsu Y, Morita T, Hara T, Endo T, Fujita H. Similar image search of breast masses by combination of mammograms and ultrasound images—study of psychophysical similarity measure based on multi-dimensional scaling. IEICE Technical Report 2016; MI2015-107:161–164. (in Japanese).
85.
Park SC, Sukthankar R, Mummert L, Satyanarayanan M, Zheng B. Optimization of reference library used in content-based medical image retrieval scheme. Med Phys. 2007;34(11):4331–9.CrossRefPubMedPubMedCentral
Metadata
Title
Overview on subjective similarity of images for content-based medical image retrieval
Author
Chisako Muramatsu
Publication date
01-06-2018
Publisher
Springer Singapore
Published in
Radiological Physics and Technology / Issue 2/2018
Print ISSN: 1865-0333
Electronic ISSN: 1865-0341
DOI
https://doi.org/10.1007/s12194-018-0461-6

Other articles of this Issue 2/2018

Radiological Physics and Technology 2/2018 Go to the issue

OriginalPaper

Body size and tube voltage-dependent guiding equations for optimal selection of image acquisition parameters in clinical X-ray imaging

OriginalPaper

Quantitative T1 and T2* carotid atherosclerotic plaque imaging using a three-dimensional multi-echo phase-sensitive inversion recovery sequence: a feasibility study

OriginalPaper

Susceptibility difference weighted imaging in vertical-field MRI

OriginalPaper

Effective-dose estimation in interventional radiological procedures

Correction

Correction to: Estimation of linear energy transfer distribution for broad-beam carbon-ion radiotherapy at the National Institute of Radiological Sciences, Japan

OriginalPaper

Evaluation of surface dose and image quality using the half-scan mode in chest computed tomography-guided interventional radiology: a phantom study