https://orcid.org/0000-0002-9141-772X
Figures
Abstract
Background
Forensic dentistry identifies deceased individuals by comparing postmortem dental charts, oral-cavity pictures and dental X-ray images with antemortem records. However, conventional forensic dentistry methods are time-consuming and thus unable to rapidly identify large numbers of victims following a large-scale disaster.
Objective
Our goal is to automate the dental filing process by using intraoral scanner images. In this study, we generated and evaluated an artificial intelligence-based algorithm that classified images of individual molar teeth into three categories: (1) full metallic crown (FMC); (2) partial metallic restoration (In); or (3) sound tooth, carious tooth or non-metallic restoration (CNMR).
Methods
A pre-trained model was created using oral-cavity pictures from patients. Then, the algorithm was generated through transfer learning and training with images acquired from cadavers by intraoral scanning. Cross-validation was performed to reduce bias. The ability of the model to classify molar teeth into the three categories (FMC, In or CNMR) was evaluated using four criteria: precision, recall, F-measure and overall accuracy.
Results
The average value (variance) was 0.952 (0.000140) for recall, 0.957 (0.0000614) for precision, 0.952 (0.000145) for F-measure, and 0.952 (0.000142) for overall accuracy when the algorithm was used to classify images of molar teeth acquired from cadavers by intraoral scanning.
Conclusion
We have created an artificial intelligence-based algorithm that analyzes images acquired with an intraoral scanner and classifies molar teeth into one of three types (FMC, In or CNMR) based on the presence/absence of metallic restorations. Furthermore, the accuracy of the algorithm reached about 95%. This algorithm was constructed as a first step toward the development of an automated system that generates dental charts from images acquired by an intraoral scanner. The availability of such a system would greatly increase the efficiency of personal identification in the event of a major disaster.
Citation: Eto N, Yamazoe J, Tsuji A, Wada N, Ikeda N (2022) Development of an artificial intelligence-based algorithm to classify images acquired with an intraoral scanner of individual molar teeth into three categories. PLoS ONE 17(1): e0261870. https://doi.org/10.1371/journal.pone.0261870
Editor: Francesco Bianconi, Universita degli Studi di Perugia, ITALY
Received: September 24, 2021; Accepted: December 10, 2021; Published: January 7, 2022
Copyright: © 2022 Eto et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the paper.
Funding: The authors received no specific funding for this work.
Competing interests: The authors have declared that no competing interests exist.
1 Introduction
Dental evidence is widely used for personal identification because the teeth exhibit age-related changes, have features that are unique to an individual, and resist decomposition after death [1, 2]. Forensic dentistry identifies deceased individuals by collecting postmortem data, such as dental charts, oral-cavity pictures and dental X-ray images, and comparing them with antemortem records [3]. A notable disadvantage of this approach is that it is a slow process due to the time needed to perform the postmortem examinations, manually interpret the findings and compare them with antemortem records. Hence, the use of conventional techniques can lead to delays in victim identification following major disasters such as earthquakes that claim many lives. The development of an automated system that produces dental charts from digitized images of the teeth would greatly speed up the process of victim identification after a large-scale disaster.
Previous studies aimed at improving personal identification methods have evaluated superposition of reconstructed images of the palatal rugae [4], superposition of computed tomography (CT) images of the skull [5, 6], and superposition of dental X-ray images [7]. However, obtaining X-ray or CT images from a huge number of victims is not only time-consuming but also difficult to achieve at the site of a disaster because it requires radiation-generating equipment that is not easily portable. Although oral-cavity pictures potentially could be used as part of an automated system for victim identification, it is often challenging to obtain postmortem pictures with a dental camera due to the presence of rigor mortis that restricts mouth opening. By contrast, real-time imaging of the teeth can be performed easily and rapidly with an intraoral scanner even when there is some restriction of mouth opening. Furthermore, an intraoral scanner is a handheld device that is highly portable, making it well suited for use after a large-scale disaster.
The development of a system that automatically creates dental charts from images acquired with an intraoral scanner would greatly facilitate the rapid creation of postmortem dental charts for personal identification in the event of a major disaster. In this study, as a component of automated dental chart filling system, we generated and evaluated an artificial intelligence (AI)-based algorithm that analyzes images of molar teeth acquired with an intraoral scanner and classifies each tooth as one of three types: (1) full metallic crown (FMC); (2) partial metallic restoration (In); or (3) sound tooth, carious tooth or non-metallic restoration (CNMR).
2 Materials and methods
2.1 Ethics
In this study, existing information was anonymized and used unidentified. The anonymized correspondence table was stored in a separate file in a separate location. We do not give individual written or oral informed consent because we are using existing information. Therefore, we have disclosed the information of this research on our website and provided contact information so that patients or their families can decline if they do not wish to be eligible. This study was approved by the Kyushu University Certified Institutional Review Board for Clinical Trials (reference no. 2020–499). All experiments were conducted in accordance with approved guidelines.
2.2 Intraoral scanning
Images were obtained using a TRIOS third-generation intraoral scanner (3Shape, Copenhagen, Denmark). The TRIOS scanner does not require opacification (powder-free) and relies on the principle of confocal microscopy to acquire a sequence of video images using structured illumination from a light-emitting diode. The imaging system produces a colored visualization of the scanned structures. The scanner’s acquisition software generates proprietary files (DICOM, Digital Imaging and Communications in Medicine format) that can be exported to an open file format (STL, Standard Tessellation Language).
Since the overarching objective of our research is to develop a system for use in disaster victim identification, this study was conducted on cadavers. The oral-cavities of 34 cadavers to be dissected were imaged with an intraoral scanner at the Department of Forensic Pathology and Sciences, Graduate School of Medical Sciences, Kyushu University, Fukuoka, Japan. First, the oral-cavity of the cadaver was cleaned to remove any contaminants (including body fluids) that might hinder the collection of accurate findings. Then, the head of the scanner was inserted into the oral-cavity between the upper and lower teeth and slowly moved along the teeth to acquire the images. The minimal opening required to allow the scanner head to be inserted into the oral-cavity was around 2 cm (the thickness of the scanner head). The three-dimensional images of the upper and lower teeth were saved as STL files, and occlusal views of the teeth were generated with a snipping tool and saved as Portable Network Graphic (PNG) files. ImageJ (National Institutes of Health, Bethesda, MD, USA) was used to generate individual images of the upper and lower molars (256×256 pixels) from the occlusal views of the teeth (examples are shown in Fig 1).
A. Occlusal view generated by intraoral scanning. Images of individual molar teeth were extracted from the occlusal view. B. full metallic crown (FMC). C. partial metallic restoration (In). D. sound tooth, carious tooth or non-metallic restoration (CNMR).
2.3 Classification of the extracted images
The individual molar teeth extracted from the scanned images were classified as FMC (full metallic crown), In (partial metallic restoration) or CNMR (sound tooth, carious tooth or non-metallic restoration). Images that were unclear on visual inspection were excluded from the analysis.
2.4 Deep learning machine
The deep learning machine used in this study was comprised of a Core i9-7920X central processing unit (2.90 GHz, 12 cores, 24 threads; Intel, Santa Clara, CA, USA), TITAN V graphics processing unit (32 GB, 5120 cores, 4096-bit bus width; Nvidia, Santa Clara, CA, USA) and Ubuntu 16.04.5 LTS software (Canonical, London, UK).
2.5 Network architecture
The convolutional neural network (CNN) was based on the LeNet architecture proposed by LeCun et al. [8]. The LeNet architecture consisted of two convolution layers, two max-pooling layers and two affine layers (Fig 2). When color molar images were provided as the input to the CNN, the output of the network classified the tooth condition as one of three types (FMC, In or CNMR). The following hyperparameters were used in this study: training epochs: 30; base learning rate: 0.001; and training / validation rate: 80% / 20%.
The LeNet architecture consisted of two convolution layers, two max-pooling layers and two affine layers.
2.6 Generation of a pre-trained model
Deep learning in general is considered to require a large number of training samples [9]. Since there was a limitation to the number of cadavers available for this research, the present study utilized the transfer learning technique, which is a method of transferring knowledge across domains to overcome problems associated with small training datasets [10]. Therefore, we carried out a pre-training process using oral-cavity pictures obtained from patients who had attended the dentistry section of Kyushu University Hospital. Training data (i.e., occlusal views of individual molar teeth) were extracted from the oral-cavity pictures using the method described in section 2.2 (Intraoral scanning) (Fig 3), and each tooth was classified as FMC, In or CNMR as described above. Images that were unclear on visual inspection were excluded. 300 images of each category were randomly divided into 10 folders of 30 images each, and training and test runs of the CNN were performed a total of 10 times for cross-validation.
A. full metallic crown (FMC). B. partial metallic restoration (In). C. sound tooth, carious tooth or non-metallic restoration (CNMR).
2.7 Generation of a classification model
The classification model was generated by applying transfer learning to the pre-trained model using images obtained from the cadavers by the intraoral scanner. 60 images of each tooth category (FMC, In or CNMR) were randomly divided into 4 folders of 15 images, and a total of 4 training and test runs of the CNN were performed for cross-validation.
2.8 Two-step cascade method for classification of tooth condition
An additional trained model was generated using a two-step cascade method to classify tooth condition. Images obtained from cadavers by intraoral scanner were used. In the first step, the tooth was classified as ‘presence or absence of metallic restoration’. In the second step, ‘metallic restoration’ was sub-classified as ‘full or partial’ (Fig 4).
FMC: full metallic crown; In: partial metallic restoration; CNMR: sound tooth, carious tooth or non-metallic restoration.
2.9 Evaluation of the models
The ability of each model to classify tooth condition was evaluated using four criteria: precision, recall, F-measure and overall accuracy. Precision was calculated as the fraction of correct predictions for a certain class. Recall was calculated as the fraction of instances of a class that were correctly predicted. F-measure was defined as the harmonic mean of precision and recall: F-measure = (2 × precision × recall) / (recall + precision). Overall accuracy was calculated as the fraction of instances that were correctly classified.
3 Results
3.1 Evaluation of the pre-trained model
Table 1 shows the recall, precision, F-measure and overall accuracy values for each of 10 cross-validated tests using oral-cavity pictures obtained from patients. The average value (variance) was 0.937 (0.00128) for recall, 0.941 (0.00108) for precision, 0.937 (0.00129) for F-measure, and 0.939 (0.00107) for overall accuracy (values given to 3 significant figures).
3.2 Evaluation of the classification model
The classification algorithm was generated through transfer learning using images obtained from cadavers by intraoral scanning. Table 2 shows the recall, precision, F-measure and overall accuracy values for each of 4 cross-validated tests. The average value (variance) was 0.952 (0.000140) for recall, 0.957 (0.0000614) for precision, 0.952 (0.000145) for F-measure, and 0.952 (0.000142) for overall accuracy.
3.3 Evaluation of the two-step cascade classification model
Table 3 (step 1 of the two-step cascade classification model) and Table 4 (step 2 of the two-step cascade classification model) present the recall, precision, F-measure and overall accuracy values for each of 4 cross-validated tests using images acquired from cadavers with the intraoral scanner. Step 1 of the two-step cascade classification model (i.e., presence or absence of a metallic restoration) achieved particularly high values (greater than 0.987) for recall, precision, F-measure and overall accuracy.
4 Discussion
Dental evidence has been used for personal identification during recent natural disasters. However, many dentists have little or no experience of collecting dental information from cadavers and experience psychological distress when faced with this task [9]. The above factors can lead to errors in judgment as well as limitations in manpower availability for personal identification of victims after a large-scale disaster. Although previous reports have described the automatic interpretation of dental findings and the collation of ante/postmortem images [11, 12], these studies were based on X-rays or CT scans of the cadavers, which are time-consuming to perform and require specialized equipment that generates radiation and lacks portability. As a method that does not use them, in the present study, images of individual molar teeth were acquired with an intraoral scanner, and an algorithm was created to categorize the tooth in each image as FMC, In or CNMR. Notably, the algorithm was able to classify molar teeth with an accuracy of around 95%.
The intraoral scanner utilized in this study has several advantageous features that make it well suited to the acquisition of dental evidence from cadavers. First, the captured images are displayed in real time on a dedicated application. Second, the images can be saved as digital data for later analysis. Third, the scanner used in this study comes with RealColor™ Technology that allows it to accurately reproduce color tones [13]. Fourth, the TRIOS scanner does not require opacification (powder-free) because it is equipped with technology that controls for the reflection of light by metals and other substances; hence, this scanner is easier to use than early-generation scanners that require the application of powder for opacification. Finally, the intraoral scanner is a highly portable, handheld device with a small scanning head that can be inserted through an opening of only 2 cm. Thus, unlike taking oral-cavity pictures with a dental camera, intraoral scanning can be performed even when mouth opening is somewhat restricted by rigor mortis or other factors. Based on the above features, we believe that the use of an intraoral scanner would improve the accuracy and speed of dental evaluation in the event of a large-scale disaster.
In this study, the number of images we were able to obtain by intraoral scanning of cadavers was limited. Possible methods to overcome this limitation include data augmentation, increasing the size of the training sample [9] and transfer learning [14]. We performed transfer learning in this study because of the availability of a large number of oral-cavity pictures that were similar to the images acquired with the intraoral scanner. First, the CNN was pre-trained using oral-cavity pictures, which were relatively easy to collect. Then, using the parameter of pre-trained model as an initial parameter of classification model, we generated the classification model using images obtained by intraoral scanning of cadavers. This approach allowed us to construct a highly accurate CNN using only a small number of intraoral scans.
The use of dental evidence for the personal identification of disaster victims requires that the dental findings are collected accurately [15]. Since teeth with composite resin restorations can sometimes be mistaken for non-restored teeth even when observed with the naked eye, the present study limited the classification of molar tooth condition to only three categories (i.e., FMC, In and CNMR) that exhibit obvious differences in color. The accuracy of the algorithm was around 95% when these three categories were used and reached about 98% when the classification was based only on the presence or absence of a metal-colored restoration (Table 3), which suggests that the accuracy of the algorithm can be improved by simplifying the conditions. However, in order to be utilized for personal identification purposes, the algorithm will need to be improved so that it can recognize non-metallic restorations and thereby provide a more complex classification. We envisage that additional research will allow the algorithm to be further developed so that it not only recognizes a wider range of tooth features (including both metallic and non-metallic restorations) but also identifies individual teeth in an occlusal view. The creation of such an algorithm would allow dental charts to be automatically generated from occlusal views obtained by intraoral scanning.
In conclusion, we have developed an AI-based algorithm that can analyze images acquired with an intraoral scanner and classify molar teeth into one of three types (FMC, In or CNMR) based on the presence/absence of metallic restorations. Furthermore, the accuracy of the algorithm reached about 95%. This algorithm was created as a first step toward the construction of a system that can automatically generate a dental chart from images obtained with an intraoral scanner. The development of such an automated system would greatly improve the efficiency of personal identification in the event of a major disaster.
Acknowledgments
The authors would like to thank Dr Kenichi Morooka, Professor, Graduate School of Natural Science and Technology, Okayama University for helpful discussion and comments on the manuscript. We are also grateful to Dr Yongsu Yoon, Assistant Professor, Department of Radiological Science, Dongseo University for help with the image editing and deep learning methods. We thank OXMEDCOMMS (www.oxmedcomms.com) for writing assistance.
References
- 1. Utsuno H. Victim identification in large-scale disasters using dental findings. IATSS Res. 2019;43:90–96.
- 2. Sable G, Rindhe D. A review of dental biometrics from teeth feature extraction and matching techniques. Int J Sci Res. 2014;3:2720–2722.
- 3. Katsumura S, Sato K, Ikawa T, Yamaura K, Ando E, Shigeta Y, et al. “High-precision, reconstructed 3D model” of skull scanned by conebeam CT: Reproducibility verified using CAD/CAM data. Leg Med. 2016;18:37–43. pmid:26832374
- 4. Gibelli D, Angelis DD, Pucciarelli V, Riboli F, Ferrario VF, Dolci C, et al. Application of 3D models of palatal rugae to personal identification: hints at identification from 3D-3D superimposition techniques. Int J Leg Med. 2018;132:1241–1245. pmid:29159511
- 5. Ishii M, Yayama K, Motani H, Sakuma A, Yasjima D, Hayakawa M, et al. Application of superimposition-based personal identification using skull computed tomography images. J Forensic Sci. 2011;56:960–966. pmid:21595689
- 6. dos Santos Rocha S, de Paula Ramos DL, de Gusmao Paraiso Cavalcanti M. Applicability of 3D-CT facial reconstruction for forensic individual identification. Pesqui Odontl Bras. 2003;17:24–28. pmid:12908055
- 7. Jain AK, Chen H. Matching of dental X-ray images for human identification. Pattern Recognit. 2015;37:1519–1532.
- 8.
LeCun Y, Haffner P, Bottou L, Bengio Y. Object recognition with gradient-based learning. In: Forsyth DA, Mundy JL, Gesu V, Cipolla R, editors. Shape, Contour and Grouping in Computer Vision, Lecture Notes in Computer Science vol 1681. Berlin, Heidelberg: Springer; 1999. pp. 319–345.
- 9. Miki Y, Muramatsu C, Hayashi T, Zhou X, Hara T, Katsumata A, et al. Classification of teeth in cone-beam CT using deep convolutional neural network. Comput Biol Med 2017;80: 24–29. pmid:27889430
- 10. Zhuang F, Qi Z, Duan K, Xi D, Zhu Y, Zhu H, et al. A comprehensive survey on transfer learning. Proc IEEE. 2021;109:43–76.
- 11. Zhou J, Abdel-Mottaleb M. A content-based system for human identification based on bitewing dental X-ray images. Pattern Recognit. 2005;38:2132–2142.
- 12. Trochesset DA, Serchuk RB, Colosi DC. Generation of intra-oral-like images from cone beam computed tomography volumes for dental forensic image comparison. J Forensic Sci. 2015;59:510–513. pmid:24328928
- 13. Ahmed KE, Wang T, Li KY, Luk WK, Burrow MF. Performance and perception of dental students using three intraoral CAD/CAM scanners for full-arch scanning. J Prosthodont Res. 2019;63:167–172. pmid:30528397
- 14. Hu W, Qian Y, Soong FK, Wang Y. Improved mispronunciation detection with deep neural network trained acoustic models and transfer learning based logistic regression classifiers. Speech Commun. 2015;67:154–166.
- 15. Aoki T, Ito K, Aoyama S. Disaster victim identification and ICT. IEICE Fundam Rev. 2015;9:119–130.