Top

BMC Medical Informatics and Decision Making

Published in:

Open Access 01-12-2020 | Tremor | Research article

Evaluation of standard and semantically-augmented distance metrics for neurology patients

Authors: Daniel B. Hier, Jonathan Kopel, Steven U. Brint, Donald C. Wunsch II, Gayla R. Olbricht, Sima Azizi, Blaine Allen

Published in: BMC Medical Informatics and Decision Making | Issue 1/2020

Abstract

Background

Patient distances can be calculated based on signs and symptoms derived from an ontological hierarchy. There is controversy as to whether patient distance metrics that consider the semantic similarity between concepts can outperform standard patient distance metrics that are agnostic to concept similarity. The choice of distance metric can dominate the performance of classification or clustering algorithms. Our objective was to determine if semantically augmented distance metrics would outperform standard metrics on machine learning tasks.

Methods

We converted the neurological findings from 382 published neurology cases into sets of concepts with corresponding machine-readable codes. We calculated patient distances by four different metrics (cosine distance, a semantically augmented cosine distance, Jaccard distance, and a semantically augmented bipartite distance). Semantic augmentation for two of the metrics depended on concept similarities from a hierarchical neuro-ontology. For machine learning algorithms, we used the patient diagnosis as the ground truth label and patient findings as machine learning features. We assessed classification accuracy for four classifiers and cluster quality for two clustering algorithms for each of the distance metrics.

Results

Inter-patient distances were smaller when the distance metric was semantically augmented. Classification accuracy and cluster quality were not significantly different by distance metric.

Conclusion

Although semantic augmentation reduced inter-patient distances, we did not find improved classification accuracy or improved cluster quality with semantically augmented patient distance metrics when applied to a dataset of neurology patients. Further work is needed to assess the utility of semantically augmented patient distances.

Campbell WW. Diagnosis and localization of neurologic disease, Chapter 53. In Dejong's The neurologic examination. 7^th edition. Lippincott Williams and Wilkins, Philadelphia, 2013, pp. 769–795.

Beaulieu-Jones B, Finlayson SG, Chivers C, Chen I, McDermott M, Kandola J, Dalca AV. Trends and Focus of Machine Learning Applications for Health Research. 2019;2:1–12. https://doi.org/10.1001/jamanetworkopen.2019.14051.CrossRef

Parimbelli E, Marini S, Sacchi L, Bellazzi R. Patient similarity for precision medicine: a systematic review. J Biomed Inform. 2018;83:87–96. https://doi.org/10.1016/j.jbi.2018.06.001.CrossRefPubMed

Xue H, Peng J, Shang X. Predicting disease-related phenotypes using an integrated phenotype similarity measurement based on HPO. BMC Syst Biol. 2019;13:1–12. https://doi.org/10.1186/s12918-019-0697-8.CrossRef

Peng J, Xue H, Shao Y, Shang X, Wang Y, J. Chen J. Measuring phenotype semantic similarity using Human Phenotype Ontology, Proc. 2016 IEEE Int. Conf. Bioinforma. Biomed. BIBM 2016. (2017) 763–766. doi:https://doi.org/10.1109/BIBM.2016.7822617.

Pai S, Bader GD. Patient similarity networks for precision medicine. J Mol Biol. 2018;430:2924–38. https://doi.org/10.1016/j.jmb.2018.05.037.CrossRefPubMedPubMedCentral

Yang S, Stansbury LG, Rock P, Scalea T, Hu PF. Linking big data and prediction strategies: tools, pitfalls, and lessons learned. Crit Care Med. 2019;47:840–8. https://doi.org/10.1097/CCM.0000000000003739.CrossRefPubMed

Raghupathi W, Raghupathi V. Big data analytics in healthcare: promise and potential. Heal Inf Sci Syst. 2014;2:1–10. https://doi.org/10.1186/2047-2501-2-3.CrossRef

Deng Y, Gao L, Wang B, Guo X. HPOSim: an r package for phenotypic similarity measure and enrichment analysis based on the human phenotype ontology. PLoS One. 2015;10:1–12. https://doi.org/10.1371/journal.pone.0115692.CrossRef

10.

Su S, Zhang L, Liu J. An effective method to measure disease similarity using gene and phenotype associations. Front Genet. 2019;10:1–8. https://doi.org/10.3389/fgene.2019.00466.CrossRef

11.

Alanazi HO, Abdullah AH, Qureshi KN. A critical review for developing accurate and dynamic predictive models using machine learning methods in medicine and health care. J Med Syst. 2017;41. https://doi.org/10.1007/s10916-017-0715-6.

12.

Köhler S, Schulz MH, Krawitz P, Bauer S, et al. Clinical diagnostics in human genetics with semantic similarity searches in ontologies. Am J Hum Genet. 2009;85:457–64. https://doi.org/10.1016/j.ajhg.2009.09.003.CrossRefPubMedPubMedCentral

13.

Girardi D, Wartner S, Halmerbauer G, Ehrenmüller M, Kosorus H, Dreiseitl S. Using concept hierarchies to improve calculation of patient similarity. J Biomed Inform. 2016;63:66–73. https://doi.org/10.1016/j.jbi.2016.07.021.CrossRefPubMed

14.

Jia Z, Lu X, Duan H, Li H. Using the distance between sets of hierarchical taxonomic clinical concepts to measure patient similarity. BMC Med. Inform. Decis. Mak. 2019;19:1–11. https://doi.org/10.1186/s12911-019-0807-y.CrossRef

15.

Sharafoddini A, Dubin JA, Lee J. Patient Similarity in Prediction Models Based on Health Data: A Scoping Review. JMIR Med Inform. (2017) 5(1):e7. Published 2017 Mar 3. doi:https://doi.org/10.2196/medinform.6730.

16.

Melton GB, Parsons S, Morrison FP, Rothschild AS, Markatou M, Hripcsak G. Inter-patient distance metrics using SNOMED CT defining relationships. J Biomed Inform. 2006;39:697–705. https://doi.org/10.1016/j.jbi.2006.01.004.CrossRefPubMed

17.

Boyack KW, Newman D, Duhon RJ, Klavans R, Patek M, Biberstine JR, Schijvenaars B, Skupin A, Ma N, Börner K. Clustering more than two million biomedical publications: Comparing the accuracies of nine text-based similarity approaches, PLoS One. 6 (2011). doi:https://doi.org/10.1371/journal.pone.0018029.

18.

L.J. Garcia Castro LJ, R. Berlanga R, A. Garcia A, In the pursuit of a semantic similarity metric based on UMLS annotations for articles in PubMed Central Open Access, J. Biomed. Inform. (2015) 57: 204–218. doi:https://doi.org/10.1016/j.jbi.2015.07.015.

19.

Mabotuwana T, Lee MC. Cohen, Solal EV. An ontology-based similarity measure for biomedical data-application to radiology reports. J Biomed Inform. 2013;46(5):857–68. https://doi.org/10.1016/j.jbi.2013.06.013.CrossRefPubMed

20.

Andrews JE, Richesson RL, Krischer J. Variation of SNOMED CT coding of clinical research concepts among coding experts. J Am Med Inform Assoc. (2007) Jul-Aug;14(4):497–506.

21.

Chiang MF, Hwang JC, Yu AC, Casper DS, Cimino JJ. Starren J. AMIA Annu Symp Proc: Reliability of SNOMED-CT Coding by Three Physicians using Two Terminology Browsers; 2006. p. 131–5.

22.

Bhattacharyya SB. Introduction to SNOMED CT. Singapore: Springer; 2016.CrossRef

23.

Bodenreider O. The Unified Medical Language System (UMLS): integrating biomedical terminology, Nucleic Acids Research. (2004) 32, issue suppl_1, Pages D267–D270, https://doi.org/10.1093/nar/gkh061.

24.

Hier DB, Brint SU. A Neuro-ontology for the neurological examination. BMC Med Inform Decis Mak. 2020;20:47. https://doi.org/10.1186/s12911-020-1066-7.CrossRefPubMedPubMedCentral

25.

Choi SS, Cha SH, Tappert CC. A survey of binary similarity and distance measures, WMSCI 2009 - 13th world multi-conference Syst. Cybern. Informatics, jointly with 15th Int. Conf. Inf. Syst. Anal. Synth. ISAS 2009 - Proc 3 (2009) 80–85.

26.

Tashkandi A, Wiese I, Wiese L. Efficient in-database patient similarity analysis for personalized medical decision support systems. Big Data Res. 2018;13:52–64. https://doi.org/10.1016/j.bdr.2018.05.001.CrossRef

27.

Haase P, Siebes R, van Harmelen F. Peer selection in peer-to-peer networks with semantic topologies. In: Bouzeghoub M., Goble C., Kashyap V., Spaccapietra S. (eds) semantics of a networked world. Semantics for grid databases. ICSNW 2004. Lecture notes in computer science. (2004) vol 3226. Springer, Berlin. Heidelberg. . https://doi.org/10.1007/978-3-540-30145-5_7.

28.

Rada R, Hafedh M, Bicknell E, Blettner M. Development and Application of a Metric on Semantic Nets. IEEE transactions on systems, Man and Cybernetics (1989) 19(1): 17–30.

29.

Wu Z, Palmer M. Verb semantics and lexical selection. In Proceedings of the 32nd Annual Meeting of the Associations for Computational Linguistics, (1994) pp 133–138.

30.

Leacock C, Chodorow M. Combining local context and WordNet similarity for word sense identification. WordNet. 1998. https://doi.org/10.7551/mitpress/7287.003.0018.

31.

Resnik P. Using information content to evaluate semantic similarity in a taxonomy. (1995) http://arxiv.org/abs/cmp-lg/9511007.

32.

Jiang JJ, Conrath DW. Semantic Similarity Based on Corpus Statistics and Lexical Taxonomy. In Proceedings of International Conference Research on Computational Linguistics (ROCLING X). (1997) Taiwan, pp 19–33, https://www.aclweb.org/anthology/O97-1002.

33.

Lin D. An Information-Theoretic Definition of Similarity, ICML 1998 Proceedings of the Fifteenth International Conference on Machine Learning. (1998) Pages 296-304, July 24–27, 1998.

34.

Lee W, Shah N, Sundlass K, Musen M. Comparison of Ontology-based Semantic-Similarity Measures. Medical College of Wisconsin, Milwaukee, WI, Symp. A Q. J. Mod. Foreign Lit. (2008) 384–388.

35.

McInnes BT, Pedersen T. Evaluating semantic similarity and relatedness over the semantic grouping of clinical term pairs. J Biomed Inform. 2015;54:329–36. https://doi.org/10.1016/j.jbi.2014.11.014.CrossRefPubMed

36.

Caviedes JE, Cimino JJ. Towards the development of a conceptual distance metric for the UMLS. J Biomed Inform. 2004;37:77–85. https://doi.org/10.1016/j.jbi.2004.02.001.CrossRefPubMed

37.

Al-Mubaid H, Nguyen HA, A cluster-based approach for semantic similarity in the biomedical domain, Annu. Int. Conf. IEEE Eng. Med. Biol. Proc. (2006) 2713–2717.

38.

Pedersen T, Pakhomov SVS, Patwardhan S, Chute CG. Measures of semantic similarity and relatedness in the biomedical domain. J Biomed Inform. 2007;40:288–99. https://doi.org/10.1016/j.jbi.2006.06.004.CrossRefPubMed

39.

The MathWorks Inc. What is machine learning?, Retrieved at https://www.mathworks.com/discovery/machine-learning.html.

40.

The Mathworks Inc. Supervised learning workflows and algorithms. Retrieved at https://www.mathworks.com/help/stats/supervised-learning-machine-learning-workflow-and-algorithms.html.

41.

The Mathworks Inc. Unsupervised learning. Retrieved at https://www.mathworks.com/discovery/unsupervised-learning.html.

42.

Al-Jabery KK, Obafemi-Ajayi T, Olbricht GR. Wunsch II DC (editors). Computational Learning Approaches to Data Analytics in Biomedical Applications: Academic Press; 2020. https://doi.org/10.1016/B978-0-12-814482-4.05001-4.CrossRef

43.

Rosenberg A, Hirschberg J. V-Measure: A conditional entropy-based external cluster evaluation measure, EMNLP-CoNLL 2007 - Proc. 2007 Jt. Conf. Empir. Methods Nat. Lang. Process. Comput. Nat. Lang. Learn. (2007) 410–420.

44.

Rand WW. Objective criteria for the evaluation of clustering methods. J Am Stat Assoc. 1971;66:846–50. https://doi.org/10.1080/01621459.1971.10482356.CrossRef

45.

Rousseeuw PJ. Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. J Comput Appl Math. 1987;20:53–65. https://doi.org/10.1016/0377-0427(87)90125-7.CrossRef

46.

Kellar SP, Kelvin EA. Munro's statistical methods for healthcare research. 6th ed. Philadelphia: Wolters Kluwer; 2013.

47.

Blumenfeld H. Neuroanatomy through clinical cases. 2nd ed. Sunderland, MA: Sinauer Associates; 2010.

48.

Macleod M. Simpson M, pal S. Neurology. Wiley-Blackwell, West Sussex UK: Clinical Cases Uncovered; 2011.

49.

Noseworthy JH. Fifty neurologic Cases from Mayo Clinic. Oxford UK: Oxford University Press; 2004.

50.

Pendlebury ST, Anslow P, Rothwell PM. Neurological case histories. Oxford UK: Oxford University Press; 2007.CrossRef

51.

Toy EC, Simpson E, Mancias P, Furr-Stimming EE. Case files neurology. 3rd ed. New York: McGraw-Hill; 2018.

52.

Waxman SG. Clinical Neuroanatomy. 28th ed. New York: McGraw Hill; 2017.

53.

Hauser SL, Levitt LP, Weiner HL. Case studies in neurology for the house officer. Baltimore: Williams and Wilkins; 1986.

54.

Liveson JA, Spielholz N. Peripheral neurology: case studies in electrodiagnosis. Philadelphia: FA Davis Company; 1979.

55.

Gauthier SG, Rosa-Netto P. Case studies in dementia. Cambridge UK: Cambridge University Press; 2011.CrossRef

56.

Erro R, Stamelou M, Bhatia K. Case studies in movement disorders. Cambridge UK: Cambridge University Press; 2017.

57.

Solomon T, Michael BD, Miller A, Kneen R. Case studies in neurological infections of adults and children. Cambridge UK: Cambridge University Press; 2019.CrossRef

58.

Howard J, Singh A. Neurology image-based clinical review. New York: Demos Publishing; 2017.

59.

Pedregosa F, Varoquaux G, Gramfort A, Michel V, et al. Scikit-learn: machine learning in Python. J Mach Learn Res. 2011;12:2825–30 http://jmlr.org/papers/v12/pedregosa11a.html.

60.

Jaccard P. The distribution of the flora in the alpine zone. New Phytol. 1912;11:37–50. https://doi.org/10.1111/j.1469-8137.1912.tb05611.x.CrossRef

61.

Jana N, Barik S, Arora N. Current use of medical eponyms--a need for global uniformity in scientific publications. BMC Med Res Methodol. (2009) 9:18. Published 2009 Mar 9. doi:https://doi.org/10.1186/1471-2288-9-18.

62.

Ward JH. Hierarchical grouping to optimize an objective function. J Am Stat Assoc. 1963;58:236–44. https://doi.org/10.1080/01621459.1963.10500845.CrossRef

63.

Xu R. Wunsch DC II. Clustering: Wiley-IEEE Press; 2008.

64.

Xu R, Wunsch DC II. Clustering algorithms in biomedical research: a review. IEEE Rev Biomed Eng. 2010;3:120–54.CrossRef

65.

Chimowitz MI, Logigian EL, Caplan LR. The accuracy of bedside neurological diagnoses. Ann Neurol. 1990;28:78–85. https://doi.org/10.1002/ana.410280114.CrossRefPubMed

66.

Liu Y, Chen PHC, Krause J, Peng L. How to read articles that use machine learning: Users' guides to the medical literature, JAMA - J. Am Med Assoc. 2019;322:1806–16. https://doi.org/10.1001/jama.2019.16489.CrossRef

67.

Aronson AR, Lang FM. An overview of MetaMap: historical perspective and recent advances. J. Am. Med. Informatics Assoc. 2010;17:229–36. https://doi.org/10.1136/jamia.2009.002733.CrossRef

68.

Savova GK, Masanz JJ, Ogren PV, Zheng J, Sohn S, Kipper-Schuler KC, Chute CG. Mayo clinical text analysis and knowledge extraction system (cTAKES): architecture, component evaluation and applications. J Am Med Informatics Assoc. 2010;17:507–13. https://doi.org/10.1136/jamia.2009.001560.CrossRef

69.

Kreimeyer K, Foster M, Pandey A, Arya N, Halford G, Jones SF, Forshee R, Walderhaug M, Botsis T. Natural language processing systems for capturing and standardizing unstructured clinical information: a systematic review. J Biomed Inform. 2017;73:14–29. https://doi.org/10.1016/j.jbi.2017.07.012.CrossRefPubMedPubMedCentral

70.

Reátegui R, Ratté S. Comparison of MetaMap and cTAKES for entity extraction in clinical notes. BMC Med Inform Decis Mak. 2018;18:74. https://doi.org/10.1186/s12911-018-0654-2.CrossRefPubMedPubMedCentral

Title: Evaluation of standard and semantically-augmented distance metrics for neurology patients
Authors: Daniel B. Hier
Jonathan Kopel
Steven U. Brint
Donald C. Wunsch II
Gayla R. Olbricht
Sima Azizi
Blaine Allen
Publication date: 01-12-2020
Publisher: BioMed Central
Keyword: Tremor
Published in: BMC Medical Informatics and Decision Making / Issue 1/2020
Electronic ISSN: 1472-6947
DOI: https://doi.org/10.1186/s12911-020-01217-8

At a glance: The STEP trials

Springer Medicine

Evaluation of standard and semantically-augmented distance metrics for neurology patients

Abstract

Background

Methods

Results

Conclusion

At a glance: The STEP trials

Springer Medicine

Abstract

Background

Methods

Results

Conclusion

Please log in to get access to this content

Other articles of this Issue 1/2020

The emerging COVID-19 research: dynamic and regularly updated science maps and analyses

The contribution of electronic health records to risk management through accreditation of residential aged care homes in Australia

Data cleaning process for HIV-indicator data extracted from DHIS2 national reporting system: a case study of Kenya

Avatar-based versus conventional vital sign display in a central monitor for monitoring multiple patients: a multicenter computer-based laboratory study

Readability of web-based sources about induced abortion: a cross-sectional study

Violence detection explanation via semantic roles embeddings