Skip to main content
Key Specialties
Cardiology Diabetology Endocrinology Gastroenterology Geriatrics and Gerontology Gynecology and Obstetrics Hematology Infectious Disease Internal Medicine Nephrology Neurology Oncology Pediatrics Respiratory Medicine Rheumatology Surgery
Congress Coverage
2024 ACC Congress 2023 ESMO Congress 2023 EASD Congress 2023 ERS Congress 2023 ESC Congress
Clinical Collections
Advances in Alzheimer’s

Keynote webinar | Spotlight on medication adherence

LIVE: Thursday 27th June 2024, 18:00-19:30 (CEST)
Join our expert panel to discover why you need to understand the drivers of non-adherence in your patients, and how you can optimize medication adherence in your clinics to drastically improve patient outcomes.
ADVANCED SEARCH
Log in
Top
BMC Medical Informatics and Decision Making
Published in: BMC Medical Informatics and Decision Making 1/2023

Open Access 01-12-2023 | Research

De-identified Bayesian personal identity matching for privacy-preserving record linkage despite errors: development and validation

Authors: Rudolf N. Cardinal, Anna Moore, Martin Burchell, Jonathan R. Lewis

Published in: BMC Medical Informatics and Decision Making | Issue 1/2023

Login to get access

Abstract

Background

Epidemiological research may require linkage of information from multiple organizations. This can bring two problems: (1) the information governance desirability of linkage without sharing direct identifiers, and (2) a requirement to link databases without a common person-unique identifier.

Methods

We develop a Bayesian matching technique to solve both. We provide an open-source software implementation capable of de-identified probabilistic matching despite discrepancies, via fuzzy representations and complete mismatches, plus de-identified deterministic matching if required. We validate the technique by testing linkage between multiple medical records systems in a UK National Health Service Trust, examining the effects of decision thresholds on linkage accuracy. We report demographic factors associated with correct linkage.

Results

The system supports dates of birth (DOBs), forenames, surnames, three-state gender, and UK postcodes. Fuzzy representations are supported for all except gender, and there is support for additional transformations, such as accent misrepresentation, variation for multi-part surnames, and name re-ordering. Calculated log odds predicted a proband’s presence in the sample database with an area under the receiver operating curve of 0.997–0.999 for non-self database comparisons. Log odds were converted to a decision via a consideration threshold θ and a leader advantage threshold δ. Defaults were chosen to penalize misidentification 20-fold versus linkage failure. By default, complete DOB mismatches were disallowed for computational efficiency. At these settings, for non-self database comparisons, the mean probability of a proband being correctly declared to be in the sample was 0.965 (range 0.931–0.994), and the misidentification rate was 0.00249 (range 0.00123–0.00429). Correct linkage was positively associated with male gender, Black or mixed ethnicity, and the presence of diagnostic codes for severe mental illnesses or other mental disorders, and negatively associated with birth year, unknown ethnicity, residential area deprivation, and presence of a pseudopostcode (e.g. indicating homelessness). Accuracy rates would be improved further if person-unique identifiers were also used, as supported by the software. Our two largest databases were linked in 44 min via an interpreted programming language.

Conclusions

Fully de-identified matching with high accuracy is feasible without a person-unique identifier and appropriate software is freely available.
Appendix
Available only for authorised users
Footnotes
1
Working: Prior probability P(H) = 1/100; prior odds 1/99. P(D | H) = 1; P(D | ¬H) = 1/99; LR = 99. Posterior odds 1; posterior P(H | D) = 0.5.
 
2
Working: Prior probability P(H) = 1/100; prior odds 1/99. P(D | H) = 1; P(D | ¬H) = 0.001 [incorrect]; LR = 1000 [incorrect]. Posterior odds 10.1 [incorrect]; posterior P(H | D) = 0.91 [incorrect].
 
3
Working: Ignoring the possibility of error: prior probability P(H) = 1/10; prior odds 1/9. P(D | H) = 1; P(D | ¬H) = 5/9 (5 other Alices among 9 other people); LR = 9/5. Posterior odds 1/5; posterior P(H | D) = 1/6.
 
4
Working: Again, without errors: prior probability P(H) = 1/10; prior odds 1/9. P(D | H) = 0; P(D | ¬H) = ?; LR = 0. Posterior odds 0; posterior P(H | D) = 0.
 
Literature
1.
2.
3.
Jones LA, Nelder JR, Fryer JM, Alsop PH, Geary MR, Prince M, et al. Public opinion on sharing data from health services for clinical and research purposes without explicit consent: an anonymous online survey in the UK. BMJ Open. 2022;12(4): e057579.PubMedPubMedCentralCrossRef
4.
Downs JM, Ford T, Stewart R, Epstein S, Shetty H, Little R, et al. An approach to linking education, social care and electronic health records for children and young people in South London: a linkage study of child and adolescent mental health service data. BMJ Open. 2019;9(1): e024355.PubMedPubMedCentralCrossRef
5.
Asher J, Resnick D, Brite J, Brackbill R, Cone J. An introduction to probabilistic record linkage with a focus on linkage processing for WTC registries. Int J Environ Res Public Health. 2020;17(18):6937.PubMedPubMedCentralCrossRef
6.
7.
Bayes T. An essay towards solving a problem in the doctrine of chances. Philosophical Transactions of the Royal Society of London. 1763;53:370–418.
8.
Newcombe HB, Kennedy JM, Axford SJ, James AP. Automatic linkage of vital records. Science. 1959;130(3381):954–9.PubMedCrossRef
9.
Fellegi IP, Sunter AB. A theory for record linkage. J Am Stat Assoc. 1969;64(328):1183–210.CrossRef
10.
Jaro MA. Advances in record-linkage methodology as applied to matching the 1985 census of Tampa, Florida. J Am Stat Assoc. 1989;84(406):414–20.CrossRef
11.
12.
Newcombe HB. Age-related bias in probabilistic death searches due to neglect of the ‘prior likelihoods’. Comput Biomed Res. 1995;28(2):87–99.PubMedCrossRef
13.
14.
DuVall SL, Kerber RA, Thomas A. Extending the Fellegi-Sunter probabilistic record linkage method for approximate field comparators. J Biomed Inform. 2010;43(1):24–30.PubMedCrossRef
15.
Xu H, Li X, Grannis S. A simple two-step procedure using the Fellegi-Sunter model for frequency-based record linkage. J Appl Stat. 2022;49(11):2789–804.PubMedCrossRef
16.
17.
Larsen MD. Multiple imputation analysis of records linked using mixture models. In: Proceedings of the Survey Methods Section. Statistical Society of Canada; 1999. p. 65–71.
18.
19.
Fortini M, Liseo B, Nuccitelli A, Scanu M. On Bayesian record linkage. Research in Official Statistics. 2001;4(1):185–98.
20.
Gutman R, Afendulis CC, Zaslavsky AM. A Bayesian procedure for file linking to analyze end-of-life medical costs. J Am Stat Assoc. 2013;108(501):34–47.PubMedPubMedCentralCrossRef
21.
Enamorado T, Fifield B, Imai K. Using a probabilistic model to assist merging of large-scale administrative records. American Political Science Review. 2019;113(2):353–71.CrossRef
22.
23.
24.
Lyons RA, Jones KH, John G, Brooks CJ, Verplancke JP, Ford DV, et al. The SAIL databank: linking multiple health and social care datasets. BMC Med Inform Decis Mak. 2009;16(9):3.CrossRef
25.
Levenshtein VI. Binary codes capable of correcting deletions, insertions, and reversals. Soviet Physics Doklady. 1966;1(10):707.
26.
Sadinle M. Bayesian estimation of bipartite matchings for record linkage. J Am Stat Assoc. 2017;112(518):600–12.CrossRef
27.
Quantin C, Binquet C, Bourquard K, Pattisina R, Gouyon-Cornet B, Ferdynus C, et al. Which are the best identifiers for record linkage? Med Inform Internet Med. 2004;29(3–4):221–7.PubMedCrossRef
28.
Goehring R. Identification of patients in medical data bases–soundex codes versus match code. Med Inform (Lond). 1985;10(1):27–34.PubMedCrossRef
29.
Brown AP, Randall SM, Boyd JH, Ferrante AM. Evaluation of approximate comparison methods on Bloom filters for probabilistic linkage. Int J Popul Data Sci. 2019;4(1):1095.PubMedPubMedCentral
30.
31.
Quantin C, Bouzelat H, Dusserre L. A computerized record hash coding and linkage procedure to warrant epidemiological follow-up data security. Stud Health Technol Inform. 1997;43Pt A:339–42.
32.
Bian J, Loiacono A, Sura A, Mendoza Viramontes T, Lipori G, Guo Y, et al. Implementing a hash-based privacy-preserving record linkage tool in the OneFlorida clinical research network. JAMIA Open. 2019;2(4):562–9.PubMedPubMedCentralCrossRef
33.
Bloom BH. Space/time trade-offs in hash coding with allowable errors. Commun ACM. 1970;13(7):422–6.CrossRef
34.
Randall SM, Ferrante AM, Boyd JH, Bauer JK, Semmens JB. Privacy-preserving record linkage on large real world datasets. J Biomed Inform. 2014;50:205–12.PubMedCrossRef
35.
Schnell R, Bachteler T, Reiher J. Privacy-preserving record linkage using Bloom filters. BMC Med Inform Decis Mak. 2009;25(9):41.CrossRef
36.
Ranbaduge T, Christen P, Schnell R. Secure and accurate two-step hash encoding for privacy-preserving record linkage. Advances in Knowledge Discovery and Data Mining. 2020;17(12085):139–51.CrossRef
37.
Christen P, Schnell R, Vatsalan D, Ranbaduge T. Efficient cryptanalysis of Bloom filters for privacy-preserving record linkage. In: Kim J, Shim K, Cao L, Lee JG, Lin X, Moon YS, editors. Advances in Knowledge Discovery and Data Mining. Cham: Springer International Publishing; 2017. p. 628–40. (Lecture Notes in Computer Science.)
38.
Blake HA, Sharples LD, Harron K, van der Meulen JH, Walker K. Probabilistic linkage without personal information successfully linked national clinical datasets. J Clin Epidemiol. 2021;1(136):136–45.CrossRef
39.
40.
Koneru K, Varol C. Privacy preserving record linkage using MetaSoundex algorithm. In: 2017 16th IEEE International Conference on Machine Learning and Applications (ICMLA). 2017. p. 443–7.
41.
Lazrig I, Ong T, Ray I, Ray I, Kahn M. Privacy preserving probabilistic record linkage using locality sensitive hashes. In: Ranise S, Swarup V, editors. Data and Applications Security and Privacy XXX. Cham: Springer International Publishing; 2016. p. 61–76. (Lecture Notes in Computer Science).
42.
43.
44.
MacKay DJC. Information Theory, Inference, and Learning Algorithms. Cambridge: Cambridge University Press; 2003.
45.
46.
47.
48.
49.
50.
51.
Smith D. Re-identification in the absence of common variables for matching. Int Stat Rev. 2020;88(2):354–79.CrossRef
52.
Gronau QF, Sarafoglou A, Matzke D, Ly A, Boehm U, Marsman M, et al. A tutorial on bridge sampling. Journal of Mathematical Psychology. 2017 Dec 1;81(Supplement C):80–97.
53.
Machado CJ, Hill K. Probabilistic record linkage and an automated procedure to minimize the undecided-matched pair problem. Cad Saude Publica. 2004;20(4):915–25.PubMedCrossRef
54.
Tancredi A, Liseo B. A hierarchical Bayesian approach to record linkage and population size problems. The Annals of Applied Statistics. 2011;5(2B):1553–85.CrossRef
55.
Larsen MD, Zhao Y. A study of factors affecting record linkage in federal statistical databases. In: FCSM Research Conference. Federal Committee on Statistical Methodology; 2012.
56.
57.
58.
59.
60.
61.
62.
63.
64.
65.
66.
67.
68.
69.
70.
71.
Bellare M, Canetti R, Krawcyk H. Keying hash functions for message authentication. Lecture Notes in Computer Sciences (Advances in Cryptology - Crypto 96 Proceedings). 1996;1109:1–15.
72.
Preneel B. The first 30 years of cryptographic hash functions and the NIST SHA-3 competition. In: Pieprzyk J, editor. Topics in Cryptology - CT-RSA 2010. Springer Berlin Heidelberg; 2010 [cited 2015 May 28]. p. 1–14. (Lecture Notes in Computer Science.) Available from: https://​doi.​org/​10.​1007/​978-3-642-11925-5_​1.
73.
Lee A, Atkison T. A comparison of fuzzy hashes: evaluation, guidelines, and future suggestions. In: ACM SE ’17: Proceedings of the SouthEast Conference. 2017.
74.
Kornblum J. Identifying almost identical files using context triggered piecewise hashing. Digit Investig. 2006;3:S91–7.CrossRef
75.
76.
77.
78.
79.
Robin X, Turck N, Hainard A, Tiberti N, Lisacek F, Sanchez JC, et al. pROC: an open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinformatics. 2011;12(1):77.PubMedPubMedCentralCrossRef
80.
81.
82.
83.
84.
85.
86.
87.
88.
Macmillan NA, Creelman CD. Detection Theory: A User’s Guide. 2nd ed. Mahwah, N.J.: Lawrence Erlbaum Associates; 2005.
89.
90.
91.
Duvall SL, Fraser AM, Kerber RA, Mineau GP, Thomas A. The impact of a growing minority population on identification of duplicate records in an enterprise data warehouse. Stud Health Technol Inform. 2010;160(Pt 2):1122–6.PubMed
92.
Elbogen EB, Lanier M, Wagner HR, Tsai J. Financial strain, mental illness, and homelessness: results from a national longitudinal study. Med Care. 2021;59(Suppl 2):S132–8.PubMedCrossRef
93.
Smartt C, Prince M, Frissa S, Eaton J, Fekadu A, Hanlon C. Homelessness and severe mental illness in low- and middle-income countries: scoping review. BJPsych Open. 2019;5(4): e57.PubMedPubMedCentralCrossRef
94.
Folsom DP, Hawthorne W, Lindamer L, Gilmer T, Bailey A, Golshan S, et al. Prevalence and risk factors for homelessness and utilization of mental health services among 10,340 patients with serious mental illness in a large public mental health system. Am J Psychiatry. 2005;162(2):370–6.PubMedCrossRef
95.
96.
Karr AF, Taylor MT, West SL, Setoguchi S, Kou TD, Gerhard T, et al. Comparing record linkage software programs and algorithms using real-world data. PLoS ONE. 2019;14(9): e0221459.PubMedPubMedCentralCrossRef
97.
Grannis SJ, Overhage JM, McDonald CJ. Analysis of identifier performance using a deterministic linkage algorithm. Proc AMIA Symp. 2002;305–9.
98.
Finney JM, Walker AS, Peto TEA, Wyllie DH. An efficient record linkage scheme using graphical analysis for identifier error detection. BMC Med Inform Decis Mak. 2011;11:7.PubMedPubMedCentralCrossRef
99.
Grannis SJ, Overhage JM, Hui S, McDonald CJ. Analysis of a probabilistic record linkage technique without human review. AMIA Annu Symp Proc. 2003;259–63.
100.
Zobel J, Dart P. Finding approximate matches in large lexicons. Software: Pract Exp. 1995;25(3):331–45.CrossRef
101.
Metadata
Title
De-identified Bayesian personal identity matching for privacy-preserving record linkage despite errors: development and validation
Authors
Rudolf N. Cardinal
Anna Moore
Martin Burchell
Jonathan R. Lewis
Publication date
01-12-2023
Publisher
BioMed Central
Published in
BMC Medical Informatics and Decision Making / Issue 1/2023
Electronic ISSN: 1472-6947
DOI
https://doi.org/10.1186/s12911-023-02176-6

Other articles of this Issue 1/2023

BMC Medical Informatics and Decision Making 1/2023 Go to the issue

Correction

Correction: The impact of data from remote measurement technology on the clinical practice of healthcare professionals in depression, epilepsy and multiple sclerosis: survey

Research

Digital communication and virtual reality for extending the behavioural treatment of obesity – the patients’ perspective: results of an online survey in Germany

Research

Pregnant women’s use and attitude toward Mobile phone features for self-management

Research

The automatic detection of diabetic kidney disease from retinal vascular parameters combined with clinical variables using artificial intelligence in type-2 diabetes patients

Research

Associations of perceived changes in work due to digitalization and the amount of digital work with job strain among physicians: a national representative sample

Research

The effect of health literacy intervention on adherence to medication of uncontrolled hypertensive patients using the M-health