Evaluating performance of electronic healthcare records and spontaneous reporting data in drug safety signal detection

Background Electronic reporting and processing of suspected adverse drug reactions (ADRs) is increasing and has facilitated automated screening procedures. It is crucial for healthcare professionals to understand the nature and proper use of data available in pharmacovigilance practice. Objectives To (a) compare performance of EU-ADR [electronic healthcare record (EHR) exemplar] and FAERS [spontaneous reporting system (SRS) exemplar] databases in detecting signals using “positive” and “negative” drug-event reference sets; and (b) evaluate the impact of timing bias on sensitivity thresholds by comparing all data to data restricted to the time before a warning/regulatory action. Methods Ten events with known positive and negative reference sets were selected. Signals were identified when respective statistics exceeded defined thresholds. Main outcome measure Performance metrics, including sensitivity, specificity, positive predictive value and accuracy were calculated. In addition, the effect of regulatory action on the performance of signal detection in each data source was evaluated. Results The sensitivity for detecting signals in EHR data varied depending on the nature of the adverse events and increased substantially if the analyses were restricted to the period preceding the first regulatory action. Across all events, using data from all years, a sensitivity of 45–73 % was observed for EU-ADR and 77 % for FAERS. The specificity was high and similar for EU-ADR (82–96 %) and FAERS (98 %). EU-ADR data showed range of PPV (78–91 %) and accuracy (78–72 %) and FAERS data yielded a PPV of 97 % with 88 % accuracy. Conclusion Using all cumulative data, signal detection in SRS data achieved higher specificity and sensitivity than EHR data. However, when data were restricted to time prior to a regulatory action, performance characteristics changed in a manner consistent with both the type of data and nature of the ADR. Further research focusing on prospective validation of is necessary to learn more about the performance and utility of these databases in modern pharmacovigilance practice.