Skip to main content
Top
Published in: BMC Psychiatry 1/2014

Open Access 01-12-2014 | Research article

Risk stratification using data from electronic medical records better predicts suicide risks than clinician assessments

Authors: Truyen Tran, Wei Luo, Dinh Phung, Richard Harvey, Michael Berk, Richard Lee Kennedy, Svetha Venkatesh

Published in: BMC Psychiatry | Issue 1/2014

Login to get access

Abstract

Background

To date, our ability to accurately identify patients at high risk from suicidal behaviour, and thus to target interventions, has been fairly limited. This study examined a large pool of factors that are potentially associated with suicide risk from the comprehensive electronic medical record (EMR) and to derive a predictive model for 1–6 month risk.

Methods

7,399 patients undergoing suicide risk assessment were followed up for 180 days. The dataset was divided into a derivation and validation cohorts of 4,911 and 2,488 respectively. Clinicians used an 18-point checklist of known risk factors to divide patients into low, medium, or high risk. Their predictive ability was compared with a risk stratification model derived from the EMR data. The model was based on the continuation-ratio ordinal regression method coupled with lasso (which stands for least absolute shrinkage and selection operator).

Results

In the year prior to suicide assessment, 66.8% of patients attended the emergency department (ED) and 41.8% had at least one hospital admission. Administrative and demographic data, along with information on prior self-harm episodes, as well as mental and physical health diagnoses were predictive of high-risk suicidal behaviour. Clinicians using the 18-point checklist were relatively poor in predicting patients at high-risk in 3 months (AUC 0.58, 95% CIs: 0.50 – 0.66). The model derived EMR was superior (AUC 0.79, 95% CIs: 0.72 – 0.84). At specificity of 0.72 (95% CIs: 0.70-0.73) the EMR model had sensitivity of 0.70 (95% CIs: 0.56-0.83).

Conclusion

Predictive models applied to data from the EMR could improve risk stratification of patients presenting with potential suicidal behaviour. The predictive factors include known risks for suicide, but also other information relating to general health and health service utilisation.
Literature
1.
go back to reference Nock MK, Green JG, Hwang I, McLaughlin KA, Sampson NA, Zaslavsky AM, Kessler RC: Prevalence, correlates, and treatment of lifetime suicidal behavior among adolescents: results from the national comorbidity survey replication adolescent supplement. JAMA Psychiatry. 2013, 70 (3): 300-310. 10.1001/2013.jamapsychiatry.55.CrossRefPubMed Nock MK, Green JG, Hwang I, McLaughlin KA, Sampson NA, Zaslavsky AM, Kessler RC: Prevalence, correlates, and treatment of lifetime suicidal behavior among adolescents: results from the national comorbidity survey replication adolescent supplement. JAMA Psychiatry. 2013, 70 (3): 300-310. 10.1001/2013.jamapsychiatry.55.CrossRefPubMed
2.
go back to reference Borges G, Nock MK, Haro Abad JM, Hwang I, Sampson NA, Alonso J, Andrade LH, Angermeyer MC, Beautrais A, Bromet E, Bruffaerts R, de Girolamo G, Florescu S, Gureje O, Hu C, Karam EG, Kovess-Masfety V, Lee S, Levinson D, Medina-Mora ME, Ormel J, Posada-Villa J, Sagar R, Tomov T, Uda H, Williams DR, Kessler RC: Twelve-month prevalence of and risk factors for suicide attempts in the world health organization world mental health surveys. J Clin Psychiatry. 2010, 71 (12): 1617-1628. 10.4088/JCP.08m04967blu.CrossRefPubMedPubMedCentral Borges G, Nock MK, Haro Abad JM, Hwang I, Sampson NA, Alonso J, Andrade LH, Angermeyer MC, Beautrais A, Bromet E, Bruffaerts R, de Girolamo G, Florescu S, Gureje O, Hu C, Karam EG, Kovess-Masfety V, Lee S, Levinson D, Medina-Mora ME, Ormel J, Posada-Villa J, Sagar R, Tomov T, Uda H, Williams DR, Kessler RC: Twelve-month prevalence of and risk factors for suicide attempts in the world health organization world mental health surveys. J Clin Psychiatry. 2010, 71 (12): 1617-1628. 10.4088/JCP.08m04967blu.CrossRefPubMedPubMedCentral
3.
go back to reference Spiller HA, Appana S, Brock GN: Epidemiological trends of suicide and attempted suicide by poisoning in the US: 2000–2008. Legal Med. 2010, 12 (4): 177-183. 10.1016/j.legalmed.2010.04.005.CrossRefPubMed Spiller HA, Appana S, Brock GN: Epidemiological trends of suicide and attempted suicide by poisoning in the US: 2000–2008. Legal Med. 2010, 12 (4): 177-183. 10.1016/j.legalmed.2010.04.005.CrossRefPubMed
4.
go back to reference Ting SA, Sullivan AF, Boudreaux ED, Miller I, Camargo CA: Trends in US emergency department visits for attempted suicide and self-inflicted injury, 1993–2008. Gen Hosp Psychiatry. 2012, 34 (5): 557-565. 10.1016/j.genhosppsych.2012.03.020.CrossRefPubMedPubMedCentral Ting SA, Sullivan AF, Boudreaux ED, Miller I, Camargo CA: Trends in US emergency department visits for attempted suicide and self-inflicted injury, 1993–2008. Gen Hosp Psychiatry. 2012, 34 (5): 557-565. 10.1016/j.genhosppsych.2012.03.020.CrossRefPubMedPubMedCentral
5.
go back to reference Centers for Disease C, Prevention: Suicide among adults aged 35–64 years - United States, 1999–2010. MMWR Morb Mortal Wkly Rep. 2013, 62 (17): 321-325. Centers for Disease C, Prevention: Suicide among adults aged 35–64 years - United States, 1999–2010. MMWR Morb Mortal Wkly Rep. 2013, 62 (17): 321-325.
6.
go back to reference Huffman LC, Wang NE, Saynina O, Wren FJ, Wise PH, Horwitz SM: Predictors of hospitalization after an emergency department visit for California youths with psychiatric disorders. Psychiatr Serv. 2012, 63 (9): 896-905.CrossRefPubMed Huffman LC, Wang NE, Saynina O, Wren FJ, Wise PH, Horwitz SM: Predictors of hospitalization after an emergency department visit for California youths with psychiatric disorders. Psychiatr Serv. 2012, 63 (9): 896-905.CrossRefPubMed
7.
go back to reference Allen MH, Abar BW, McCormick M, Barnes DH, Haukoos J, Garmel GM, Boudreaux ED: Screening for suicidal ideation and attempts among emergency department medical patients: instrument and results from the psychiatric emergency research collaboration. Suicide Life Threat Behav. 2013, 43 (3): 313-323. 10.1111/sltb.12018.CrossRefPubMed Allen MH, Abar BW, McCormick M, Barnes DH, Haukoos J, Garmel GM, Boudreaux ED: Screening for suicidal ideation and attempts among emergency department medical patients: instrument and results from the psychiatric emergency research collaboration. Suicide Life Threat Behav. 2013, 43 (3): 313-323. 10.1111/sltb.12018.CrossRefPubMed
8.
go back to reference Choi JW, Park S, Yi KK, Hong JP: Suicide mortality of suicide attempt patients discharged from emergency room, nonsuicidal psychiatric patients discharged from emergency room, admitted suicide attempt patients, and admitted nonsuicidal psychiatric patients. Suicide Life Threat Behav. 2012, 42 (3): 235-243. 10.1111/j.1943-278X.2012.00085.x.CrossRefPubMed Choi JW, Park S, Yi KK, Hong JP: Suicide mortality of suicide attempt patients discharged from emergency room, nonsuicidal psychiatric patients discharged from emergency room, admitted suicide attempt patients, and admitted nonsuicidal psychiatric patients. Suicide Life Threat Behav. 2012, 42 (3): 235-243. 10.1111/j.1943-278X.2012.00085.x.CrossRefPubMed
9.
go back to reference Perry IJ, Corcoran P, Fitzgerald AP, Keeley HS, Reulbach U, Arensman E: The incidence and repetition of hospital-treated deliberate self harm: findings from the world’s first national registry. PLoS ONE [Electronic Resource]. 2012, 7 (2): e31663-10.1371/journal.pone.0031663.CrossRef Perry IJ, Corcoran P, Fitzgerald AP, Keeley HS, Reulbach U, Arensman E: The incidence and repetition of hospital-treated deliberate self harm: findings from the world’s first national registry. PLoS ONE [Electronic Resource]. 2012, 7 (2): e31663-10.1371/journal.pone.0031663.CrossRef
10.
go back to reference Gonda X, Fountoulakis KN, Kaprinis G, Rihmer Z: Prediction and prevention of suicide in patients with unipolar depression and anxiety. Ann Gen Psychiatry. 2007, 6: 23-10.1186/1744-859X-6-23.CrossRefPubMedPubMedCentral Gonda X, Fountoulakis KN, Kaprinis G, Rihmer Z: Prediction and prevention of suicide in patients with unipolar depression and anxiety. Ann Gen Psychiatry. 2007, 6: 23-10.1186/1744-859X-6-23.CrossRefPubMedPubMedCentral
11.
go back to reference Gomez-Duran EL, Martin-Fumado C, Hurtado-Ruiz G: Clinical and epidemiological aspects of suicide in patients with schizophrenia. Actas Esp Psiquiatr. 2012, 40 (6): 333-345.PubMed Gomez-Duran EL, Martin-Fumado C, Hurtado-Ruiz G: Clinical and epidemiological aspects of suicide in patients with schizophrenia. Actas Esp Psiquiatr. 2012, 40 (6): 333-345.PubMed
12.
go back to reference Gonda X, Pompili M, Serafini G, Montebovi F, Campi S, Dome P, Duleba T, Girardi P, Rihmer Z: Suicidal behavior in bipolar disorder: epidemiology, characteristics and major risk factors. J Affect Disord. 2012, 143 (1–3): 16-26.CrossRefPubMed Gonda X, Pompili M, Serafini G, Montebovi F, Campi S, Dome P, Duleba T, Girardi P, Rihmer Z: Suicidal behavior in bipolar disorder: epidemiology, characteristics and major risk factors. J Affect Disord. 2012, 143 (1–3): 16-26.CrossRefPubMed
13.
go back to reference Haw C, Hawton K: Living alone and deliberate self-harm: a case–control study of characteristics and risk factors. Soc Psychiatry Psychiatr Epidemiol. 2011, 46 (11): 1115-1125. 10.1007/s00127-010-0278-z.CrossRefPubMed Haw C, Hawton K: Living alone and deliberate self-harm: a case–control study of characteristics and risk factors. Soc Psychiatry Psychiatr Epidemiol. 2011, 46 (11): 1115-1125. 10.1007/s00127-010-0278-z.CrossRefPubMed
14.
go back to reference Sapyta J, Goldston DB, Erkanli A, Daniel SS, Heilbron N, Mayfield A, Treadway SL: Evaluating the predictive validity of suicidal intent and medical lethality in youth. J Consult Clin Psychol. 2012, 80 (2): 222-231.CrossRefPubMedPubMedCentral Sapyta J, Goldston DB, Erkanli A, Daniel SS, Heilbron N, Mayfield A, Treadway SL: Evaluating the predictive validity of suicidal intent and medical lethality in youth. J Consult Clin Psychol. 2012, 80 (2): 222-231.CrossRefPubMedPubMedCentral
15.
go back to reference Waern M, Sjostrom N, Marlow T, Hetta J: Does the suicide assessment scale predict risk of repetition? A prospective study of suicide attempters at a hospital emergency department. Eur Psychiatry: the J Assoc Eur Psychiatrists. 2010, 25 (7): 421-426. 10.1016/j.eurpsy.2010.03.014.CrossRef Waern M, Sjostrom N, Marlow T, Hetta J: Does the suicide assessment scale predict risk of repetition? A prospective study of suicide attempters at a hospital emergency department. Eur Psychiatry: the J Assoc Eur Psychiatrists. 2010, 25 (7): 421-426. 10.1016/j.eurpsy.2010.03.014.CrossRef
16.
go back to reference Fountoulakis KN, Pantoula E, Siamouli M, Moutou K, Gonda X, Rihmer Z, Iacovides A, Akiskal H: Development of the risk assessment suicidality scale (RASS): a population-based study. J Affect Disord. 2012, 138 (3): 449-457. 10.1016/j.jad.2011.12.045.CrossRefPubMed Fountoulakis KN, Pantoula E, Siamouli M, Moutou K, Gonda X, Rihmer Z, Iacovides A, Akiskal H: Development of the risk assessment suicidality scale (RASS): a population-based study. J Affect Disord. 2012, 138 (3): 449-457. 10.1016/j.jad.2011.12.045.CrossRefPubMed
17.
go back to reference Stefansson J, Nordstrom P, Jokinen J: Suicide intent scale in the prediction of suicide. J Affect Disord. 2012, 136 (1–2): 167-171.CrossRefPubMed Stefansson J, Nordstrom P, Jokinen J: Suicide intent scale in the prediction of suicide. J Affect Disord. 2012, 136 (1–2): 167-171.CrossRefPubMed
18.
go back to reference Bolton JM, Spiwak R, Sareen J: Predicting suicide attempts with the SAD PERSONS scale: a longitudinal analysis. J Clin Psychiatry. 2012, 73 (6): e735-e741. 10.4088/JCP.11m07362.CrossRefPubMed Bolton JM, Spiwak R, Sareen J: Predicting suicide attempts with the SAD PERSONS scale: a longitudinal analysis. J Clin Psychiatry. 2012, 73 (6): e735-e741. 10.4088/JCP.11m07362.CrossRefPubMed
19.
go back to reference Ryan CJ, Large MM: Suicide risk assessment: where are we now?. Med J Aust. 2013, 198 (9): 462-463. 10.5694/mja13.10437.CrossRefPubMed Ryan CJ, Large MM: Suicide risk assessment: where are we now?. Med J Aust. 2013, 198 (9): 462-463. 10.5694/mja13.10437.CrossRefPubMed
20.
go back to reference Sakinofsky I: Attendance at accident and emergency for deliberate self harm predicts increased risk of suicide, especially in women. Evid Based Ment Health. 2005, 8 (4): 97-10.1136/ebmh.8.4.97.CrossRefPubMed Sakinofsky I: Attendance at accident and emergency for deliberate self harm predicts increased risk of suicide, especially in women. Evid Based Ment Health. 2005, 8 (4): 97-10.1136/ebmh.8.4.97.CrossRefPubMed
21.
go back to reference Da Cruz D, Pearson A, Saini P, Miles C, While D, Swinson N, Williams A, Shaw J, Appleby L, Kapur N: Emergency department contact prior to suicide in mental health patients. Emerg Med J. 2011, 28 (6): 467-471. 10.1136/emj.2009.081869.CrossRefPubMed Da Cruz D, Pearson A, Saini P, Miles C, While D, Swinson N, Williams A, Shaw J, Appleby L, Kapur N: Emergency department contact prior to suicide in mental health patients. Emerg Med J. 2011, 28 (6): 467-471. 10.1136/emj.2009.081869.CrossRefPubMed
22.
go back to reference Luoma JB, Martin CE, Pearson JL: Contact with mental health and primary care providers before suicide: a review of the evidence. Am J Psychiatr. 2002, 159 (6): 909-916. 10.1176/appi.ajp.159.6.909.CrossRefPubMed Luoma JB, Martin CE, Pearson JL: Contact with mental health and primary care providers before suicide: a review of the evidence. Am J Psychiatr. 2002, 159 (6): 909-916. 10.1176/appi.ajp.159.6.909.CrossRefPubMed
23.
go back to reference Liu HL, Chen LH, Huang SM: Outpatient health care utilization of suicide decedents in their last year of life. Suicide Life Threat Behav. 2012, 42 (4): 445-452. 10.1111/j.1943-278X.2012.00103.x.CrossRefPubMed Liu HL, Chen LH, Huang SM: Outpatient health care utilization of suicide decedents in their last year of life. Suicide Life Threat Behav. 2012, 42 (4): 445-452. 10.1111/j.1943-278X.2012.00103.x.CrossRefPubMed
24.
go back to reference Qin P, Webb R, Kapur N, Sorensen HT: Hospitalization for physical illness and risk of subsequent suicide: a population study. J Intern Med. 2013, 273 (1): 48-58. 10.1111/j.1365-2796.2012.02572.x.CrossRefPubMed Qin P, Webb R, Kapur N, Sorensen HT: Hospitalization for physical illness and risk of subsequent suicide: a population study. J Intern Med. 2013, 273 (1): 48-58. 10.1111/j.1365-2796.2012.02572.x.CrossRefPubMed
25.
go back to reference Galfalvy HC, Oquendo MA, Mann JJ: Evaluation of clinical prognostic models for suicide attempts after a major depressive episode. Acta Psychiatr Scand. 2008, 117 (4): 244-252. 10.1111/j.1600-0447.2008.01162.x.CrossRefPubMedPubMedCentral Galfalvy HC, Oquendo MA, Mann JJ: Evaluation of clinical prognostic models for suicide attempts after a major depressive episode. Acta Psychiatr Scand. 2008, 117 (4): 244-252. 10.1111/j.1600-0447.2008.01162.x.CrossRefPubMedPubMedCentral
26.
go back to reference Elliott AJ, Pages KP, Russo J, Wilson LG: A profile of medically serious suicide attempts. J Clin Psychiatry. 1996, 57 (12): 567-571. 10.4088/JCP.v57n1202.CrossRefPubMed Elliott AJ, Pages KP, Russo J, Wilson LG: A profile of medically serious suicide attempts. J Clin Psychiatry. 1996, 57 (12): 567-571. 10.4088/JCP.v57n1202.CrossRefPubMed
27.
go back to reference Elixhauser A, Steiner C, Harris DR, Coffey RM: Comorbidity measures for use with administrative data. Med Care. 1998, 36 (1): 8-27. 10.1097/00005650-199801000-00004.CrossRefPubMed Elixhauser A, Steiner C, Harris DR, Coffey RM: Comorbidity measures for use with administrative data. Med Care. 1998, 36 (1): 8-27. 10.1097/00005650-199801000-00004.CrossRefPubMed
28.
go back to reference Archer KJ, Williams AA: L1 penalized continuation ratio models for ordinal response prediction using high-dimensional datasets. Stat Med. 2012, 31 (14): 1464-1474. 10.1002/sim.4484. doi: 10.1002/sim.4484 [published Online First: Epub Date]CrossRefPubMedPubMedCentral Archer KJ, Williams AA: L1 penalized continuation ratio models for ordinal response prediction using high-dimensional datasets. Stat Med. 2012, 31 (14): 1464-1474. 10.1002/sim.4484. doi: 10.1002/sim.4484 [published Online First: Epub Date]CrossRefPubMedPubMedCentral
29.
go back to reference Ananth CV, Kleinbaum DG: Regression models for ordinal responses: a review of methods and applications. Int J Epidemiol. 1997, 26 (6): 1323-1333. 10.1093/ije/26.6.1323.CrossRefPubMed Ananth CV, Kleinbaum DG: Regression models for ordinal responses: a review of methods and applications. Int J Epidemiol. 1997, 26 (6): 1323-1333. 10.1093/ije/26.6.1323.CrossRefPubMed
30.
go back to reference Fowler JC, Piers C, Hilsenroth MJ, Holdwick DJ, Padawer JR: The rorschach suicide constellation: assessing various degrees of lethality. J Pers Assess. 2001, 76 (2): 333-351. 10.1207/S15327752JPA7602_13. doi: 10.1207/S15327752JPA7602_13[published Online First: Epub Date]CrossRefPubMed Fowler JC, Piers C, Hilsenroth MJ, Holdwick DJ, Padawer JR: The rorschach suicide constellation: assessing various degrees of lethality. J Pers Assess. 2001, 76 (2): 333-351. 10.1207/S15327752JPA7602_13. doi: 10.1207/S15327752JPA7602_13[published Online First: Epub Date]CrossRefPubMed
31.
go back to reference Tibshirani R: Regression shrinkage and selection via the Lasso. J Roy Stat Soc B Met. 1996, 58 (1): 267-288. Tibshirani R: Regression shrinkage and selection via the Lasso. J Roy Stat Soc B Met. 1996, 58 (1): 267-288.
32.
go back to reference Moons KG, Donders AR, Steyerberg EW, Harrell FE: Penalized maximum likelihood estimation to directly adjust diagnostic and prognostic prediction models for overoptimism: a clinical example. J Clin Epidemiol. 2004, 57 (12): 1262-1270. 10.1016/j.jclinepi.2004.01.020. doi: 10.1016/j.jclinepi.2004.01.020 [published Online First: Epub Date]CrossRefPubMed Moons KG, Donders AR, Steyerberg EW, Harrell FE: Penalized maximum likelihood estimation to directly adjust diagnostic and prognostic prediction models for overoptimism: a clinical example. J Clin Epidemiol. 2004, 57 (12): 1262-1270. 10.1016/j.jclinepi.2004.01.020. doi: 10.1016/j.jclinepi.2004.01.020 [published Online First: Epub Date]CrossRefPubMed
33.
go back to reference Ishii N, Terao T, Araki Y, Kohno K, Mizokami Y, Arasaki M, Iwata N: Risk factors for suicide in Japan: a model of predicting suicide in 2008 by risk factors of 2007. J Affect Disord. 2013, 147 (1–3): 352-354.CrossRefPubMed Ishii N, Terao T, Araki Y, Kohno K, Mizokami Y, Arasaki M, Iwata N: Risk factors for suicide in Japan: a model of predicting suicide in 2008 by risk factors of 2007. J Affect Disord. 2013, 147 (1–3): 352-354.CrossRefPubMed
34.
go back to reference Bilen K, Ponzer S, Ottosson C, Castren M, Owe-Larsson B, Ekdahl K, Pettersson H: Can repetition of deliberate self-harm be predicted? A prospective multicenter study validating clinical decision rules. J Affect Disord. 2013, 149 (1–3): 253-258.CrossRefPubMed Bilen K, Ponzer S, Ottosson C, Castren M, Owe-Larsson B, Ekdahl K, Pettersson H: Can repetition of deliberate self-harm be predicted? A prospective multicenter study validating clinical decision rules. J Affect Disord. 2013, 149 (1–3): 253-258.CrossRefPubMed
35.
go back to reference Challis S, Nielssen O, Harris A, Large M: Systematic meta-analysis of the risk factors for deliberate self-harm before and after treatment for first-episode psychosis. Acta Psychiatr Scand. 2013, 127 (6): 442-454. 10.1111/acps.12074.CrossRefPubMed Challis S, Nielssen O, Harris A, Large M: Systematic meta-analysis of the risk factors for deliberate self-harm before and after treatment for first-episode psychosis. Acta Psychiatr Scand. 2013, 127 (6): 442-454. 10.1111/acps.12074.CrossRefPubMed
36.
go back to reference Hawton K, Casanas ICC, Haw C, Saunders K: Risk factors for suicide in individuals with depression: a systematic review. J Affect Disord. 2013, 147 (1–3): 17-28.CrossRefPubMed Hawton K, Casanas ICC, Haw C, Saunders K: Risk factors for suicide in individuals with depression: a systematic review. J Affect Disord. 2013, 147 (1–3): 17-28.CrossRefPubMed
37.
go back to reference de Achaval S, Feudtner C, Palla S, Suarez-Almazor ME: Validation of ICD-9-CM codes for identification of acetaminophen-related emergency department visits in a large pediatric hospital. BMC Health Serv Res. 2013, 13: 72-10.1186/1472-6963-13-72.CrossRefPubMedPubMedCentral de Achaval S, Feudtner C, Palla S, Suarez-Almazor ME: Validation of ICD-9-CM codes for identification of acetaminophen-related emergency department visits in a large pediatric hospital. BMC Health Serv Res. 2013, 13: 72-10.1186/1472-6963-13-72.CrossRefPubMedPubMedCentral
Metadata
Title
Risk stratification using data from electronic medical records better predicts suicide risks than clinician assessments
Authors
Truyen Tran
Wei Luo
Dinh Phung
Richard Harvey
Michael Berk
Richard Lee Kennedy
Svetha Venkatesh
Publication date
01-12-2014
Publisher
BioMed Central
Published in
BMC Psychiatry / Issue 1/2014
Electronic ISSN: 1471-244X
DOI
https://doi.org/10.1186/1471-244X-14-76

Other articles of this Issue 1/2014

BMC Psychiatry 1/2014 Go to the issue