Evaluation of Veterans’ Suicide Risk With the Use of Linguistic Detection Methods

In 2010, suicide was the tenth leading cause of death in the United States and among the top four leading causes of death for Americans between ages ten and 54 (1). Recent use of the health care system by those who die from suicide has long been seen as an opportunity for intervention (2). However, in the context of clinical care, suicide continues to be difficult both to predict and to prevent (3). Despite efforts to screen and monitor for suicide risk as well as to boost awareness, education about risk factors, and outreach initiatives, suicide remains a significant public health problem (4,5).

Clinical identification of patients who later die from suicide has long been limited by the low base rate and high false-positive rate inherent in the use of any single predictor or even combinations of recognized risk factors (6,7). This limitation has spurred the development of quantitative models for suicide prediction, including both multivariate and nonlinear modeling of structured data elements from questionnaires, rating scales, and demographic factors (8,9). However, limiting analyses to known risk factors may occur at the cost of foregoing untapped valuable information in the written record of the clinical encounter.

Even with direct questioning, patients may not disclose suicidal intent (10–12). Recent qualitative work indicates that patients may see disclosure of suicidal thoughts as shameful and private and feel that such disclosures could expose them to risks of receiving unwanted treatment (13). Consistent with prior research indicating patients’ desire to have a nuanced discussion around suicide attempts (14), patients found the use of questionnaires about suicide risk “perfunctory and disrespectful.” Therefore, building a suicide risk prediction model reliant on direct inquires of and mentions about suicide may not be a rational strategy. Linguistic analysis seeks to tap into another potential source of knowledge. It attempts to glean knowledge regarding suicide risk not from overt keywords related to suicide (for example, “suicide,” “despair,” “gun,” or “agitation”) but rather from more subtle changes in language that may indicate clinicians’ feelings about their patients (15).

Linguistic analysis has been previously applied to documents written by people who later died by suicide. Investigators found that suicidal individuals’ writing in diaries, correspondence, suicide notes, and online venues has unique semantic characteristics, including unique patterns of pronoun use (16,17). Such research has expanded to emerging social environments, such as publicly accessible, open, forum-type online support groups (18,19). Notably, Won and colleagues (20) developed a model for predicting suicide rates in South Korea by using two social constructs developed from text in Weblog entries in social media: suicide-related text and dysphoria-related text.

There has been no prior research using linguistic analysis of notes written by clinicians about patients who later die from suicide. One area of potential interest is measuring the emotional distance between patient and clinician. The concept of emotional distancing was initially described in Schopenhauer’s (21) “Hedgehog’s Dilemma,” which Freud (22) later incorporated into a description of how individuals relate to others in society. Human intimacy can be limited by the potential for emotional harm that can occur in relationships. This results in cautious interpersonal behavior and emotional distance. Prior research using clinician self-report addresses the function of emotional distancing in minimizing and mitigating stress when working with patients who have mental health problems. Van Sant and Patterson (23) found that nurses working with patients in mental health crises used emotional distancing as a protective maneuver; the investigators also described emotional damage to caregivers who did not use such protective maneuvers to establish patient-caregiver boundaries. Kintzle and colleagues (24) explored rates of secondary traumatic stress in a sample of 70 military primary and mental health care providers, who commonly felt “emotionally numb” as a result of serving this population. Thus, in our context, emotional distancing can be conceptualized as a largely unconscious effort that clinicians use to spare themselves from the emotional trauma associated with the suicide of one of their patients.

The advent of computational linguistics has led to approaches of measuring emotional content, including distancing, through analysis of text. Linguistic studies typically measure distancing language by assessing pronoun use and referencing. Frequent use of first-person and second-person pronouns signifies interpersonal closeness, whereas frequent use of the third-person pronouns characterizes interpersonal distance (25,26). “Referencing” describes a situation where a clinician assigns the source of the content of the note to someone other than the clinician him- or herself. Examples of referencing are when a clinician quotes a patient or an authority such as a mental health consultant, cites protocol or guidelines, or otherwise restates comments made by another person involved with the patient’s care. Van De Mieroop (27) found that use of the “quotative” (attributing a communication’s content to a third party) increased when bad news was communicated between doctor and patient.

Our research is conceptually driven by prior research on emotional distancing by mental health providers, which we quantify by measuring the use of distancing language in text written by clinicians. We focus specifically on measures of distancing language that might be used, consciously or not, by clinicians encountering patients at high risk of suicide. We had two hypotheses. Our primary hypothesis was that there would be more distancing language in notes documenting clinical encounters with patients who died from suicide than in notes documenting clinical encounters with patients who remained alive. For our primary hypothesis, we theorized that differences could be observed at least three months prior to the suicide. Our exploratory hypothesis attempted to triangulate the possible finding of increased use of distancing language. For this hypothesis we examined trends in word use in the year preceding suicide. We hypothesized that words related to distancing language would emerge as keywords as the suicide date drew closer.

Methods

Overview

There were four main steps to completing this project. [The steps are outlined in the online supplement to this article.] First, we identified our cohorts on the basis of whether they died from suicide and whether they used mental health services. Second, we created text corpora (collections of text files to be analyzed) for each quarter. Third, using linguistic software, we created sets of our independent variables from each of the text corpora. Fourth, we analyzed our independent variables, with cohort membership as our dependent variable. This study, which received a waiver for obtaining informed consent, was approved by the White River Junction Veterans Affairs (VA) Research and Development Committee, the Dartmouth College Center for the Protection of Human Subjects, and the VA Office of Mental Health Operations.

Cohort Identification and Corpora Construction

We used a sample of 63 U.S. military veterans who died from suicide during 2009 and who also had received outpatient VA health services in the year prior to suicide. Cases were selected from a national random sample used in prior work (28).

Patients who died from suicide were divided into two cohorts, those who had received mental health services and those who had not. They were matched to 63 VA outpatients who were living in 2009. The match was based on age, gender, VA priority group, and whether they received mental health services within the prior year. For all patients, we obtained one year of deidentified VA outpatient notes. For each patient who died from suicide, we obtained all clinical notes for the year prior to the suicide event, ending the day before the suicide. The period of observation for veterans in the control group was consistent with the period for veterans who died from suicide.

We identified notes related to mental health services by using the clinic designation field accompanying each note and the contents of the clinical notes. We categorized patients with any note related to mental health services as being in the “mental health service user” cohort and the remaining patients as the “non–mental health service user” cohort. Within each of these cohorts, patients who died from suicide were classified into the “suicide” group and other patients were classified into the “alive” group, who constituted the control group for the study.

In order to examine any trends in notes over time (as the date of the suicide became nearer), we then categorized the notes according to the quarter of the year prior to the suicide date in which they occurred. Quarter 1 began one year prior to the index date, and quarter 4 ended the day prior to the index date (date of the suicide). Corpora from quarters 1–3 were combined for the primary analysis. We excluded the fourth quarter from our primary analytic corpora so that findings would support early identification of suicide risk and to prevent analyses from being dominated by text entries just prior to the suicide events.

Linguistic Analysis

There were two parts to our linguistic analysis. First, using preidentified constructs related to referencing and third-person pronoun use, we tested for differences between the groups. Constructs are groups of words that belong together, whether structurally or contextually. Constructs used in this analysis included third-person pronouns and referencing. We used Linguistic Inquiry and Word Count (LIWC) software version 1.12 to identify constructs of interest in each text corpora. Second, using keyword analysis, we tested for longitudinal differences within the suicide groups. Keyword analysis is a method of identifying individual words that occur at statistically different frequencies between two collections of text. Keywords emerge from the text and are not predefined. We used Wordsmith Version 6.0.137 to identify keywords in the text corpora.

To test our primary hypothesis—that terms related to referencing and third-person pronoun use in clinical notes would be used more frequently in the suicide cohorts versus the nonsuicide cohorts more than three months prior to the index date—we subjected the quarters 1–3 corpora of all four cohorts to LIWC analysis. We used a user-defined dictionary containing two constructs specifically designed to test our hypotheses. The first construct was a group of words related to third-person pronoun use, and the second was a group of words related to referencing. LIWC counts the number of words related to the construct as defined in a dictionary, then computes, using embedded algorithms, a score for each construct relative to the total number of words in the document or collection of documents (29).

To test our exploratory hypothesis—that keywords related to third-person pronoun use and referencing would emerge with increasing frequency the nearer the suicide date—we used Wordsmith to compare keywords in the third-quarter corpora with those in the first-quarter quarter for the suicide group in each cohort. To validate our finding, we also compared fourth-quarter results with first-quarter results for the suicide group in each cohort, expecting similar results. Because there was not sufficient power to perform a regression across time periods, our primary focus was on a comparison between quarters 1 and 3. This a priori comparison was selected for several reasons. First, the comparison between quarters 1 and 2 was felt to be too close in time to allow for sufficient changes in the characteristics of notes. Second, although we were interested in the occurrences in quarter 4, we were concerned that many of the notes during that period would specifically address suicidal behavior, masking implicit reference to emotional distancing with explicit references to suicide. Therefore, we selected a primary comparison between quarter 1 and quarter 3. However, we also conducted an exploratory comparison between quarter 1 and quarter 4 to determine the validity of our concern that quarter 4 notes would contain a high number of references to specific interventions for suicide prevention.

Statistical Analysis

We performed statistical analysis and managed data with SAS version 9.3. Using one-tailed t tests to identify increases in referencing and third-person pronoun use, we compared LIWC category scores for the first- and third-quarter corpora between the suicide and alive groups within each cohort.

For our secondary analysis identifying keywords, we used a measure of “keyness” produced by the Wordsmith software. Keyness is a relative measure of the frequency of word occurrence in a text corpus when compared with a reference corpus, with higher keyness indicating a higher relative frequency. By default, Wordsmith uses the log likelihood test to identify significant keywords (29). Because of the exploratory nature of keyword identification, Wordsmith’s default p value for positive keyness is set to p≤.000001.

Results

Cohort Identification

The clinical notes analyzed for quarters 1–4 are described in Table 1. The number of patients was well balanced across cohorts and groups. The number of notes increased over time among the mental health service users who died from suicide but stayed relatively stable or declined among the other groups.

Corpora Construction

We excluded ten patients from our primary analysis (four in the suicide group and six in the alive group). In all ten cases, the patient used VA outpatient services only in quarter 4. The text corpora from quarters 1 through 3 that we used to evaluate our primary hypothesis indicated that mental health service users had over twice as many notes within both the suicide and alive groups (Table 2).

Linguistic Analysis

As shown in Table 3, we observed a statistically significant difference in third-person pronoun use between the suicide and alive groups within the mental health service user cohort. Specifically, the third-person pronoun use construct included the concepts of “veteran” and “he/she.” We did not observe a significant difference in our custom LIWC category for referencing. We observed no differences between the suicide and alive groups in the non–mental health service user cohort.

Table 4 shows the results of the keyword analysis used to evaluate our exploratory hypothesis. Each row indicates a keyword that differs significantly between the primary and reference corpora when adjusted for the number of words in each corpus. The frequency measures identify the number of times a word appeared in a text corpus. When we compared quarter 3 to quarter 1 in the mental health service user cohort, four keywords emerged, one of which was related to third-person pronoun use (“vet”) and one of which was related to referencing (“stated”). Our comparison of quarter 4 to quarter 1 in the mental health service user cohort replicated these keywords with a higher keyness value and added terms related directly to suicide, including “suicidal” and “SPC,” an acronym used for the suicide prevention coordinator stationed at each VA facility. In the non–mental health service user cohort, there were no significant keywords related to our hypotheses when comparing the quarter 3 or quarter 4 corpora with the quarter 1 corpora.

Discussion

We performed a linguistic analysis comparing clinical notes of VA health service users who died from suicide with those who did not die from suicide. We found differences in text corpora among mental health service users, but not among non–mental health service users, that indicate more frequent use of distancing language by clinicians treating patients who later died from suicide. In our primary analyses we confirmed our hypothesis that there would be more third-person pronoun use in the suicide group than in the nonsuicide group, but we were unable to find an increase in referencing. In our exploratory analysis, we confirmed our hypothesis that keywords related to distancing language emerged as suicide date neared among mental health service users.

These findings suggest that more frequent use of distancing language by clinicians is a predictor of suicide among mental health service users. If replicated in additional studies, this finding could have important implications in the identification of suicide risk. For example, while linguistic trends are likely invisible to a practicing clinician, Hoffman and colleagues (30) suggested that routine linguistic analysis of clinical notes could be shown to clinicians, perhaps via a clinician “dashboard,” to help guide psychotherapy practice. Further, linguistic signals could be combined with other demographic information and risk factors in creating a more comprehensive risk profile to guide clinical decision making and to help individual clinicians tailor their interpersonal style to their patients’ needs.

There were several limitations to this work. First, predictive power in this analysis was at the level of the group rather than at the level of a patient or provider. Larger samples would be needed to establish and validate individual and clinician-level predictors of risk. Second, we were unable to find suicide risk signals in the group of non–mental health service users. This group had far fewer visits and related notes, limiting the power of the analysis. Different linguistic hypotheses may be needed to identify suicide risk signals in this group. In addition, it is possible that this group lacked repeated notes and visits, which would have made emotional distancing more apparent.

Third, we did not report demographic information about patients and clinicians or diagnostic information about patients. In order to facilitate the use of custom software not available on VA servers, this data set was stripped of identifying information in the parent study in order to ensure data security (28). Therefore, we were unable to rematch patients in order to examine the effect of potentially important clinician-patient dyad factors, such as gender and race-ethnicity, and patient factors, such as diagnosis, on interpersonal distancing. Oversampling patients on the basis of diagnostic information and patient-clinician demographic information represents an area for future investigation. Given that no associations were found among non–mental health service users, limiting our analyses to mental health clinician-patient dyads may have strengthened our results.

Fourth, no standardized measure of psychological distress was available in the data set. It is possible that increased use of distancing language by clinicians was a reflection of greater distress overall rather than an increase in suicide risk specifically. Fifth, it is possible that our findings in this small sample are spurious. However, our sample of veterans who died from suicide was drawn at random from all VA users who died from suicide in 2009, and our primary analysis was both parsimonious and hypotheses driven. Our exploratory keyword analysis used an extremely low threshold for significance. Regardless, validation of our findings in larger samples is indicated. Finally, our analysis was based on users of VA outpatient services. Validation of our findings in other patient populations is needed.

Conclusions

Linguistic analysis is a promising approach to identify use of distancing language by clinicians, which appears to be a marker of suicide risk. This pilot work indicates that additional analysis and validation with larger cohorts are warranted.