Implementation of Evidence-Based Psychotherapies for Posttraumatic Stress Disorder in VA Specialty Clinics

Posttraumatic stress disorder (PTSD) is a common and often debilitating condition (1,2). The disorder occurs in the aftermath of an event or events involving actual or threatened death or serious injury or threat to physical integrity (3). The treatment of PTSD is a major objective at U.S. Department of Veterans Affairs (VA) medical facilities. A large percentage of veterans returning from the wars in Iraq and Afghanistan suffer from PTSD, and many veterans from prior conflicts still seek treatment for PTSD (4,5).

Over the past 20 years, multiple specific treatments for PTSD have been developed. These treatments, including various types of both medication and psychotherapy, have shown effectiveness in rigorous clinical trials (6,7). Convergent recommendations from clinical practice guidelines and other published reviews suggest that medications, including selective serotonin reuptake inhibitors and noradrenergic antidepressants, and psychotherapies, including several forms of cognitive-behavioral therapy, are effective treatments for PTSD (8–10). Evidence suggests that cognitive-behavioral therapies, including prolonged exposure (PE) (11) and cognitive-processing therapy (CPT) (12), are efficacious for treatment of PTSD among veterans and can be implemented in routine practice (13,14). The VA has advocated the use of a range of evidence-based treatments for PTSD, including specific types of cognitive-behavioral therapy, such as PE and CPT. The VA has ensured that evidence-based medications and psychotherapy are widely available to veterans with PTSD.

In addition to generally advocating the use of PE and CPT, the VA has taken active steps to promote the spread and use of these treatments, including requiring each VA facility to provide the treatments, hiring additional psychotherapy staff, and training psychotherapists to perform the treatments (15–17). Between 2005 and 2010, the VA hired more than 7,000 new mental health professionals, bringing the total mental health staff to more than 20,000 (17). The VA has developed clear clinical guidelines and administrative policies that advocate and even require that veterans have access to specific evidence-based treatments for PTSD (10,16). An administrative mentoring program assists clinical leaders in implementing these treatments (18). In addition, a large-scale and well-resourced national clinical training program trains VA therapists in the use of PE and CPT. As of 2010, 2,300 therapists had been trained in CPT and 1,100 in PE (15,19). The training includes attendance at a several-day course conducted by experts in the treatments and an ongoing program of mentorship and supervision of treatment. Clinicians who complete the course and program may obtain certification in the use of the treatments.

Although the VA’s efforts to develop policies that promote the use of evidence-based psychotherapy for PTSD have been described, little is known about the effectiveness of those interventions. Moreover, the subject of promoting the use of evidence-based treatments in mental health settings is not well understood (20–23). As an example, it is unclear if training staff to perform evidence-based therapy for PTSD actually leads to increased use of these treatments. Furthermore, there is no clear understanding of the steps necessary to promote use of evidence-based practices in mental health settings. These questions apply both to training and to other attempts to promote use of evidence-based treatments for PTSD, such as increasing staffing and adopting policies that improve access to the treatments. Overall, two problems have limited exploration of these questions. First, previous examinations were not able to distinguish evidence-based psychotherapies from psychotherapy in general (24,25). Second, examination of implementation of mental health efforts has been limited because implementation frameworks have been developed largely for use in general medical settings (26,27). As such, their applicability in mental health settings has not been rigorously tested.

The Promoting Action on Research Implementation in Health Services (PARiHS) framework may have particular relevance in understanding VA efforts to increase the use of PE and CPT. The PARiHS framework was designed to aid those attempting to implement a research finding into clinical practice. It examines three independent and important domains for the implementation process—evidence for the practice, clinical context, and the facilitation efforts aimed at promoting use (28,29). Notably, the PARiHS framework has been used both as a road map to guide implementation efforts and as a tool to understand and contextualize the results of improvement efforts that were planned without a formal framework (20–22).

The overall goal of this study was to examine the effectiveness of VA efforts to promote use of evidence-based psychotherapies to treat veterans with PTSD. We determined the use of these treatments in one region and, through staff interviews, explored factors and approaches that facilitated or impeded the use—and spread—of these treatments. We used the PARiHS framework to develop a rubric for evaluating how aspects of implementation affected use of evidence-based psychotherapies.

Methods

The study and protocol were reviewed and approved by the Dartmouth College Committee for the Protection of Human Subjects.

Data source

We used data obtained from the VA New England Data Warehouse. The data warehouse contains clinical and administrative data for all inpatient and outpatient care delivered in the New England VA. We analyzed data from six specialized VA PTSD clinics, three located at large, academically affiliated VA medical centers; two at medium-sized, academically affiliated VA medical centers; and one at a large VA medical center not closely affiliated with a medical school.

Current use of PE and CPT

Our method for determining use of PE and CPT consisted of four steps: identification of patients, automated coding of PE and CPT notes, validation, and determination of the rate of use. The method is described in greater detail elsewhere (30).

Identification of patients.

We identified patients who were presenting for the first time to one of the clinics during the first half of fiscal year (FY) 2010. We obtained administrative data and progress notes for these patients from the data warehouse. We used the administrative data to identify patients who received a new diagnosis of PTSD and who were seen at least once in a specialized PTSD clinic. We selected this patient population, given that they are the ideal candidates for evidence-based psychotherapy for PTSD. This diagnosis of PTSD was based on clinical examination by PTSD clinic staff. Overall, 16,121 patients who were seen in VA New England Health Care facilities during this time period received a primary diagnosis of PTSD during one or more outpatient encounters. During that period, the specialized outpatient PTSD clinics saw 5,379 outpatients.

Automated coding.

We used machine learning to develop a text-based algorithm that differentiated psychotherapy notes, CPT notes, and PE notes by using the automated retrieval console (ARC) (31,32). All progress notes for new patients of specialized PTSD clinics were analyzed by using this algorithm.

Validation.

Manual raters performed blinded coding of a 10% sample of the notes. When the agreement among reviewers on the manual rating team and the ARC was greater than .8, we considered the automated ratings valid. The remaining 90% of notes were coded only by ARC. Agreement between manual ratings and ARC was calculated by using information retrieval measures called recall (the proportion of relevant notes identified by the ARC), precision (the proportion of notes identified by the ARC that were relevant), and “f measure” (a balanced measure of the recall and precision) (28). The recall was .9; the precision, 1; and the f measure, .95.

Rates of PE and CPT use.

Using the coded notes, we calculated the rate of use of CPT and PE at each of the six specialized PTSD clinics. The rate represents the percentage of new patients receiving at least one session of PE or CPT in the six months following clinic enrollment.

Qualitative assessment of implementation

Interview development.

We developed an interview guide to conduct semistructured interviews by telephone with staff members at each clinic. We used the PARiHS framework to develop overarching questions about the implementation of CPT and PE (29) (Table 1). These overarching questions guided the development of the more specific questions used in the interview guide. Phone interviews were conducted with the mental health service line chief, the PTSD clinic leader, and PTSD therapists.

The phone interviews (N=30) were recorded and transcribed. The transcriptions were reviewed, and the contents were coded by domain and element of the PARiHS framework. Evaluators were blinded to the use of treatments at the site.

Interview scoring.

We developed a scoring rubric that transformed qualitative data for each element in the PARiHS framework into a numeric value (Table 1). First, the interview transcripts from each staff member at each site were reviewed, and each PARiHS element was assigned a score from 1 to 10, indicating a range of possible responses. Using this method, we developed a site score for each element. As an example, if all staff members interviewed described highly organized and systematic care and referral processes, we scored the clinic as having a highly organized system of care.

Data analysis

The goal of the data transformation was to develop a continuous measure for each domain and element (in contrast to the use of dichotomous scoring [present or absent]) (33,34). We then conducted a Poisson linear regression equation that used element scores from each facility as independent variables and the percentage of patients at each site receiving any evidence-based therapy as the dependent variable.

Results

Rate of psychotherapy use

The review of administrative data revealed that 1,924 newly diagnosed patients with PTSD were seen at one of the VA New England specialty PTSD clinics in the first six months of FY 2010. The use of evidence-based psychotherapy for PTSD varied by site (Table 2). Overall, there was greater variation in the use of CPT (1%–13%) than in the use of PE (0%–3%). The overall use of either therapy among the sites varied from 4% to 14%. Patients who received at least one session of either therapy received a median of five sessions. Again, there was considerable variation among sites in the median number of sessions (range 2–9).

Psychotherapy and implementation domains

Next we used a Poisson linear regression to correlate the proportion of patients receiving at least one session of evidence-based psychotherapy at each specialty PTSD clinic and the scores on each element of the PARiHS implementation framework. Five of the 18 elements showed a statistically significant association with use of at least one of the evidence-based psychotherapies for PTSD (Table 3). The strongest predictor in terms of both the z score of the association and the r² of the correlation was prior clinical experience having used an evidence-based therapy for PTSD. This suggests that attempts to use these treatments as part of training and other experiences strongly influence eventual adoption of the treatment.

The next strongest predictor was a sustained involvement with the implementation team. Those sites with a more sustained connection with the facilitation team were more likely to use evidence-based therapy for PTSD. Similarly, the therapy was more likely to be used clinically if training was customized to the therapist.

Two elements were negatively correlated with use of evidence-based therapy for PTSD. Contrary to the prediction of the conceptual framework, training that focused on overall clinician development resulted in less use of evidence-based therapy for PTSD than training that focused narrowly on the use of the treatments (purpose of facilitation). Last, organized and highly systematic mental health systems with clearly defined processes were slightly less likely than less organized systems to use evidence-based therapy for PTSD (leadership: system of care). This finding also contradicted the predictions of the PARiHS framework.

Discussion

Summary of findings

During the period of study, delivery of evidence-based psychotherapy for PTSD among newly diagnosed patients of specialty PTSD clinics was quite low. From 4% to 14% of new patients with PTSD received an evidence-based psychotherapy for PTSD in their first six months of enrollment in specialized PTSD clinics.

Most patients who received any evidence-based psychotherapy for PTSD received a modest number of sessions (median=5). Typically, in clinical trials, a full course of CPT was 12 sessions, and a full course of PE ranged from eight to 18 sessions. Although a full course may not be necessary for all patients, it is unlikely that fewer than five sessions of either psychotherapy would constitute an adequate dose. Even when the analysis used a minimal threshold of eight sessions to indicate an adequate dose, the results suggest that 2% of veterans newly presenting with PTSD received an adequate trial of either PE or CPT over their initial six months of care.

This suggests that the VA efforts to promote use of the treatments were not fully successful. Our work examining the factors that promoted use of psychotherapy suggests that, overall, the implementation framework that we applied may need to be modified for implementations in mental health settings or, perhaps more specifically, for implementations of psychotherapy interventions. Even in our small sample, a number of elements of the implementation framework robustly predicted the use of evidence-based therapy for PTSD. Predictive elements included previous experience with and use of the treatments, sustained contact with the facilitation team, and customization of training experiences.

Systems of care that were highly structured and organized and facilitation that focused on overall professional growth rather than on the specific treatment protocols were actually negatively correlated with use of evidence-based therapy for PTSD. These findings contradict the PARiHS framework, which advocates highly organized clinics and facilitation efforts with broad goals. Therefore, we must consider that organized clinics and broad facilitation goals may not be useful to promote mental health implementation efforts. Moreover, the findings of negative associations have some face validity. It is easy to understand how more focused and specific implementation efforts may have more effect than broad ones. Similarly, one can imagine that highly structured clinics may also be more difficult to change.

Limitations

There were a substantial number of limitations to our work. First, we examined only six clinics in a narrow geographic region. It is possible that the use of evidence-based therapy for PTSD and the features associated with implementation are unique to these clinics or this region. More important, our limited sample size likely diminished the study’s power to detect other elements of the framework that were important and valid predictors of use. Thus, in a larger sample, other elements could emerge as significant. Second, we examined treatment use during a narrow, cross-sectional sample of time. A longitudinal approach to examine trends in use would also be desirable. In addition, this study did not evaluate more recent strategies by the VA to implement evidence-based therapy. For example, the VA recently augmented an initiative for hiring mental health providers through the addition of piloted metrics that encourage delivery of evidence-based therapy for treatment of PTSD among veterans returning from Iraq and Afghanistan. Our interview strategy and structured questions were targeted toward a single implementation model. Ideally, we would have considered multiple competing frameworks simultaneously in this work. Last, we did not consider the important effect of patients’ preferences on the use of these treatments.

Implications

The overall results of this study suggest that the uptake and use of evidence-based therapy for PTSD were limited, despite a thoughtful, well-resourced, and highly motivated implementation effort. The scope of the VA efforts to promote use of these treatments was likely larger and more developed than most implementation efforts. Based on our work, we believe that a number of aspects of the facilitation efforts could be further emphasized. Our results suggest that during training, providing therapists with an opportunity to participate in trial treatment is essential. This step also requires that patients are willing and available to participate. Clinical experience with the treatments was the most important predictor of the delivery of evidence-based therapy for PTSD. Designing training to provide a more prolonged contact with the training and facilitation team seems warranted. In this study, sites that reported more than a year of contact were more likely to use evidence-based therapy than sites that had less than a year of contact. We suggest that training activities focus relatively specifically on the desired treatments. Other aspects of professional growth and development seem to detract from the effect. Last, customizing the training to therapists is important. Applying a single approach to all therapists may, for example, bore those with experience with other forms of CBT and overwhelm those with no CBT exposure.

Guidance based on our findings can be of immediate help for those undertaking similar efforts to implement evidence-based therapy in mental health settings. However, the overall lack of effectiveness of these implementation strategies suggests that additional investment in researching implementation in mental health settings is warranted. Efforts to determine optimal methods to promote the use of new types of mental health services deserve similar investments. An important first step would be a detailed description of successful implementation in mental health settings. These successes may provide important clues about the steps needed for effective spread. Ultimately, implementation and dissemination methods need testing in rigorous trials. Unsuccessful practices should be eliminated, and promising approaches should be compared directly. Last, the relative ease and difficulty of spreading different types of mental health interventions deserve more study. Understanding which types of intervention are easier and harder to spread could foster greater investment in approaches that are easier to disseminate.

This study was unable to validate most of the domains of the PARiHS framework, and further research regarding validation of this framework is warranted. Other implementation frameworks, such as the Combined Framework for Implementation Research, show promise in behavioral health implementation research and may have revealed more about factors predictive of implementation (35). More work is needed to determine the applicability of this framework and other implementation models to other mental health settings.

Conclusions

Our work suggests that a major implementation effort aimed at promoting use of evidence-based psychotherapies for PTSD had a modest effect in one region. Nevertheless, some factors predicted the extent to which clinics used the treatments. Overall, it seems that the implementation framework did not predict the clinics’ use of evidence-based therapy for PTSD. Further work is needed to determine the factors that might drive use of these treatments in clinics. Patient preferences were also not explored in this study and could be a focus of future work.