Assessing the Effectiveness of Recovery-Oriented ACT in Reducing State Psychiatric Hospital Use

Assertive community treatment (ACT), an evidence-based practice for persons with severe and persistent mental illness, has been intensively studied for the past four decades to determine whether it is effective, for whom, and under what circumstances. Key components of the model include a mobile, multidisciplinary team with a psychiatrist and a psychiatric nurse; a shared caseload among team members; direct service provision by team members; a high frequency of consumer contact; low consumer-to-staff ratios; and round-the-clock outreach in the community (1). ACT was developed for consumers with severe mental illness who have significant difficulty with independent living, high service needs, and repeated psychiatric hospitalizations (2)

Since the early 1970s, more than two dozen randomized trials have been conducted throughout the United States and abroad to evaluate the effectiveness of ACT (3–6). Most of the early U.S. trials found that ACT did not consistently reduce symptoms and improve quality of life. However, ACT was effective at engaging hard-to-reach consumers, improving housing stability, and reducing hospitalization (7). The evidence for ACT as an effective alternative to psychiatric hospitalization led to its gradual dissemination throughout the United States, Canada, Western Europe, and Australia.

In 1999, Latimer (8) published the first meta-analysis of ACT economic outcomes, which synthesized findings from 19 randomized studies and 15 nonrandomized studies of ACT in the United States, Australia, Canada, England, and Sweden. Findings indicated that the most reliable cost offset of ACT treatment was reduced psychiatric hospital use, with high-fidelity ACT teams achieving the greatest reductions. Latimer also suggested that if ACT is to be cost-effective it should target consumers with at least 50 days of hospitalization in a calendar year.

Concurrently, findings from randomized trials of ACT conducted in the United Kingdom failed to show the superiority of ACT over standard case management as in the United States (9–15), leading to commentaries about the utility of ACT (16–18). Proponents and skeptics offered several explanations to account for these discrepancies, but the commentaries overlooked Latimer’s work, which suggested that high baseline rates of psychiatric hospitalization were key to the effectiveness of ACT compared with usual care.

Two meta-analyses have reexamined these issues. First, Burns and colleagues (19) conducted a meta-regression of randomized trials involving 42 community treatment teams for persons with severe mental illness. In a major step forward, this study obtained data from the Cochrane Registry of individuals who participated in the trials, as well as more complete information directly from the investigators about treatment program characteristics. All of the teams provided intensive case management (ICM), a category that included ACT as well as case management services with larger staff-to-consumer ratios (up to 1:20). The main finding was that reduced hospitalization was most likely to occur when trial participants had high psychiatric hospital use at baseline (before the trial) or when the control group had high use. Other findings suggested that ACT teams were most successful in reducing hospitalization but that these effects did not require teams to have the enriched staffing associated with ACT.

Second, Dieterich and colleagues (20) undertook a sequel to the study by Burns and colleagues (19) as part of a Cochrane Collaborative systematic review. The objective was to compare ICM (caseload <20) with each of two control groups—usual care and traditional case management (caseload >20). This study used pooled individual-level data (N=3,595) from 24 randomized trials and expanded the meta-analysis to evaluate baseline hospitalization rates, fidelity to the ACT model, and several other outcomes. The main results indicated that ICM significantly reduced hospital days compared with usual care, that ICM overall did not have a significant advantage over traditional case management in reducing hospital days, and that the more that ICM adhered to the ACT model, the better it was at reducing hospital days, especially if baseline hospital use was high.

The first and third findings support Latimer’s earlier assessments about the effectiveness of ACT. However, in a recent commentary, Burns (21) singled out the second finding as the most relevant for the United Kingdom, where there has been much greater use of community mental health teams with larger caseloads and leaner staffing than specified in the ACT model. Burns concluded that the extra staffing and smaller caseload costs of ACT are not justified given the lack of clear benefits of ACT in reducing hospitalization when it is compared with similarly organized but more economically staffed community mental health teams. All things considered, Burns suggested that ACT is no longer needed in today’s mental health services array.

In the U.S. context, however, we believe that an obituary for ACT is premature. Spurred on in part by the evidence-based practice movement (22–26), ACT teams have continued to expand throughout the United States during the past ten years, often on a statewide basis in places such as New York, Indiana, North Carolina, and Washington. An important development during this period was the growth of recovery-oriented approaches to community care for persons with severe mental illness as showcased in the influential final report of the President’s New Freedom Commission in 2003 (27). ACT teams have begun to embrace many of these recovery principles (28,29).

We had the opportunity to examine the outcomes of recovery-oriented ACT in the context of Washington State’s PACT program, a statewide network of ten new teams launched in 2007 to reduce the use of state psychiatric hospitals (30). (Sponsors named the program PACT after the original Stein and Test [2] project in Madison, Wisconsin, and we use PACT in this article when referring specifically to the ACT teams in Washington State). State funding provided $2.4 million for development and training in state fiscal year 2007 and an additional $10.4 million per year for operational expenses.

The Washington State Mental Health Authority adopted the National Program Standards for ACT (31). In addition to ACT training, the teams underwent both intensive training and continuing education in person-centered and recovery-oriented services provided by the University of Washington’s Institute for Mental Health Research and Training and a panel of national experts (32). These sessions focused on strengths-based assessment, individualized and person-centered planning processes, promotion of a culture of recovery within ACT, and use of peers as recovery mentors. Training also centered on helping teams to develop consumer choice and independence while avoiding coerciveness. Within 18 months of start-up (33), the teams achieved relatively high ratings on the Tool for Measuring ACT (32), an update of the Dartmouth Assertive Community Treatment Scale fidelity measure (34) that incorporates recovery-oriented items.

To date, the emerging literature in regard to ACT and its alignment with recovery principles is largely conceptual, with few empirical studies of outcomes associated with recovery-oriented ACT teams (35). Our study of Washington State’s PACT experience addressed two as yet unanswered questions: Is the combination of ACT and recovery-oriented practices more effective in reducing state psychiatric hospital use than usual care arrangements? Do ACT teams that embrace recovery principles and practices generate any greater reductions in state psychiatric hospital use than traditional ACT teams as portrayed in the published literature? Personal recovery is thought to promote greater responsibility for one’s own care and a greater alliance between the consumer and ACT staff in clinical decision making. If so, recovery-oriented practices may affect decisions about when consumers need inpatient care and, once they are hospitalized, decisions about length of stay and discharge readiness. Companion papers address the timing of PACT effects (33) and their relationship to fidelity on the Tool for Measuring ACT (36).

Methods

Setting

The State Mental Health Authority in Washington operates two psychiatric hospitals for adults. Western State Hospital, an 806-bed facility located in Takoma, serves the 19 counties west of the Cascade Mountains, the most densely populated region, which accounts for 78% of the state’s 6.5 million population. Eastern State Hospital, a 287-bed facility located in Medical Lake, a small community 20 miles southwest of Spokane, serves 21 predominantly rural counties in the eastern section of the state. At the time of this study, both hospitals were accredited by the Joint Commission and certified by the federal Centers for Medicare and Medicaid Services. Seven of the new PACT teams were situated in the western region, and three were in the eastern region. Nine of the teams were created de novo; the tenth was an expansion of a preexisting program. Each team was operated by a private agency under contract with one of the 13 Regional Support Networks that are responsible for funding and managing local community mental health services in Washington State (37).

Design and data

The research was conducted with the approval of institutional review boards at the Washington State Department of Social and Health Services and at the University of North Carolina at Chapel Hill. It was based exclusively on statewide, linked administrative data obtained from the Research and Data Analysis Division of Washington State’s Department of Social and Health Services (DSHS). The database included DSHS client service contacts since 2000, Medicaid claims data, demographic characteristics, diagnostic information, and service costs (38).

We analyzed the administrative data as a quasi-experiment using a pre-post case-control (difference-in-differences) design with propensity score–matched consumers as the control group (39,40). A total of 450 PACT consumers with a history of state psychiatric hospital use were compared with a usual-care control group of 450 propensity score–matched consumers who had a history of state psychiatric hospital use but were not enrolled in PACT. The two study cohorts were assembled in several steps. First, the intervention group (N=636) was limited to PACT recipients (N=450) who had any state hospital use in the six years before PACT enrollment (July 2007 or later). Individuals who were enrolled in PACT (N=186) but who did not have any prior state hospital use were excluded from the intervention group. Second, we identified a potential comparison group (N=6,665) of people with a diagnosis of psychotic or affective disorders who were also state psychiatric hospital users between January 2001 and June 2007. Following Estee and colleagues (41), we randomly assigned index (start) dates each month during the study period according to the proportions observed in the PACT sample. That is, if 4% of the PACT sample was enrolled in PACT in July 2007, we “enrolled” a randomly selected 4% of the control sample in July 2007. These start dates were used to create the pre-post time periods for our analyses.

Third, we ran separate logistic regression models to estimate the predicted probabilities or propensity scores of participating in PACT in the eastern and western regions. Covariates in the propensity score (logit) model included more than 30 baseline measures of demographic characteristics, diagnoses, hospital and other service use, and arrests. For measures of hospital use, we created quarterly measures of days spent in each of the two state psychiatric hospitals in the year before the index date (annual measures were used instead for some of the less common measures in the eastern cohort), then annual measures of hospital days beyond the first year. Selected variable interactions were included in the region-specific models until each model passed three specification tests: the Hosmer-Lemeshow test, the link test, and the correlation test between residuals and predicted values (42).

We examined results from two sets of nearest-neighbor matches on propensity scores. In the first set of analyses, we required that matched persons in the control group be within one-fourth of the standard deviation of the propensity score of each PACT participant. Because a number of PACT participants had no “neighbor” within this caliper radius, our sample dropped from 450 to 364 PACT participants. Most of the PACT participants who did not have a close match were from the high end of the propensity distribution, which meant, ironically, that those who looked most like PACT participants were dropped from the analyses. Because results from this sampling approach may not generalize to all PACT participants, we also ran analyses using the full set of PACT participants and their nearest match, regardless of the distance in propensity scores. A Hotelling test of differences between the set of variable means detected no difference in variable means between the control and PACT participants in the expanded sample. Chi square tests of proportions and t tests of individual variable means were also statistically insignificant with one exception: the annual number of medication management minutes before PACT had a significant mean difference between PACT participants and control group members (113 versus 76 minutes; t=–2.36, df=1, p=.019).

Measures

The following dichotomous (0 or 1) variables were created: male, white, African American, Asian, Native American, other race, Hispanic, schizophrenia, affective disorder, and substance use disorder. Each outcome measure is defined below. Costs are reported in 2010 dollars, having been adjusted for inflation by the Gross Domestic Product deflator.

State psychiatric hospital utilization.

Hospital days were derived directly from admission and discharge dates, and costs were determined by multiplying the number of hospital days in each year by an annual, hospital-specific per diem rate.

Local general hospital psychiatric utilization.

Hospital days were derived from admission and discharge dates.

Local crisis stabilization utilization.

Admissions to local crisis stabilization units were derived from admission and discharge dates. These short-term, community-based, psychiatric crisis stabilization units were developed several years earlier in the western part of the state to serve consumers in need of acute psychiatric care and to reduce the utilization of Western State Hospital. These units were not available in the eastern part of the state during the study period.

Local hospital emergency department utilization.

Visits to local general hospital emergency departments for psychiatric and medical reasons were included.

Arrests.

Gross misdemeanor and felony arrests were enumerated from data supplied by the Washington State Patrol.

Analyses

All data were pooled across the ten PACT teams and collapsed to the person-year level, with each observation reflecting the use of services during that year. Years were defined based on the PACT enrollment date or simulated start date for participants in the control group. Partial years were included; a covariate controlled for the actual number of months in each year that were observable. Up to three annual observations in each of the pre and post periods were included in the analysis.

Generalized estimating equations (GEE) assessed the marginal effects for PACT participants and usual care participants on each of the outcomes defined above. In a second set of analyses, we examined the effects of PACT versus usual care on a subsample of PACT participants and those in the control group who were high utilizers of the state hospital, which we defined, following Dieterich and colleagues (20) in their Cochrane review, as consumers who had, on average, at least 96 days in the hospital during the two years before their start date (an average of four days per month). We reran the GEE models described above with an additional indicator of high or low use that interacted with the PACT and post period variables. We report the average marginal effects of PACT by high and low baseline state hospital use.

We conducted exploratory analyses on a composite cost measure that sums up costs of psychiatric stays in state hospitals, local general hospitals, and crisis stabilization units. Detailed cost data were not available for these measures except for state hospital costs, and approximations were used from available data. In particular, we used the per diem rate ($1,166) for Harborview Medical Center, the county general hospital serving the greater Seattle area, from February 10, 2010, to estimate costs of inpatient psychiatric stays; because this per diem is relatively high compared with other hospital per diem rates in Washington, in separate analyses we used the statewide modal per diem of $863. We used an estimate of $600 for visits to crisis stabilization units (David Mancuso, personal communication, March 2011). Information on outpatient service use and medication use were not available in our data and thus represent important exclusions to our cost estimate. We ran the composite cost models in each sample and generated estimates by high and low baseline use.

Outcome models were run on the matched samples. Continuous variables, such as state hospital costs, were run with a log link and a gamma distribution; binary variables used a logit link and a binomial distribution; and count variables, such as hospital days, used a log link and a Poisson distribution. All models were run with an autoregressive correlation structure using Stata, version 11, to account for the dependence of repeated (annual) observations on individuals. Although there was some evidence of clustering at the team level (intraclass correlation coefficient=.01121; 95% confidence interval=.00003–.02239), the explained variance (R²=.01) based on a large sample of person-months (N=13,343) was extremely small and essentially ignorable for these analyses.

Results

Table 1 summarizes data on baseline characteristics and service use for PACT and control group participants before PACT enrollment; data are presented for both the restricted samples (N=364) and the full nearest-neighbor samples (N=450). Overall, the four samples were closely balanced on these observed variables. The overall mean age was about 41, two-thirds were male, and slightly more than three-quarters were white (consistent with the general population of Washington State). Nearly all participants had a schizophrenia diagnosis, slightly more than half had a substance use diagnosis, and nearly three-quarters had also been diagnosed as having an affective disorder. In the two years before baseline, nearly two-thirds of participants had used a state psychiatric hospital, and the average number of state hospital days ranged between 100 and 113. The use of local hospital services at baseline was much less frequent than use of state hospital services. Some 13%–16% of the sample used local hospital psychiatric units, 9%–20% used crisis stabilization units, and 23%–25% used emergency departments. In addition, 28%–35% had been arrested at some point in the two-year baseline period.

State hospital use

GEE models showed substantial reductions in use and costs of state psychiatric hospitals as a result of PACT participation (Table 2). Annual reductions associated with PACT were estimated to be $12,699 (in 2010 dollars) in the restricted sample and $11,257 in the full sample. This reduction was achieved not by keeping PACT participants entirely out of the hospital but by a reduction in the days spent in the hospital in each year. That is, we found no evidence of a difference between PACT participants and control participants in the probability of having any state hospital use; however, we found a reduction of 33 days per person per year in hospital utilization (a 32-day reduction for the full sample) (p<.01) (Table 2).

Local service use

In general, reductions in state hospital use were not offset by increases in local hospital use. Although we found evidence of a small increase (3%) among PACT participants in the probability of using local hospitals for psychiatric admissions, it was significant only in the full sample, as shown in Table 2. The estimated increase in the number of local hospital days associated with PACT was less than one per year (p<.01). We did find a fairly large increase of about 13% (p<.01) in the probability of using emergency departments each year after PACT enrollment; however, the estimated increase in the number of emergency department visits each year associated with PACT was only about one (.87 to 1.17; p<.01). Further, we found a moderate 6% (p<.01) increase in the probability of using a crisis stabilization unit each year after PACT enrollment, and the estimated increase in the number of days of use each year of crisis stabilization units associated with PACT was about two (2.14 to 2.41; p<.01). We found no evidence that PACT affected the annual probability of arrest or the number of arrests for gross misdemeanors or felonies per person.

High versus low state hospital users

The average marginal effects from the GEE models of state hospital utilization status at baseline (high utilizers versus low utilizers) are reported in Table 3. Expenditure reductions in state psychiatric hospital use were concentrated among the PACT participants who had high state hospital use at baseline, and the reductions ranged from $16,719 to $19,872 per person per year. In contrast, we did not find evidence of these expenditure reductions among PACT participants who had low state hospital use at baseline, and we estimated that the number of state hospital days among these participants may have increased by 4.8 to 9.1 days per year on average (p<.01) during the PACT period. However, we found greater increases in local hospital use among those with high state hospital use at baseline, although the magnitude of the difference was small (.93 to 1.25 more days in a local hospital each year compared with .37 to .73 more days for those with low baseline use; p<.01). The effects of PACT on use of crisis stabilization units also varied by baseline status, with a larger increase in use of these units among individuals who had low baseline hospital use (2.86–3.55 days per year) than among those who had higher levels of baseline hospital use (1.84–1.90 days per year; p<.01). Arrests were largely homogeneous between the two baseline status measures.

Composite costs for psychiatric stays

Our exploratory analysis of the composite costs of stays in state hospitals, local hospital inpatient psychiatric units, emergency departments, and crisis stabilization units indicated that the reductions in state hospital costs were partially offset by increases in the costs of local services. Although our findings indicated a net reduction in the composite costs for PACT participants compared with matched control group participants, the magnitude of the findings decreased to just under $6,000 (data not shown) in the full sample and just over $2,100 and insignificant in the nearest-neighbor sample. When we examined the results by high and low state hospital use at baseline, we found that the magnitude of the cost reductions was somewhat smaller than the estimate of state hospital cost reductions alone for participants with high hospital use at baseline in the nearest-neighbor sample (–$15,762; p<.01) but somewhat greater than reductions in state hospital costs alone in the full sample (–$19,061; p<.01). Among participants with lower levels of state hospital use at baseline, PACT appears to have led to greater composite costs, ranging from $12,218 in the nearest-neighbor sample to $9,525 in the full sample (both p<.01; data not shown in table).

Discussion

Our findings contribute to the mental health services research literature in several ways. First, they allow us to update the cost offsets of ACT on the basis of the Washington State context. Second, they affirm the conclusions of prior meta-analyses that ACT is most cost-effective when targeted to high users of psychiatric hospitals. Third, they demonstrate the ways that administrative data can be used to assess intervention outcomes and, specifically, how they can provide a framework for future impact assessments of ACT. And fourth, they illustrate the positive effects that accrue from recovery-oriented ACT.

Overall, without regard to psychiatric hospital use at baseline, PACT led to an average reduction of 32–33 days per year, for a reduction in state hospital costs of $11,257 to $12,699 per person per year or a total of $4 million to $5.7 million per year when weighted by the size of our study samples. These overall effects were calculated on the basis of a caseload composed of a substantial number of consumers who were low users of state psychiatric hospitals at baseline (42%–44% in the nearest-neighbor and full samples, respectively). These estimates provide useful guidelines for state mental health authorities and other community agencies about expenditure reductions that might be realized from deploying ACT teams in circumstances where teams enroll persons who have severe mental illness but who also have a wide range of hospitalization histories.

The magnitude of these savings was diminished overall, however, when costs of community alternatives to state hospitalization, including local hospital psychiatric inpatient, emergency departments, and crisis stabilization units, were considered in exploratory analyses. Analysis by levels of hospital use before PACT enrollment indicated that increases in use of these alternative services came almost entirely from PACT participants with low levels of state hospital use at baseline. This finding suggests that after PACT enrollment, some consumers with low use of state hospitals at baseline who needed intensive services received them in alternative community treatment settings rather than in state hospitals.

Our PACT findings confirm meta-analyses (8,19,20) indicating that ACT has its greatest effects and is most efficient when targeted to high users of state psychiatric hospitals. Indeed, high users at baseline accounted for all of the reductions in hospital days and for the overall reductions in expenditures, because low users at baseline increased their use of state hospitals, on average, after PACT enrollment. These estimates are helpful in gauging the cost effects of a carefully targeted implementation of ACT that would enroll only consumers who have 96 or more days of state psychiatric hospitalization in the two years before enrollment. Even with this level of expenditure reductions, however, PACT did not break even during our study period. With an annual cost of $10.4 million and an annual savings of $20,000 per person (full sample), PACT would have had to serve 520 high users of state psychiatric hospitals each year to break even, more than twice the number identified in our study (52%, or 234 of 450).

One caveat regarding these dollar reductions is that they refer only to the savings realized from reductions in use of state psychiatric hospitalization minus the costs of increased use of local services; they do not subtract the costs of outpatient service use, medications, housing, public protection, and other community costs from the savings in use of state hospitals. Clearly, a much more sensitive cost-benefit analysis is required to identify the true societal value of PACT.

Several other limitations of our data and methods should also be acknowledged. The administrative data used in this study did not allow us to create a head-to-head comparison between PACT as recovery-oriented ACT and traditional ACT because there were no traditional ACT teams in Washington State. However, we can compare our findings with results reported in a recent Cochrane Collaborative review by Dieterich and colleagues (20) that was based on two dozen randomized trials of ACT conducted mostly between the mid-1980s and mid-1990s. The review found that ACT reduced the average number of psychiatric hospital days per person per month by .86 over 24 months. Our estimated PACT reduction of 32–33 days per person per year translates into an average of 2.67 and 2.75 days per person per month for the reduced and full samples, respectively. Thus the effects of Washington State’s PACT on psychiatric hospital use were more than three times greater than the average ACT effect reported in the recent Cochrane Collaborative review.

This comparison with the Cochrane review does not provide definitive evidence that the performance increments reported here can be uniquely attributed to PACT’s recovery orientation. Yet, what makes this comparison plausible and suggestive is our use of propensity matching. This approach helped control for the effects of other advances during the past decade as alternative explanations for our findings (such as improved psychotropic medications), because both PACT participants and control participants would have been exposed to these influences.

Further, because our findings are based on nonrandomized data, threats to internal validity cannot be totally eliminated. The administrative data we used were limited to demographic characteristics, diagnoses, and a variety of service use counts, and thus there may be important unobserved personal and situational differences between PACT and comparison groups that we were not able to adjust for with propensity matching. PACT assignment was voluntary, and therefore biases associated with self-selection may also exist. Under these circumstances, effects estimated with propensity methods can produce larger differences than those estimated from true experiments (43,44).

Other research has demonstrated that ACT can serve as a platform for additional interventions that effectively address issues of substance abuse (45), involvement with the criminal justice system (46), and employment (47). Further research is needed to demonstrate the value-added benefits of combining recovery-oriented practices with the traditional ACT model (28). Future studies of recovery-oriented ACT must also consider outcomes beyond reductions in psychiatric hospital use that address a range of consumer interests, including wellness, a self-directed life, stable and safe housing, meaningful daily activities, and supportive community relationships.

Conclusions

Washington State’s PACT experience demonstrates the continued relevance of ACT for U.S. communities, especially in settings where state psychiatric hospitals are overused. Future research needs to confirm these findings and consider whether recovery-oriented ACT teams also promote consumer growth and independence.