Cost-Effectiveness of Critical Time Intervention to Reduce Homelessness Among Persons With Mental Illness

Homelessness has become a seemingly permanent part of the social landscape in the United States, with more than half a million persons sleeping on the streets or in shelters on any given night and millions more having had some homelessness experience during their lifetimes (1,2,3). Epidemiological estimates have indicated that about one-quarter of homeless individuals have a mental disorder (4). Although various community treatment programs have shown promise for serving homeless persons with severe mental illness (5,6,7,8,9,10), controlled studies of attempts to prevent homelessness are scarce (11,12,13,14,15).

Since Weisbrod and colleagues' groundbreaking cost-benefit analysis (16) of the Program for Assertive Community Treatment, a number of cost-effectiveness studies have been conducted with mentally ill populations. With two exceptions, such evaluations have focused on housed persons and have measured only mental health treatment costs (17,18,19,20,21,22). These studies also have tended to analyze costs independently from outcome. More recent studies have extended the cost definition to social costs (23,24,25) and have adopted statistical techniques that allow for cost-effectiveness analysis (23,24). An even smaller number of cost-effectiveness studies have focused on evaluating treatments that reduce homelessness among persons with severe mental illness (17,26,27). Reviewing these articles, Rosenheck (28) found that the interventions were generally associated with improved housing outcomes but also resulted in higher costs—or, in the absence of cost data, higher service utilization.

The critical time intervention program, a nine-month case management program, was designed to prevent homelessness by enhancing the continuity of care for persons in their transition from a shelter to community living. Over an 18-month follow-up period, those receiving critical time intervention services spent an average of 30 nights homeless, compared with 91 nights for those receiving usual services (29). Furthermore, participants in the intervention program had greater improvement in negative psychiatric symptoms (30).

Employing an approach of Weisbrod and colleagues (17), which was refined by Wolff and colleagues (31), the study reported here compared the cost-effectiveness of the two conditions in this randomized clinical trial—critical time intervention services and usual services. We hypothesized that the critical time intervention program would prove to be cost-effective in that any additional costs associated with this innovation would be offset by better outcomes, such as an increase in nonhomeless nights for individuals in the experimental group.

Methods

Participants

A total of 102 enrollees of a psychiatric program in a men's shelter in New York City were approached from 1991 to 1993 to participate in the study, which was approved by the institutional review boards of the New York State Psychiatric Institute-Columbia University department of psychiatry and the Columbia Presbyterian Medical Center and had a federal certificate of confidentiality. After providing a complete study description to the participants, written informed consent was obtained. All participants had severe mental illness, and all had a prearranged housing placement before random assignment to a treatment condition.

From 1991 to 1993, a total of 96 men were recruited and randomly assigned to the two treatment conditions—the critical time intervention program or usual care. As shown in Table 1, most were African Americans (74 percent), more than half were older than 35 years (60 percent), two out of five (41 percent) had completed high school, and more than two-thirds (68 percent) had a diagnosis of schizophrenia according to DSM-III-R criteria (32) on the basis of a research diagnostic interview. All study participants had experienced at least one psychiatric hospitalization, and 36 percent had five or more hospitalizations. Differences in demographic and diagnostic characteristics between the critical time intervention and usual care participants were not statistically significant.

Interventions

Critical time intervention is a time-limited case management program for individuals with severe mental illness. Designed for transitions from various institutions to the community, the intervention, as used in this study, was intended to prevent recurrent homelessness for individuals discharged from a shelter to community living.

The goal of the critical time intervention program is to enhance continuity of care through two means: by strengthening the individual's long-term ties to services, family members, and friends and by providing emotional and practical support during the critical time of transition. Rather than providing services long term, critical time intervention workers create or coordinate systems of care among various service providers, family members, and others in the community. In this study, the intervention was offered for a maximum of nine months. More detailed information about the intervention has been provided by Valencia and colleagues (33).

Participants assigned to the control condition received usual care, which consisted of referrals to mental health, rehabilitation, and other programs in the community. To assist in their transition to community living, study participants who received usual care could contact their shelter case managers for assistance during the 90 days immediately after community placement. These case managers did not visit participants in the community and did not actively arrange for community services beyond the initial discharge plan.

Collection and preparation of data

An attempt was made every 30 days over the 18-month study period to interview each study participant about his housing, income, service utilization, and employment during the previous month. Most interviews occurred in the research office, but some were conducted in hospitals, private homes, jails, or other public places or over the telephone. If the time since the last interview was longer than a month, the participant was asked to reconstruct his residential situation for the entire period. In such cases, detailed information was obtained on where an individual spent each night but not on other activities, such as use of outpatient services or case management or receipt of income transfers. Missing data on income, service utilization, and employment were imputed by using information from an individual's existing interviews, conditioned on his place of residence. Key informants were contacted for the few participants who were lost to follow-up and were questioned as to the participant's whereabouts. On the basis of these responses, reasonable estimates of the total number of nights homeless were made. Participants' reports of the use of high-cost services—such as stays in municipal hospitals, public shelters, or jails or enrollment in an intensive case management program—were validated through registries and by contacting providers.

Cost estimation

A total cost measure was constructed for each participant. The volume for each resource category used by a participant was multiplied by its estimated unit cost. To find the total cost for a participant, these products were summed.

Before this operation, unit costs for each resource category had to be estimated. The wide spectrum of resources included in the analysis demanded that many sources be used in the unit cost estimation. For all resource categories, unit costs were based on costs experienced by providers in New York City for services reported by study participants. If necessary, data were adjusted to reflect 1992 constant dollars.

Accounting data from published reports, such as the New York State Office of Mental Health consolidated fiscal report and Empire Blue Cross/Blue Shield institutional cost reports, were used to estimate service unit costs. In such cases, the unit cost was calculated as the ratio of total operating expenditures to total service units. This approach was used to find the average cost for specialty mental health services and supported housing programs, inpatient psychiatric stays, medical and surgical stays, and psychiatric emergency services.

The types of cost documents described above were not available for medical or substance abuse services. Thus unit costs for these services were calculated from Medicaid total expenditures and number of claims. Unit costs for arrests were based on utilization data for criminal justice resources, such as booking, court appearances, pleas, sentencing, and postsentencing supervision or probation, that are associated with misdemeanor and nonviolent felony offenses stemming from drug use or drug dealing (34)—the types of crimes many participants were charged with during the study period. An estimate of the cost per night in jail was provided by the New York City Department of Corrections.

Data on Supplemental Security Income, Home Relief, and Social Security Disability Income were based on administrative entitlement rules and included administrative costs. Participants living in supported housing used these government benefits to help pay rent. To avoid including both housing costs and the full entitlement amounts, data used in the total cost measure in such cases were adjusted by subtracting rent and, when relevant, board expenses from the entitlement amount. This adjustment resulted in equivalent housing and income costs across all housing possibilities. Private housing costs were obtained from the American Housing Survey for the New York-Nassau-Suffolk Metropolitan Area (35).

Donations of money from friends and family members, food stamps, and pensions were included as reported in the interview. Earned income, although negligible, was treated as a resource and subtracted from costs.

The per-unit cost measure for the critical time intervention program was based on the actual program budget modified to reflect a staffing pattern necessary to conduct the intervention with the stated number of clients. These modifications entailed removing costs for specific research activities unnecessary for replicating the model in other settings, as well as resources shared with a shelter-located mental health program.

The main analysis used an intent-to-treat approach and included data for all 96 participants in the trial. Costs for five participants who did not have sufficient service utilization and income data to make accurate calculations were imputed as average costs conditioned on the group to which they were randomly assigned and on their reported amount of homelessness. Secondary analyses were restricted to the 91 participants for whom more complete income and service utilization data were obtained.

Housing outcomes

Nonhomeless nights, the primary client outcome measure, was defined as the total number of study nights minus the number of nights homeless, that is, spent on the street, in a shelter, or in other public areas. Thus nonhomeless nights reflected the sum of the number of nights spent in supported housing programs, rental apartments, friends' and family members' residences, residential treatment facilities, hospitals, and jail. Because the length of follow-up varied somewhat, housing and cost variables were adjusted to represent a constant of 540 nights in the 18-month study period.

Statistical analysis

Three time periods were considered independently—the first nine months, during which the critical time intervention was delivered; the second nine months, during which all participants received usual services only; and the entire 18-month study period. For the cost measures, two-tailed z-score statistics were used to test the hypotheses that differences in the group means were not zero. This approach was applied to log transformations of the cost variables to correct for their positively skewed distributions (36). Housing outcome measures were compared by using t tests.

The statistical cost-effectiveness analyses were based on net housing stability benefit, the net benefit expected for a treatment (37). In this study, the net housing stability benefit compared the gain expected to result from implementing the critical time intervention program with gains required to justify its cost. Net housing stability benefit can be expressed mathematically as follows: the number of nonhomeless nights minus cost divided by the willingness-to-pay value, in which the willingness-to-pay value is the additional price that society is willing to expend for an additional nonhomeless night. The group with the largest positive net housing stability benefit at each willingness-to-pay value was determined by statistical methods described by Laska and associates (38,39). These methods controlled the error at each willingness-to-pay value (the pointwise error) at ≤.05. The familywise error, .18, was the probability of making a pointwise error at one or more willingness-to-pay values.

In addition, we conducted sensitivity analyses by omitting the data requiring imputations and varying the definition of nonhomeless nights by removing jail time.

Results

Costs across resource categories

Table 2 presents average earnings and costs across selected resource categories for the two nine-month periods among the 91-member subsample with more detailed data. Earned income, representing an addition of resources, was low for both groups.

Study participants incurred the greatest costs in the cost categories of outpatient mental health services (excluding direct costs of the critical time intervention program), acute services, supported housing, criminal justice costs, and public transfers. Participants in the critical time intervention program incurred lower costs than those who received usual care in four of these categories, the exception being use of the criminal justice system. However, these cost differences were not statistically significant. In the first nine months, criminal justice costs for critical time intervention participants averaged $2,373. Participants who received usual care had an average cost of $1,385. In the second nine months of the study, critical time intervention participants averaged $4,098 in criminal justice costs and usual care participants averaged $2,200. Highlighting the difficulty of testing for statistical differences in this context, the standard deviations for criminal costs were more than twice the mean and disproportionately larger for the critical time intervention group, especially in the first period. Even so, the differences were not statistically significant.

In the remaining four high-cost categories, usual care participants incurred higher average costs than critical time intervention participants. Of the remaining categories, shelter costs during the first nine months were significantly greater for the usual care group. Usual care participants averaged $1,270 in shelter costs, compared with $502 for critical time intervention participants, for a difference of $768 (z=−2.363, p=.02). In the second nine months, this trend continued, with usual care participants' shelter costs averaging $1,500 and critical time intervention participants' shelter costs averaging $617, but this difference was not statistically significant.

Total costs and housing outcomes

Total costs and housing outcomes for all 96 participants are presented in Table 3. During the first nine-month period, critical time intervention participants' costs averaged $28,206—including the added cost of the critical time intervention ($3,769 per participant), and those for the usual care participants averaged $25,943. In the second nine months, critical time intervention participants' costs averaged $24,168, and usual care participants' costs averaged $25,781. Over the entire study period, the costs of resources used by the critical time intervention participants averaged $52,374, and the costs of resources used by usual care participants averaged $51,649. None of these cost findings indicated a statistically significant difference between groups.

Critical time intervention participants averaged a total of 259 nonhomeless nights in the first nine months, and usual care participants averaged 235.4 nonhomeless nights, for a difference of 23.6 nights (t=2.17, df=73, p=.033). In the second nine months, critical time intervention participants averaged 248.9 nonhomeless nights, and usual care participants averaged 213.8 nonhomeless nights, for a difference of 35.1 nights (t=2.31, df=89, p=.024). Overall, critical time intervention participants averaged 508 nonhomeless nights, and usual care participants averaged 450 nonhomeless nights, a difference of 58 nights (t=2.64, df=89, p=.01).

Cost-effectiveness analysis

For the first nine months, the second nine months, and the entire study period, for willingness-to-pay values greater than $457, $120, and $152 per nonhomeless night, respectively, the critical time intervention group had significantly greater net housing stability benefit values than did the usual care group. Below these willingness-to-pay values, the net housing stability benefit values for the critical time intervention and usual care groups were indistinguishable. There was no willingness-to-pay value for any period in which the usual care group had a significantly greater net housing stability benefit than critical time intervention group.

The results of the sensitivity analyses were similar for the two nine-month periods and the entire study period, although in most cases the willingness-to-pay values above which the critical time intervention was cost-effective increased relative to the analyses reported above. For the first nine months, the critical time intervention was not more cost-effective than usual care for any willingness-to-pay value.

Discussion

The critical time intervention program helped homeless men with severe mental illness stay in housing (29). However, given the substantial amounts of societal resources used by this population, adoption of a new innovation must also be cost-effective. Over 18 months, the usual care and critical time intervention groups had a mean of $51,649 and $52,374, respectively, in expenditures for acute care services, outpatient services, housing and shelter services, criminal justice, and transfer income. Concurrently, they experienced a substantial amount of homelessness—an average of 90 days for the usual care group and 32 days for the critical time intervention group—despite beginning the study with newly obtained housing. The overall results showed that if a decision maker is willing to spend an additional $152 per nonhomeless night, critical time intervention is cost-effective relative to usual care.

The first nine months of the study period was considered separately to test the cost-effectiveness of the critical time intervention program while the intervention cost of $3,769 per person was incurred. It is no surprise that no cost savings were observed for the critical time intervention group during that period. Even considering that all study participants received housing assistance, it was the improved housing stability of the critical time intervention group that explained the finding that usual care was not significantly more cost-effective than the critical time intervention. If only this period was considered, the critical time intervention program would not be considered cost-effective unless a decision maker's willingness to pay was more than $457 per nonhomeless night.

The second nine months of the study period are of interest because data from that period can be used to consider whether the impact on costs of a time-limited program can extend beyond the intervention period. In combination, cost and outcome data for the second nine months provided evidence of a sustained impact from participation in the critical time intervention program. During the second nine months, costs averaged $1,613 less per person in the critical time intervention group than in the usual care group. This difference may be the result of improved linkage to community services for the critical time intervention group, resulting in fewer episodes of extreme emotional duress and thus reduced reliance on acute care services.

Cost studies inevitably include data with substantial variation, reflecting the wide variety of possible resource use patterns. In this study, reported resource use reflected a range of experiences, from those of individuals who lived on the streets and had never used services to those of individuals who became appropriately engaged in services or who spent practically the entire study period in the hospital or in jail. Such variation together with skewness argues against the statistical assumption of normality. However, use of the net housing stability benefit for cost-effectiveness eliminates this problem, even for modest-sized samples, and allows an unbiased estimation of the true net benefit for the critical time intervention group and the usual care group across different possible willingness-to-pay values (37).

The critical time intervention group was "unlucky" in that two of the randomly assigned participants experienced prolonged prison stays, although the incidence of incarceration was similar across the two groups. Even with these high costs factored in, the critical time intervention program was found to be cost-effective. The two individuals were incarcerated shortly after enrollment and remained so until the end of their study participation. Their criminal charges were similar to those of other participants who received substantially shorter sentences. Perhaps controlling for parole status at baseline—a variable for which information was not collected in this study—would have resulted in a more equal distribution of such individuals. These longer-than-usual jail stays exacerbated costs and made the results of the sensitivity analyses weaker.

In addition, if the costs of being homeless on the street had been included in the analysis, the finding of the intervention's cost-effectiveness may have been more dramatic, because more participants in the usual care group experienced this type of homelessness. However, apart from the shelter costs and criminal justice costs that were included in the analysis, we were unable to assign a monetary cost to this being homeless on the streets.

Three limitations must be noted. First, the cost variable was dependent on self-reported economic activity rather than drawn from computerized data sources. Use of self-reported data for this variable permitted the development of a more comprehensive cost measure but raises concern about data reliability. Second, the study was limited to one setting characterized by a rich array of services—New York City. Future studies will need to examine this model in other settings. Third, time has elapsed since the data were collected. Therefore, policy makers and program planners may want to assess whether the delivery system is sufficiently unaltered for our results to remain pertinent. However, the critical time intervention is now being used and evaluated in a number of different settings. Newer data will inform us if it remains cost-effective.

Conclusions

Unlike previous cost studies, this study evaluated a program for individuals with severe mental illness and homelessness histories, used a social-cost perspective, included outcome measures reflecting homelessness, and analyzed cost-effectiveness by using a net benefit approach. The critical time intervention program both reduced homeless nights and achieved this goal at lower costs than did usual treatment. Although difficult to conduct, such cost analyses—especially those that include a comprehensive accounting of resource utilization—can yield rich information for policy makers and program planners.

Acknowledgments

This work was supported by National Institute of Mental Health grants MH-48041, MH-51359, and MH-56299-01; the Center for the Study of Issues in Public Mental Health; the Aaron Diamond Foundation; and the Lieber Center for Schizophrenia Research. The authors thank Eugene M. Laska, Ph.D., Morris Meisner, Ph.D., Carole Siegel, Ph.D., Joseph Wanderling, M.S., Robert A. Rosenheck, M.D., Carol T. Mowbray, Ph.D., and Ya-Chi Wu for their contributions. The authors dedicate the paper to the memory of Richard Jed Wyatt, M.D., whose long-standing interest in this topic inspired their analysis

Dr. Jones is a research scientist with the Center for the Study of Issues in Public Mental Health at the Nathan S. Kline Institute for Psychiatric Research, 140 Old Orangeburg Road, Orangeburg, New York 10962 (e-mail, [email protected]). Dr. Colson is program director of the Charles P. Felton National Tuberculosis Center at Harlem Hospital in New York City. Dr. Holter is assistant professor in the School of Social Work at the University of Michigan in Ann Arbor. Dr. Lin is a research scientist with the statistical science and epidemiology laboratory at the Nathan S. Kline Institute for Psychiatric Research. Mr. Valencia is a senior researcher with the Parnassia Psychiatric Institute in The Hague. Dr. Susser is with the department of epidemiology at the Joseph L. Mailman School of Public Health at Columbia University in New York City. The late Dr. Wyatt was chief of the neuropsychiatry branch of the National Institute of Mental Health in Bethesda, Maryland.

Enlarge table

Enlarge table

Enlarge table