Homeless Veterans' Satisfaction With Residential Treatment

The use of client satisfaction as a measure of quality of mental health services is receiving increasing emphasis. In addition to being a desirable result of treatment, client satisfaction has been associated with clinical measures of treatment outcomes (1). Satisfaction with services is influenced by characteristics of both the client—for example, demographic and clinical variables (2)—and the treatment program (3,4).

Currently, relatively little is known about homeless individuals' satisfaction with mental health services or the association between satisfaction and measures of treatment outcome. In a study of homeless mentally ill individuals receiving community-based case management, Morse and colleagues (5,6) showed that similar treatment variables—more client contacts with the program and the provision of supportive services—were associated with both higher client satisfaction and better outcomes, as measured by an increase in the number of days clients were in stable housing and a reduction in psychiatric symptoms. The extent to which these findings generalize to homeless persons in other types of programs and to studies using other outcome measures is not known.

This study had two purposes: to examine demographic, clinical, and environmental correlates of client satisfaction and to assess the association of client satisfaction with treatment outcomes in a group of homeless veterans receiving residential treatment services.

Methods

Sample and data collection

The survey sample included 1,106 veterans treated in residential treatment facilities under contract with the Department of Veterans Affairs Health Care for Homeless Veterans (HCHV) program (7), a national outreach and case management program. VA clinicians distributing the survey at each facility were instructed to give the survey to the first 30 veterans admitted to residential treatment in 1996. Veterans were given the survey after approximately two weeks in treatment and completed it independently. Only 20 female veterans completed the survey, and thus the analyses focused on the 1,048 male veterans for whom both intake and discharge information was available.

Measures

Client satisfaction.

Veterans' satisfaction with residential treatment services was assessed by four items based on the Client Satisfaction Questionnaire (8). They were "How would you rate the quality of the services you have received in this facility?" (with answers ranging from excellent to poor); "If another veteran or a friend were in need of similar help, would you recommend the program at this facility to him or her?" (ranging from definitely not to definitely yes); "How satisfied are you with the amount of help you have received here?" (ranging from completely satisfied to very dissatisfied); and "If you needed help again and had a choice of where to go at no cost to you, would you return to the program at this facility?" (ranging from definitely not to definitely yes).

All items were recoded to reflect a 5-point scale, from least satisfaction to most satisfaction. The four scores were averaged and treated as a single score for analyses. The Cronbach's alpha for the satisfaction scale was .87.

Perceived environment.

Veterans in the survey sample also completed the Community-Oriented Programs Environment Scale (COPES) (9,10). This well-validated 100-item instrument consists of ten subscales on which they rate their perceptions of the treatment environment. The reliabilities of the subscales for the sample of 1,048 veterans are shown in parentheses. The involvement subscale rates how active participants are in the day-to-day functioning of the program (Cronbach's alpha=.83). The support subscale rates how much participants help and support each other and how supportive staff members are toward participants (Cronbach's alpha=.79). The spontaneity subscale assesses how much the program encourages open expression of feelings by participants and staff (Cronbach's alpha=.58)

The autonomy subscale measures how self-sufficient and independent participants are in decision making and how much they are encouraged to take leadership in the program (Cronbach's alpha=.34). The practical orientation subscale assesses the degree to which participants learn practical skills and are prepared for program release (Cronbach's alpha=.71). The personal problem orientation subscale measures the extent to which participants are encouraged to understand their feelings and personal problems (Cronbach's alpha=.78).

The anger and aggression subscale measures how much program participants argue with each other and with staff, become openly angry, and display aggressive behavior (Cronbach's alpha=.66). The order and organization subscale rates how important order and organization are in the program (Cronbach's alpha=.74). The program clarity subscale rates the extent to which participants know what to expect in the day-to-day routine of the program and the explicitness of program rules and procedures (Cronbach's alpha=.68). The staff control subscale rates the extent to which staff members use measures to keep participants under the necessary level of control (Cronbach's alpha=.41).

Background characteristics and residential treatment outcomes.

Background characteristics and clinical diagnoses of the veterans were collected at the time of intake to the HCHV program in a structured interview conducted by trained program clinicians, predominantly social workers and nurses. Demographic characteristics included gender, ethnicity, military history, economic situation, and length of homelessness. Veterans' self-reports of alcohol, drug, and psychiatric problems were based on the Addiction Severity Index (ASI) (11). Two items measuring recent alcohol use were combined to form a single alcohol use score, two items measuring recent drug use were similarly combined, and eight items measuring recent psychiatric symptoms were combined to create a single psychiatric problems score.

The interview also included a section to record clinical psychiatric diagnoses according to DSM-IV criteria (12). These diagnoses were derived from unstructured assessments and were therefore based on clinical judgment.

Measures of each veteran's participation in residential treatment were taken from discharge reports completed by each veteran's case manager. The measures included length of stay; discharge status (successfully discharged, discharged for rule violations, or left the program without staff consultation); housing status at discharge; employment status at discharge; clinicians' ratings of improvement (improved or not improved) in the areas of alcohol, drug, psychiatric, and social-vocational problems; and arrangements for follow-up treatment in these areas.

Data analysis

To determine the associations between client satisfaction and demographic and clinical characteristics and perceived environmental factors, linear multiple regression analyses were conducted on the satisfaction scale. The regression model contained several covariates, including age at the time of the interview (continuous) and ethnicity (separate dummy variables were coded 1 for African American and Hispanic).

Other demographic and clinical covariates were included by forward selection (p<.10) from a set of variables. They included three dichotomous demographic variables—currently married, receiving public support payments, and income of $500 or more in the month before intake; three continuous variables—days homeless, days institutionalized, and days worked in the month before intake; the abbreviated ASI scores for alcohol, drug, and psychiatric problems (all continuous variables); and diagnoses of substance abuse or psychiatric disorders made by clinicians at intake, including alcohol abuse or dependence, drug abuse or dependence, schizophrenia, other psychotic disorder, mood disorder, personality disorder, posttraumatic stress disorder, and adjustment disorder (all dichotomous variables).

After selection of demographic and clinical covariates, the ten COPES subscale scores were put into the model.

To determine the associations between client satisfaction and treatment outcomes, linear multiple regression was conducted on the continuous measure of residential treatment (length of stay), and logistic regression was conducted on dichotomous measures of residential treatment participation (successfully discharged from the program; housed at discharge; employed at discharge; improvement in alcohol, drug, psychiatric, and social-vocational problems; and follow-up arrangements made for alcohol, drug, psychiatric, and social-vocational problems). All models included the main independent variable (the client satisfaction scale) plus covariates for age and ethnicity. The control variables listed above were included in the model for each dependent variable based on statistical significance (p<.10) following forward selection.

Results

Sample characteristics

Table 1 compares demographic and clinical characteristics and treatment outcomes of the 1,048 veterans who completed the survey and 1,248 veterans who were treated at the facilities during the same time period but were not given the survey. Consistent with the larger HCHV population (7), about 50 percent of the veterans in this study were African Americans, and 44 percent were whites. The mean age was approximately 44 years for both the survey participants and the nonparticipants. Fewer than five percent of the veterans in either group were married at the time of intake, and on average they had worked less than four days in the month before intake.

At intake, about three-quarters of both the survey participants and the nonparticipants were diagnosed as having alcohol dependence or abuse, and approximately 60 percent were given a diagnosis of drug dependence or abuse at intake. A large proportion of veterans had both drug and alcohol diagnoses. Over half of the veterans in both groups had at least one of the psychiatric disorders listed in Table 1. Collectively, the characteristics of the two groups reflect the high level of psychiatric and substance abuse problems among veterans in the HCHV program and their severe economic disadvantages.

Survey participants were generally similar to nonparticipants in demographic and clinical characteristics. Compared with nonparticipants, the survey participants had a longer mean length of stay in the program and uniformly better treatment outcomes. This bias occurred because the survey was not given until a veteran had been in treatment for two weeks. Therefore, all veterans with stays shorter than two weeks were nonparticipants. The veterans with very short lengths of stay were less likely to have successful treatment outcomes. Although it does not undermine the validity of the current analyses, this bias must be noted.

Associations with satisfaction

Satisfaction with residential treatment services was reasonably high among the homeless veterans who completed the survey. For example, 64.9 percent of the veterans reported that they definitely would recommend the program to a friend, and an additional 30 percent said they probably would recommend it. In addition, 56.9 percent said they definitely would return to the program if they needed help in the future, and an additional 33.1 percent said they probably would return. The mean±SD overall score on the 5-point satisfaction scale was 4.06±.80.

The results of the regression model of satisfaction predicted by client characteristics and their perceptions of the treatment environment are shown in Table 2. No demographic characteristics were significantly associated with client satisfaction. Three of four clinical characteristics related to drug use and previous institutional stays were significantly associated with satisfaction. When the analysis controlled for these characteristics of veterans, several dimensions of the perceived environment as measured by the COPES were related to satisfaction. Client ratings of policy clarity, emphasis on order, client involvement, practical orientation, and peer support were all positively associated with satisfaction. Staff control was negatively associated with satisfaction.

Satisfaction and treatment outcomes

The results of the regression analyses of the association between satisfaction and residential treatment outcomes are shown in Table 3. Only the contribution of the satisfaction scale is presented for each outcome model. Client satisfaction was significantly related to three of the 12 residential treatment outcomes. The strongest association was with length of stay. Greater satisfaction was correlated with longer lengths of stay. Greater satisfaction was also correlated with positive ratings of clinical improvement of drug problems and mental health problems. Greater satisfaction was correlated with being housed after discharge, although this correlation did not achieve statistical significance (p=.0669).

Discussion and conclusions

The results of this study suggest a high level of satisfaction among veterans treated in residential treatment facilities under contract with the Department of Veterans Affairs. Although response bias, or perceived pressure to give favorable answers, cannot be estimated from the current data, the large proportion of responses indicating willingness to recommend the program to a friend and to return to the program in this large sample of homeless veterans is encouraging.

Demographic and clinical characteristics positively associated with satisfaction were related to drug use and recent institutional treatment. The presence of a psychotic disorder other than schizophrenia was negatively associated with satisfaction, although the correlation did not achieve statistical significance (p<.09). The greater satisfaction of drug-using veterans compared with seriously mentally ill veterans is likely due to the situational aspect of drug use and the chronic nature of serious mental illness (4). However, even controlling for these influences on satisfaction stemming from veterans' characteristics, clients' perceptions of the treatment environment were found to bear a strong relation to satisfaction.

The pattern of associations between satisfaction and clients' perceptions of the treatment environment suggests that the veterans viewed as satisfactory programs those that combine clear policy and client support and involvement but that are not overly controlling. The analysis of the association between perceived environment and satisfaction is useful because it offers a framework for how a program might be modified to increase the satisfaction of its clients.

Client satisfaction was associated with longer length of stay but was not significantly related to most other outcomes of residential treatment. It is not surprising that length of stay was strongly associated with client satisfaction scores, in that veterans must have a reasonable level of satisfaction to continue participating in a voluntary program. Veterans' reports of satisfaction agreed with case managers' reports of clinical improvement in the areas of drug and mental health problems. Satisfaction was not related to other measures of treatment outcome, with the exception of a marginally significant association with being housed at discharge. At minimum, these findings show that client satisfaction is not an adequate substitute for other measures of treatment progress and outcome (1).

Limitations of this study should be mentioned. First, it appears that veterans who completed the survey had more favorable outcomes, which may be due to the survey's being given two weeks following admission to the program, precluding survey responses from veterans with very short lengths of stay. This bias may have resulted in higher satisfaction scores than would have been observed in a more representative sample. Moreover, variability in all measures may have been reduced due to this bias, limiting the sensitivity of the analyses.

Second, the satisfaction scale used in this study depends mainly on face validity. For example, we do not know how many veterans who said they would recommend the program to a friend actually did so. In addition, this measure addresses only global satisfaction with services rather than separable dimensions, such as access, physical environment, cost, and value. A multidimensional scale might have been more sensitive (1).

Finally, this study used relatively coarse outcome measures that were taken only at discharge from residential treatment. Although VA case managers all received the same training in completing the evaluation instruments, variability among clinicians, such as in judging whether a patient improved in a particular clinical area, may have impeded our ability to detect associations between client satisfaction and treatment outcomes.

Despite these limitations, the main finding—that clients' perceptions of the treatment environment are related to their satisfaction—has implications for program practice. The individual items that constitute the COPES are generally based on readily observable program features. For example, an item measuring policy clarity states "If a resident's program is changed, the staff tell him why." Thus a survey of this type can provide feedback to program staff on the dimensions that clients view as contributing to their satisfaction. Although considerations of client satisfaction must necessarily be subordinate to considerations of clinical effectiveness, they should have a prominent place in program design.

Acknowledgments

The ongoing operation of the Health Care for Homeless Veterans program is under the guidance of Thomas Horvath, M.D., and Gay Koerber, M.A., of the Department of Veterans Affairs mental health strategic health care group. Funding for the program evaluation also comes from the Department of Veterans Affairs. The authors thank Diane DiLella, M.P.H., and Janine Chapdelaine, M.A., for assistance with data collection.

The authors are affiliated with the department of psychiatry of Yale School of Medicine in New Haven, Connecticut. Dr. Kasprow and Dr. Rosenheck are also affiliated with the Department of Veterans Affairs Northeast Program Evaluation Center in West Haven, Connecticut. Dr. Frisman is with the department of psychology at the University of Connecticut in Storrs. Send correspondence to Dr. Kasprow at the Northeast Program Evaluation Center (182), 950 Campbell Avenue, West Haven, Connecticut 06516 (e-mail, [email protected]).

Enlarge table

Enlarge table

Enlarge table