Multiple Stakeholder Agreement on Desired Outcomes for Adolescents' Mental Health Services

For the past decade health care, particularly mental health care, has paid significantly increased attention to outcome measurement and data-driven accountability (1,2) and to consumer perspectives on the goals and impact of services (3,4). In the context of competition for limited resources, most publicly and privately funded systems now mandate some form of outcome measurement, and many seek consumers' perspectives on their satisfaction with care (4,5). The interaction of these two forces requires increased knowledge of the types of outcomes that consumers and other key stakeholders desire most.

Although discussion about data-driven accountability in mental health is ubiquitous, relatively limited attention has been paid to the methodological complexity of outcome measurement. Many potential outcomes could be measured, and multiple sources of information, or perspectives, on these outcomes could be obtained. Little is known about the outcome priorities of various stakeholder groups, the extent to which these groups share similar outcome priorities, or the level of agreement between triads of the stakeholders—the individual patient, family member, and provider—on their outcome priorities for any given patient.

The limited research that is available on adult mental health services suggests that patients, family members, and providers rarely agree on desired outcomes or their relative importance (3,6). In addition, providers' awareness of their clients' outcome priorities is generally rather poor (6). For child and adolescent services, no studies have compared desired outcomes across stakeholder groups. However, two related studies have examined perceptions of target problems for treatment among parents and children (7) and among parents, children, and therapists (8). Both studies reported relatively poor agreement; for example, only 23 percent of 315 parent-child-therapist triads agreed on any target problems (8).

Disagreement, or even differential prioritization, on desired treatment outcomes obviously presents a methodological dilemma for outcome research—that is, which outcomes should be measured and according to which informant. This phenomenon also presents a clinical dilemma, which may limit the effectiveness of treatment. Consensual goal-setting is a key component of the therapeutic alliance (9,10). Thus a lack of consensus about treatment goals may undermine the patient's or family's alliance with the therapist and have a negative impact on the patient's engagement in, or benefit from, therapy (6,8). Agreement on treatment goals may be particularly challenging in treatment with adolescents, because adolescent patients are usually not seeking treatment on their own initiative. Some evidence suggests that adolescents underutilize services, have high dropout rates, and exhibit fewer benefits of mental health services compared with other age groups (11). The reasons for these discrepancies in service effectiveness are largely unknown, but differences in stakeholders' goals or treatment expectations may be a contributing factor.

Unfortunately, little is known about factors that are associated with stakeholder agreement on the desired outcomes of treatment. In our study, we examined a variety of potential correlates of agreement, including characteristics of stakeholders—adolescents, parents, and therapists—and the treatment. Our study also built on the existing literature by expanding the array of potential desired outcomes beyond symptoms and functioning, so that potential outcomes also included satisfaction, environmental factors (for example, family and school), and service utilization (2). The primary goals of our study were to describe and compare the frequency of desired outcomes for adolescent outpatient mental health services among the three groups of stakeholders; to examine agreement among stakeholder dyads and triads on the desired outcomes for individual youths; and to identify correlates of agreement for stakeholder triads.

Methods

Participants

A total of 170 adolescents aged 11 to 18 (mean±SD age, 13.5±2 years) were interviewed when they entered outpatient mental health treatment at one of two publicly funded mental health clinics in San Diego. All clients who were English speaking and entering a new episode of treatment—defined as not having had outpatient psychotherapy treatment in the previous six months—between 2000 and 2002 were recruited for our study. Of the 223 sequentially solicited participants, 76 percent (N=170) agreed to participate in a longitudinal study examining the measurement of youth mental health outcomes. Data presented here were collected exclusively at the intake interview. Clients who declined participation primarily cited busy schedules and time constraints. Limited data on nonparticipants showed that they did not differ from the participants in age or gender distribution.

Of the 170 participating adolescents, 63 percent (N=107) were male; 45 percent (N=77) identified themselves as white, 18 percent (N=30) as Latino, 18 percent (N=31) as biracial, 14 percent (N=24) as African American, and 2 percent (N=3) as other. Almost all the caregivers were mothers (158 caregivers, or 93 percent), including biological, foster, and adoptive mothers. Half the caregivers (83 caregivers, or 49 percent) reported an annual family income of less than $15,000, and 75 percent of the remainder (N=65) reported an annual family income of $15,000 to $45,000.

In terms of clinical characteristics, the youths in our sample were representative of those in similar clinical samples. For example, the mean T score on the Child Behavior Checklist total problem scale (12) for our sample was 67.3±9.7, compared with a score of 67.9±9.4 for an outpatient treatment sample (13). The mean is 50±10, with higher scores indicating greater symptom severity. The most prevalent diagnoses in our study were mood disorders, attention-deficit hyperactivity disorder, and oppositional defiant and conduct disorder.

Of the 57 therapists who participated in our study, 75 percent (N=43) were women, with a mean age of 32±7.2 years (range, 24 to 53 years). Sixty percent (N=34) identified themselves as white, 18 percent (N=10) as Latino, 14 percent (N=8) as Asian American, 4 percent (N=2) as African American, and 5 percent (N=3) as other. Therapists' mean number of years of experience was 6±5 (range, .6 to 34 years). Forty-four percent (N=25) of the therapists held a master's degree in counseling or social work, 26 percent (N=15) held a doctoral degree in psychiatry or psychology, and 30 percent (N=17) held a bachelor's degree and were currently in master's or doctoral-level training programs. Family systems therapy and eclectic psychotherapy were the most frequent primary theoretical orientations endorsed by therapists (19 therapists, or 33 percent, for family systems therapy; 16 therapists, or 28 percent, for eclectic psychotherapy), followed by psychodynamic and cognitive-behavioral psychotherapy (nine therapists, or 16 percent, each).

Procedures

Individual interviews were conducted after the adolescent had completed one or two therapy sessions. Youths and parents were interviewed at the family's home or at the research facility. Information about the therapists was collected at the therapist's office. All parents and therapists provided written informed consent, and all adolescents provided written informed assent. Each participant was compensated $20 for participating in the study. The protocols were approved by human subjects protection committees at the University of California, San Diego; Children's Hospital and Health Center; and the San Diego County Department of Health and Human Services.

Measures

During the interview, we asked respondents to describe the three most important outcomes or changes that they hoped would be achieved as a result of mental health services for the adolescent. The open-ended responses were coded into 30 outcome categories based on pilot interviews with adolescents and therapists (14,15) and based on examples from the five outcome domains defined by Hoagwood and colleagues (2). These domains include symptoms (for example, reduce anger and aggression and improve mood), functioning (for example, have more friends), consumer perspectives (for example, enjoy therapy), environment (for example, have a more peaceful home life), and systems (for example, be placed in an appropriate school). Two trained raters coded each response and agreed on more than 80 percent of the initial ratings. Discrepancies between the coders were discussed by the first three authors and resolved by consensus.

Ninety-six percent (N=381) of all respondents reported three desired outcomes. However, multiple responses from a respondent could be coded within the same outcome category. For example, if a parent reported, "I want him to work on his anger" and "I want him to be less hostile," both responses were coded as "reduce anger and aggression." Additional measures that were used in our study included both the parent and the youth versions of the Child Behavior Checklist (12) as well as the Child and Adolescent Functional Assessment Scale (16). The parent and youth versions of the Child Behavior Checklist are well-established self-report measures of youths' emotional and behavioral problems. The Child and Adolescent Functional Assessment Scale was completed by therapists to assess the youths' functional impairment across multiple domains. Both measures have strong psychometric characteristics.

Statistical analyses

The frequencies were computed for each of the desired outcomes by stakeholder group for the five outcome domains (2) and for each of the 30 individual outcomes. Then a test for correlated proportions was used to determine whether a significant difference was seen among all three stakeholder groups for selected outcome categories. An outcome category was selected if at least 10 percent of any of the three stakeholder groups reported it as a desired outcome. We used generalized estimating equations (GEEs) to compare the proportions of desired outcomes across the three groups of stakeholders, while taking into account the correlation, or clustering, of responses. This GEE was constructed by using a logistic model with a logit link and a binomial error and by declaring the triad as a cluster of observations. If the overall Wald chi square test that had two degrees of freedom was significant at the .05 level, then pairwise comparisons were examined in the GEE logistic model by using individual Wald tests, each with one degree of freedom, and by applying a Bonferroni correction.

Agreement among stakeholders at the individual client level was assessed by counting the number of desired outcomes on which the triad or dyads of stakeholders "matched." Kappa statistics were also used to examine agreement on desired outcomes: traditional kappa statistics were used to represent dyadic agreement, and three-way kappa statistics were used to represent triadic agreement (17).

To identify potential correlates of agreement, we tested for statistically significant bivariate relationships between potential correlates and triadic agreement, which was defined as all three stakeholders reporting at least one of the same desired outcomes.

Results

Desired outcomes by stakeholder group

Figure 1 presents the frequency of desired outcomes by outcome domain for each of the three stakeholder groups (p<.01 for the differences between the stakeholder groups for the symptoms, functioning, and environment domains). A Bonferroni correction (p=.017) was applied to pairwise difference tests. As seen in Figure 1, significantly fewer adolescents than either parents or therapists reported desired outcomes in the symptoms and functioning domains, whereas significantly fewer parents than either adolescents or therapists reported desired outcomes in the environment domain. Few respondents in any stakeholder group reported desired outcomes in the consumer perspectives (satisfaction) or systems domains.

Table 1 lists the frequencies for the most common desired outcomes—that is, desired outcomes reported by at least 10 percent of any stakeholder group—and the p value, which represents the significance of the difference between the three stakeholder groups. Significant pairwise comparisons are displayed in the last column. "Reduce anger and aggression" was the most frequently reported desired outcome for adolescents, parents, and therapists alike. In contrast, "improve family relationships" was reported as a desired outcome more frequently by both therapists and adolescents than by parents. Therapists also reported "talk about feelings" as a desired outcome significantly more than either parents or adolescents. "Obey parents" and "improve self-esteem" were among the top three most frequent desired outcomes reported by parents, who reported these desired outcomes significantly more than adolescents.

Agreement on desired outcomes

Triadic agreement, which was defined as all three stakeholders reporting at least one of the same desired outcomes, was moderate to poor. Only 65 of the triads (38 percent) agreed on even one desired outcome, and none of the triads agreed on all three outcomes. Agreement on at least one desired outcome was higher for dyads: 70 percent (N=119) for the parent-therapist dyad, 64 percent (N=109) for the parent-adolescent dyad, and 62 percent (N=105) for the therapist-adolescent dyad. As seen in Table 2, dyadic and triadic kappa statistics were also calculated for the 12 most common desired outcomes. Although several kappa statistics were statistically significant, the magnitude of agreement was not strong. A majority of dyadic and triadic kappa statistics were under .30, reflecting slight to fair agreement (18). Reduction of anxiety showed the highest level of agreement, with a kappa of .53, reflecting moderate agreement.

Correlates of agreement

Variables that were tested as potential correlates included characteristics of the youth, family, and therapist as well as the treatment. Characteristics of the youth and family included the age and gender of the youth, family income, diagnosis of the youth, Child Behavior Checklist externalizing score, and youth version of the Child Behavior Checklist internalizing score. Characteristics of the therapist and the treatment included the therapist's self-reported theoretical orientation, discipline, and number of years of experience, as well as the number of sessions before the interview. Only two variables demonstrated significant point biserial correlations with triadic agreement. Youth anxiety disorder was associated with significantly greater agreement (r=.187, p<.05, N=169), whereas therapist cognitive-behavioral orientation was associated with poorer agreement (r=−.213, p<.01, N=167).

Discussion

Overview

Our findings indicate that stakeholders tend to report many of the same types of desired outcomes for youths who are entering treatment—that is, reduce anger and aggression as well as improve family and peer relationships—at the aggregate level. However, individual adolescent-parent-therapist triads did not have a high rate of agreement on reported desired outcomes. Sixty-two percent of triads did not agree on any of their reported desired outcomes. Dyadic agreement on at least one desired outcome was higher and relatively similar for each of the three types of stakeholder dyads (range, 62 to 70 percent). Dyad and triad agreement at the individual desired outcome category level was often better than what could have occurred by chance, but the magnitude of the agreement was generally poor. The only individual desired outcome with relatively strong agreement was "reduce anxiety." Similarly, if the youth had a diagnosis of anxiety, triadic agreement was more likely to occur. Interestingly, if the therapist's self-reported primary theoretical treatment orientation was cognitive-behavioral psychotherapy, triadic agreement was less likely to occur.

Desired outcomes by stakeholder group

Overall, a majority of desired outcomes across stakeholder groups were classified in the symptom and functioning outcome domains. Symptom reduction has been reported to be the highest desired outcome priority across stakeholder groups for adult services (6). Symptoms and functioning are the outcome domains most commonly assessed in treatment efficacy and effectiveness research (2,13). However, Kazdin and Wassell (13) have argued that the exclusive assessment of child symptom and functioning outcomes may underestimate the impact of child mental health services and that desired changes in parent and family functioning can also be demonstrated. In our study, both therapists and youths were more likely than parents to report desired outcomes in the environmental domain, reflecting primarily family functioning outcomes. In contrast, adolescents were less likely than either parents or therapists to report desired outcomes in the youth symptom and functioning domains. Our findings are consistent with those of Hawley and Weisz (8), who found that parents reported more individual youth symptoms as target problems and that children reported more family and environmental issues as target problems. The desired outcome that was reported most frequently by all stakeholder groups was the reduction of aggressive and angry behavior, which is consistent with reports that disruptive behavior problems are the most common presenting problems for mental health services for youths (8,13).

Youths in our study were very willing to identify desired outcomes in treatment, acknowledging their need to be in treatment and identifying a variety of issues to address. Thus our data do not support the myth that youths are particularly reluctant to enter psychotherapy (19). Fewer than ten youths had difficulty identifying desired treatment outcomes; however, a similar number of parents also had difficulty.

Agreement within stakeholder triads

Although perhaps surprising given the general agreement across all the stakeholder groups on desired outcomes, the relatively low rates of agreement on desired outcomes for individual triads that were found in our study are generally consistent with the limited related research that is available. Specifically, using a different coding system, Hawley and Weisz (8) found that only 23 percent of stakeholder triads agreed on at least one target problem for youth psychotherapy. In a sample of 60 adults with schizophrenia, their family members, and their therapists, agreement on desired outcomes was also relatively poor, with only a third of the pairs agreeing on at least one desired outcome (6).

It is important to note that a lack of agreement within individual stakeholder triads on reported desired outcomes does not necessarily imply active disagreement; rather, it may imply differential prioritization, or preferences. The lack of agreement among triads may reflect the complex, comorbid problems challenging these youths and their families. Each stakeholder may be reporting equally valid desired outcomes for a youth and family with multiple challenges. This lack of agreement may also reflect different attributions that individuals make for behavior and emotional problems or different linguistic labels for similar behaviors (7). For example, as indicated above, therapists were significantly more likely than parents to report a desired outcome that was related to improving family functioning; thus therapists may have attributed a youth's behavioral problems to family issues.

A variety of potential correlates of agreement that ranged from characteristics of the adolescent and family to characteristics of the therapist and treatment were tested, but very few demonstrated any significant association with agreement, which is consistent with previous research (8). A diagnosis of anxiety disorder was associated with a greater likelihood of triadic agreement, perhaps because these disorders are less common in this type of treatment setting. Therefore, the symptoms of anxiety may be more distinct and lend themselves to easily identifiable outcome priorities, compared with more diffuse and common disruptive behavior problems.

Limitations

The major limitation or methodological challenge of our study is related to the qualitative method that was used to assess desired outcomes. Open-ended responses are subject to a variety of validity challenges, including differential verbal abilities, retrieval biases, and perceived response demands. Given these potential limitations we also assessed desired outcomes by using a forced-choice method that was presented later in the interview. For the forced-choice method we presented the list of potential desired outcome codes to participants and asked them to choose the three desired outcomes of most importance to them. The results from the forced choice method did not differ substantially from those of the open-ended method in terms of the most commonly endorsed desired outcomes or agreement among participants.

The timing of data collection may have also affected the results. Clients had met with the therapist just one or two times when desired outcomes were assessed. Agreement may increase substantially over the course of treatment as participants have more time to discuss problems and goals. However, we chose to focus on the initial agreement in our investigation, because it may play a role in a family's decision to engage in treatment or to drop out immediately, as is common in community-based settings (20).

Conclusions

Attention to clients' treatment goals and expectations has been emphasized by psychotherapy researchers for decades (21), but the extent to which therapists explicitly discuss goals and expectations with clients or family members at the outset of treatment is unknown. Public and private care systems often require explicit documentation of treatment goals, but such documentation may be perceived by therapists as an administrative task that is quite distinct from the therapeutic process with the client and family (15). Also, a growing amount of empirical evidence shows that increased systematic attention to patients' treatment goals and expectations may improve their engagement in services (22). Finally, some evidence suggests that agreement on desired outcomes across stakeholder groups may be associated with improved outcomes of care (23). Thus valuable directions for future research include examinations of the extent to which therapy goals are discussed and consensus is reached between therapists and clients, as well as whether agreement on those goals is associated with greater client engagement, improved outcomes in general, and improved outcomes for the targeted goals in particular.

Although a great deal of research and administrative attention has been given to outcome measurement in recent years, most of the methodological work has addressed the statistical reliability and the validity of measures or technological advances in data collection and analysis. We need to pay increased attention to the ecological validity of outcome measures, including the variability in meanings and in prioritization of outcomes for various stakeholders. Such attention may improve the clinical utility of outcome measurement in the field, which could ultimately improve the effectiveness of mental health services.

Acknowledgments

This study was supported by grant K01-MH-01544 from the National Institute of Mental Health. The authors thank Donald Slymen, Ph.D., for statistical consultation.

The authors are affiliated with the Child and Adolescent Services Research Center at Children's Hospital in San Diego. Dr. Garland and Ms. Lewczyk-Boxmeyer are also with the department of psychiatry at the University of California, San Diego, 3020 Childrens Way (MC 5033), San Diego, California 92123 (e-mail, [email protected]). Ms. Lewczyk-Boxmeyer and Dr. Hawley are also with the department of psychology at San Diego State University.

Enlarge table

Enlarge table