The American Psychiatric Association (APA) has updated its Privacy Policy and Terms of Use, including with new information specifically addressed to individuals in the European Economic Area. As described in the Privacy Policy and Terms of Use, this website utilizes cookies, including for the purpose of offering an optimal online experience and services tailored to your preferences.

Please read the entire Privacy Policy and Terms of Use. By closing this message, browsing this website, continuing the navigation, or otherwise continuing to use the APA's websites, you confirm that you understand and accept the terms of the Privacy Policy and Terms of Use, including the utilization of cookies.

×
Published Online:

The Evidence-based Practices National Demonstration project, based at Dartmouth University, is centered on facilitating the implementation of clinical practices for persons with severe mental illness with consistent empirical support for their effectiveness ( 1 , 2 ). Whereas the Dartmouth focus is on clinical interventions and practices at the clinician-consumer nexus ( 3 ), we were interested in understanding the facilitators and barriers to implementation encountered at the state level and the role of the state in the implementation process ( 4 ). As state mental health systems increasingly move toward adopting more clinical practices with established evidence bases, it is important to understand the state's role in facilitating those practices and what potential roadblocks exist for implementation.

In a related paper we identify three common areas where state mental health authorities played a critical role for the implementation of high-fidelity evidence-based practices (Isett KR, Burnam MA, Coleman-Beattie B, et al., unpublished manuscript, 2006). These areas are financing and regulations (creating a supportive legal and fiscal environment), leadership (ensuring that critical actors are paying attention to and are actively soliciting support for the initiative), and training and quality (providing resources for frontline workers to obtain the clinical skills necessary for a given practice). Here, we focus on the dynamics of those three critical areas for each of the five evidence-based practices included in this project. For each practice, we identify the specific issues that arise during implementation as observed at the demonstration sites (see box on this page) ( 5 ). The five practices are described in more detail in a compendium of articles published in Psychiatric Services in 2001 as part of a series on evidence-based practices ( 5 ).

Key elements of the five evidence-based practices from the Evidence-based Practices National Demonstration Project

Assertive community treatment

Multidisciplinary team approach to comprehensive care, with psychiatrist and nurse participation

Targeted toward consumers with the most severe needs

Low consumer-to-clinician ratio (10:1)

Continuous, personalized, and flexible care

Services are provided where needed; community outreach is expected

Care includes intensive treatment, out-of-office care, and collateral visits

Time-unlimited support

Round-the-clock availability for crisis care

Team may include a substance abuse specialist and an employment specialist

Family psychoeducation

For families and other people who provide support to a consumer

Collaboration between family and clinicians optimizes the outcome of mental illness

Family education helps relatives support their loved ones in pursuing recovery goals

Reducing family stress and improving coping can improve the quality of all members' lives

Family support can validate the experiences of all members and facilitate shared problem solving

Family collaboration is long term, not short term

Structure: joining sessions to develop an alliance with the family, educational workshop and ongoing education about the illness, and multifamily group with a focus on problem solving

Illness management and recovery

Centered on the consumer's taking responsibility for his or her own life and treatment

Based on the concept of recovery, as defined by the consumer

Information-based curriculum about the illness

Importance of medications to symptom management and quality of life; information about their effects

Education provides a foundation for informed and shared decision making

The stress-vulnerability model provides a blueprint for illness management

Collaboration with professional caregivers and significant others helps consumers to attain goals

Individualized plans to reduce relapses and rehospitalizations

Emphasizes how to interact with clinical caregivers to get the most from services

Builds skills to manage illness effectively through practicing techniques

Integrated dual diagnosis treatment

Substance abuse treatment is integrated into comprehensive mental health treatment for persons with serious mental illness

Mental health clinicians have knowledge of substances of abuse

Treatment for substance abuse and mental illness delivered by the same clinician or treatment team

Special features include stagewise treatment, an integrated treatment plan, motivational interviewing, cognitive-behavioral treatments, substance abuse counseling, harm reduction services, self-help liaison, and relapse prevention services

Multiple formats for services should be available for the individual, group, and family

Medication services are integrated with other services and sensitive to the substance abuse

Supported employment

Flexibility of services is based on consumer choice

Vocational services are integrated with mental health services

Competitive employment is the goal

Includes benefits counseling and planning

Job search starts soon after a consumer expresses interest in working

Follow-along supports are continuous

Consumer preferences for type of work are important

Includes the use of job developers and job coaches

Mental health teams provide additional support to consumers once they enter the workforce

Our focus here is not on the provider role in implementation of the evidence-based practices, but rather the role that the state played in implementing the practices. The provider aspects of the Evidence-based Practices National Demonstration Project are part of the larger "toolkit" project ( 1 , 2 ) and are a platform for our project. This article represents a policy and administration perspective that is complementary to the clinical work being done through Dartmouth.

Methods

This section presents a condensed version of the approach used by the investigatory team. Here we outline our methods to give a basic rendering of what we did and how we did it. [An expanded Methods section is available as an online supplement to this article at ps.psychiatryonline.org.]

Members of the MacArthur Foundation's Network on Mental Health Policy Research visited the eight states involved in the Evidence-based Practices National Demonstration Project ( 2 ) to assess the role played by state mental health authorities with regard to implementation of evidence-based practices. The investigators used grounded case study methods to identify the critical barriers and facilitators related to implementing evidence-based practices ( 6 ).

We conducted two site visits in each state approximately one year apart, with the first wave between September 2002 and March 2003 and the second wave between March and April 2004. Site visit teams of two individuals focused on one of the five evidence-based practices included in this study: assertive community treatment, family psychoeducation, illness management and recovery, integrated dual diagnosis treatment, and supported employment. Each team consisted of a senior investigator who led the interview and a junior member who provided additional questions and primarily took notes. Teams used a semistructured interview protocol that covered a wide variety of general topics such as financing and organization by the state mental health authority as well as issues specific to the Evidence-based Practices Project, such as planning, site selection, regulations, financing, workforce, training, and leadership. Interviews were conducted over two days with groups of participants in four 120-minute slots per day. Consistent with grounded case study methods, after the first set of interviews we completed a thematic content analysis, and the interview protocol was narrowed to three dominant themes for the follow-up visits: financing and regulations, leadership, and training and quality management.

Respondents in each state were identified by the mental health commissioner's office to be knowledgeable about evidence-based practice implementation and came from various backgrounds to capture a broad range of perspectives ( Table 1 ). Our interviews included key members of the state mental health authority staff responsible for implementing the evidence-based practices, consumers, family members, representatives of other relevant state agencies (such as substance abuse and vocational services), and local mental health authorities (or their equivalent). More than 30 individuals in each of the eight states participated in our interviews at both time points. We were interested in interviewing individuals who held specific roles in each state, rather than the individuals themselves. As such, our interview respondents were consistent across both periods, although the specific individual holding those positions may have changed.

Table 1 Interview participants and key stakeholders in the Evidence-based Practices Project
Table 1 Interview participants and key stakeholders in the Evidence-based Practices Project
Enlarge table

After the site visits, interview teams produced a report for each state visited. To ensure validity of the information contained in the reports, state mental health authority staff received a copy of their site visit report for review and factual corrections. After the report on the second site visit was completed, the two reports were synthesized into a final report for each state. The eight final reports can be found at www.nri-inc.org/cmhqa.cfm.

Results

We present and discuss our observational findings about implementation issues related to each of the five evidence-based practices, which we organized around the three areas of critical state involvement identified earlier. Unlike most qualitative studies, however, this article does not present direct quotations from our respondents because of the specificity of the interviews we conducted and our agreement with the states about the use of the information collected. Instead, we analyze what we found in our interview data and support our findings with specific examples. We highlight similarities and differences among the five practices throughout this process ( Table 2 ).

Table 2 Implementation issues for the five evidence-based practices
Table 2 Implementation issues for the five evidence-based practices
Enlarge table

Assertive community treatment

Because assertive community treatment is perhaps the most expensive of the evidence-based practices to operate, financing strategies are crucial to its viability. Assertive community treatment has extremely high start-up costs because of a long lag time between when teams are formed and when they have enough capacity to see and bill for sufficient numbers of clients and collateral visits. Thus agencies starting assertive community treatment teams must find prospective funds for payment until the team is certified for Medicaid reimbursement (where the state plan permits) and can be potentially self-sustaining. Further, an appropriate rate structure that covers team meeting time and partial follow-up with clients is important. Overall, adequate funding strategies and appropriate targeting of the practice to only the consumers with the highest need are keys to rolling out assertive community treatment on a statewide basis, according to state implementation staff.

There are two important leadership issues for assertive community treatment: the perceived coerciveness of the practice and the availability of assertive community treatment in rural areas. First, consumers often fear and sometimes fight the implementation of assertive community treatment because of a concern about loss of autonomy over treatment direction ( 7 ). However, empirical evidence suggests that such perceived coercion is not universally experienced by service users and that the long-term benefits of assertive community treatment include fewer hospital days and crisis episodes, longer periods of stability for consumers, and greater consumer satisfaction than with other treatment methods ( 8 ). Second, challenges related to implementing assertive community treatment in rural settings include low client density, workforce shortages, and travel distances that make frequent face-to-face contacts difficult ( 8 ). State leadership must develop a feasible plan for implementation to avoid the perception of an urban bias and geographic disparities for assertive community treatment, as noted by several provider and consumer respondents in our study.

Training and quality issues are particularly difficult for assertive community treatment because this practice has the special issue of retraining clinicians already performing assertive community treatment-like functions. In some cases assertive community treatment teams operate in the field with little or no fidelity. So, as was the case with both of our sites, clinicians must be retaught elements of a practice that they thought they already were doing appropriately. The most successful implementations of assertive community treatment require high standards for fidelity (that is, a score of 4 or better out of 5 points on most elements of the fidelity scale) with rigorous monitoring systems ( 9 , 10 ).

Family psychoeducation

In contrast to assertive community treatment, family psychoeducation's main contingency for rollout is leadership. Consensus building is particularly important for family psychoeducation because the practice requires consumers and their families to be actively involved in the service. Overall, stakeholder consensus was that progress on implementing family psychoeducation at our three sites was associated with how soon these stakeholders became involved in and committed to the practice. Another leadership issue for family psychoeducation is the perception by many advocates of peer programs that this practice is in competition with the National Alliance for the Mentally Ill's Family-to-Family program. Advocates and consumers in our study suggested that clear communication about the details of both programs is needed to facilitate understanding that the practices are complementary rather than mutually exclusive. Thus early and effective communication with stakeholders is essential for the successful implementation of family psychoeducation.

Issues of financing and training for family psychoeducation center on "who" questions. For financing the "who" question is whether a consumer needs to be present in order to bill for family psychoeducation; specifically whether sessions with collateral individuals without the consumer can be reimbursed. For training the "who" question centers on the most appropriate and competent person to provide psychoeducation services. Advocates of consumer-based services assert that only persons with a particular illness or persons who are directly affected by that person's illness, such as family members, can effectively and empathically communicate what the illness is like and what to expect from it, not professionals or clinicians, as with this evidence-based model.

Illness management and recovery

As with family psychoeducation, illness management and recovery's main contingency was leadership. First, illness management and recovery implementers reported many of the same tensions as in family psychoeducation with regard to consumer and family member involvement and education about the model. There was concern from consumers and advocates about funding for this practice relative to existing funding for the Wellness Recovery Action Plan ( 11 ) and other consumer-driven practices. There was also a need for early education for consumers about the relative strengths and weaknesses of the illness management and recovery model in comparison with consumer-driven models. However, leadership is most critical for illness management and recovery with regard to the philosophical shift that it represents. Under illness management and recovery, the view of a consumer changes from a passive or marginally involved treatment recipient to a full partner in decision making about treatment. Given this fundamental shift, providers suggested that entire organizations need to be recruited and trained to ensure support for consumers' use of new skills within the treatment context, rather than recruitment of just organizational leaders and specific clinicians.

The financing issues related to illness management and recovery stem from the problematic differentiation of illness management and recovery activities from usual care activities in billing systems. In addition to these differentiation issues and in contrast to many of the other evidence-based practices, there is not a clear, single funding stream to pay for illness management and recovery, according to state commissioners' staffs and providers. As a result, funding streams used in the demonstration project were diverse, ranging from Medicaid, to block grant funding, to one-time start-up funds—lending no clear guidance as to funding likely to be sustainable for this practice.

Illness management and recovery had a subtle but important workforce issue. Some providers reflected that clinicians must possess a high level of clinical skill in order to use illness management and recovery effectively. This necessity is problematic for a workforce with high turnover rates and low initial skill sets.

Integrated dual diagnosis treatment

Like assertive community treatment, integrated dual diagnosis treatment's biggest contingency is financing and regulations. However, whereas both of these evidence-based practices require team-based funding for intensive services, integrated dual diagnosis treatment has the added barrier of delivering integrated mental health and substance abuse treatment when different regulatory and administrative rules apply to distinct funding streams and there is a relative scarcity of substance abuse funding. Start-up funding in the demonstration project often used block grant or other flexible funding. However, state mental health authority staff viewed Medicaid reimbursement as key to longer-term sustainability. Adding to the difficulty of using Medicaid for integrated dual diagnosis treatment, though, is the fact that not all states have substance abuse treatment for adults as a Medicaid-reimbursable service.

Medicaid reimbursement and developing appropriate regulations for integrated dual diagnosis treatment is tricky, as reflected by our respondents. Traditional providers of substance abuse services often hit a stumbling block in seeking Medicaid reimbursement because most of these providers are not certified to work with Medicaid. In the demonstrations, integrated dual diagnosis treatment was often done through the mental health system, where services are performed by mental health professionals (sometimes without specialized substance abuse training) and reimbursement rates are adequate for additional services. In terms of regulations, state policy makers and regulators need to be creative about who can provide services, when, and to whom. Some examples of creativity for implementation of integrated dual diagnosis treatment that emerged in our interviews include dual certification for mental health and substance abuse providers and coordinated treatment plans.

Leadership issues for integrated dual diagnosis treatment center on the coordination of services across agency boundaries. There is a question among commissioner-level staff of how to link appropriately with the substance abuse agency for this practice because there is a lack of clarity about the role of the state's substance abuse agencies in the toolkit. In most interviews, the substance abuse agency was tacitly supportive of or completely lacking involvement in the implementation (most likely because of the focus on the population with serious mental illness). Leadership is required to overcome these coordination issues with the state substance abuse agency. Although coordination problems also exist in assertive community treatment, the coordination issues in integrated dual diagnosis treatment are much more complex because of the cross-agency nature of the evidence-based practice.

Coordination is the crux of training and quality concerns for integrated dual diagnosis treatment as well. First, state quality assurance staff noted that there is difficulty in creating links between data systems among agencies to allow monitoring of need for and use of co-occurring services. This lack of coordination results in suboptimal tracking and outcomes assessment for this practice. Second, state implementation staff noted that cross-training clinicians to effectively deliver both substance abuse and mental health services is an important workforce development concern for integrated treatment implementation. There are very few practitioners that are certified in providing care for both mental health and substance abuse and an equally limited number of substance abuse providers that are Medicaid certified. Thus, although integrated dual diagnosis treatment can be implemented, the necessary skill sets of those implementing the evidence-based practice are difficult to find.

Supported employment

Similar to integrated dual diagnosis treatment, supported employment must coordinate funding for services with another agency—here, the state's rehabilitation services agency. Where it is available, Medicaid reimbursement for employment services will cover many of the clinical services under the supported employment umbrella. However, supplemental funding from the rehabilitation services agency is needed to pay for job acquisition and job coaching services, according to finance staff of state mental health authorities. Another important consideration pointed out by providers for supported employment is to ensure adequate reimbursement rates for the practice. Agencies will want to ensure that under new supported employment and mental health funding arrangements, reimbursement rates are not lower than with traditional funding mechanisms, which would undermine services to the mental health community. Of interest, although regulation related to licensure, payment, and fidelity measures is helpful in the early stages of implementation, it is not necessary. States in our study developed special arrangements to get supported employment projects started or used resources from another demonstration on supported employment ( 12 ). Regulations would be essential for statewide rollout, however.

The leadership issues for supported employment are somewhat different from those for the other evidence-based practices. On the one hand, there are few "political" issues related to the implementation of supported employment in that there is almost universal stakeholder approval for it. On the other hand, leadership efforts need to be concentrated on coordinating regulatory and financing schemes with an outside agency, as with integrated dual diagnosis treatment. An area that may need to be finessed in implementing supported employment is the bias toward a mental health model in the toolkits. The broader population to which supported employment services might be offered is much more general and larger than the mental health consumers targeted in the demonstration. This narrowness could make the supported employment toolkit less attractive to the rehabilitation agency partner, as noted by staff of state mental health authorities. Although these two leadership issues are similar to integrated dual diagnosis treatment, they are less complex in that rehabilitation services agencies are actively involved in delivery of supported employment and the funding streams available for this practice are more exploitable than with substance abuse agencies.

Quality issues for supported employment are unique among the evidence-based practices in that competitive employment outcomes are clear and easily measurable. These outcomes were typically emphasized in our local implementation sites over the clinical outcomes in the toolkit. However, state staff and providers noted that success in competitive employment is somewhat independent of those factors controlled by clinicians. For example, the processes of care, job development, and coaching and the relevant measured outcomes are outside the usual realm of the mental health discipline and create a locus of control for mental health treatment outside of the mental health system. Another unique factor in supported employment is that participation creates a disincentive to consumers to be successful in the program: work and higher income may result in consumers' no longer meeting criteria for disability payments or medical care provided by Medicaid. This dynamic creates powerful counterforces in the willingness and ability of consumers to take full advantage of the benefits of this practice.

Discussion

If the implementation process was proceeding well in a state, participants were doing three specific things. First, regulations were being aligned to be commensurate with the needs of an evidence-based system of care. States also found or created new ways to finance these practices that compensated providers fairly for their efforts, both for ongoing implementation and at start-up. Second, states were providing leadership to the mental health system to move toward dynamic and effective treatment centered on the consumer. Third, states were providing training to frontline clinicians to implement the evidence-based practices with high fidelity, and they were linking those training efforts to quality assessments and infrastructure needs.

The results from our thematic analyses and interviews are consistent with much of the literature on implementation and organizational change ( 13 , 14 , 15 ). Specifically, leadership for change and champions of the practices were identifiable, sites were empowered to act on and adapt change initiatives, and resources necessary to accomplish change were present. Each of these aspects of change is important for long-term sustainability.

There are several implications from our findings that have relevance for other states interested in implementing evidence-based practices with high fidelity statewide. First, evidence-based practices must be carefully selected because each of the evidence-based practices mobilizes a different set of stakeholder groups, requires different regulations, and encounters different implementation obstacles. There are unique challenges associated with each evidence-based practice that need to be assessed for fit within a given context. States will likely find that their current environment is suited to a particular set of the evidence-based practices.

Second, a systematic rollout of the selected evidence-based practice or practices is recommended. This strategy may take longer to accomplish statewide access to an array of evidence-based practices, but ultimately it will be more effective in facilitating high-quality standards and the necessary leadership focus to implement the practices successfully. In the demonstration project one of the two selected practices always lagged in implementation. Invariably, states became burdened by the workforce requirements and resources needed to implement multiple practices. States found that it was more effective to focus first on one set of details related to the implementation of one of the evidence-based practices and then on the other set of details for the second evidence-based practice.

Our research reported here is not without several important limitations. First, our respondents for the interviews were all identified by the mental health commissioner's office. This could cause some bias in our data if respondents were chosen to present a positive view of the implementation process. However, a review of our notes from the site visits suggests that local respondents were not prone to providing "rosy views" of the implementation. Further, our research was intended explicitly to capture the state mental health authority's point of view, so we do not see this potential bias as a particular problem because it is consistent with our research objectives. Second, the qualitative design of this research may potentially cause some interpretation biases, because our number of cases was very small. This generalizability issue is not to be minimized. However, the depth of information collected and insight gained in this study could not be gained by a more quantitative approach. We took all appropriate methodological precautions to ensure reliability in our study, such as multiple interviewers, interrater consensus, review of reports by respondents, and group review and analysis. Thus we feel the benefits of our qualitative design outweigh the costs.

One aspect of our research that is unfortunate is the abstraction of our reporting. Although we had a sizable number of respondents, there were only eight states involved in implementing the evidence-based practices in this project and only five practices were implemented. When the states and practices are arrayed in a grid, there are virtually no degrees of freedom, meaning direct quotes or specific details of an implementation would be identifiable to anyone involved in the demonstration project and anyone knowledgeable about it. Thus we had to abstract our findings in order to protect our respondents' anonymity, at both the individual and state levels. Therefore, we report only generalizations that reflect the overall gist of what happened in the states we visited. These generalizations are made to inform the field but not to reflect a unanimous consensus of all stakeholders in all contexts.

Conclusions

In summary, each of the evidence-based practices requires the dedication of a specific set of resources for effective implementation that includes the acquisition of appropriate legal, political, and human capital and supports. These resources include time, attention, and money, as well as different skills and stakeholder groups that are needed to form a consensus for implementation. Overall, the important message here is that the benefits of implementing multiple evidence-based practices simultaneously may not be additive. States should be careful to understand the resource constraints of each evidence-based practice and where economies may or may not be achieved before beginning implementation.

Acknowledgments and disclosures

The authors thank Kathryn Falls, Wilma Townsend, and Eileen Hansen for their invaluable assistance in data collection and report writing and Greg McHugo for ongoing availability to answer questions about the Evidence-based Practices Demonstration Project.

The authors report no competing interests.

Dr. Isett is affiliated with the Department of Health Policy and Management, Columbia University, 600 W. 168th St., 6th Floor, New York, NY 10032 (e-mail: ki2129@colum bia.edu). Dr. Burnam is with RAND Corporation, Santa Monica, California. Ms. Coleman-Beattie is a family member and consultant in Austin, Texas. Ms. Hyde is with the Department of Human Services, State of New Mexico. Dr. Morrissey is with the Cecil G. Sheps Center for Health Services Research, University of North Carolina at Chapel Hill. Dr. Magnabosco is with the Leavey Center for the Study of Los Angeles, Loyola Marymount University. Dr. Rapp is with the Department of Social Work, University of Kansas. Dr. Ganju is with Mental Health Transformation, Texas Department of State Health Services. Dr. Goldman is with the Department of Psychiatry, University of Maryland.

References

1. Drake RE, Goldman HH, Leff HS, et al: Implementing evidence-based practices in routine mental health service settings. Psychiatric Services 52:179–182, 2001Google Scholar

2. Torrey WC, Drake RE, Dixon L, et al: Implementing evidence-based practices for persons with severe mental illnesses. Psychiatric Services 52:45–50, 2001Google Scholar

3. Lipsky M: Street-Level Bureaucracy. New York, Russell Sage Foundation, 1980Google Scholar

4. Magnabosco JL: Innovations in mental health services implementation: a report on state-level data from the National Evidence-Based Practices Project. Implementation Science 1(13), 2006Google Scholar

5. Drake RE, Goldman HH (eds): Evidence-Based Practices in Mental Health Care. Arlington, Va, American Psychiatric Publishing, 2003Google Scholar

6. Yin RK: Case Study Research: Design and Methods, 3rd ed. Thousand Oaks, Calif, Sage, 2003Google Scholar

7. Phillips SD, Burns BJ, Edgar ER, et al: Moving assertive community treatment into standard practice. Psychiatric Services 52:771–779, 2001Google Scholar

8. Meyer PS, Morrissey JP: A comparison of assertive community treatment and intensive case management for patients in rural areas. Psychiatric Services 58:121–127, 2007Google Scholar

9. McGrew JH, Bond GR: The association between program characteristics and service delivery in assertive community treatment. Administration and Policy in Mental Health 25:175–189, 1997Google Scholar

10. Bond GR, Salyers MP: Prediction of outcomes from the Dartmouth Assertive Community Treatment Fidelity Scale. CNS Spectrums 9:937–942, 2004Google Scholar

11. Copeland ME: Wellness Recovery Action Plan. Dummerston, Vt, Peach Press, 1997Google Scholar

12. Drake RE, Becker DR, Goldman HH, et al: The Johnson & Johnson-Dartmouth community mental health program: disseminating evidence-based practice. Psychiatric Services 57:302–304, 2006Google Scholar

13. Kotter JP: Leading change: why transformation efforts fail. Harvard Business Review, Mar–Apr 1995, pp 59–67Google Scholar

14. Greenhalgh T, Robert G, Macfarlane F, et al: Diffusion of innovations in service organizations: systematic review and recommendations. Milbank Quarterly 82:581–629, 2004Google Scholar

15. Stetler CB: The role of organization in translating research into evidence-based practice. Outcomes Management 7(3):97–103, 2003Google Scholar