The State Policy Context of Implementation Issues for Evidence-Based Practices in Mental Health

The Evidence-based Practices National Demonstration project, based at Dartmouth University, is centered on facilitating the implementation of clinical practices for persons with severe mental illness with consistent empirical support for their effectiveness ( 1 , 2 ). Whereas the Dartmouth focus is on clinical interventions and practices at the clinician-consumer nexus ( 3 ), we were interested in understanding the facilitators and barriers to implementation encountered at the state level and the role of the state in the implementation process ( 4 ). As state mental health systems increasingly move toward adopting more clinical practices with established evidence bases, it is important to understand the state's role in facilitating those practices and what potential roadblocks exist for implementation.

In a related paper we identify three common areas where state mental health authorities played a critical role for the implementation of high-fidelity evidence-based practices (Isett KR, Burnam MA, Coleman-Beattie B, et al., unpublished manuscript, 2006). These areas are financing and regulations (creating a supportive legal and fiscal environment), leadership (ensuring that critical actors are paying attention to and are actively soliciting support for the initiative), and training and quality (providing resources for frontline workers to obtain the clinical skills necessary for a given practice). Here, we focus on the dynamics of those three critical areas for each of the five evidence-based practices included in this project. For each practice, we identify the specific issues that arise during implementation as observed at the demonstration sites (see box on this page) ( 5 ). The five practices are described in more detail in a compendium of articles published in Psychiatric Services in 2001 as part of a series on evidence-based practices ( 5 ).

Our focus here is not on the provider role in implementation of the evidence-based practices, but rather the role that the state played in implementing the practices. The provider aspects of the Evidence-based Practices National Demonstration Project are part of the larger "toolkit" project ( 1 , 2 ) and are a platform for our project. This article represents a policy and administration perspective that is complementary to the clinical work being done through Dartmouth.

Methods

This section presents a condensed version of the approach used by the investigatory team. Here we outline our methods to give a basic rendering of what we did and how we did it. [An expanded Methods section is available as an online supplement to this article at ps.psychiatryonline.org.]

Members of the MacArthur Foundation's Network on Mental Health Policy Research visited the eight states involved in the Evidence-based Practices National Demonstration Project ( 2 ) to assess the role played by state mental health authorities with regard to implementation of evidence-based practices. The investigators used grounded case study methods to identify the critical barriers and facilitators related to implementing evidence-based practices ( 6 ).

We conducted two site visits in each state approximately one year apart, with the first wave between September 2002 and March 2003 and the second wave between March and April 2004. Site visit teams of two individuals focused on one of the five evidence-based practices included in this study: assertive community treatment, family psychoeducation, illness management and recovery, integrated dual diagnosis treatment, and supported employment. Each team consisted of a senior investigator who led the interview and a junior member who provided additional questions and primarily took notes. Teams used a semistructured interview protocol that covered a wide variety of general topics such as financing and organization by the state mental health authority as well as issues specific to the Evidence-based Practices Project, such as planning, site selection, regulations, financing, workforce, training, and leadership. Interviews were conducted over two days with groups of participants in four 120-minute slots per day. Consistent with grounded case study methods, after the first set of interviews we completed a thematic content analysis, and the interview protocol was narrowed to three dominant themes for the follow-up visits: financing and regulations, leadership, and training and quality management.

Respondents in each state were identified by the mental health commissioner's office to be knowledgeable about evidence-based practice implementation and came from various backgrounds to capture a broad range of perspectives ( Table 1 ). Our interviews included key members of the state mental health authority staff responsible for implementing the evidence-based practices, consumers, family members, representatives of other relevant state agencies (such as substance abuse and vocational services), and local mental health authorities (or their equivalent). More than 30 individuals in each of the eight states participated in our interviews at both time points. We were interested in interviewing individuals who held specific roles in each state, rather than the individuals themselves. As such, our interview respondents were consistent across both periods, although the specific individual holding those positions may have changed.

Enlarge table

After the site visits, interview teams produced a report for each state visited. To ensure validity of the information contained in the reports, state mental health authority staff received a copy of their site visit report for review and factual corrections. After the report on the second site visit was completed, the two reports were synthesized into a final report for each state. The eight final reports can be found at www.nri-inc.org/cmhqa.cfm.

Results

We present and discuss our observational findings about implementation issues related to each of the five evidence-based practices, which we organized around the three areas of critical state involvement identified earlier. Unlike most qualitative studies, however, this article does not present direct quotations from our respondents because of the specificity of the interviews we conducted and our agreement with the states about the use of the information collected. Instead, we analyze what we found in our interview data and support our findings with specific examples. We highlight similarities and differences among the five practices throughout this process ( Table 2 ).

Enlarge table

Assertive community treatment

Because assertive community treatment is perhaps the most expensive of the evidence-based practices to operate, financing strategies are crucial to its viability. Assertive community treatment has extremely high start-up costs because of a long lag time between when teams are formed and when they have enough capacity to see and bill for sufficient numbers of clients and collateral visits. Thus agencies starting assertive community treatment teams must find prospective funds for payment until the team is certified for Medicaid reimbursement (where the state plan permits) and can be potentially self-sustaining. Further, an appropriate rate structure that covers team meeting time and partial follow-up with clients is important. Overall, adequate funding strategies and appropriate targeting of the practice to only the consumers with the highest need are keys to rolling out assertive community treatment on a statewide basis, according to state implementation staff.

There are two important leadership issues for assertive community treatment: the perceived coerciveness of the practice and the availability of assertive community treatment in rural areas. First, consumers often fear and sometimes fight the implementation of assertive community treatment because of a concern about loss of autonomy over treatment direction ( 7 ). However, empirical evidence suggests that such perceived coercion is not universally experienced by service users and that the long-term benefits of assertive community treatment include fewer hospital days and crisis episodes, longer periods of stability for consumers, and greater consumer satisfaction than with other treatment methods ( 8 ). Second, challenges related to implementing assertive community treatment in rural settings include low client density, workforce shortages, and travel distances that make frequent face-to-face contacts difficult ( 8 ). State leadership must develop a feasible plan for implementation to avoid the perception of an urban bias and geographic disparities for assertive community treatment, as noted by several provider and consumer respondents in our study.

Training and quality issues are particularly difficult for assertive community treatment because this practice has the special issue of retraining clinicians already performing assertive community treatment-like functions. In some cases assertive community treatment teams operate in the field with little or no fidelity. So, as was the case with both of our sites, clinicians must be retaught elements of a practice that they thought they already were doing appropriately. The most successful implementations of assertive community treatment require high standards for fidelity (that is, a score of 4 or better out of 5 points on most elements of the fidelity scale) with rigorous monitoring systems ( 9 , 10 ).

Family psychoeducation

In contrast to assertive community treatment, family psychoeducation's main contingency for rollout is leadership. Consensus building is particularly important for family psychoeducation because the practice requires consumers and their families to be actively involved in the service. Overall, stakeholder consensus was that progress on implementing family psychoeducation at our three sites was associated with how soon these stakeholders became involved in and committed to the practice. Another leadership issue for family psychoeducation is the perception by many advocates of peer programs that this practice is in competition with the National Alliance for the Mentally Ill's Family-to-Family program. Advocates and consumers in our study suggested that clear communication about the details of both programs is needed to facilitate understanding that the practices are complementary rather than mutually exclusive. Thus early and effective communication with stakeholders is essential for the successful implementation of family psychoeducation.

Issues of financing and training for family psychoeducation center on "who" questions. For financing the "who" question is whether a consumer needs to be present in order to bill for family psychoeducation; specifically whether sessions with collateral individuals without the consumer can be reimbursed. For training the "who" question centers on the most appropriate and competent person to provide psychoeducation services. Advocates of consumer-based services assert that only persons with a particular illness or persons who are directly affected by that person's illness, such as family members, can effectively and empathically communicate what the illness is like and what to expect from it, not professionals or clinicians, as with this evidence-based model.

Illness management and recovery

As with family psychoeducation, illness management and recovery's main contingency was leadership. First, illness management and recovery implementers reported many of the same tensions as in family psychoeducation with regard to consumer and family member involvement and education about the model. There was concern from consumers and advocates about funding for this practice relative to existing funding for the Wellness Recovery Action Plan ( 11 ) and other consumer-driven practices. There was also a need for early education for consumers about the relative strengths and weaknesses of the illness management and recovery model in comparison with consumer-driven models. However, leadership is most critical for illness management and recovery with regard to the philosophical shift that it represents. Under illness management and recovery, the view of a consumer changes from a passive or marginally involved treatment recipient to a full partner in decision making about treatment. Given this fundamental shift, providers suggested that entire organizations need to be recruited and trained to ensure support for consumers' use of new skills within the treatment context, rather than recruitment of just organizational leaders and specific clinicians.

The financing issues related to illness management and recovery stem from the problematic differentiation of illness management and recovery activities from usual care activities in billing systems. In addition to these differentiation issues and in contrast to many of the other evidence-based practices, there is not a clear, single funding stream to pay for illness management and recovery, according to state commissioners' staffs and providers. As a result, funding streams used in the demonstration project were diverse, ranging from Medicaid, to block grant funding, to one-time start-up funds—lending no clear guidance as to funding likely to be sustainable for this practice.

Illness management and recovery had a subtle but important workforce issue. Some providers reflected that clinicians must possess a high level of clinical skill in order to use illness management and recovery effectively. This necessity is problematic for a workforce with high turnover rates and low initial skill sets.

Integrated dual diagnosis treatment

Like assertive community treatment, integrated dual diagnosis treatment's biggest contingency is financing and regulations. However, whereas both of these evidence-based practices require team-based funding for intensive services, integrated dual diagnosis treatment has the added barrier of delivering integrated mental health and substance abuse treatment when different regulatory and administrative rules apply to distinct funding streams and there is a relative scarcity of substance abuse funding. Start-up funding in the demonstration project often used block grant or other flexible funding. However, state mental health authority staff viewed Medicaid reimbursement as key to longer-term sustainability. Adding to the difficulty of using Medicaid for integrated dual diagnosis treatment, though, is the fact that not all states have substance abuse treatment for adults as a Medicaid-reimbursable service.

Medicaid reimbursement and developing appropriate regulations for integrated dual diagnosis treatment is tricky, as reflected by our respondents. Traditional providers of substance abuse services often hit a stumbling block in seeking Medicaid reimbursement because most of these providers are not certified to work with Medicaid. In the demonstrations, integrated dual diagnosis treatment was often done through the mental health system, where services are performed by mental health professionals (sometimes without specialized substance abuse training) and reimbursement rates are adequate for additional services. In terms of regulations, state policy makers and regulators need to be creative about who can provide services, when, and to whom. Some examples of creativity for implementation of integrated dual diagnosis treatment that emerged in our interviews include dual certification for mental health and substance abuse providers and coordinated treatment plans.

Leadership issues for integrated dual diagnosis treatment center on the coordination of services across agency boundaries. There is a question among commissioner-level staff of how to link appropriately with the substance abuse agency for this practice because there is a lack of clarity about the role of the state's substance abuse agencies in the toolkit. In most interviews, the substance abuse agency was tacitly supportive of or completely lacking involvement in the implementation (most likely because of the focus on the population with serious mental illness). Leadership is required to overcome these coordination issues with the state substance abuse agency. Although coordination problems also exist in assertive community treatment, the coordination issues in integrated dual diagnosis treatment are much more complex because of the cross-agency nature of the evidence-based practice.

Coordination is the crux of training and quality concerns for integrated dual diagnosis treatment as well. First, state quality assurance staff noted that there is difficulty in creating links between data systems among agencies to allow monitoring of need for and use of co-occurring services. This lack of coordination results in suboptimal tracking and outcomes assessment for this practice. Second, state implementation staff noted that cross-training clinicians to effectively deliver both substance abuse and mental health services is an important workforce development concern for integrated treatment implementation. There are very few practitioners that are certified in providing care for both mental health and substance abuse and an equally limited number of substance abuse providers that are Medicaid certified. Thus, although integrated dual diagnosis treatment can be implemented, the necessary skill sets of those implementing the evidence-based practice are difficult to find.

Supported employment

Similar to integrated dual diagnosis treatment, supported employment must coordinate funding for services with another agency—here, the state's rehabilitation services agency. Where it is available, Medicaid reimbursement for employment services will cover many of the clinical services under the supported employment umbrella. However, supplemental funding from the rehabilitation services agency is needed to pay for job acquisition and job coaching services, according to finance staff of state mental health authorities. Another important consideration pointed out by providers for supported employment is to ensure adequate reimbursement rates for the practice. Agencies will want to ensure that under new supported employment and mental health funding arrangements, reimbursement rates are not lower than with traditional funding mechanisms, which would undermine services to the mental health community. Of interest, although regulation related to licensure, payment, and fidelity measures is helpful in the early stages of implementation, it is not necessary. States in our study developed special arrangements to get supported employment projects started or used resources from another demonstration on supported employment ( 12 ). Regulations would be essential for statewide rollout, however.

The leadership issues for supported employment are somewhat different from those for the other evidence-based practices. On the one hand, there are few "political" issues related to the implementation of supported employment in that there is almost universal stakeholder approval for it. On the other hand, leadership efforts need to be concentrated on coordinating regulatory and financing schemes with an outside agency, as with integrated dual diagnosis treatment. An area that may need to be finessed in implementing supported employment is the bias toward a mental health model in the toolkits. The broader population to which supported employment services might be offered is much more general and larger than the mental health consumers targeted in the demonstration. This narrowness could make the supported employment toolkit less attractive to the rehabilitation agency partner, as noted by staff of state mental health authorities. Although these two leadership issues are similar to integrated dual diagnosis treatment, they are less complex in that rehabilitation services agencies are actively involved in delivery of supported employment and the funding streams available for this practice are more exploitable than with substance abuse agencies.

Quality issues for supported employment are unique among the evidence-based practices in that competitive employment outcomes are clear and easily measurable. These outcomes were typically emphasized in our local implementation sites over the clinical outcomes in the toolkit. However, state staff and providers noted that success in competitive employment is somewhat independent of those factors controlled by clinicians. For example, the processes of care, job development, and coaching and the relevant measured outcomes are outside the usual realm of the mental health discipline and create a locus of control for mental health treatment outside of the mental health system. Another unique factor in supported employment is that participation creates a disincentive to consumers to be successful in the program: work and higher income may result in consumers' no longer meeting criteria for disability payments or medical care provided by Medicaid. This dynamic creates powerful counterforces in the willingness and ability of consumers to take full advantage of the benefits of this practice.

Discussion

If the implementation process was proceeding well in a state, participants were doing three specific things. First, regulations were being aligned to be commensurate with the needs of an evidence-based system of care. States also found or created new ways to finance these practices that compensated providers fairly for their efforts, both for ongoing implementation and at start-up. Second, states were providing leadership to the mental health system to move toward dynamic and effective treatment centered on the consumer. Third, states were providing training to frontline clinicians to implement the evidence-based practices with high fidelity, and they were linking those training efforts to quality assessments and infrastructure needs.

The results from our thematic analyses and interviews are consistent with much of the literature on implementation and organizational change ( 13 , 14 , 15 ). Specifically, leadership for change and champions of the practices were identifiable, sites were empowered to act on and adapt change initiatives, and resources necessary to accomplish change were present. Each of these aspects of change is important for long-term sustainability.

There are several implications from our findings that have relevance for other states interested in implementing evidence-based practices with high fidelity statewide. First, evidence-based practices must be carefully selected because each of the evidence-based practices mobilizes a different set of stakeholder groups, requires different regulations, and encounters different implementation obstacles. There are unique challenges associated with each evidence-based practice that need to be assessed for fit within a given context. States will likely find that their current environment is suited to a particular set of the evidence-based practices.

Second, a systematic rollout of the selected evidence-based practice or practices is recommended. This strategy may take longer to accomplish statewide access to an array of evidence-based practices, but ultimately it will be more effective in facilitating high-quality standards and the necessary leadership focus to implement the practices successfully. In the demonstration project one of the two selected practices always lagged in implementation. Invariably, states became burdened by the workforce requirements and resources needed to implement multiple practices. States found that it was more effective to focus first on one set of details related to the implementation of one of the evidence-based practices and then on the other set of details for the second evidence-based practice.

Our research reported here is not without several important limitations. First, our respondents for the interviews were all identified by the mental health commissioner's office. This could cause some bias in our data if respondents were chosen to present a positive view of the implementation process. However, a review of our notes from the site visits suggests that local respondents were not prone to providing "rosy views" of the implementation. Further, our research was intended explicitly to capture the state mental health authority's point of view, so we do not see this potential bias as a particular problem because it is consistent with our research objectives. Second, the qualitative design of this research may potentially cause some interpretation biases, because our number of cases was very small. This generalizability issue is not to be minimized. However, the depth of information collected and insight gained in this study could not be gained by a more quantitative approach. We took all appropriate methodological precautions to ensure reliability in our study, such as multiple interviewers, interrater consensus, review of reports by respondents, and group review and analysis. Thus we feel the benefits of our qualitative design outweigh the costs.

One aspect of our research that is unfortunate is the abstraction of our reporting. Although we had a sizable number of respondents, there were only eight states involved in implementing the evidence-based practices in this project and only five practices were implemented. When the states and practices are arrayed in a grid, there are virtually no degrees of freedom, meaning direct quotes or specific details of an implementation would be identifiable to anyone involved in the demonstration project and anyone knowledgeable about it. Thus we had to abstract our findings in order to protect our respondents' anonymity, at both the individual and state levels. Therefore, we report only generalizations that reflect the overall gist of what happened in the states we visited. These generalizations are made to inform the field but not to reflect a unanimous consensus of all stakeholders in all contexts.

Conclusions

In summary, each of the evidence-based practices requires the dedication of a specific set of resources for effective implementation that includes the acquisition of appropriate legal, political, and human capital and supports. These resources include time, attention, and money, as well as different skills and stakeholder groups that are needed to form a consensus for implementation. Overall, the important message here is that the benefits of implementing multiple evidence-based practices simultaneously may not be additive. States should be careful to understand the resource constraints of each evidence-based practice and where economies may or may not be achieved before beginning implementation.

Acknowledgments and disclosures

The authors thank Kathryn Falls, Wilma Townsend, and Eileen Hansen for their invaluable assistance in data collection and report writing and Greg McHugo for ongoing availability to answer questions about the Evidence-based Practices Demonstration Project.

The authors report no competing interests.