Research, Data, and Evidence-Based Treatment Use in State Behavioral Health Systems, 2001–2012
Abstract
Objective:
Empirical study of public behavioral health systems’ use of data and their investment in evidence-based treatments (EBTs) is limited. This study describes trends in state-level EBT investment and research supports from 2001 to 2012.
Methods:
Data were from National Association for State Mental Health Program Directors Research Institute (NRI) surveys, which were completed by representatives of state mental health authorities (SMHAs). Multilevel models examined change over time related to state adoption of EBTs, numbers of clients served, and penetration rates for six behavioral health EBTs for adults and children: supported housing, supported employment, assertive community treatment, therapeutic foster care, multisystemic therapy, and functional family therapy. State supports related to research, evaluation, and information management were also examined.
Results:
Increasing percentages of states reported funding an external research center, promoting the adoption of EBTs through provider contracts, and providing financial incentives for EBTs. Decreasing percentages of states reported promoting EBT adoption through stakeholder mobilization, monitoring fidelity, and specific budget requests. There was greater reported use of adult-focused EBTs (65%−80%) compared with youth-focused EBTs (25%−50%). Overall penetration rates of EBTs were low (1%−3%) and EBT adoption by states showed flat or declining trends. SMHAs’ investment in data systems and use of research showed little change.
Conclusions:
SMHA investment in EBTs, implementation infrastructure, and use of research has declined. More systematic measurement and examination of these metrics may provide a useful approach for setting priorities, evaluating success of health reform efforts, and making future investments.
Over the past several decades, a steady stream of articles, reports, and national calls to action have concluded that public behavioral health systems are characterized by ineffectiveness (1–3). Many of these reports called for increasing the availability of evidence-based treatments (EBTs) in public behavioral health systems and systematic use of data and research for continuous quality improvement (1–5). States are in a clear position to lead mental health service and system reform efforts, including investment in EBTs and application of research to improve outcomes (6–8). Thus states represent a logical focus for research on public system investments in research and evidence. Overall, however, empirical study of state support of EBT implementation and data use is scant (9).
One potential metric for gauging use of research evidence is the extent to which states implement EBTs. EBT implementation has been widely promoted as a partial solution to improving efficiency and increasing the likelihood of effectively addressing the behavioral health needs of children and adults (10,11). Because commitment to EBTs requires a commensurate commitment to practitioner and organizational capacity to deliver EBTs, state infrastructural supports (for example, policy and fiscal incentives, workforce development supports, data systems, and research centers or centers of excellence) represent a second potential metric (12–14).
In 1987, a not-for-profit research center for the National Association of State Mental Health Program Directors (NASMHPD) was established. In 1993, the NASMHPD Research Institute (NRI) initiated regular surveys of state representatives about the characteristics of state mental health authorities (SMHAs), including (starting in 2001) the nature and extent of their investment in EBTs and support of research, evaluation, and data use. To date, these data have not been used to examine state trends in EBT implementation, EBT implementation supports, or methods for promoting data and research use—all potential indicators of research-based decision making.
Using the NRI data, this study examined the degree to which state systems invest in EBTs and research-based supports and how such investments have changed from 2001 to 2012. This period represents the initiation of NRI tracking of specific EBTs as well as a time of increased awareness of—and federal initiatives to support—behavioral health EBTs (3,15–17). We addressed three research questions. First, how have the rates of use and penetration of EBTs by SMHAs changed from 2001 to 2012? Second, what kinds of support for EBTs are provided by SMHAs, and how has this changed over time? Third, what infrastructure is in place to support use of data by SMHAs, and how has this changed over time?
Methods
Data Source and Elements
NRI data originate from two publicly available sources. The State Profiles System (SPS) asks questions such as whether the SMHA conducted research on client outcomes, implemented a statewide client outcomes–monitoring system, engaged in initiatives to build awareness of EBTs, and other similar items. These questions referred to the state’s public mental health system in general and were not specifically tied to individual EBTs. Data elements from the Uniform Reporting System (URS) focus on counts of individuals receiving specific EBTs and estimates of youths with serious emotional disturbance and adults with serious mental illness. Prevalence estimates are overall estimates, not specific to the number of people eligible for each EBT. Additional details can be found in recent reports (18,19). Because data are all publicly available, formal review by an institutional review board was not required.
Sample
Information was provided by SMHA representatives in all 50 states, the District of Columbia, Puerto Rico, and the Virgin Islands. Response rates by states and territories were high, ranging from 87% (46 of 53) in 2001 to 98% (52 of 53) in 2005.
EBTs tracked.
EBTs tracked include three interventions for children with serious emotional disturbance— therapeutic foster care (20), multisystemic therapy (21), and functional family therapy (22)—and three treatments for adults with serious mental illness—supported housing (23), supported employment (24), and assertive community treatment (25). Because the NRI survey is intended to provide federal funding agencies with information on populations of specific interest (adults with serious mental illness and children with serious emotional disturbance), all of these interventions are intended for individuals with complex needs, and most are multimodal (that is, they include multiple strategies that address a range of factors that may influence individual and contextual needs).
Years examined.
Data were collected for most variables in 2001, 2002, 2004, 2007, 2009, 2010, and 2012. Data on numbers and rates of individuals served by EBTs, however, are available only from 2007 to 2012, when the URS was implemented as a state-level accountability mechanism by the Substance Abuse and Mental Health Services Administration.
Outliers and Missing Data
Continuous variables were examined for possible outliers or data entry errors, and 115 outliers (out of 1,543 total responses, 7%) were deleted or replaced. [More information on identification and handling of outliers is included in an online supplement to this article.]
Data analysis
Two-level multilevel models (MLMs) (26), with time and year nested within state, were used to examine change over time. Three models were run for each dependent variable to test linear, quadratic, and cubic time trends, and the best-fitting parsimonious model was selected for each. [More information on distributions used, determination of model fit, and software used is included in the online supplement.]
Results
State Use of Evidence and Activities to Promote EBTs
Table 1 presents data on the states that endorsed each specific practice concerning EBT implementation, training and workforce support, or data and research support. Time trend estimates are presented in Table 2. Significant linear time trends were found for five variables. Increasing percentages of states reported funding an external research center or institute, funding initiatives to increase EBT awareness, promoting the adoption of EBTs through provider contracts, providing financial incentives for offering EBTs, and conducting research or evaluation studies examining client change in functioning. The two most dramatic increases (based on the slope size) were in conducting research on change in client functioning and promoting adoption of EBTs through contracts.
Question and activity | 2001 | 2002 | 2004 | 2007 | 2009 | 2010 | 2012 | |||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
N | % | N | % | N | % | N | % | N | % | N | % | N | % | |
Does the state mental health authority (SMHA) do or has it done the following? | ||||||||||||||
Conducted research on or evaluations of client outcomesb | 13 | 29 | 32 | 84 | 36 | 75 | 42 | 88 | 40 | 83 | — | — | — | — |
Implemented a statewide client outcomes-monitoring system | — | — | — | — | — | — | — | — | — | — | 29 | 58 | 29 | 64 |
Integrated its client data sets with client data sets from other agencies | — | — | 23 | 49 | 28 | 58 | — | — | 26 | 54 | 25 | 50 | 24 | 53 |
Produced a directory of research or evaluation projects | 11 | 23 | 11 | 23 | 7 | 15 | 9 | 19 | 9 | 19 | 11 | 23 | — | — |
Operated a research center or institute | 8 | 17 | 8 | 17 | 7 | 15 | 7 | 15 | 7 | 15 | 4 | 8 | — | — |
Funded a research center or institutec | 6 | 13 | 11 | 23 | 9 | 19 | 13 | 27 | 8 | 17 | 15 | 31 | — | — |
What initiatives are you implementing to promote the adoption of EBTs? | ||||||||||||||
Awareness and trainingc | — | — | 38 | 75 | 34 | 71 | 41 | 84 | 43 | 86 | 44 | 88 | 37 | 86 |
Consensus building among stakeholdersd | — | — | 38 | 79 | 36 | 75 | 44 | 90 | 42 | 84 | 41 | 82 | 31 | 72 |
Incorporation in contractsc | — | — | 20 | 42 | 21 | 44 | 30 | 61 | 29 | 58 | 37 | 74 | 29 | 67 |
Monitoring of fidelityd | — | — | 25 | 52 | 27 | 56 | 34 | 69 | 36 | 72 | 35 | 70 | 29 | 67 |
Financial incentivesc | — | — | 8 | 17 | 15 | 31 | 14 | 29 | 18 | 37 | 19 | 38 | 15 | 35 |
Modification of information technology systems and data reports | — | — | 20 | 42 | 22 | 46 | 27 | 55 | 25 | 50 | 29 | 58 | 22 | 51 |
Specific budget requestsd | — | — | 14 | 29 | 19 | 40 | 25 | 51 | 19 | 39 | 19 | 38 | 12 | 28 |
Does the SMHA conduct research on or evaluations of the following? | ||||||||||||||
Utilization rates | — | — | 33 | 87 | 36 | 78 | 40 | 83 | 39 | 81 | — | — | — | — |
Change in functioningc | — | — | 26 | 68 | 29 | 63 | 35 | 73 | 37 | 77 | — | — | — | — |
Penetration rates | — | — | 28 | 74 | 30 | 65 | 35 | 73 | 36 | 75 | — | — | — | — |
Question and activity or EBT | Time trend | Intercept | Linear slope | Quadratic slope | Cubic slope | ||||
---|---|---|---|---|---|---|---|---|---|
Coefficient | p | Coefficient | p | Coefficient | p | Coefficient | p | ||
Activities to support research, data use, and EBTs | |||||||||
Does the state mental health authority (SMHA) do or has it done the following? | |||||||||
Conducted research on or evaluations of client outcomes | Fixed | –.681 | .014 | 1.577 | <.001 | –.327 | .008 | .021 | .030 |
Implemented a statewide client outcomes-monitoring system | Random | –.721 | .656 | .113 | .484 | ||||
Integrated its client data sets with client data sets from other agencies | Random | .242 | .372 | –.012 | .744 | ||||
Produced a directory of research or evaluation projects | Random | –1.303 | <.001 | .004 | .878 | ||||
Operated a research center or institute | Random | –1.412 | <.001 | –.028 | .188 | ||||
Funded a research center or institute | Random | –1.464 | <.001 | .054 | .031 | ||||
What initiatives are you implementing to promote the adoption of EBTs? | |||||||||
Awareness and training | Random | 1.049 | <.001 | .073 | .029 | ||||
Consensus building among stakeholders | Random | .765 | .009 | .255 | .019 | –.023 | .022 | ||
Incorporation in contracts | Random | –.419 | .141 | .119 | .008 | ||||
Monitoring of fidelity | Random | –.277 | .420 | .264 | .019 | –.016 | .066 | ||
Financial incentives | Random | –1.287 | <.001 | .081 | .032 | ||||
Modification of information technology systems and data reports | Random | –.269 | .318 | .470 | .220 | ||||
Specific budget requests | Random | –1.131 | .002 | .311 | .019 | –.027 | .013 | ||
Does the SMHA conduct research on or evaluations of the following? | |||||||||
Utilization rates | Random | 1.012 | .014 | .041 | .502 | ||||
Change in functioning | Random | .089 | .779 | .134 | .005 | ||||
Penetration rates | Random | .449 | .136 | .070 | .115 | ||||
Numbers of people receiving specific EBTs | |||||||||
Therapeutic foster care | Random | 6.309 | <.001 | .107 | .019 | –.021 | .016 | ||
Multisystemic therapy | Random | 5.611 | <.001 | .089 | .020 | ||||
Functional family therapy | Random | 6.508 | <.001 | –.004 | .938 | ||||
Supported housing | Random | 7.727 | <.001 | –.259 | .029 | .104 | .019 | –.011 | .021 |
Supported employment | Random | 6.948 | <.001 | .037 | .394 | ||||
Assertive community treatment | Random | 7.212 | <.001 | .033 | .163 | ||||
Rates of adults with serious mental illness and children with serious emotional disturbance receiving EBTs | |||||||||
Therapeutic foster care | Fixed | –3.284 | <.001 | .303 | .015 | –.132 | .014 | .014 | .055 |
Multisystemic therapy | Fixed | –4.013 | <.001 | .552 | .010 | –.293 | .045 | .039 | .057 |
Functional family therapy | Fixed | –3.874 | <.001 | .737 | .042 | –.311 | .051 | .035 | .066 |
Supported housing | Fixed | –.53 | <.001 | .004 | .849 | .045 | .004 | –.009 | .002 |
Supported employment | Random | –3.309 | <.001 | .005 | .906 | ||||
Assertive community treatment | Random | –3.41 | <.001 | –.013 | .459 |
Multilevel models testing longitudinal trends in state endorsement of activities to support research, data use, and evidence-based treatments (EBTs) and receipt of six EBTs
Significant quadratic time trends were found for three variables. Significantly increasing and then decreasing percentages of states reported promoting EBT adoption through consensus building among stakeholders, through monitoring fidelity, and through specific budget requests. Reported use of these practices peaked in 2007–2009. A significant cubic trend was found for states conducting research and evaluation on client outcomes, which showed a sharp increase from 2001 to 2002 and then a slight decrease and slight increase from 2002 to 2009. No significant change over time was found for implementing a statewide outcomes-monitoring system, integrating client data sets with client data sets from other agencies, producing a directory of research or evaluation projects, operating a research center or institute, modifying information technology (IT) systems and data reports, conducting research on or evaluations of utilization rates, or conducting research on or evaluations of penetration rates.
States With EBT Services Available
Figure 1 shows the percentage of states that reported availability of the six EBTs tracked. On average across 2001–2012, EBTs serving adults were more commonly available. On average, 72% of states reported the availability of assertive community treatment, 70% reported supported employment, and 66% reported supported housing. For children and youths, on average, 54% of states reported that therapeutic foster care was available, 39% reported multisystemic therapy, and 27% reported functional family therapy.
Model fitting confirmed that piecewise linear time trends fit the data better than exponential time trends. Table 3 presents the results of individual piecewise MLMs. For all EBTs with valid data from 2001 to 2012, there were significant increases in the proportion of states using therapeutic foster care, multisystemic therapy, supported employment, and assertive community treatment from 2001 to 2005 and then no significant increases or decreases from 2007 to 2012. Data on functional family therapy and supported housing were not collected until 2005; tests for these practices found no significant increases or decreases from 2005 to 2012.
EBTb | Slope 2001–2005 | Slope 2007–2012 | Slope 2005–2012 | ||||
---|---|---|---|---|---|---|---|
Time trend | Coefficient | p | Coefficient | p | Coefficient | p | |
Therapeutic foster care | Random | .347 | <.001 | .051 | .285 | ||
Multisystemic therapy | Fixed | .390 | .004 | .021 | .653 | ||
Supported employment | Random | .458 | <.001 | .043 | .349 | ||
Assertive community treatment | Random | .133 | .020 | –.081 | .119 | ||
Functional family therapy | Random | –.056 | .202 | ||||
Supported housing | Random | –.010 | .826 |
Number of Clients Served by Specific EBTs
Because of positive skew (a few states reporting large numbers served), we examined state medians for clients served and used data only from states that reported any availability of the salient EBT. On average across 2007–2012, a median of 1,029 clients per year per state used supported housing, 950 used assertive community treatment, and 669 used supported employment. For EBTs serving children and youths, a median of 371 clients used functional family therapy, 279 used therapeutic foster care, and 230 used multisystemic therapy.
Figure 2 shows the median number served per state each year, and Table 2 provides MLM estimates for best-fitting time trends. Results indicate a significant but small linear increase in the number of multisystemic therapy clients, a significant quadratic change in therapeutic foster care clients (an increase followed by a decrease), and a significant cubic change in the number of supported housing clients (a decrease followed by a flattening). No time trends were found for functional family therapy, supported employment, or assertive community treatment.
Rates of Clients Served
Rates of clients served by EBTs were calculated by dividing the reported numbers of clients served by the reported number of adults with serious mental illness or children and youths with serious emotional disturbance in the state. Averaging across 2007–2012, the median rates of clients served were, in descending order, 3% for supported housing, 2% for functional family therapy, 2% for assertive community treatment, 2% for supported employment, 1% for therapeutic foster care, and 1% for multisystemic therapy.
Table 2 presents the MLM estimates for best-fitting time trends for rates of clients served. Results indicated no significant changes for supported employment or assertive community treatment and significant cubic changes for therapeutic foster care, multisystemic therapy, functional family therapy, and supported housing. Therapeutic foster care and supported housing both showed flat trends followed by a substantial decrease and then another flat trend. Multisystemic therapy showed an increase from 2007 to 2009, followed by a decrease and a brief increase. Functional family therapy showed a flat trend, a sharp increase, and then a slow downward trend from 2009 to 2012.
Discussion
Although there is evidence that many states have invested in infrastructure supports to encourage and support EBT implementation, the impact on availability and reach of EBTs has been relatively small. In 2012, between 65% and 80% of states reported using the EBTs for adults examined in this study. However, the median numbers of clients served by states were only about 1,000 for each EBT, and penetration rates ranged from only 2% to 3%. Uptake of child EBTs was lower: only 25% to 50% of states reported use of the interventions studied, and median numbers of clients served were only 250 to 400, reaching only 1%−3% of youths with serious emotional disturbance. It is not possible from these data to explain the reasons for the discrepancy in use of adult- versus child-focused EBTs; however, the discrepancy could result from the long-standing trend in state behavioral health systems to invest more heavily in services for adults (27).
The results also pointed to large gaps between need and available EBTs for both adults and children. Some of this observed gap may be the result of study limitations. Because the NRI survey focuses on a small number of multimodal interventions for individuals with complex needs, data for a range of well-known interventions for individuals with less complex or intensive behavioral health needs (for example, cognitive-behavioral therapies and behavioral parenting interventions) were not available. Thus the actual number of EBTs being used across states is likely higher. In addition, there have been significant changes since 2001 in states’ authority; for example, the increasing penetration of managed care has led to less direct state control over services (18,28). Thus states have less authority to oversee EBT initiatives or may be unaware of EBTs that are being used through managed care.
Nonetheless, the scant uptake of EBTs found in this study is consistent with other studies of specific EBT implementation, which demonstrate significant challenges in getting research-based services installed in real-world systems (29–32). Results highlight the fact that the population entrusted to public mental health care is not deriving benefits from receipt of evidence-based therapeutic services. Although each of the interventions examined has been proposed as a strategy to meet the behavioral health needs of individuals with the most complex presentations, and all are tailored to clients served by specific public systems (for example, foster care for therapeutic foster care and juvenile justice–involved youths for multisystemic therapy), only a very small portion of individuals actually receive one of the EBTs.
Perhaps more worrisome, adoption by states, numbers served, and penetration rates were found to be flat or declining. Even if these interventions are merely considered indicators or proxies of EBT uptake in states, trends examined here suggest that after a burst of initial adoption and expansion in the early 2000s, EBT uptake has leveled off, if not diminished. This study also found similar “inverted U”-shaped trends in state EBT adoption drivers, including consensus building among stakeholders, state-led fidelity monitoring, and specific budget requests for EBTs.
Given that research-based strategies require investment in resources, both types of downward trends are not surprising given the magnitude of budget shortages that hit most states from 2007 to 2012. In fiscal year 2013, 13 SMHAs still had expenditures below 2007 levels (33), and these recent cuts occurred on top of historical and long-standing budget shortfalls that have consistently affected SMHAs (27).
Regardless, the trend in reduced state support for EBTs and their implementation is troubling, given that resources beyond standard fee-for-service payments are typically necessary for providers to implement EBTs well (34–36). Moreover, the goal of reducing overall public expenditures “downstream” (for example, corrections services, emergency room care, and hospitalization) (37) is theoretically linked to maintaining investment in EBTs that reduce the need for these expensive services. Thus the trend to limit EBT investment may ultimately result in increased state spending. In future analyses, we plan to use NRI data along with other data to examine the relationship between fluctuations in state fiscal investments and use of research-based strategies.
Some EBT implementation support strategies increased across the study period. Building EBTs into provider contracts showed the single greatest increase among the variables examined, from 43% of states in 2001–2004 to 70% in 2009–2012. This phenomenon was also found in a study by Finnerty and colleagues (38), which found that in 2007–2009 the majority of states opted to promote substance use disorder–related EBTs by encouraging providers to deliver these practices with contract funds.
States also reported increased use over the study period of financial incentives for provider EBT use. This may be an encouraging trend, because it has been highlighted as a proactive fiscal policy that encourages EBT adoption (38–40). However, the number of states reporting such incentives in 2012 was only 35%, barely more than the 31% that reported using the strategy in 2004.
Despite substantial improvements in IT since the turn of the millennium, the percentage of states implementing a statewide outcomes-monitoring system, integrating client data sets across agencies, producing directories of research or evaluation projects, modifying IT systems to support EBTs, and conducting research and evaluation did not change from 2001 to 2012. In 2012, only half of states reported having integrated data systems or a client outcomes-monitoring system. This is very troubling given broad consensus that true accountability is achieved not merely through investment in EBTs (41) but also through investment in data systems, data integration, and outcomes monitoring that promotes feedback, understanding of system functioning, and continuous improvement (2,42,43).
Unfortunately, however, to preserve services, SMHAs reported making their cuts in “administrative” expenses, such as data and research use, IT systems, outcomes monitoring, evaluation, and fidelity monitoring. Among evaluation and data use variables, only “funding an external research center or institute” showed an increase over time, from 13% to 31%. Although this could be interpreted as a positive trend, it is mirrored by a decrease in SMHAs’ internal operation of research centers, from 17% to 8%.
The study had some limitations. SMHAs are not the only systems that provide EBTs in a state. Vocational rehabilitation agencies may provide supported employment, and child welfare or juvenile justice agencies may support therapeutic foster care or multisystemic therapy. When responding to NRI-administered surveys, SMHA respondents may not have been fully informed about all state behavioral health initiatives or about localized efforts or pilot projects. In addition, estimates of the penetration rates for EBTs were based on general estimates provided by SMHA respondents of the overall number of youths with serious emotional disturbance or adults with serious mental illness, which may not accurately reflect actual rates of individuals eligible for these EBTs.
Another limitation is that because of the NRI’s funding mandate to focus on services funded by federal block grants, EBT data focus only on services for adults with serious mental illness or children with serious emotional disturbance, not on populations with other conditions or less intensive needs. The extent to which these findings are applicable to EBT implementation for other conditions, less intensive needs, or early intervention and prevention is unknown. In addition, although most of the EBTs studied are well defined and manualized, therapeutic foster care is a broader service type, which may have affected the accuracy of the findings and interpretation of utilization trends. Similarly, other important drivers and indicators of research and EBT use were not surveyed and thus could not be examined, including workforce support and training, the role of changing state leadership, differential costs associated with implementing EBTs, and shifts over time to managed care. These are all important areas for future research. Finally, the study aimed to examine national trends, and thus we present aggregate results across all states. Although beyond the scope of this study, future analyses should examine patterns of individual state trends and predictors of these variations.
Conclusions
State investment in EBT implementation infrastructure and use of data are critical and are only likely to grow in importance as health care reforms call for more effective services and greater quality and accountability. Results of this study suggest, however, that SMHA investment in EBTs and implementation infrastructure is flat or declining in many areas. Consistent, reliable, and valid measurement of these constructs will assist states in ensuring that their systems of services are capable of meeting the needs of the populations for which they are responsible. Such data may also serve as a basis for renewed investment and a metric for reform efforts.
1 : A meta-systems approach to evidence-based practice for children and adolescents. American Psychologist 65:85–97, 2010Crossref, Medline, Google Scholar
2 : Ethical Considerations for Research Involving Prisoners. Washington, DC, National Academies Press, 2006Google Scholar
3 Mental Health: A Report of the Surgeon General. Rockville, Md, US Department of Health and Human Services, US Public Health Service, 1999Google Scholar
4 : To Err Is Human: Building a Safer Health System. Washington, DC, National Academies Press, 1999Google Scholar
5 : Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, DC, National Academies Press, 2001Google Scholar
6 : Adoption of clinical and business trainings by child mental health clinics in New York State. Psychiatric Services 65:1439–1444, 2014Link, Google Scholar
7 : State Mental Health Commissions: Recommendations for Change and Future Directions. Washington, DC, National Technical Assistance Center for State Mental Health Planning, 2002Google Scholar
8 : The ARC organizational and community intervention strategy for implementing evidence-based children’s mental health treatments. Mental Health Services Research 7:243–259, 2005Crossref, Medline, Google Scholar
9 : Innovation adoption: a review of theories and constructs. Administration and Policy in Mental Health and Mental Health Services Research 41:480–502, 2014Crossref, Medline, Google Scholar
10 : Improving community-based mental health care for children: translating knowledge into action. Administration and Policy in Mental Health and Mental Health Services Research 40:6–22, 2013Crossref, Medline, Google Scholar
11 : Evidence-based practice in adult mental health; in Handbook of Evidence-Based Practice in Clinical Psychology. Edited by Hersen M, Sturmey P. New York, Wiley, 2012Crossref, Google Scholar
12 : Improving programs and outcomes: implementation frameworks and organization change. Research on Social Work Practice 25:477–487, 2015Crossref, Google Scholar
13 : State implementation of evidence-based practice for youths, part I: responses to the state of the evidence. Journal of the American Academy of Child and Adolescent Psychiatry 47:369–373, 2008Crossref, Medline, Google Scholar
14 : Core implementation components. Research on Social Work Practice 19:531–540, 2009Crossref, Google Scholar
15 : Defining empirically supported therapies. Journal of Consulting and Clinical Psychology 66:7–18, 1998Crossref, Medline, Google Scholar
16 : Implementing evidence-based practices in routine mental health service settings. Psychiatric Services 52:179–182, 2001Link, Google Scholar
17 Achieving the Promise: Transforming Mental Health Care in America. Pub no SMA-03-3832. Rockville, Md, Department of Health and Human Services, President’s New Freedom Commission on Mental Health, 2003Google Scholar
18 2011 Medicaid Managed Care Enrollment Report. Baltimore, Center for Medicare and Medicaid Services, 2012Google Scholar
19 : The research potential of administrative data from state mental health agencies. Psychiatric Services 61:546–548, 2010Link, Google Scholar
20 : Assessing conformity to standards for treatment foster care. Journal of Emotional and Behavioral Disorders 10:213–222, 2002Crossref, Google Scholar
21 : Efficacy studies to large-scale transport: the development and validation of multisystemic therapy programs. Annual Review of Clinical Psychology 7:351–381, 2011Crossref, Medline, Google Scholar
22 : Functional family therapy: a mature clinical model for working with at-risk adolescents and their families; in Handbook of Family Therapy: The Science and Practice of Working With Families and Couples. Edited by Sexton TL, Weeks GR, Robbins MS. New York, Brunner-Routledge, 2003Google Scholar
23 : Studies of “supported” and “supportive” housing: a comprehensive review of model descriptions and measurement. Evaluation and Program Planning 33:446–456, 2010Crossref, Medline, Google Scholar
24 : Results of a multisite randomized trial of supported employment interventions for individuals with severe mental illness. Archives of General Psychiatry 62:505–512, 2005Crossref, Medline, Google Scholar
25 : Assertive community treatment for patients with co-occurring severe mental illness and substance use disorder: a clinical trial. American Journal of Orthopsychiatry 68:201–215, 1998Crossref, Medline, Google Scholar
26 : Applied Longitudinal Data Analysis. New York, Oxford University Press, 2003Crossref, Google Scholar
27 : Better but Not Well: Mental Health Policy in the United States Since 1950. Baltimore, Johns Hopkins University Press, 2006Google Scholar
28 : Improving Medicaid Managed Care for Youth With Serious Behavioral Health Needs: A Quality Improvement Toolkit. New Brunswick, NJ, Center for Health Care Strategies, 2009Google Scholar
29 : Worldwide application of prevention science in adolescent health. Lancet 379:1653–1664, 2012Crossref, Medline, Google Scholar
30 : Unpacking the black box of implementation: the next generation for policy, research and practice. Administration and Policy in Mental Health and Mental Health Services Research 40:451–455, 2013Crossref, Medline, Google Scholar
31 : Partnerships for the design, conduct, and analysis of effectiveness, and implementation research: experiences of the prevention science and methodology group. Administration and Policy in Mental Health and Mental Health Services Research 39:301–316, 2012Crossref, Medline, Google Scholar
32 : Facing reality and jumping the chasm. Administration and Policy in Mental Health and Mental Health Services Research 40:1–5, 2013Crossref, Medline, Google Scholar
33 Neylon K, Shaw R, Lutterman T: State Mental Health Agency–Controlled Expenditures for Mental Health Services, State Fiscal Year 2013. Alexandria, Va, National Association of State Mental Health Program Directors, 2014Google Scholar
34 : A measurement feedback system (MFS) is necessary to improve mental health outcomes. Journal of the American Academy of Child and Adolescent Psychiatry 47:1114–1119, 2008Crossref, Medline, Google Scholar
35 : Routine outcome monitoring in the Netherlands: practical experiences with a web-based strategy for the assessment of treatment outcome in clinical practice. Clinical Psychology and Psychotherapy 18:1–12, 2011Crossref, Medline, Google Scholar
36 : The Computer-based Health Evaluation Software (CHES): a software for electronic patient-reported outcome monitoring. BMC Medical Informatics and Decision Making 12:126, 2012Crossref, Medline, Google Scholar
37 : Benefits and Costs of Prevention and Early Intervention Programs for Youth. Olympia, Wash, Washington State Institute for Public Policy, 2004Google Scholar
38 : The State Health Authority Yardstick (SHAY). Community Mental Health Journal 45:228–236, 2009Crossref, Medline, Google Scholar
39 : Patient outcomes and evidence-based medicine in a preferred provider organization setting: a six-year evaluation of a physician pay-for-performance program. Health Services Research 42:2140–2159, discussion 2294–2323, 2007Crossref, Medline, Google Scholar
40 : State implementation of evidence-based practice for youth: recommendations for research and policy. Journal of the American Academy of Child and Adolescent Psychiatry 47:5, 2008Google Scholar
41 : From data to wisdom: quality improvement strategies supporting large-scale implementation of evidence-based services. Child and Adolescent Psychiatric Clinics of North America 14:329, 2005Crossref, Medline, Google Scholar
42 : Effect of the exclusion of behavioral health from health information technology (HIT) legislation on the future of integrated health care. Journal of Behavioral Health Services and Research 42:534–539, 2015Crossref, Medline, Google Scholar
43 Federal Health Information Technology Strategic Plan, 2011–2015. Washington, DC, Department of Health and Human Services, Office of the National Coordinator for Health Information Technology, 2011Google Scholar