A Literature Review of Learning Collaboratives in Mental Health Care: Used but Untested

Recently, a tremendous emphasis has been placed on the integration of evidence-based practices into routine mental health care. Substantial budget cuts to mental health funding at the state and national levels have forced policy makers to seek out efficient and effective ways to scale up training in evidence-based practices (1). States, counties, and national organizations have turned to learning collaboratives (LCs) as a method for large-scale training with ongoing support.

This collaborative approach has clearly become a priority in the field. The Substance Abuse and Mental Health Services Administration recently issued a call for applications for State Adolescent Treatment Enhancement and Dissemination grants totaling $30 million over three years to help states develop “learning laboratories” focused on shared provider experiences during the implementation of new evidence-based practices (2). Similarly, through the National Council for Community Behavioral Healthcare, 35 states are now using LCs to change health care provider practices (personal communication, Salerno A, July 2012). LCs represent a significant investment in the field as a potentially viable approach to large-scale implementation and dissemination of new treatment practices. However, there has been little research on the effectiveness of LCs for evidence-based practices in mental health care.

LCs as they are implemented in mental health care are adapted from quality improvement collaborative (QIC) models used in health care. One of the most widely cited and adapted QIC models is the Breakthrough Series (BTS) collaboratives of the Institute for Healthcare Improvement (IHI) (3–9). The quality improvement processes at the core of the IHI and other approaches are rooted in industrial improvement practices and the work of W. Edwards Deming and Joseph Juran, statisticians who advocated for process improvement driven by ongoing data collection and analysis and an assumption of workers’ interest in learning and improvement (10–12).

Although some evidence exists for the effectiveness of QICs in health care, there is a need for rigorous research in this area. A systematic literature review by Schouten and colleagues (13) identified nine controlled studies of health care QICs and concluded that the QICs showed promise in changing provider practices. However, the authors found less evidence in support of an impact on patient-level outcomes. Although the review included two randomized controlled trials (RCTs), a majority of the studies used matched control sites or compared administrative data from similar sites in a larger provider network.

Building on these findings, a more recent review included 24 articles, with the goal of updating the original literature review and developing a deeper understanding of the core components of QICs as they are reported in the literature (14). This review included additional RCTs (five distinct studies); however, as with the earlier review (13), a vast majority of studies used matched controls. Of the 14 crosscutting components identified as common ingredients, the QICs reported including, on average, six or seven components—most commonly, in-person learning sessions, plan-do-study-act (PDSA) cycles, multidisciplinary quality improvement teams, and data collection for quality improvement. As in the earlier review (13), outcome data suggested that the greatest impact of the QICs was on provider-level process-of-care variables; patient-level findings were less robust. Because of the imprecise reporting on specific components, it was not possible to link any specific components with improved care.

Of note, neither of these systematic reviews included collaboratives focused on mental health issues because when they were undertaken there had been no controlled studies targeting mental health care. LCs have been applied in mental health care to a wide range of practices, including the process of care (for example, engagement in services, care integration, and use of a screening tool) (15–18) and implementation of complex evidence-based practices (7,19). The focus on evidence-based practices is notable given the complexity of the patient outcomes and the substantial skill development required of providers.

This systematic literature review focused on peer-reviewed studies of mental health LCs that included any patient or provider pre-to-post outcome data. Given the differences between mental health and general health care settings in terms of their structure, types of interventions and patient issues addressed, and data systems available, there is a critical need for a better understanding of how LCs are implemented in mental health care. The primary goal of this review was to identify the components of LCs as reported in mental health studies and to characterize the existing data on LCs (for example, patient-level data, reports of changed provider practices, and analyses of feasibility or acceptability in real-world care settings).

Methods

This literature search on LCs focused on individual empirical articles published from January 1995 to October 2013. The database search included Ovid MEDLINE, ProQuest, PsycINFO, and PubMed. Search terms included “learning collaborative,” “quality improvement collaborative,” “Breakthrough Series,” and “NIATx.” These terms were refined after several preliminary searches and are similar to those used in earlier reviews (13,14). NIATx (Network for the Improvement of Addiction Treatment) was included in order to capture the NIATx process improvement approach used in the substance abuse literature, which draws on conceptual models similar to the predominant approach to collaboratives, specifically the IHI’s BTS (20).

Articles that met inclusion criteria were peer reviewed, written in English, and included a pre- and postintervention comparison of the impact of an LC. To define LCs in mental health care, we searched the theoretical literature on QICs (3–5,9,21–24) and reviewed the definition used by Schouten and colleagues (13). We then conducted informational interviews with a subset of LC purveyors to elicit more detail. This review defined LCs as organized, structured group learning initiatives in which organizers took the following steps: convened multidisciplinary teams representative of different levels of the organization; focused on improving specific provider practices or patient outcomes; included training from experts in a particular practice or in quality improvement methods; included a model for improvement with measurable targets, data collection, and feedback; engaged multidisciplinary teams in active improvement processes wherein they implemented “small tests of change” or engaged in PDSA activities; and employed structured activities and opportunities for learning and cross-site communication (for example, in-person learning sessions, phone calls, and e-mail Listservs) (3,5–7,9,25,26). We assessed the ways in which the 14 components identified by Nadeem and colleagues (14), including in-person learning sessions, phone meetings, data reporting, feedback, training in quality improvement methods, and use of process improvement methods, were reported in these studies.

Two authors (EN and LCH) reviewed all abstracts generated by the initial search to select articles that merited a full-text review. These two authors also reviewed each article retrieved to determine whether it met final inclusion criteria. In the event of a discrepancy or if inclusion was unclear, the two authors conferred with the other authors to make a final determination. Once article selection was finalized, each article was coded by using a standardized table to summarize study details (for example, targets for improvement, study design, setting, study sample, and LC components). A primary coder was assigned to each article, and a secondary coder reviewed the primary coder’s work. Disagreements were resolved by consensus.

The initial search generated 421 unique articles across several disciplines (primarily mental health, education, and health care). From a review of the 421 abstracts, 52 were determined to be related to mental health or substance abuse, 28 of which met criteria for full-text article review (that is, they appeared to be focused on learning collaboratives). Articles were excluded after the full-text review if they did not report any pre-post LC quantitative data. After a review of those articles and their references, 20 articles were selected for final inclusion (7,15–18,27–41). [A figure illustrating article selection is included in an online data supplement to this article.]

Results

The 20 articles selected for inclusion encompass 16 distinct studies. Table 1 provides a summary of the study type, LC model, and LC components reported in each study. Table 2 provides definitions of the study characteristics and LC components tracked in this review. The LC features were categorized into components, quality improvement processes, and organizational involvement. LC components refer to LC features that constituted the structure of the model. Quality improvement processes include available details about PDSAs and other quality improvement activities. The organizational involvement section includes indicators of the ways in which the LC penetrated various levels of the organization.

Description of LC components

Ten of the 16 distinct studies were explicitly based on the IHI BTS model, three of which also noted using the chronic care model, a model originally used as part of a joint effort by the IHI and the Robert Wood Johnson Foundation (42). One additional study cited the chronic care model without the BTS model, four studies reported using the NIATx model for process improvement (43), and one study reported using the National Assembly on School-Based Health Care’s (NASBHC) QIC model, which is based on nationally recognized models for quality improvement (39). On average, each of the 16 studies reported implementing seven LC components. The most commonly reported components included in-person learning sessions (16 of 16), multidisciplinary quality improvement teams (12 of 16), PDSAs (12 of 16), and quality improvement team calls (12 of 16). In addition, 11 of the 16 studies reported doing some leadership outreach or engagement. Across articles, there was great variability in the level of detail provided in descriptions of the components of each LC.

Overall LC structure.

The LCs lasted an average of 14 months (range nine to 27 months), with a modal length of 12 months. LCs typically began with an in-person learning session; LC faculty hosted the sessions, and multidisciplinary quality improvement teams attended. Follow-up occurred via additional in-person learning sessions, regular phone meetings for the quality improvement teams, and e-mail or Web-based support. Sites conducted quality improvement projects between quality improvement team calls and in-person learning sessions. All in-person learning sessions and most phone meetings involved multiple sites.

Content of in-person learning sessions.

All studies reported including in-person learning sessions throughout the course of the LC. The most common number of sessions was three (range one to four). In-person sessions were typically two days long, ranging from a half-day to three day-long sessions. One of the studies was an RCT; the four conditions compared were interest circle calls (group teleconference calls), individual site coaching (that is, in-person, telephone, and e-mail support provided to each site), in-person learning sessions, and a combination of all three (34). All studies appear to have included in their sessions some didactic training in a particular care process or specific practice. One study, which focused on care for attention-deficit hyperactivity disorder in primary care clinics, used a combination of shorter in-person sessions (four 90-minute sessions focused on didactic lectures and quality improvement methods) and office visits (28–30).

In the National Child Traumatic Stress Network model, all LC participants had already received standard training by the treatment developer in trauma-focused cognitive behavioral therapy before the LC began (7). Participants in a NIATx collaborative took part in a two-day workshop on an evidence-based practice, Seeking Safety (19), in addition to LC activities. Similarly, participants in the NASBHC collaborative learned core components from evidence-based treatment elements for depression, anxiety, disruptive behavior disorders, and substance abuse, along with selected manualized interventions (39). Participants in an LC on engagement strategies received training for agency staff in addition to the standard learning sessions (37).

All of the studies that included descriptions of the in-person sessions also reported that the LC faculty provided training in quality improvement techniques, such as engaging in PDSA cycles or improvement projects. Very few details were provided on the techniques that were taught. In some studies, the LC purveyors had already identified potential areas for improvements that sites should consider for their quality improvement projects (for example, domains in the chronic care model, system improvements, and known implementation barriers) (7,18,31,33). In addition to didactic training related to practices and quality improvement methods, four of the studies reported that individual sites presented information to other participating quality improvement teams during the in-person sessions (7,15,27,37). Few specific details were included about the structure of these cross-site collaborative efforts. Some studies reported having individual site presentations, breakout sessions among “affinity groups,” or the use of “storyboards” (7).

PDSAs.

Twelve studies reported use of PDSAs between in-person sessions during “action periods” (7,17–19,28–31,34–38,41). However, it was largely unclear what occurred during the PDSA cycles, how they were used, or how the ongoing data collection informed the quality improvement process. However, a few studies provided some detail about use of quality improvement methods. In those LCs, the faculty set forth possible improvement areas from which a site could develop its PDSAs or provided hands-on coaching and support (7,17,18,29–31,34,37,38). One study did not include PDSAs but instead provided teams with a template to develop “work plans” to facilitate the integration of mental health and primary care in school-based health centers (39).

Quality improvement team calls.

Twelve studies reported that there were calls between in-person sessions for the quality improvement teams (7,15–18,27,31–39). The calls were typically held monthly with the goal of allowing sites to share progress and solve problems together. Few details were provided on the content or structure of the calls. Two studies reported holding “all collaborative” calls to facilitate sharing and problem solving (7,37). Others described “affinity group” calls targeted toward clinical supervisors, change leaders, or executive leadership or calls focused on specific clinical issues and other special topics (7,38). Studies using the NIATx model also described holding individual site coaching calls focused on the use of process improvement methods (34,38).

E-mail or Web support.

Six studies reported e-mail or Web-based support for the LC participants (7,17,33–37). Articles did not provide information about the extent to which LC participants used e-mail Listservs or Web-based support to communicate with other LC participants or LC faculty.

Quality improvement processes.

Eleven studies reported some type of ongoing data collection for the purposes of the LC (for example, performance indicators and ongoing reporting on target outcomes) (7,17,18,28–30,32–36,38,39,41); eight reported that the LC faculty provided sites with data-based feedback (18,28–30,33–36,38,39,41). Nine studies reported external support with data collection and feedback (17,18,28–30,33–36,38,39,41). With a few exceptions (7,30,33,34,38), most articles provided very little information about the data collected, how data were used, or how the data informed quality improvement activities.

Organizational involvement.

Ten studies reported that the organization’s leadership was involved in the LC (7,15–17,19,27–30,34–38). However, it was unclear whether the organizational leadership was included as a part of the quality improvement team or was engaged through other outreach efforts. We also examined indicators of the LCs’ penetration into the broader organization by tracking the training provided to staff who were not members of the quality improvement team, either by LC faculty or by local quality improvement team members themselves. No studies reported providing expert training (conducted by LC faculty or treatment developers) for frontline staff members who were not already on the quality improvement team. Five studies reported that quality improvement team members trained additional staff in the organization (16,17,31,32,39).

Pre-LC activities.

Finally, we tracked “prework” activities, which we defined as planning activities delineated in the original IHI BTS model (8,9). Only five studies reported that the LC used an expert panel during this prework phase—that is, a planning group that identifies targets for improvement and that plans the LC (7,35–39). Eight studies reported requiring formal commitments, application criteria, or readiness activities before the start of the LC (7,15,16,18,27,32,35,36,38,39).

Study goals and findings

Study goals.

The primary intent of 19 of the 20 articles was either to explore general feasibility and acceptability of the LC model or to examine pre-post LC changes at the patient and provider level. The only RCT was designed to test various components of the LC to determine which were most related to change (34). In this study, sites were randomly assigned to receive interest circle calls (group teleconference), individual site coaching, in-person learning sessions, or a combination of all three components with the intent of examining which components were related to study outcomes. The study’s use of individual site coaching is somewhat unique. Individual site coaching was described in some studies of the NIATx model (35,36), but most articles did not specify the use of coaching.

Across the studies, ten examined provider-level variables (7,17–19,30–32,37,39,41), 11 examined patient-level variables (15–17,19,28,29,34–36,38,41), nine examined acceptability of the LC model to providers (7,17–19,31,32,37,39,41), and eight examined sustainability of the changes achieved (7,19,27,30,31,34,36,39). One study examined the relation between LC components and study outcomes in an RCT (34). Three studies examined how elements of the LC process may have contributed to the findings from the LC by exploring issues such as the relation between reported barriers and facilitators (31), social networks (31), and theoretically or empirically derived attitudinal and contextual factors (for example, team effectiveness) (33,40) and changes in outcomes. In addition, two articles provided cost estimates for participation in the collaborative (31,34) (Table 3).

Study findings.

There was wide variability in study designs and methods, quality of the methodology, and methodological details provided in the articles. Moreover, with the exception of one RCT (34), the strength of the outcomes was difficult to judge across studies because of the lack of control groups and the variability in the reporting of the LC elements. Therefore, we were unable to draw conclusions about the overall effectiveness of the LC within the mental health context.

However, the study by Gustafson and colleagues (34) suggested that certain LC elements may be more potent in predicting patient outcomes. Specifically, the authors found that waiting times declined for clinics in the individual site coaching, in-person learning sessions, and a combination of three LC components (in-person learning sessions, individual site coaching, and group calls). They also found that the number of new patients increased for the combination and coaching-only groups and that interest circle group teleconferences had no impact on outcomes. Although individual site coaching and the combination intervention were considered to be similarly effective, individual site coaching was more cost-effective over the 18-month study period ($2,878 per clinic versus $7,930) (34).

Of the 19 other articles that were not RCTs, most reported positive findings with respect to patient, provider, or sustainability variables. Each of the ten articles that reported on provider-level variables reported positive trends from pre- to post-LC, suggesting improvements in areas such as process of care and uptake of new practices (7,17–19,30–32,37,39,41). Similarly, although there were some mixed findings, each of the 11 articles that reported on patient-level variables reported positive pre- to post-LC changes in areas such as symptoms and engagement in services (15–17,19,28,29,34–36,38,41). Six of the eight articles that reported on sustainability reported sustained use of new practices or procedures after the conclusion of the LC (7,27,30,31,36,39). In addition, the LC model was reported to be feasible and acceptable to providers in each of the nine articles that assessed these variables (7,17–19,31,32,37,39,41).

Discussion

The use of LCs in the mental health context is an important area for research as policy makers seek to scale up evidence-based practices and improve the quality of care. LCs are being widely used as an attractive alternative to traditional developer training models because they hold promise for achieving sustained change in a way that typical treatment developer trainings may not (7,44–46). LCs can help sites build local capacity and address organization- and provider-level implementation barriers (44,47,48). They have the potential to foster local ownership of the implementation process, promote transparency and accountability, create a culture of continuous learning, provide an infrastructure for addressing barriers, and cultivate support networks (7,44).

The major challenge for the mental health field is the lack of rigorous studies of LCs. In our previous review, we found 20 studies of LCs in other areas of health care that used comparison groups (14), but only one study in mental health care was an RCT (34). In the review reported here, we identified 20 articles that reported data on LC outcomes. Although we can be encouraged by the positive trends reported in these studies with respect to provider, patient, and sustainability outcomes, the findings must be interpreted with caution given the lack of comparison data. In addition, because of the variability in methods and rigor used in these studies, it was not possible to come to any broad conclusions about the effects of LCs on provider- or patient-level outcomes.

It is critical that future research on LCs include more studies with comparison conditions—ideally with randomized designs that can examine the impact of different implementation strategies. There are a number of quality improvement approaches to implementation of new practices that could be tested against LCs. Evidence has been found for several approaches in terms of improving the quality of care: audit and feedback methods from health care (49); individual site–focused quality improvement initiatives that involve training of local quality improvement teams, leadership support, coaching, and audit and feedback (50,51); and the availability, responsiveness, and continuity model, an organizational-level quality improvement intervention (52). In addition, a review of Six Sigma and Lean continuous improvement approaches borrowed from industry and applied in health care suggest that these are promising strategies that could be further tested (53). Of particular importance are studies such as the one conducted by Gustafson and colleagues (34) that can identify which structural and theoretical components of LCs contribute to favorable outcomes.

Recent studies provide insights into active components that could be directly tested. These include cross-site and local learning activities (for example, staff education, PDSAs, and team effectiveness) (31,41,48,54,55), local leadership support, sites’ ability to address common implementation barriers, expert support, ongoing data collection, and the visibility of local changes achieved through quality improvement methods (3,33,48,56–59). In addition, there is a great need to continue to examine the costs associated with LCs and the incremental cost-benefit of using this approach, compared with traditional developer trainings and other quality improvement methods. This type of information is critical for decision makers because LCs can be costly. One study of an LC for depression care reported that the average cost of participation was more than $100,000 per site (31). Another study suggested that the added cost of in-person learning sessions may not bring much incremental cost-benefit with respect to patient outcomes, compared with individual site coaching (34).

With respect to the reporting of LC components, we found patterns similar to those found in previous research. Prior reviews have highlighted the variation in implementation of the LC model and inconsistent reporting of components (4,13,14,25). Across studies, the LCs in this review had a similar structure. However, insufficient detail was provided with respect to the presence of LC components and how they were implemented in most studies. Moreover, because the original QIC models in health care were based on management theory (10–12), the lack of specificity on how process improvement was conducted, how quality improvement data were collected, and how data were used is striking. It is essential to carefully describe how quality improvement methods are being used in mental health care because previous studies have suggested that LC participants perceive instruction in quality improvement methods to be useful (31,48,59) and because the innovations implemented in mental health are often complex evidence-based treatments that may require adaptations from the original QIC models in health care. This review provides one potential template for the reporting of specific LC components, each of which should be reported in sufficient detail that others could replicate the activities and processes (that is, “dosage” provided, engagement of participants, details on how quality improvement was taught, how data were used, and how teams and leadership were engaged). In addition, it will be important for future research to report on and explore theoretically driven active ingredients of LC by examining not only structure but also LC processes.

Some limitations should be considered in interpreting these findings. As with any systematic review, it is possible that relevant studies were omitted. By searching multiple databases, reviewing the reference lists of key articles, and cross-checking with free-text search terms, we minimized the possibility of such omissions. In addition, negative findings are generally not published, potentially biasing our results. Despite these potential limitations, our review provides an important assessment of the state of the evidence for use of LCs in mental health care. Uses of LCs that focus on processes of care (for example, implementation of engagement practices and depression guidelines) align more closely with the targets of collaboratives that have been applied in other areas of health care. The applicability of LCs for disseminating and implementing more complex mental health evidence-based practices remains unknown; in the mental health field, such efforts often require additional specialized trainings to develop provider skills in implementing these evidence-based practices. The cost-effectiveness or added value of such an approach must thus be carefully assessed.

Conclusions

As LCs continue to grow in popularity among policy makers and national organizations, there is great need for rigorous research that evaluates the utility of these costly endeavors. Moreover, research focused on active components of LCs is vital to the replication of successful LCs, ensuring quality and fidelity to the model, guiding future adaptations, and identifying the types of innovations and improvements for which the model is most appropriate.