Enhanced Fidelity to Treatment for Bipolar Disorder: Results From a Randomized Controlled Implementation Trial

Translating effective behavioral treatment models from research to routine practice has been identified by the National Institutes of Health as a public health priority. A number of psychosocial interventions have been shown to improve treatment adherence, quality of care, and health outcomes among persons with mental illnesses (1,2). However, few of these evidence-based practices have been successfully translated into community-based practices.

Even when evidence-based practices are delivered, treatment fidelity is suboptimal, especially in community-based practices, where resources are more limited than in traditional academic settings. Treatment fidelity is a process to ensure that the core treatment components of an intervention are delivered as intended (3,4). Fidelity can falter over time because mental health providers are not given the tools necessary to overcome barriers to implementation or they resist buying in to the new treatment (5–7).

Hence, a growing body of literature focuses on developing implementation interventions that can assist mental health providers in maintaining treatment fidelity. The Centers for Disease Control and Prevention established an implementation strategy called Replicating Effective Programs (REP) to translate effective HIV prevention strategies for community-based organizations (8,9). REP focuses on strategies that maximize fidelity to treatments through the development of treatment manual “packages” that are supported by provider training and limited technical assistance (10,11). Because these strategies mainly focus on the steps required to implement new programs, REP may not fully support the ongoing fidelity or sustainability of these practices, notably by encouraging provider buy-in and organizational support.

Few studies in the literature compare implementation strategies that are designed to enhance treatment fidelity. In one study of a community-academic partnership (12–14), investigators enhanced REP by adding facilitation, a process in which expert consultants and local stakeholders collaborate to implement and sustain new programs (15,16). Facilitation was adapted from the Promoting Action on Research Implementation in Health Services (PARiHS) framework. Facilitators help providers address organizational barriers, such as clinical flow issues and staff turnover, and foster ongoing relationships with organizational leadership to promote buy-in to the program (14,15,17).

This article reports six-month treatment fidelity outcomes from the Recovery-Oriented Collaborative Care study, a randomized controlled trial comparing the effectiveness of enhanced versus standard REP for implementing Life Goals Collaborative Care (LGCC) at community-based practices (15,18). LGCC is an evidence-based, manualized psychosocial intervention for individuals with bipolar disorder (19–23) that is based on the chronic care model (24,25). The intervention has been found in five randomized controlled trials to improve quality of life related to both general medical health and mental health and to reduce symptoms among patients with bipolar disorder (19–23). The core components of LGCC are described in Table 1.

The study employed a hybrid effectiveness-implementation model that allowed for the comparison of both clinical and fidelity outcomes at practices that received standard or enhanced REP to support uptake of LGCC (26). We hypothesized that enhanced REP would increase treatment fidelity to LGCC implementation. We also examined patient factors associated with LGCC treatment fidelity.

Methods

The Recovery-Oriented Collaborative Care study has been described elsewhere (15,18). In brief, community-based practices in Michigan or Colorado were enrolled between March 2010 and December 2011 and were assigned at random to receive enhanced versus standard REP to help implement LGCC. This study was reviewed and approved by local institutional review boards.

Study population and treatment

Five community-based clinical practices agreed to participate, two from Colorado and three from Michigan. These practices serve a majority of individuals with mental disorders in their respective regions, and they represent a diverse patient population. To be eligible, practices were required to care for at least 200 unique patients with bipolar disorder and to have available at least one mental health provider, such as a social worker, nurse, or psychologist, to implement LGCC.

Randomization was stratified by state, and practices in each state were assigned at random to receive enhanced (N=3) or standard (N=2) REP. All practices had administrative buy-in to participate in the study before randomization to implementation strategy.

Implementation intervention

Providers at practices using standard REP (18) were provided with three components to support preimplementation and implementation activities: an LGCC manual package that included an outline and a treatment manual and implementation guide that were not customized to the practice; a standard training program; and program support given through as-needed technical assistance and intervention monitoring and feedback reports (Table 2).

Enhanced REP included key steps to further adapt the LGCC package to the local practices on the basis of provider and consumer input. In addition, a study staff member provided facilitation for six months to enhance provider buy-in and uptake of LGCC over time (Table 2). We chose to provide facilitation for six months, balancing the potential cost of the implementation intervention with the desire to provide a more intensive intervention than REP (9).

The first step at the practices that received enhanced REP was a needs assessment to identify the practices’ current priorities and initial organizational barriers and facilitators to implementing LGCC. An additional step included customization of the LGCC package and training program for each practice on the basis of feedback from the needs assessment. For example, consumer workbooks were tailored to reflect local community resources, and group meeting frequency was limited to four sessions in both arms on the basis of enhanced REP feedback in step 1 indicating a desire to minimize travel burden.

The next three steps involved facilitation support (Table 2). Two study staff members based in Michigan and Colorado served as external facilitators, providing technical assistance to practices in implementing LGCC. In addition, external facilitation required identifying an internal facilitator at each practice who was an employee with a direct reporting line to leadership. Subsequently, external facilitators worked with internal facilitators to set measureable goals, problem-solve implementation challenges through scheduled telephone consultations, and develop plans to market and sustain the program. Internal facilitators did not provide direct LGCC services; instead, they assisted providers to implement LGCC by addressing organizational barriers unique to each practice and met with practice leadership on a regular basis. As detailed in Table 2, internal facilitators actively worked to ensure the program fit within the work flows, priorities, and long-term plans of their local practice. The internal and external facilitators worked together to identify opportunities to enhance LGCC implementation.

Procedures

Health specialists from each practice were responsible for implementing LGCC. They were jointly hired by research and clinical staff and had a clinical background in mental health care and previous experience in addressing suicidal ideation, severe manic episodes, and other emergent issues. Patients were eligible if they had a diagnosis in their medical record of bipolar disorder (bipolar I, II, or not otherwise specified) or schizoaffective disorder, bipolar subtype; received outpatient care from the participating practices; and lived in the community rather than a nursing home or other institution. Patients were excluded if they were not able to provide informed consent because of serious illness or evidence of intoxication at enrollment.

After identifying eligible patients, the health specialists approached them by mail, by phone, or in person to request that they participate. Patients were enrolled after providing informed consent and completing a baseline assessment survey. Participants were asked to attend weekly group sessions for four weeks, followed by monthly individual care management phone calls for six months. Health specialists recorded patient attendance, clinical status, and length of care management phone contacts in an electronic registry. Participants in both implementation groups received $5 in remuneration for each group session that they attended to cover transportation costs and $20 for each survey assessment completed at baseline, six, 12, and 24 months.

LGCC training, supervision, and fidelity

Health specialist training, supervision, and fidelity monitoring were similar across enhanced and standard REP practices. Training lasted two days and covered the content of the four self-management sessions, registry use, and care management contacts. All health specialists participated in biweekly conference calls with the research team lasting one hour, when recruitment progress and other study issues were discussed.

Study investigators provided additional education and training to the health specialists at the three practices that received enhanced REP. This training was designed to address the specific organizational factors and barriers identified by each practice in its needs assessment. Health specialists at the practices that received enhanced REP also participated in six monthly calls with the external facilitator, who employed a problem-solving approach to discuss implementation barriers and offer support to overcome these barriers. A detailed mixed-methods description related to fidelity to these enhanced REP implementation strategy processes will be published separately.

Measures

Fidelity outcomes.

We used a practical approach for assessing treatment fidelity that was based on the National Institutes of Health’s Behavioral Change Consortium fidelity framework for psychosocial treatments (3,4) and previous LGCC intervention studies (19,27). Fidelity measures were developed a priori on the basis of core LGCC theoretical components (Table 1). Our two primary outcomes for fidelity included number of group self-management sessions completed and number of care management contacts completed.

In addition, study staff observed a random sample of 25% of group self-management sessions and used the LGCC session fidelity checklist to record information about the session (Table 1). The checklist is part of the LGCC package and is used by an independent rater during direct observations of all four self-management sessions (28). Among the multiple group cohorts of participants managed by each health specialist, the rater randomly selected a fourth of these cohorts to observe and rate (29,30). The first section allow raters to record session length, number of participants, location, potential sources of delivery problems, and the overall degree to which the LGCC session’s educational objectives were met. Section 2 rates characteristics of the provider, such as whether he or she was prepared and organized and ensured that participants understood the didactic content, and how well he or she met process goals, such as facilitating participation of discussion and interaction, avoiding judgmental feedback, and displaying empathy. Section 3 rates the degree to which the session covered designated focus points. We used the checklist to collect information about the number of focus points covered and process goals achieved in each session as well as the total amount of time for each session.

Cutpoints for minimum treatment fidelity were based on previously established minimum necessary standards (19,27) and consisted of attending at least three self-management sessions and having at least four care management contacts. For optimal fidelity, a cutpoint of attending all four group sessions and having at least six care management contacts was used (although additional care management contacts could be delivered as needed).

Covariates.

Independent variables that are thought to influence fidelity and also differ between practices that received enhanced or standard REP were ascertained from the patient baseline survey. Variables included demographic characteristics (age, gender, race-ethnicity, education, and living arrangement) and lifetime history of homelessness, defined by a standard question ascertaining whether the patient ever spent at least one night in a shelter, park, abandoned building, or street. Other covariates included clinical indicators at baseline, including depressive symptoms, alcohol use, and comorbid general medical illnesses. Depressive symptoms were ascertained by the Patient Health Questionnaire nine-item survey (PHQ-9) (31). Alcohol use, particularly hazardous drinking, was defined by the question on binge drinking from the Alcohol Use Disorders Identification Test alcohol consumption questions (32,33). The number of comorbid psychiatric and general illnesses was ascertained from patient self-report and was based on the question, “Has the doctor ever told you that you have one or more of the following?” Patients were asked to check boxes for hypertension or high blood pressure, arthritis or chronic pain, angina or coronary heart disease, heart attack or myocardial infarction, depression, posttraumatic stress disorder, diabetes or high blood sugar, or high cholesterol or parents with high cholesterol.

Data analysis.

Multiple regression models were used to compare treatment receipt fidelity (number of group sessions, number of care management contacts, and total number of sessions and contacts) between standard and enhanced REP treatment arms. Logistic regression models were used to compare minimum and optimal treatment receipt fidelity between patients at the practices that received enhanced versus standard REP. All models were adjusted for age, gender, race (white versus nonwhite), college education (yes or no), living arrangement (alone versus with others), lifetime history of homelessness (yes or no), depression symptoms (PHQ-9 score), hazardous drinking (yes or no), and number of comorbid general medical illnesses. For all fidelity models, an alternative analysis was conducted by including individual sites as fixed effects. All analyses were conducted by using SAS, version 9.2.

Results

A total of 2,019 potentially eligible patients were identified from medical record reviews as having a diagnosis of bipolar disorder, of whom 1,158 were found after consultation with providers to be ineligible. Another 477 patients declined to participate or could not be contacted, leaving 384 patients who consented and were enrolled in the study. Between enrollment and the start of LGCC group sessions, 67 patients were excluded from program participation because of dropout (N=51), physical illness (N=5), time conflict (N=4), incarceration (N=4), relocation (N=2), and cognitive impairment (N=1). Thus 317 patients started LGCC group sessions, 140 at the practices that received standard REP and 177 at the practices that received enhanced REP, and were included in the analyses. The patient mean±SD age was 42±11 years, 67% (N=256) were female, and 30% (N=115) were nonwhite, including 64 (17%) African Americans, 34 (9%) Hispanics, eight (2%) Native Americans, seven (2%) persons of multiracial background, and two (1%) Asians. Eighteen (5%) patients declined to divulge their race-ethnicity.

Patients attended 3.0±1.2 self-management group sessions (range 1–4), with 49% attending all four group sessions (Table 3). Group session length averaged 114 minutes (range=112 to 120 minutes). The group sessions conducted by all health specialists covered at least 80% of focus points.

Participants had 4.0±2.4 care management contacts, and 57% had at least four contacts. Each care management contact lasted 22.6±12.1 minutes (range three to 80 minutes).

The total number of sessions and contacts was higher at practices that received enhanced versus standard REP (8.1±2.9 versus 5.5±2.1, p<.001). Patients at practices that received enhanced REP had significantly more care management contacts (5.0±2.4 versus 2.6±1.7, p<.001), but there was no significant difference in the number of group sessions attended at practices that received standard or enhanced REP.

Logistic regression analyses revealed that patients of practices that received enhanced REP were 7.2 times more likely to achieve minimum treatment receipt fidelity and 22.3 times more likely to achieve optimal treatment receipt fidelity (Table 4).

Regression analyses also revealed that participants with a history of homelessness were less likely to have care management contacts (β=–.72, p<.05). Younger age (β=.03, p<.05) and living alone (β=.72, p<.05) were associated with fewer total sessions and contacts. Additional analyses in which practices were added as fixed effects produced similar results (Table 5).

Discussion

The Recovery-Oriented Collaborative Care study was one of the first comparative effectiveness trials of two different implementation strategies. One of its goals was to determine whether an enhanced versus standard version of a well-established implementation strategy improved fidelity to a psychosocial treatment for bipolar disorder. There is a growing demand for specific implementation interventions that are theory based and that are proven to enhance and maintain the fidelity of evidence-based practices. However, few implementation frameworks have been rigorously tested in health services trials for use as implementation strategies.

We found that compared with standard REP implementation, enhanced REP improved overall fidelity to LGCC independent of patient factors. The differences in fidelity were most pronounced for the care management component of LGCC. The level of treatment receipt fidelity to LGCC among patients of practices that used enhanced REP was similar to estimates reported in previous randomized controlled trials in more tightly controlled clinical settings (19,27). As in to these previous LGCC studies, we used practical fidelity assessments that were easy to deliver, such as the fidelity rater checklist. We were able to demonstrate good treatment delivery and fidelity of treatment receipt of the LGCC intervention.

Overall, high patient-level treatment receipt fidelity for number of group sessions attended and care management contacts was achieved at all practices. Health specialists across all of the practices demonstrated good fidelity to LGCC focus points and process goals in the group sessions. These findings could be due to the small incentives given to patients to cover transportation costs to group sessions. LGCC group sessions were also limited to four, which may have encouraged participation.

Compared with standard REP, enhanced REP was associated with attendance at significantly more total sessions and contacts and was more likely to be associated with optimal fidelity. The association of enhanced REP and optimal fidelity was due mainly to increased receipt of care management contacts. Often the hardest component to maintain, care management is a crucial component of LGCC and similar chronic care models because it facilitates assessment of clinical status over time and encourages positive changes in health behavior. Enhanced REP might have improved sustainability of care management because the external and internal facilitators were able to help the health specialists to secure the resources required for group sessions and provider follow-up. The added facilitation components, such as leadership engagement and customization, may have contributed to improved fidelity overall by promoting greater overall acceptance of LGCC in the practices. Greater use of care management in the practices that used enhanced REP also suggests that participants were more engaged with the health specialist and more focused on their wellness goals.

Additionally, we examined patient factors that influenced treatment delivery fidelity. Younger participants and those who were living alone participated in fewer group sessions. We also found that women participated in fewer group sessions and that study participants who had a history of homelessness had fewer care management contacts. Younger individuals and persons who live alone may have difficulty attending group sessions because of job constraints or transportation issues. Women may have difficulty attending group sessions because of child or other family caretaking responsibilities and may need different options for group times or child care. Similarly, participants with housing instability might benefit from more flexibility in delivery of care management contacts, such as face-to-face encounters or use of other technologies that enable more cost-efficient communication.

The study had limitations that may affect generalizability to other community settings. We were unable to assess contacts with other providers, either by the health specialists or by participants, or assess whether enthusiasm for LGCC at the sites varied across individual providers or over time. We were also unable to describe the specific components of enhanced REP, including the facilitator’s actions, that led to improved fidelity. We were unable to monitor fidelity to provider guideline support, including use of guidelines by other frontline providers in the practice, or to comprehensively assess patient factors, including co-occurring psychiatric conditions, that may have influenced fidelity. Also, the health specialists who delivered LGCC were all master’s-level clinicians with prior clinical experience, who may not represent the providers available elsewhere, especially in smaller community-based clinics.

Conclusions

Implementation interventions, such as enhanced REP, that include ongoing technical assistance to help providers implement evidence-based practices may improve treatment fidelity. However, future research should consider whether the added costs of facilitation can be absorbed by health care organizations because they contribute to the overall cost-efficiency of delivering effective psychosocial treatments in routine practice. Finally, measures that assess fidelity to the implementation intervention may be useful tools to improve the translation of research into practice. Such fidelity measures should be further developed in order to better assess the uptake of evidence-based practices.