Fidelity of a Strengths-Based Intervention Used by Dutch Shelters for Homeless Young Adults

Homelessness among young adults is a serious problem worldwide (1,2). Even though the estimation of homeless young adults in the Netherlands varies widely, the minimum estimate is 9,000 (2). Although this population experiences a wide range of problems, including abuse and trauma (3), addiction (4), and general medical and mental health problems (5,6), scientific evidence for effective interventions targeted at improving quality of life for this group is scarce (7). Moreover, there is a gap between homeless young adults’ expressed needs and the support provided by professionals (8,9). In the Netherlands, a strengths-based intervention for these youths, called Houvast (the Dutch word for “grip”), has been developed in collaboration with professionals and homeless young adults and manualized for standardized training and supervision of mental health professionals. The intervention is based on the strengths model as developed by Rapp and Goscha (10) and has been tailored to the homeless young adult population. The main aim of Houvast is to improve the quality of life of these youths by focusing on their strengths and stimulating their capacity for self-reliance.

Fidelity is the degree to which an intervention that is being applied in practice adheres to the model (11). For adequate implementation, a well-defined description of the intervention is required and includes an operational definition of the critical components and the tools to be used (12). Without this level of specificity, dissemination of a model is vulnerable to deviations in practice. This vulnerability highlights the importance of assessing the degree of fidelity when examining the effectiveness of an intervention.

Assessment of model fidelity also can be used as a tool for internal quality assurance. Based on its results, an action plan with concrete suggestions for improving further implementation of an intervention can be formulated (10). In addition, fidelity assessments help team leaders and management to satisfy the need for external accountability requirements. Also, funders and other stakeholders show increasing interest in fidelity assessments to ensure quality of service.

Over the past decade, there has been considerable methodological progress in the measurement of fidelity (10), as demonstrated by the development of fidelity scales for different models, such as assertive community treatment and the strengths model (10,13). Several studies have revealed that effective interventions compared with others tend to be characterized by higher fidelity scores that produce better client outcomes (11,12,14–19). Houvast is a strengths-based intervention for which a fidelity scale is available (10). This article describes the fidelity assessment of the Houvast intervention in five shelters for homeless young adults.

Methods

Participants

As part of a study of the effectiveness of Houvast, fidelity assessment of this strengths-based intervention was conducted among five Dutch shelters for homeless young adults. These facilities provide ambulant care (one facility), residential care (two facilities), or both (two facilities) to homeless young adults ages 18 to 26. More details on the effectiveness study can be found elsewhere (20).

Informed consent was obtained from homeless young adults as well as professionals. The study fulfilled the criteria for approval by an accredited Medical Review Ethics Committee within the region Arnhem-Nijmegen. Upon consultation, the Ethics Committee stated that because of the behavioral character of the intervention, the study (registration number 2011/260) was exempt from formal review.

Introduction and Implementation of Houvast

Houvast is based on the principles of the strengths model and uses the tools specified for the model. A theoretical framework and trajectory were elaborated, and training material, including new tools, was developed for working with homeless young adults. [Further information about Houvast and the implementation activities we used is presented in an online supplement to this article. This supplement also contains baseline demographic characteristics of the 117 participants and additional information about a progress evaluation held six months after the audit.]

In fall 2011, we introduced Houvast at all five shelters that were randomly allocated to the experimental condition of the study. Upon consenting to participate, two to three managers of each facility attended a meeting in which additional information on the study was provided, as well as guidelines to optimize the implementation of Houvast. Team leaders, who are responsible for team work and the daily organization of tasks at the shelters (such as creating work schedules), attended a two-day training session provided by certified trainers. Team leaders were taught how to support professionals in adhering to Houvast and to maintain quality in implementation, for example through group supervision meetings. In addition, all professionals who are responsible for the daily support of homeless young adults received a four-day training course in the same period (October 2011–January 2012). Six days of training (April 2012–May 2012) were provided to supervisors, who were part of the team and were responsible for providing feedback to professionals to attain fidelity to the Houvast model. Besides training in the basic aspects and competencies of supervision, supervisors were taught to review the Houvast tools, such as the strengths assessment and the personal recovery plan. Finally, in September 2012, all professionals attended a one-day follow-up training session.

Strengths Model Fidelity Scale

Fidelity was measured with the strengths model fidelity scale, developed and validated by Rapp and Goscha (21). Table 1 shows the Dutch strengths model fidelity scale, the data sources, and the instruments we used for assessment (10). The scale consists of ten indicators distributed across three subscales—structure, supervision, and clinical practice—and includes professionals’ responsibilities, caseload ratio, group supervision, supervisor duties and caseload, strengths assessment and its integration, use of personal recovery plans, community contact, use of naturally occurring resources, and use of hope-inducing behavior. Quality requirements are listed for each indicator.

Each indicator is rated on a 5-point scale, as is commonly used in other fidelity research (22). For example, the item “Strengths assessment is used to help clients develop treatment plan goals” is rated as 1, up to 60% of the files; 2, 61%−70% of the files; 3, 71%−80% of the files; 4, 81%−90% of the files; or 5, 91%−100% of the files. A total fidelity score was obtained by averaging the scores on the ten indicators. A score of ≥4 indicates sufficient fidelity and <4 indicates insufficient fidelity (13,21).

Fidelity Assessment

To acquire a fidelity score, we translated and adjusted several fidelity assessment tools and methods, such as interview outlines, an observation scheme for group supervision, and questionnaires. Two trained assessors who had no relation to the teams conducted a one-day audit at each of the five facilities. During a four-day training course, assessors were taught how to conduct the audit, obtain measurements, and write a standardized report. During this training, the assessors’ scores on indicators were discussed until consensus was reached, and during the audit they obtained complete interrater agreement on the rating of group supervision.

Each audit included the following activities: observation of a group supervision meeting, a file analysis of a maximum of 12 files of six randomly chosen professionals whose clients received care for at least one month (N=46), five focus group interviews with three to four homeless young adults (N=19) who received care in the shelter for at least one month, and a separate interview with the team leader (N=5) and the supervisor (N=4) to gather additional information about their respective roles and the implementation of Houvast. Two weeks before the audit, the professionals, supervisors, and team leaders were asked to fill out a questionnaire on the use of Houvast (such as their use of personal recovery plans and naturally occurring resources). The homeless young adults who participated in the focus groups received care at the shelter for five to 47 weeks on average. Audits were conducted between June and September 2012.

Each shelter received a report on the audit, which included scores on the indicators, the total fidelity score, a summary of the results, and specific recommendations for improving model fidelity. Within six weeks after the audit, we held a conference call with the team leader, the supervisor, and the manager to evaluate the audit and to discuss the fidelity results and recommendations.

Booster Sessions and Progress Evaluation

On the basis of results of the fidelity scales, booster sessions were organized for the whole team (April–June 2013). During a booster session at each of the five shelters, a certified trainer discussed the recommendations of each fidelity report, and attendees practiced key elements of Houvast, for example by conducting role plays. Before the booster sessions were held, we evaluated the progress in fidelity since the audit. Team leaders and supervisors were asked whether the indicators of the fidelity scale (except hope-inducing behavior) had improved, deteriorated, or stayed the same compared with the audit in 2012. [Results are presented in the online supplement.]

Results

Characteristics of Participants in Fidelity Assessment

Each shelter had between five and 11 professionals using the Houvast method. In total, 43 professionals filled out the questionnaire. Nine professionals did not participate because of illness or for unknown reasons. At one shelter, all professionals completed the questionnaire. The average nonresponse rate at the other four facilities was 24% (range 9%−33%). Most of the professionals were women, and most had completed higher professional education for social work. The supervisors (N=4) had a mean±SD age of 47.6±16.3 years, were on contract to work 29 hours per week, and had worked in the organization for an average of 4.8 years. All team leaders (N=5) completed higher professional or university education and had a mean±SD age of 40.6±8.2 years.

Fidelity Scores

Table 2 shows the fidelity scores per indicator for the three subscales and the total fidelity score for all five shelters. The scores for the indicators were taken either during or after the audit, depending on indicator type. In addition, given that fidelity scores were not normally distributed, the median for the five facilities is reported. No sufficient fidelity scores of ≥4 were obtained on the total scale or on the three subscales for structure, supervision, and clinical practice; median scores, respectively, were 2.6, 3.0, 2.8, and 2.3.

At all five shelters, the fidelity scores were sufficient on the following three indicators: caseload ratio, group supervision, and strengths assessment. Caseload ratio and group supervision were sufficient at four of the five facilities, and strengths assessment was sufficient at three of the five facilities. Two shelters had sufficient scores on hope-inducing behavior. Furthermore, the execution of personal recovery plans and supervision of professionals was insufficient at all of the facilities. Only one facility used any naturally occurring resources. Moreover, all facilities received the lowest possible score on the indicator integration of the strengths assessment, and there was no evidence that professionals worked with young adults in the community (community contact). Finally, the insufficient fidelity score on the indicator professionals’ responsibilities indicates that almost all professionals had other responsibilities (completing intakes, for example) besides working with homeless young adults.

Recommendations for Improvement

On the basis of its lowest fidelity scores on the ten indicators, each shelter received a personalized report that gave three to five recommended priorities. Table 3 presents the most frequent suggestion for improvement per indicator. The progress evaluation of fidelity six months after the audit showed some improvement on use of the strengths assessment and personal recovery plans and on supervision.

Discussion

This study is the first to report on the assessment of fidelity to Houvast, a strengths-based intervention for homeless young adults. The scores on three of the ten indicators of fidelity were sufficient six months after the introduction of Houvast at the shelters and after professionals and team leaders received training in the intervention: caseload ratio, group supervision, and strengths assessment. The total score for the five participating shelters for homeless young adults showed insufficient fidelity to Houvast, but the fidelity assessment provided the participating facilities with concrete guidelines for improvement.

This study showed that much still needs to be done to attain fidelity to the Houvast model. How can we explain the apparent gap between the application of Houvast in practice and model fidelity? This study marked the first time that the Dutch version of the strengths model fidelity scale was used, and its validity for use in the Netherlands may need further analysis. Although in this study the data were too limited to investigate the quality of the scale, the results constitute a first indication for face validity because the participating professionals, team leaders, and supervisors acknowledged the scale’s indicators and perceived the fidelity scores as credible. Furthermore, the auditors considered the fidelity scale to be a useful tool for assessing fidelity to the Houvast model, they reached consensus on the indicators’ scores, and they did not encounter any problems in applying the scale despite differences between the American and Dutch context.

From questionnaires filled out by professionals and from interviews with the supervisors and team leaders, we found that professionals were using Houvast tools inconsistently at the primary process level and seemed to have some difficulty with integrating the process with other training and as part of their daily routine. Some professionals, for example, regarded the personal recovery plan as less useful or as just another form to fill out instead of as something meaningful for their homeless young clients. Also, during the focus groups with homeless young adults and on the basis of information extracted from their completed questionnaires, we found that many professionals did not model hope-inducing behavior as Houvast requires. Perhaps they needed more time to adopt and express hope-inducing competencies, given that doing so requires a shift in attitude and not merely a behavioral change to be authentic. Moreover, supervisors mentioned that their workload was too high; thus not every professional received field mentoring. Also, because supervisors received their training after the professionals and team leaders in the first four months, supervisors’ skills may not have been optimal for ensuring implementation. Furthermore, professionals reported using institutionalized resources instead of naturally occurring resources. Institutionalized resources are services or facilities that are not available for every citizen in the community, such as mental health care and substance use treatment. In contrast, naturally occurring resources are services or facilities available for every citizen in the community, such as neighbors, friends, and clubs.

The paradigm shift professionals needed to make became clear from information retrieved from the interviews with supervisors and team leaders and the focus groups with homeless young adults: Houvast requires professionals to focus on the strengths and talents of homeless young adults rather than on their problems, even though this was not part of professionals’ formal training or daily routine. This shift required professionals also to let go of their control over service delivery and the working relationship and to become a coparticipant rather than the expert in the recovery process of homeless young adults (23,24). The questions professionals asked their supervisors, such as “What [can I] do with personal goals of homeless young adults that seem unrealistic?” indicate that the essentials of Houvast had not yet been fully adopted.

From questionnaires, interviews, and conference calls with supervisors and team leaders, we learned that there were also factors at the organizational level that inhibited the adoption of Houvast and that may have negatively influenced the scoring on all subscales. First, responses often indicated that time constraints made obtaining a sufficient fidelity score on indicators such as supervision and community contact nearly unachievable. Consequently, most supervisors provided supervision when professionals experienced difficulties in their work. Also, professionals were not always able to work in the community because the work schedule for the shelter facilities required them to work on site.

Second, in most organizations only the professionals working with homeless young adults at the same facilities received training in Houvast, whereas other professionals in the same organization working at other facilities did not receive Houvast training and worked according to a more problem-oriented approach. These differences, in turn, led to conflicting demands and expectations of the professionals using the Houvast approach. At one shelter, professionals had to work with Houvast tools as well as with the tools of the organization.

Third, some team leaders or supervisors mentioned reorganizations and financial reductions, which may have caused teams to be unstable and experience high turnover of employees. Although training was offered to new staff, few Houvast-trained professionals remained part of some teams.

Fourth, the Houvast tools were incorporated in the electronic file systems of only a few organizations. At facilities where the tools were not incorporated, professionals needed additional registration time for updating client records, and this time was not always available. More registration time was often necessary because the health insurers require problem-oriented registration to fund care, whereas the Houvast tools do not generate this type of information because of the strengths base of the approach.

For these reasons, the adoption of Houvast by all those involved in the implementation process was difficult to achieve in the six-month period between the introduction of Houvast and the fidelity assessment. What can be learned about implementing a strengths model? This model is more than a collection of tools; it is a philosophical approach that requires organizationwide adoption (25). Providing supervision of implementation of new practices is essential; to lead the effort, supervisors should be trained before professionals. Team leaders should also receive training (25).

In the six-month period, the implementation of Houvast focused mainly on the internal infrastructure of the facility (working according to the tools of Houvast) and captured neither the entire organization nor relevant external relations. As confirmed by supervisors and team leaders, training should be conducted not only for all professionals working with homeless young adults in the same organization but also for colleagues in the organization who are working with other groups and for those performing other tasks, such as working at the reception desk (25). An infrastructure that supports the implementation and maintenance of Houvast over time is essential to improve model fidelity (25,26). In addition, the availability of financial resources and the willingness of organizations to adopt the strengths-based approach and to take the necessary measures to make that happen (such as investing in supervision) are factors that could boost all fidelity indicators (27–29). Previous studies have corroborated the value of supportive leadership at multiple levels (30–32), and agency and program directors who can facilitate implementation (32) are essential for model fidelity. Furthermore, greater perceived benefit (33) and a high level of congruence between organizational values and characteristics of an intervention would facilitate implementation (32,34).

Conclusions

The Dutch strengths model fidelity scale appears to be a useful tool with face validity for assessing model fidelity. In this study, much effort was invested in a comprehensive plan for the introduction and implementation of Houvast at all levels of the shelter facilities, including management, team leaders, and professionals, and for the maintenance and strengthening of Houvast via supervision. Furthermore, training proceeded according to plan, and professionals, team leaders, and supervisors were enthusiastic about the training and the Houvast intervention. The low fidelity ratings may have resulted from the timing of the fidelity assessment, which was performed six months after the introduction of Houvast and the training of professionals. This period probably was too short for professionals to fully adopt the Houvast intervention into daily practice (19,25). Furthermore, to address the challenges that professionals and organizations faced when implementing and adopting Houvast, a more comprehensive approach is needed. Important components of such an approach are building an infrastructure, training the whole organization, and ensuring supportive leadership.