Abstract
Purpose
To ensure that more people will benefit from integrated care initiatives, scaling-up of successful initiatives is the way forward. However, new challenges present themselves as knowledge on how to achieve successful large-scale implementation is scarce. The EU-funded project SCIROCCO uses a step-based scaling-up strategy to explore what to scale-up, and how to scale-up integrated care initiatives by matching the complementary strengths and weaknesses of five European regions involved in integrated care. The purpose of this paper is to describe a multi-method evaluation protocol designed to understand what factors influence the implementation of the SCIROCCO strategy to support the scaling-up of integrated care.
Design/methodology/approach
The first part of the protocol focuses on the assessment of the implementation fidelity of the SCIROCCO step-based strategy. The objective is to gain insight in whether the step-based strategy is implemented as it was designed to explore what works and does not work when implementing the scaling-up strategy. The second part concerns a realist evaluation to examine what it is about the SCIROCCO’s strategy that works for whom, why, how and in which circumstances when scaling-up integrated care.
Findings
The intended study will provide valuable information on the implementation of the scaling-up strategy which will help to explain for what specific reasons the implementation succeeds and will facilitate further improvement of project outcomes.
Originality/value
The expected insights could be useful to guide the development, implementation and evaluation of future scaling-up strategies to advance the change towards more sustainable health and care systems.
Keywords
Citation
Grooten, L., Alexandru, C.-A., Alhambra-Borrás, T., Anderson, S., Avolio, F., Valia Cotanda, E., Gütter, Z., Henderson, D., Kassberg, A.-C., de Manuel Keenoy, E., Lange, M., Lundgren, L., Pavlickova, A., Txarramendieta Suarez, J., Whitehouse, D., Fullaondo Zabala, A., Zabala Rementeria, J.I. and Vrijhoef, H.J.M. (2019), "A scaling-up strategy supporting the expansion of integrated care: a study protocol", Journal of Integrated Care, Vol. 27 No. 3, pp. 215-231. https://doi.org/10.1108/JICA-04-2018-0029
Publisher
:Emerald Publishing Limited
Copyright © 2018 Liset Grooten, Cristina-Adriana Alexandru, Tamara Alhambra-Borrás, Stuart Anderson, Francesca Avolio, Elisa Valia Cotanda, Zdenek Gütter, Donna Henderson, Ann-Charlotte Kassberg, Esteban de Manuel Keenoy, Marc Lange, Lisa Lundgren, Andrea Pavlickova, Jon Txarramendieta Suarez, Diane Whitehouse, Ane Fullaondo Zabala, Joseba Igor Zabala Rementeria and Hubertus Johannes Maria Vrijhoef
License
Published by Emerald Publishing Limited. This article is published under the Creative Commons Attribution (CC BY 4.0) licence. Anyone may reproduce, distribute, translate and create derivative works of this article (for both commercial & non-commercial purposes), subject to full attribution to the original publication and authors. The full terms of this licence may be seen at http://creativecommons.org/licences/by/4.0/legalcode. This study has received funding from the European Union’s Health Programme (2014-2020).
Introduction
European countries are dealing with both an evident increase in the proportion of elderly citizens and a rapid rise in the number of people with multiple health and care needs (Busse et al., 2010). These changes place severe pressure on Europe’s society, economy and healthcare systems (European Commission, 2013). The need to transform fragmented health and social care systems towards people-centred and integrated health and social care is widely supported (WHO, 2016).
Already, in 2001, the World Health Organization (WHO) defined integration of care as “bringing together inputs, delivery, management and organization of services related to diagnosis, treatment, care, rehabilitation and health promotion wherein integration is regarded as a means to improve the services in relation to access, quality, user satisfaction and efficiency” (Gröne and Garcia-Barbero, 2001). More recently, the WHO (2015) highlighted a people-centred and integrated health services approach (see Box 1) presented in the form of a global strategy and offering a way forward for comprehensive health system design. There has been a considerable evolution in the definition of integrated care. A shift can be recognised from a definition focussed on services towards a focus on people-centred care, including a comprehensive perspective of people’s needs and a sensitivity to the context-specific nature of health systems.
Over the years, various integrated care initiatives have been developed and implemented in European healthcare systems (Busse and Stahl, 2014; Nolte et al., 2008, 2016; Raak et al., 2003). To ensure that the wider population in Europe can benefit from integrated care, scaling-up of successful initiatives is desirable. In 2012, the European Commission launched the European Innovation Partnership on Active and Healthy Ageing (hereafter referred to as “the Partnership”), which was designed to connect and engage stakeholders across sectors and to facilitate the scaling-up of innovations for active and healthy ageing (The European Innovation Partnership on Active and Healthy Aging, 2015). Within the Partnership, it is assumed that, by sharing experiences of the development and implementation of good practices in European regions, lessons for stakeholders in other regions can be provided that will help to simplify and speed up the process of adaptation and implementation in their regions. This assumption is reflected in the Partnership Scaling-up Strategy (The European Innovation Partnership on Active and Healthy Aging, 2015). The first three steps of this strategy focus on what to scale-up, while the other two steps focus on how to scale-up.
To know what to scale-up is, however, a challenging task as evidence of successful (elements of) integrated care initiatives is not readily available. A number of challenges abound. Notwithstanding reviews focussing on the effectiveness of integrated care interventions reported positive outcomes, heterogeneity in outcomes is also found (Bongaerts et al., 2017; Busetto et al., 2015; Elissen et al., 2012; Nolte and Pitchforth, 2014; Pimouguet et al., 2011). The explanation of the heterogeneity in outcomes partly lies in the different approaches used for the implementation of interventions (Nolte and Pitchforth, 2014), the diversity in components used in interventions (Elissen et al., 2012; Nolte and Pitchforth, 2014) and the evaluation or research paradigms used (Busetto et al., 2015; Nolte and Pitchforth, 2014). The scaling-up of these interventions is yet another challenge since elements specific to a pilot site, which enabled a pilot to work in that particular context, might not be present in a new context where the initiative is planned to be transferred to (i.e. another care setting, community or country). An understanding of what elements are related with successful implementation on a wider scale remains largely absent (Bardsley et al., 2013; Darker, 2014; Liebman et al., 2007; Low et al., 2011; Nolte et al., 2016; Wodchis et al., 2015).
WHO’s definition on integrated health services and people-centred care
Definition of integrated health services: health services that are managed and delivered so that people receive a continuum of health promotion, disease prevention, diagnosis, treatment, disease-management, rehabilitation and palliative care services, coordinated across the different levels and sites of care within and beyond the health sector, and according to their needs throughout the life course Definition of people-centred care: an approach to care that consciously adopts individuals’, carers’, families’ and communities’ perspectives as participants in, and beneficiaries of, trusted health systems that are organised around the comprehensive needs of people rather than individual diseases, and respects social preferences. People-centred care also requires that patients have the education and support they need to make decisions and participate in their own care and that carers are able to attain maximal function within a supportive working environment. People-centred care is broader than patient and person-centred care, encompassing not only clinical encounters, but also including attention to the health of people in their communities and their crucial role in shaping health policy and health services
Source: WHO (2015)
Acknowledging the need to know how to make use of the experiences and knowledge of current integrated care initiatives and to make the learning easily available and accessible to potential adopters, one of the six Action Groups within the Partnership (i.e. the B3 Action Group on Integrated Care) has thought of a structured approach to facilitate upscaling of successful initiatives. It is in this light, that the B3-Maturity Model (MM) was developed to obtain a more standardised approach for scaling-up integrated care throughout Europe (Henderson et al., 2016). Testing and validation of the MM is needed to demonstrate its full potential as a tool for supporting regions in Europe to understand the preconditions for successful scaling-up. Building on the extensive experience of the Partnership, the EU-subsidised project SCIROCCO aims to test the model to facilitate the successful scaling-up and transfer of good practices in integrated care across European regions. In doing so, SCIROCCO is implementing a step-based scaling-up strategy to explore what to scale-up in integrated care initiatives and how to scale-up these initiatives in five European regions. To support the implementation and scaling-up of integrated care in these five European regions, SCIROCCO is interested to match regions with complementary strengths and weaknesses in integrated care and facilitate twinning and coaching activities between these regions to promote shared learning.
The SCIROCCO scaling-up strategy is a promising approach intended to guide system changes towards integrated care in European regions. The strategy consists of multiple interacting components which will be implemented in different setting and can be regarded as a complex intervention or strategy. To understand what factors have contributed to the outcomes of the strategy, it is important to obtain insight in whether the strategy was implemented as planned (Damschroder et al., 2009) and to obtain insight in the different regional contextual factors which might have an interaction with the strategy (Craig et al., 2008; Kernick, 2006; Lamont et al., 2016; Tsiachristas and Rutten-van Mölken, 2017). This paper describes a two-folded evaluation study protocol focussing on the implementation of the SCIROCCO strategy to obtain a thorough understanding of how the strategy operates in its intended context. Implementing the strategy might be challenging as the strategy may be modified depending on the context and there is a risk that not all planned elements will be implemented with high fidelity in the different regions. Therefore, the first part of the evaluation protocol describes the assessment of the implementation fidelity of the SCIROCCO step-based strategy. The anticipated study aims to provide information on the extent to which the envisaged activities within the SCIROCCO project have been implemented in line with expectations and if, how, and how far relevant initiatives have been developed between 2016 and 2018. This information is needed to draw accurate conclusions from the outcomes of the strategy.
The second part of the evaluation protocol concerns a method with a vital focus on the context and describes a realist evaluation design. Realist evaluation is an evaluation method using a theory-driven approach including multi-method approaches to provide an explanation for why study outcomes occur in complex interventions (Pawson and Tilley, 1997). The evaluation method is becoming widely used in evaluation concerning the implementation of complex interventions within health systems (Busetto et al., 2015; Jeffs et al., 2017; Nurjono et al., 2018). Realist evaluation endeavours to explain what it is about an intervention that has worked, for whom, how, why and under what conditions by exploring the interactions observed between the context, mechanisms and outcomes. These relationships are presented as context–mechanisms–outcomes (CMO) configurations (Pawson and Tilley, 1997). The SCIROCCO strategy includes specific actions which change contexts. These modified contexts will trigger mechanisms, leading to possible scaling-up outcomes. Realist evaluation will be used to explore what it is about the SCIROCCO’s strategy that works for whom, why, how and in which circumstances.
The evaluation of the specific SCIROCCO scaling-up strategy will be guided by the following research questions:
What factors influence the implementation of the SCIROCCO strategy to support the scaling-up of integrated care in the healthcare regions involved in SCIROCCO?
How have the contexts and mechanisms by which SCIROCCO’s strategy is implemented affected its outputs and outcomes achieved in the healthcare regions involved in the SCIROCCO project?
Methods
In the next sections, the details of the SCIROCCO strategy; study methods per research question including data collection methods; state of the evaluation; and the discussion and conclusion will be presented.
SCIROCCO’s strategy
SCIROCCO’s overarching objective is to facilitate the scaling-up of integrated care interventions at local, regional or country level by recognising the maturity requirements of good practices and health systems in order to achieve scaling-up and knowledge transfer among European Member States. SCIROCCO uses a step-based approach. Several work packages (WPs) have been designed to implement SCIROCCO’s objective from its start (April 2016) till the end of the project (November 2018). Partners from several participating regions (Basque Country (Spain), Norrbotten Lans Landsting (Sweden), Puglia region (Italy), Olomouc region (Czech Republic), Scotland (UK) and organisations (ETHEL, UVEG) including one independent evaluator (Vrije Universiteit Brussels) conduct the work in the several WPs. The step-based approach described below is derived from the grant proposal HP-PJ-2016 Grant Agreement of the SCIROCCO project (which was approved by the Third Health Programme (Regulation (EU) No. 282/2014 of the European Parliament).
Step 1: validity and reliability assessment of the MM
In the first step, the validity and reliability of the MM will be assessed to optimise the instrument. The MM instrument intends to show how healthcare systems are attempting to deliver more integrated care services for their citizens. It consists of 12 dimensions including a 0–5 scale specific to every dimension. The current status within a healthcare system is assessed by considering each of the 12 dimensions and allocating a measure of progress or “maturity” on the scale. From the assessment, a simple graphical representation (a radar diagram) of the status is developed, to reveal areas of strength as well as areas of lower maturity in integrated care. A literature review and Delphi study were performed to test the content-validity of the tool (Grooten et al., 2018). Furthermore, the underlying structure, test-retest reliability, internal consistency and convergent validity of MM will be assessed.
Step 2: maturity assessment of integrated care interventions by applying MM
The objective of the second step is to test the MM as a tool to support evaluation of good practices in terms of the maturity of integrated care and to filter and identify potentially adoptable good practices for health and care systems. A minimum of 15 local integrated care interventions (i.e. good practices) in five participating EU regions have been selected. The precondition for the selection of these good practices is that they are strategic initiatives that can contribute to the transformation of health and care systems and that there is an existing commitment to the practice. In the first phase, a viability assessment will be performed to identify the potential of good practices for scaling-up. In the next phase, the maturity requirements for successful implementation of the selected good practices will be defined using the MM. By considering each dimension of the MM, and allocating a measure of maturity to that domain, it becomes possible to assess the maturity requirements for transfer of the practice. Potential adopters will be informed about the good practice of any preconditions regarding the context in which the good practice has emerged to maximise the chances of successful transfer and scaling-up. These good practices will contribute to an online database and the objective is to enrich the existing collection of good practices compiled by the Action Groups and Reference Sites (within the Partnership) to showcase additional, inspiring, bottom-up innovation in active and healthy ageing and the benefits of moving towards community-based health and care.
Step 3: refinement of the MM
This step includes: refinement of the dimensions and maturity indicators of the MM based on the outcomes of Step 2. This might comprise splitting or merging the dimensions depending on the clarity of the distinctions and salience for maturity. This step also includes the development of the approach to provide a rating for each dimension of the MM. This will consist of the development of a series of questions that provide a score that relates to a position on the dimension. Finally, this step will include the development of a methodology on how to carry out the self-assessment process (comparison of the regions) using the MM.
Step 4: the self-assessment of European regions
Five European regions will be assessed in terms of their capacity and readiness for adoption of good practices. For the purpose of the SCIROCCO project, the validated MM as an outcome of Step 3 will be applied as the baseline for the self-assessment process. The five participating European regions will assess their maturity in the adoption of integrated care, using the validated tool, to identify strengths, gaps and areas for improvement. A “radar diagram” is developed which presents areas of strengths and weakness in each dimension of the MM, thereby identifying any gaps between the maturity required to implement a practice (as identified in Step 2) and the maturity status of the potential adopting region (this step). The self-assessment process provides a measure of the capacity of the system to accommodate the new practices. The SCIROCCO project will, during its lifespan, see the development of Action Plans to address the gaps; however, the actual implementation of the plans and monitoring of their progress (i.e. ongoing self-assessment) is not within the scope of the project due its limited duration.
Step 5: further refinement of the MM
Further refinement of the MM is envisaged based on the experience of regions using the MM as a self-assessment tool during Step 4.
Step 6: knowledge transfer
Step 6 involves the exploration of the extent to which an approach of matching regions that have complementary strengths and weaknesses (as a result of Steps 4 and 5) can provide both a strong basis for twinning and coaching activities and be useful in facilitating the process of information sharing between the regions to speed up adoption and scaling-up of good practices. The goal of this step is to support the creation of teaming and coaching relationships on a sound basis that uses evidence to construct these relationships (Steps 1–4 of SCIROCCO’s strategy). By comparing the radar diagram of a region with those of other regions/countries that have conducted the same exercise, the MM facilitates two activities: the capacity to offer others knowledge and experience from the regions’ areas of strengths, and the opportunity to find learning/share expertise to fill any gaps in capabilities. The outcomes of Step 6 will also inform whether the benchmarking of good practices, in terms of their maturity, can be used to promote more short-term relationships between regions (where there is a need to “fix” a particular part of the context), and other regions (that are deemed to have demonstrated strengths in those areas). The outcomes of Step 6 will result in the final refinement of the MM as a tool, facilitating the knowledge transfer and flow of appropriate information to achieve scaling-up and implementation of good practices.
Step 7: analysis of experience of scaling-up
In this step, lessons learned from using the MM to facilitate the process of knowledge transfer of the multidimensional maturity requirements of good practices and health and care systems will be retrieved. The lessons learned will inform the development of evidence-based policy recommendations on the challenges of scaling-up, the volume and relevance of knowledge gained from other partners progressing towards implementation, and how policy can facilitate this process of knowledge transfer. In addition, the experience and satisfaction of the MM users will be analysed in order to understand the usefulness of the MM as a tool for facilitating the scaling-up and exchange of good practices in Europe. The outcomes of this step will inform the final development of the MM, which will be referred to as the SCIROCCO tool.
Step 8: final SCIROCCO tool and exploitation of its findings
In the final step, the SCIROCCO tool will be provided as an online version. The tool will become publicly available for the use of interested countries, regions or organisations in Europe. SCIROCCO will also identify support actions to promote and accelerate the use of the tool, and, implicitly, the implementation or scaling-up of good practices in and across Europe. Among the supportive actions that could be considered are: education and training workshops on the MM for local stakeholders in regions and match-making activities to facilitate knowledge transfer through twinning and coaching to transfer or scale-up good practices. In the following section, a description of the methods, which will be used to answer the two research questions, is provided.
Research question 1: assessment of implementation fidelity
The modified conceptual framework for implementation fidelity (Hasson, 2010), based on the conceptual framework of Carroll et al. (2007), will be used to evaluate the implementation of the SCIROCCO step-based strategy. Data for the present study will be collected from the start of the SCIROCCO project (April 2016) until August 2018. The measurement of implementation fidelity is a measurement of adherence, with its subcategories content, frequency, duration and coverage (dose) (Carroll et al., 2007). According to Fixsen et al. (2005), fidelity assessment should focus on all intervention activities if no analyses have been made of active ingredients of an intervention.
To systematically evaluate the implementation fidelity of the SCIROCCO project, a stepwise approach is used. To operationalise implementation fidelity, we first identified the “main” programme components of the SCIROCCO project. The description of the implementation of SCIROCCO’s step-based approach, as described above, is summarised in a logic model, which is available upon request by the corresponding author. In the second step, we formulated research questions for each subcategory of adherence based on the framework for fidelity, as well as for the two moderating factors “availability of facilitation strategies” and “participant responsiveness”. A table is developed where each element of the implementation fidelity is presented in the first column, followed by the programme component of the SCIROCCO project, research questions, start of the programme component, measurement method for the programme component and planning of data collection. A short version of the table is presented in the Appendix, the full version is available upon request by the corresponding author.
Data collection
To assess the fidelity of each SCIROCCO component, we will collect data from each of the WPs involved in the SCIROCCO project and collect data from five SCIROCCO regions during the entire intervention period using a multi-method approach. Data collection methods include key informant interviews, questionnaire studies and the collection of project documents (i.e. progress reports, deliverables, presentations and notes of meetings).
Setting and subjects
To investigate the content and find factors that moderate adherence, we will use qualitative data collection methods. Semi-structured interviews will be conducted alongside the intervention with WP leaders/members, SCIROCCO partners in the five regions and external members in the regions (referred to as “local stakeholders”). We will include the following topics in the interview for the WP leaders and SCIROCCO partners in the regions (per implementation fidelity concept): content (work practices, changes in activities), facilitation strategies (quality and usefulness of the protocol, project coordination, guidance/collaboration), context (barriers and facilitators in carrying out the activities), participant recruitment procedures and participant responsiveness, and in addition: points of improvement. We will include the following topics for the local stakeholders in the regions: participant recruitment procedures, participant responsiveness and points of improvement. Interviewees will be requested to sign an informed consent form. In this form they are asked if they agree with participation, with the interview being audio-taped and to confirm that they understand the purpose of the study. Participation in the study is on a voluntary basis, and interviewees can drawback from participation at any time. All interviews will be audio-recorded and transcribed. To enhance the quality of the qualitative data, subjects will be asked to approve a summary of the interview (member check). In addition, the researcher who carries out the interviews will keep a research log in which she/he reflects on methodological decisions and her/her own role in the research process.
WP leaders in SCIROCCO
The WP leaders are responsible for the detailed management of their WP. This includes monitoring and control of their WP, production of WP deliverables and contribution to other cross-cutting activities including preparation of SCIROCCO meetings, presentations and other awareness-raising and dissemination activities. We will collect data from the WP leaders on undertaking the activities as part of their WP using mixed methods.
Partners in the SCIROCCO regions
Consortium partners in the five SCIROCCO regions will be involved in facilitating the self-assessment process, matching and process of knowledge transfer and information sharing for each participating region. We will collect data from these SCIROCCO partners with regard to their activities in facilitating these processes in their regions by conducting interviews with a minimum of 1, and a maximum of 3, intensively involved partner(s) per region and by collecting their work-related documents. In the Summer of 2018, we plan to conduct interviews with the same subjects to collect their experiences after all the SCIROCCO project activities have been undertaken.
Members in the SCIROCCO regions
Members in the five regions (i.e. coordinators, practitioners, policy officers, etc.), the local stakeholders, will be involved in the self-assessment process, the matching and process of knowledge transfer and information sharing. We will collect data from these local stakeholders with regard to their involvement in the SCIROCCO activities in their regions by conducting interviews with a minimum of one stakeholder per SCIROCCO region (a total of five subjects) and surveys with all the stakeholders who are involved in the SCIROCCO activities (resulting in a total of approximately 30–50 subjects). These local stakeholders will be selected for the interviews by each of the SCIROCCO partners in the regions on basis of the following criteria: being a professional in the region which is highly involved in the twinning and coaching sessions.
Data analysis
The interview and document data collected will be analysed using content analysis. To organise the large amount of collected text, a basic coding process is used for content analysis in order to achieve fewer content categories (Weber, 1990). A coding scheme, including the implementation fidelity concepts, and each intervention component will be used during the coding process. The scheme will be tested independently by two researchers prior to implementing the coding process. The results from the coding process will be analysed and discussed among the researchers and any disagreement will be resolved until consensus is reached. The planned surveys will be analysed using both descriptive and analytical methods.
Research question 2: realist evaluation
Data collection
To collect data for the realist evaluation, interviews will be conducted with WP leaders involved in the five SCIROCCO regions and minimum one of the local stakeholders per region (N (total)=minimum 10 subjects). During the interview, we are interested to collect the experiences of the regional stakeholders with implementing the SCIROCCO strategy. The purpose is to reveal and define the key functions/actions of the strategy and to understand the interactions between the pre-planned activities and context and perceived outcomes. Furthermore, we will collect documents (e.g. notes on meetings and deliverables of the WPs) which will be used during the implementation of the SCIROCCO strategy.
Data analysis
Realist evaluation will guide the development, testing and refining of middle-range theories through the analysis of the relationships between the context, mechanisms and outcomes. The data analyses will be guided by the recently developed programme theory of Willis et al. (2016). They conducted a realist synthesis to increase understanding of how and under what conditions complex public health interventions may be scaled up to benefit more people and populations. In this theory, contextual details were considered as the characteristics of individuals, organisations, communities or systems which were influenced by deliberate actions or activities by any actor to scale-up a complex intervention. The mechanisms were regarded as the unseen response by people to the specific changes made to the context through actions (e.g. excitement, fear, commitment), and outcomes as the proximal results of activated mechanisms (e.g. adding new communities, secured financial resources for scaling-up, increased knowledge/skills necessary for scaling-up). The outcomes, actions and contexts identified in Willis et al.’s programme theory will be used to extract the outcomes, actions and contexts observed from the interviews and the collected documents of the implementation of SCIROCCO strategy. Direct text quotations will be extracted from the interviews and the collected documents from each individual SCIROCCO region into an Excel spreadsheet and coded as contextual details, mechanisms or outputs and outcomes. An initial coding exercise will be conducted among two researchers for the first five documents and any disagreement will be resolved by discussion. One researcher will complete the data extraction, consulting the other researcher if needed. After the data extraction, two researchers will identify scaling-up outcomes, and the contextual details linked to each outcome in the extracted data per SCIROCCO region If quoted, the linked mechanisms will also be extracted. Where the mechanism is not stated, the two researchers will discuss a derivative mechanism. The individual CMO configurations extracted per region will be presented in a Word file. Where outcomes cannot be linked to contexts, neither these nor any linked data will be extracted. The extracted CMO configurations for each region will be summarised and sent to the subjects in the regions for verification of their accuracy.
Current status of the SCIROCCO project and evaluation study
As of August 2016, a literature review and Delphi study were performed to test the content-validity of the tool and the manuscript was submitted to a peer reviewed journal (Step 1) (Grooten et al., 2018). Step 2 of the SCIROCCO step-based approach, the maturity assessment performed by WP4, was completed. An interview was held with the members responsible for the work within WP4 for the implementation fidelity assessment, and the interview with members of WP6 will be scheduled. The SCIROCCO project has finished the self-assessment processes in the five regions (June 2017 to November 2017). The process for the matching of regions and the twinning and coaching processes is currently being designed within the consortium.
Discussion
This paper describes a research protocol to comprehensively evaluate the SCIROCCO scaling-up strategy by using implementation fidelity and realist evaluation, two approaches which are sensitive to the context of the intervention.
To advance our understanding on what works in integrated care initiatives and how successful ones can be scaled up, evaluation methods need to be sensitive to the fact that the success or failure of initiatives is associated with the complexity of implementing integrated care. There is the recognition in the field of integrated care that the extensive focus and use of quantitative evidence on outcomes should be revised (Nolte, 2017). Among others, Nurjono et al. (2018) argued the need “to develop comprehensive, rigorous and practical methods to evaluate people-centred integrated care programmes, to inform the selection of effective and efficient interventions and to facilitate improvement and scaling-up”. An important part of the SCIROCCO strategy focusses on matching five participating European regions, based on their complementary strengths and weaknesses, and exploring whether this matching can provide a strong basis for successful twinning and coaching to facilitate both shared learning and a practical support for the scaling-up of good practices. The context of the different participating regions will play an important role in this exercise. The intended shared learning and the practical support for the scaling-up of good practices needs to adequately match within the context of that specific region. The implementation of the scaling-up strategy in integrated care requires adaptation to the context, the needs, capacity and capability of (local) stakeholders. To adequately evaluate the strategy, methods should consider the process of implementation and be sensitive to the context of the intervention. The use of the evaluation approaches described in this paper to evaluate the scaling-up strategy will support us in not only exploring the outcomes of the strategy, but will also provide an understanding when, why and how these outcomes are achieved. This could provide important insights in what will be useful to scale-up and how to reach upscaling in integrated care, which will, in turn, contribute to overcoming the prevailing knowledge gap on what elements are associated with successful implementation of integrated care initiatives on a wider scale.
Another evaluation approach which has been widely used in process evaluation of many complex interventions is the RE-AIM framework (Altpeter et al., 2015; Glasgow et al., 2006). The framework can be used to assess the reach, efficacy, adoption, implementation and maintenance of a complex intervention evaluated at an individual, organisational and/or community level (Glasgow et al., 1999). Despite the usefulness of the model to evaluate interventions in real-world settings, implementation fidelity is not being measured systematically. To know if an intervention was implemented as was intended it is important to know what factors contribute to the outputs (Damschroder et al., 2009). Therefore, in this protocol we make use of the most complete framework for implementation fidelity, which was originally developed by Carroll et al. (2007) and slightly extended by Hasson (2010). Measuring implementation fidelity and analysing results using a theoretical framework will help us to document the process by which the strategy will be implemented and understand the level of implementation fidelity achieved. These insights will yield conclusions about the characteristics of the strategy and contexts that are favourable for implementation, will assist us to improve the strategy, inform adaptation of the process in the different contexts and will help to manage the risks with implementation of the strategy in different European countries in the future. Furthermore, similar European projects that are based on collaboration between several European regions can learn from the lessons learned in SCIROCCO and can become more aware of the facilitating factors and pitfalls of implementing such projects. The evaluation will be undertaken in a European context defined by five participating healthcare regions. To test the external validity of study findings it would be recommended to also test the SCIROCCO strategy in other countries inside and outside Europe.
In addition to investigating the implementation of the project, in this study realist evaluation will be used to explore what it is about the SCIROCCO’s strategy that works for whom, why, how and in which circumstances. The implementation fidelity framework will split the implementation of the SCIROCCO strategy into the concepts of the framework and will help us to obtain an understanding of the fidelity of the implementation of the strategy. Realist evaluation, on the other hand, views the process of change within natural contexts leading to certain outcomes. To complement and further interpret the results of implementation fidelity we expect that realist evaluation will help us to gain insight in the impact of different components of the strategy on anticipated outcomes to obtain a more complete overview by linking the mechanisms to the outcomes of the strategy. This will provide an understanding of what factors are involved, such as the benefits and unintended consequences, in SCIROCCO’s strategy to facilitate the scaling-up of integrated care among the five regions. These insights could provide support to other regions or good practices involved in integrated care that are interested to increase the implementation of integrated care initiatives (by using elements of the SCIROCCO strategy).
The SCIROCCO project fits nicely with other past and present European project focussing on the implementation and scaling-up of integrated care initiatives. For example, completed in 2016, the INTEGRATE project aimed to define what constitutes good quality integrated care provision, by gaining valuable insights into integrated care especially in terms of care process design, service delivery, the professional skills mix, patient involvement, funding flows, regulatory conditions and enabling information communication technology, in order to create connectivity, alignment and collaboration within and between the cure and care sectors (Cash-Gibson and Rosenmoller, 2014). The results of the project provide valuable insights into those elements of integrated care that are useful to scale-up. SCIROCCO will complement the work of INTEGRATE by addressing the issues of knowledge transfer and scaling-up of innovative initiatives in integrated care in Europe and by an evaluation of its own work.
Another European project, ACT@Scale, started simultaneously with the SCIROCCO project in Spring 2016. It has the objective to develop, test and consolidate “best practice” Care Coordination and Telehealth concepts. They can be leveraged by the participating healthcare regions to expedite the scaling-up of their services but can also be transferred to other regions through Europe and beyond. ACT@Scale is targeting different good practices across Europe. It is tracking and evaluating changes in the process, structure and outcomes of the services during two one-year intervention cycles, thereby generating knowledge about upscaling across programmes and health issues. The two projects, ACT@Scale and SCIROCCO, complement each other by gaining knowledge on scaling-up interventions in Europe. The MM can potentially be used to assess the maturity of healthcare care system structure in a particular region and monitor changes in the healthcare system over time, as well as similarities or differences in the healthcare system between services. For ACT@Scale, the MM could provide insight into the context in which certain interventions did have certain outcomes. In return, it is interesting to compare these outcomes with the ACT@Scale outcomes of the assessment using MM, to see if and how the MM can monitor the changes in services.
A few limitations of the study and the SCIROCCO methods need to be mentioned. First, several WP leaders in the SCIROCCO project are responsible for the activities of their WP and are also involved in the self-assessment process and twinning and coaching activities in their regions, which may cause bias in the implementation of the SCIROCCO strategy. Second, since the evaluation study is part of the SCIROCCO project, the planning done in conducting the research activities is dependent on the progress of the project, which could lead to delays since all the five regions need undertake their activities. Regarding the feasibility of collecting data within a larger EU project, including several WPs focussing on their own tasks, the data collection needs to be aligned with other work and feasible within the project time. Third, the participants in the SCIROCCO regions are already involved in the activities of the project. Therefore, the planning of the evaluation activities needed to take into consideration not to overburden the subjects. Hence, choices need to be made regarding data collection. The study presents a systematic way of evaluating implementation fidelity, and factors potentially affecting fidelity and performing a realist evaluation. However, the study does not cover all the potential factors influencing the implementation of complex interventions. Furthermore, it would be interesting to gain insight in the experiences on implementation of the plans and the monitoring of their progress in the five participating regions. However, since the evaluation study is part of the SCIROCCO project, which will not address the implementation and monitoring of these plans, due to its limited durations, this is, unfortunately, out of the scope for the evaluation study. The goal of the evaluation study is to contribute to the knowledge base of what factors are associated with the implementation of a scaling-up strategy concerning integrated care initiatives.
Another limitation concerns a risk of bias in the interviews with the partners of SCIROCCO. This is because the evaluation study is part of the SCIROCCO project, and the researchers conducting the study are partners involved in the SCIROCCO consortium. However, being a partner in the consortium provides a unique opportunity to be able to follow the project closely (which is important for implementation fidelity assessment) and to obtain sound cooperation to conduct the research activities (i.e. conducting interviews and recruiting subjects for the measurements for reliability and validity testing).
Conclusion
There is an evident need to know what elements of successful integrated care initiatives should be scaled up, and how the initiatives themselves can be scaled up. This study describes an evaluation protocol of the SCIROCCO step-based scaling-up strategy interested to explore what to scale-up, and how to scale-up, integrated care initiatives by matching the complementary strengths and weaknesses of regions involved in integrated care. One part of the evaluation study focusses on assessing the implementation fidelity of the SCIROCCO strategy to explore whether the SCIROCCO scaling-up strategy is implemented as intended. The other part of the evaluation is a realist evaluation which examines what it is in the SCIROCCO’s strategy that works and in which circumstances. The multi-method design of the evaluation protocol will be useful to guide the development, implementation and evaluation of future scaling-up strategies to accelerate change towards more sustainable health and care systems.
Implementation fidelity components (adherence subcategory and potential moderating factors), research question, data collection procedure/source and planning
| Adherence subcategory | Research question(s) | Data collection procedures/sources | Measurement planning |
|---|---|---|---|
| Content | How are the different programmes components of SCIROCCO delivered? | Semi-structured interview with project leader(s) of WP 1, 4, 5, 6, 7, 8 Semi-structured interview with regional partners of SCIROCCO in the five participating regions Work documents (i.e. progress reports, interim reports) of WP 1, 4, 5, 6, 7, 8 Deliverables of WP 1, 4, 5, 6, 7, 8 |
After the respective programme component is finished After the matching of regions, first and last twinning/coaching sessions Alongside SCIROCCO project Alongside SCIROCCO project |
| Frequency/Duration | Are the programme components of SCIROCCO implemented as often and as long as planned? | Semi-structured interview with project leader(s) of WP 1, 4, 5, 6, 7 Semi-structured interview with regional partners of SCIROCCO in the five participating regions Work documents (i.e. progress reports, interim reports) of WP 1, 4, 5, 6, 7 and regional partners Deliverables of WP 1, 4, 5, 6, 7 |
After the respective programme component is finished After the matching of regions, first and last twinning/coaching sessions Alongside SCIROCCO project Alongside programme |
| Coverage (Reach) | What proportion of target group participated in the different activities of the SCIROCCO programme? | Work documents of WP 4, 6, 7 Deliverables of WP 4, 6, 7 |
Alongside programme Alongside programme |
| Potential moderating factors | |||
| Participant responsiveness | How did the participants get engaged with the SCIROCCO project? How satisfied were the participants with their participation in SCIROCCO? How did the participants perceive the outcomes and relevance of participating in SCIROCCO? |
Semi-structured interview work package leader op WP 1, 4, 5, 6, 7 Semi-structured interview with SCIROCCO partners in the participating regions Semi-structured interviews with local stakeholders of the five participating regions Work documents of WP 1, 4, 5, 6, 7 and regional partners Deliverables of WP 1, 4, 5, 6, 7 Short survey with the local stakeholders of the five regions on experience in the self-assessment process in their region Short survey with the local stakeholders of the five regions on experience in the twinning and coaching activities in their region |
After the respective programme component is finished After the matching of regions, first and last twinning/coaching sessions After the last twinning/coaching sessions Alongside SCIROCCO programme Alongside SCIROCCO programme After the self-assessment process in the regions After the first twinning/coaching sessions |
| Participant recruitment | What recruitment procedures were used to attract regions to the SCIROCCO project? What constituted barriers to maintaining involvement of regions? |
Semi-structured interview work package leader op WP 1, 4, 5, 6, 7 Semi-structured interview with SCIROCCO partners 1–5 Semi-structured interviews with local stakeholders of the five participating regions Work documents of WP 1, 4, 5, 6, 7 and regional partners Deliverables of WP 1, 4, 5, 6, 7 |
After the respective programme component is finished After the matching of regions, first and last twinning/coaching sessions After the last twinning/coaching sessions Alongside SCIROCCO programme Alongside SCIROCCO programme |
| Strategies to facilitate implementation | What strategies were used to support implementation of the SCIROCCO project? How were these strategies perceived by SCIROCCO partners and local stakeholders involved in the project? | Semi-structured interview with project leader(s) of WP 1, 4, 5, 6, 7, 8 Semi-structured interview with regional partners of SCIROCCO in the five participating regions Semi-structured interviews with local stakeholders of the five participating regions Work documents (i.e. progress reports, interim reports) of WP 1, 4, 5, 6, 7, 8 Deliverables of WP 1, 4, 5, 6, 7, 8 |
After the respective programme component is finished After the matching of regions, first and last twinning/coaching sessions After the last twinning/coaching sessions Alongside SCIROCCO project Alongside SCIROCCO project |
| Context | What factors at political, economical, organisational and work group level affected the implementation of the SCIROCCO project? | Semi-structured interview with project leader(s) of WP 1, 4, 5, 6, 7, 8 Semi-structured interview with regional partners of SCIROCCO in the five participating regions Semi-structured interviews with local stakeholders of the five participating regions Work documents (i.e. progress reports, interim reports) of WP 1, 4, 5, 6, 7, 8 Deliverables of WP 1, 4, 5, 6, 7, 8 |
After the respective programme component is finished After the matching of the regions, first and last twinning/coaching sessions After the last twinning/coaching sessions Alongside SCIROCCO project Alongside SCIROCCO project |
Note: WP, work package
Appendix
References
Altpeter, M., Gwyther, L.P., Kennedy, S.R., Patterson, T.R. and Derence, K. (2015), “From evidence to practice: using the RE-AIM framework to adapt the REACHII caregiver intervention to the community”, Dementia, Vol. 14 No. 1, pp. 104-113.
Bardsley, M., Steventon, A., Smith, J. and Dixon, J. (2013), “Evaluating integrated and community based care”, The Nuffield Trust, London, available at: www.nuffieldtrust.org.uk/files/2017-01/evaluating-integrated-community-care-web-final.pdf (accessed 26 July 2017).
Bongaerts, B.W.C., Müssig, K., Wens, J., Lang, C., Schwarz, P., Roden, M. and Rathmann, W. (2017), “Effectiveness of chronic care models for the management of type 2 diabetes mellitus in Europe: a systematic review and meta-analysis”, BMJ Open, Vol. 7 No. 3, pp. 1-16, available at: https://doi.org/10.1136/bmjopen-2016-013076
Busetto, L., Luijkx, K.G., Mathilda, A., Elissen, J., Johannes, H. and Vrijhoef, M. (2015), “Context, mechanisms and outcomes of integrated care for diabetes mellitus type 2: a systematic review”, BMC Health Services Research, Vol. 16 No. 18, pp. 1-14, available at https://doi.org/10.1186/s12913-015-1231-3
Busse, R. and Stahl, J. (2014), “Integrated care experiences and outcomes in Germany, the Netherlands, and England”, Health Affairs, Vol. 33 No. 9, pp. 1549-1558.
Busse, R., Blümel, M., Scheller-Kreinsen, D. and Zentner, A. (2010), “Tackling chronic disease in Europe: strategies, interventions and challenges”, WHO Regional Office Europe on behalf of the European Observatory on Health Systems and Policies, Copenhagen.
Carroll, C., Patterson, M., Wood, S., Booth, A., Rick, J. and Balain, S. (2007), “A conceptual framework for implementation fidelity”, Implement Science, Vol. 2, p. 40.
Cash-Gibson, L. and Rosenmoller, M. (2014), “Project INTEGRATE – a common methodological approach to understand integrated health care in Europe”, International Journal of Integrated Care, Vol. 14 No. 4, pp. 1-17.
Craig, P., Dieppe, P., Macintyre, S., Mitchie, S., Nazareth, I. and Petticrew, M. (2008), “Developing and evaluating complex interventions: the new Medical Research Council guidance”, British Medical Journal, Vol. 337 No. 7676, pp. 979-983.
Damschroder, L.J., Aron, D.C., Keith, R.E., Kirsh, S.R., Alexander, J.A. and Lowery, J.C. (2009), “Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science”, Implementation Science, Vol. 4 No. 50.
Darker, C. (2014), “Integrated healthcare in Ireland – a critical analysis and a way forward”, available at: www.tcd.ie/medicine/public_health_primary_care/assets/pdf/Integrated-Care-Policy-LR.pdf (accessed 26 July 2017).
Elissen, A.M.J., Steuten, L.M.G., Lemmens, L.C., Drewes, H.W., Lemmens, K.M.M., Meeuwissen, J.A.C., Baan, C.A. and Vrijhoef, H.J.M. (2012), “Meta-analysis of the effectiveness of chronic care management for diabetes: investigating heterogeneity in outcomes”, Journal of Evaluation in Clinical Practice, Vol. 19 No. 5, pp. 753-762, available at: https://doi.org/10.1111/j.1365-2753.2012.01817.x
European Commission (2013), “Public health: improving health for all EU citizens”, EC Report, Luxembourg, available at: https://doi.org/10.2775/79603
Fixsen, D.L., Naoom, S.F., Blase, K.A., Friedman, R.M. and Wallace, F. (2005), “Implementation research: a synthesis of the literature”, Louis de la parte Florida Mental Health Institute, The National Implementation Research Network, University of South Florida, Tampa, FL.
Glasgow, R.E., Vogt, T.M. and Boles, S.M. (1999), “Evaluating the public health impact of health promotion interventions: the RE-AIM framework”, Diabetes Care, Vol. 89 No. 9, pp. 2090-2092.
Glasgow, R.E., Nelson, C.C., Strycker, L.A. and King, D.K. (2006), “Using RE-AIM metrics to evaluate diabetes self- management support interventions”, American Journal of Preventive Medicine, Vol. 30 No. 1, pp. 67-73.
Grooten, L., Borgermans, L. and Vrijhoef, H.J.M. (2018), “An instrument to measure maturity of integrated care: a first validation study”, International Journal of Integrated Care, Vol. 18 No. 1, pp. 1-20, available at: http://doi.org/10.5334/ijic.3063
Gröne, O. and Garcia-Barbero, M. (2001), “Integrated care: a position paper of the WHO European Office for Integrated Health Care Services”, International Journal of Integrated Care, Vol. 1 No. e21, pp. 1-26.
Hasson, H. (2010), “Systematic evaluation of implementation fidelity of complex interventions in health and social care”, Implementation Science, Vol. 5 No. 67, pp. 1-21.
Henderson, D., Pavlickova, A. and Lewis, L. (2016), “Scalability and transferability of good practices in Europe: What does it take?”, Annual Conference Supplement 2016 of the International Conference in Barcelona, Spain, International Journal of Integrated Care, Utrecht.
Jeffs, L., Jain, A.K., Man, R.H., Onabajo, N., Desveaux, L., Shaw, J., Hensel, J., Agarwal, P., Saragosa, M., Jamieson, T., Wong, I., Maione, M. and Bhatia, R.S. (2017), “Exploring the utility and scalability of a telehomecare intervention for patients with chronic kidney disease undergoing peritoneal dialysis – a study protocol”, BMC Nephrology, Vol. 18 No. 155, pp. 1-23.
Kernick, D. (2006), “Wanted – new methodologies for health service research: is complexity theory the answer?”, Family Practice, Vol. 23 No. 3, pp. 385-390.
Lamont, T., Barber, N., Pury, J., de, Fulop, N., Garfield-Birkbeck, S., Lilford, R., Mear, L., Raine, R. and Fitzpatrick, R. (2016), “New approaches to evaluating complex health and care systems”, British Medical Journal, Vol. 352 No. 154, pp. 1-5.
Liebman, J., Heffernan, D. and Sarvela, P. (2007), “Establishing diabetes self-management in a community health centre serving low-income Latinos”, Diabetes Educator, Vol. 33 No. 6, pp. 132S-138S.
Low, L.L.-F., Yap, M. and Brodaty, H. (2011), “A systematic review of different models of home and community care services for older persons”, BMC Health Services Research, Vol. 11 No. 93, pp. 1-15.
Nolte, E. (2017), “Evidence supporting integrated care”, in Amelung, V., Stein, V., Goodwin, N., Balicer, R., Nolte, E. and Suter, E. (Eds), Handbook Integrated Care, Springer International Publishing, Cham, pp. 25-38.
Nolte, E. and Pitchforth, E. (2014), “What is the evidence on the economic impacts of integrated care?”, WHO Regional Office Europe on behalf of the European Observatory on Health Systems and Policies, Copenhagen.
Nolte, E., Knai, C. and McKee, M. (2008), “Managing chronic conditions: experience in eight countries”, WHO Regional Office Europe on behalf of the European Observatory on Health Systems and Policies, Copenhagen.
Nolte, E., Frølich, A., Hildebrandt, H., Pimperl, A., Schulpen, G.J. and Vrijhoef, H.J. (2016), “Implementing integrated care: a synthesis of experiences in three European countries”, International Journal of Care Coordination, Vol. 19 No. 2, pp. 5-19.
Nurjono, M., Shrestha, P., Lee, A., Lim, X.Y., Shiraz, F., Tan, S., Wong, S.H., Foo, K.M., Wee, T., Toh, S., Yoong, J. and Vrijhoef, H.J.M. (2018), “Realist evaluation of a complex integrated care programme: protocol for a mixed methods study”, British Medical Journal Open, Vol. 8 No. e017111, pp. 1-11, available at: https://doi.org/10.1136/bmjopen-2017-017111 (accessed 19 June 2018).
Pawson, R. and Tilley, N. (1997), Realistic Evaluation, Sage Publications, London.
Pimouguet, C., Le Goff, M., Thiébaut, R., Dartigues, J.F. and Helmer, C. (2011), “Effectiveness of disease-management programs for improving diabetes care: a meta-analysis”, Canadian Medical Association Journal, Vol. 183 No. 2, available at: https://doi.org/10.1503/cmaj.091786
Raak, A., van, Mur-Veeman, I., Hardy, B., Steenbergen, M. and Paulus, A. (2003), Integrated Care in Europe: Description and Comparison of Integrated Care in Six EU Countries, Elsevier Gezondheidszorg, Maarssen.
The European Innovation Partnership on Active and Healthy Aging (2015), “European scaling-up strategy in active and healthy aging”, available at: https://ec.europa.eu/research/innovation-union/pdf/active-healthy-ageing/scaling_up_strategy.pdf (accessed 9 June 2017).
Tsiachristas, A. and Rutten-van Mölken, M.P.M.H. (2017), “Evaluating complex interventions”, in Amelung, V., Stein, V., Goodwin, N., Balicer, R., Nolte, E. and Suter, E. (Eds), Handbook Integrated Care, Springer International Publishing, Cham, pp. 297-313.
Weber, R.P. (1990), Basic Content Analysis, Sage, Beverly Hills, CA.
WHO (2015), “WHO global strategy on people-centred and integrated health services”, available at: www.who.int/servicedeliverysafety/areas/people-centred-care/global-strategy/en/ (accessed 2 December 2016).
WHO (2016), “Integrated care models: an overview, health services delivery programme”, available at: www.euro.who.int/__data/assets/pdf_file/0005/322475/Integrated-care-models-overview.pdf (accessed 4 August 2017).
Willis, C.D., Riley, B.L., Stockton, L., Abramowicz, A., Zummach, D., Wong, G., Robinson, K.L. and Best, A. (2016), “Scaling up complex interventions: insights from a realist synthesis”, Health Research Policy and Systems, Vol. 14, No. 88, pp. 1-16, available at: https://doi.org/10.1186/s12961-016-0158-4 (accessed 24 February 2017).
Wodchis, W.P., Dixon, A., Anderson, G.M. and Goodwin, N. (2015), “Integrating care for older people with complex needs: key insights and lessons from a seven-country cross-case analysis”, International Journal of Integrated Care, Vol. 15 No. 6, pp. 1-15, available at: https://doi.org/10.5334/ijic.2249
Acknowledgements
Competing interest: the authors have no competing interests to declare. Authors’ contributions: Cristina-Adriana Alexandru, Tamara Alhambra, Stuart Anderson, Francesca Avolio, Elisa Vaila Cotanda, Zdenek Gütter, Donna Henderson, Ann-Charlotte Kassberg, Esteban de Manuel Keenoy, Marc Lange, Lisa Lundgren, Andrea Pavlickova, Jon Txarramendieta Suarez, Diane Whitehouse, Ane Fullaondo Zabala, Joseba Igor Zabala Rementeria are all highly active partners in the SCIROCCO consortium contributing to the execution of SCIROCCO project. The partners provided feedback to the study protocol. Liset Grooten and Hubertus JM Vrijhoef are active partners in the SCIROCCO consortium and are responsible for the evaluation activities within the SCIROCCO project. Both authors wrote the manuscript for the study protocol. Funding: this study is part of the project 710033/SCIROCCO which has received funding from the European Union’s Health Programme (2014-2020).