Introduction

Care pathways were originally introduced as a means of controlling costs and reducing unnecessary variation in care delivered by physicians in America.1 An essential part of this approach involved a focus on 'outcomes', or the 'end result' of care. This emphasis was based on a recognition that (if a clinical intervention was effective), each intervention produces a change in the health status of the patient, and that change can be measured. Until relatively recently there was all too little evidence about the end results for patients who receive a particular procedure,2 but this is changing. Outcome measurement is now considered to be an essential tool in quality improvement and cost containment.3

Previously there has been an emphasis on monitoring processes of care (things that happen to the patient) to gauge whether NHS providers have met contractual obligations. However, there is now a recognition that focusing too heavily on these can lead to a distortion of clinical priorities and risks, creating a whole system of accountability that is more concerned with the means than the result – thus losing sight of the purpose of the NHS.4 Consequently, the recent government's White paper Equity and excellence: liberating the NHS5 places a new emphasis on holding the NHS to account for improving healthcare outcomes. Proposals for a new NHS Outcomes Framework have been outlined as a means to realise this new approach, the framework consisting of a focused set of national outcomes against which the NHS Commissioning Board can be held to account. Measurement of performance through a focus on outcomes therefore seems set to be an important agenda for both purchasers and providers of NHS care in the next few years.

Care pathways are recognised to be multifaceted tools.6 They have a primary purpose of supporting clinical processes, but they have also been used for secondary purposes including monitoring activity and commissioning services. De Luc conceptualises a distinction between models of care pathway development concerned with achieving clinical effectiveness through their primary use (auditing clinical care, variance reporting) and also their secondary use (benchmarking, developing clinical protocols, discussions with purchasers about identifying health need).6

Although care pathways in dentistry are relatively uncommon compared to other branches of healthcare, an earlier paper outlined their use in a pilot scheme involving three dental practices in the North West of England.7 This paper reports on data and learning resulting from the implementation of care pathways for prevention implemented in these practices. While involvement of the practitioners in developing the care pathways did lead to some focus on the primary purposes of care pathways in these practices (setting and implementing standards of care), the focus of this report is on the secondary use of care pathways (such as analysing data on patient need, activity and outcomes), which will inform new models of commissioning dental services incorporating quality indicators into a 'basket' of performance measures.8

Methods

Detail of the three dental practices and the care protocols involved is described in an earlier paper.7 In summary, one of the three dental practices (Practice 1) was a newly established (two surgery) dental practice sited within a new build (LIFT) premises which included other primary care medical facilities. The health centre was situated in the Oldham area, an area with an IMD (Index of Multiple Deprivation) score of 56.44 (indicating a relatively high level of deprivation compared to other areas of England9). The local population also comprised a relatively high proportion of people with an Asian ethic background.10

Practices 2 and 3 were both established dental practices situated in the Salford area, which is about 12 kilometres from Oldham. Practice 2 was a large (11 surgery) vocational training practice situated in an even lower socio-economic area (IMD score = 68.43) than practice 1. Practice 3 was a medium-sized (four surgery) dental practice (also a vocational training practice) and also situated in a relatively disadvantaged area when compared to the rest of the country (IMD = 30.04), although this area was relatively better off than the locale of Practices 1 and 2.

Care pathways for prevention were introduced in Practice 1 in January 2008, and in the other two practices, gradually between 2008 and 2009. Between 2008 and 2010 data were regularly extracted from the computerised clinical databases used in the three practices. Data were also retrieved from electronically transmitted centrally monitored data (VITAL signs). Data collection was limited to adding additional fields to existing electronic systems to avoid over-burdening the practices with additional administrative tasks. Between 2008 and 2010, over the period of implementation of the scheme, semi-structured interviews were undertaken involving participating members of the dental teams, commissioners and other stakeholders. These interviews were audio-taped and transcribed. A content analysis of transcripts identified key words related to measurement of performance, with quotes then grouped thematically. Ethical approval and NHS Research Governance approval was obtained before data collection.

Results

Key performance indicators

A set of key performance indicators (KPIs) were identified at the outset of the pilot by a steering group which included both practitioners and commissioners (Table 1). Although contracts between the primary care trust and practices were on the basis of a contract value conditional on achievement of a target number of units of dental activity (UDAs), the attainment of the UDA value was only associated with a proportion of the contract value. The remainder of the contract value was linked to adherence to the care pathway model and agreed KPIs, and conditional on attending service re-design workshops and the use of skill mix within the practice. As such this contractual agreement represented a type of 'blended contract'.8 Of the 25 KPIs identified initially by the steering group, three described the population, 17 were process indicators and four were outcome indicators.

Table 1 Feasibility of key performance indicator measurement
Full size table

Interviews revealed that although a set of initial KPIs had been agreed, commissioners and practitioners held different perspectives about the number of KPIs that might be useful, underpinned by a difference in approach as to how performance might be monitored and improvements achieved. The commissioner experience and view was informed by the use of performance indicators in primary medical care; for these were numerous, and well established as a means for improving as well as monitoring performance and to justify value for money within contractual agreements.

'I think there's about 20 or so KPIs on the sheet at the moment. How does that number compare with your experience of measuring KPIs?' (Interviewer)

'20 is I think in the right region. Yeah I would say, I mean I have no knowledge of what I can base that on other than the fact that I have been involved very heavily with the new personal medical services [PMS] specification. We have got probably about 40 to 50 KPIs within that, maybe more, so I think. I think we need to stretch people as well. You know there's always that.' (Commissioner)

'And what sort of indicators are they in the PMS?' (Interviewer)

'We have clinical areas we have identified that we want to see improvements on. We then have some basic general practice management type issues that we want to see improvements on.' (Commissioner)

'We want PMS patients to be receiving the same treatment as the practice down the road that maybe is a general medical services [GMS] practice and the whole reason why we put this specification in place ... is because when you compare GMS practices with the PMS practices, the PMS practices were getting paid a lot more per patient and not delivering anything over and above a general medical services contract.' (Commissioner)

Practitioners were more circumspect. Whether this was influenced by a preference for a looser, less specified contract form, concern about feasibility and time taken to monitor progress towards these additional performance targets, or whether through naivety, is conjecture; but when asked how many KPIs might be appropriate, one practitioner replied:

'....Erm I mean five's a nice round figure isn't it?' (Practitioner)

'Right so you'd feel more comfortable with five than 20? I mean, I think on the document at the moment there must be 20 or so?' (Interviewer)

'It would be great to have a handful of things, half a dozen or so to look at so, if you include your UDAs it is half a dozen indicators of performance and I think that's more than enough actually.... Erm ... If you go to a practice and you look at record keeping, that'll tell you. I think a pretty good indication as to a whole variety of things that could occur to the patient. ...You can look at the quality of bitewings. When we have VTs [vocational trainees] start, even just looking at what their X-rays are like right at the beginning of the year it gives you a very good idea as to what sort of clinicians they are. So actually I think we need the PCT almost to be less uptight and more supportive of practitioners. So I think having a huge range of KPIs is something that will cloud the issue rather than encouraging it [performance] in a practice.' (Practitioner)

Feasibility of KPIs

It became apparent, as the pilot progressed, that the number and range of potential KPIs needed to be considerably reduced because the computer software used in the practices had only a limited capacity to provide summary data on a practice population. Regular reporting was thus reduced to a limited number of fields, with some KPIs having to be monitored by other means (using audits of patient records, using electronically transmitted centrally monitored data). After two years of the pilot scheme, it was agreed that three KPIs should be removed from the initial list of 25 KPIs because the information generated did not justify the work involved in adapting the clinical software system to generate the data. It was also agreed that in future a further seven KPIs were to be monitored by periodic audits mainly undertaken by a Dental Reference Officer rather than using the surgery IT system. In the two years since the initial list of KPIs had been drawn together, a system of electronic transmission to a central organisation undertaking data analysis had been set up (VITAL signs data), and it was agreed that this central system could be used to capture data relating to ten other KPIs in the future. This left five KPIs to be monitored by extracting data from clinical databases periodically (Table 1).

Describing the practice population

One of the more useful indicators used during the pilot related to the summaries of the proportions of red/amber/green patients per practice (risk assessment based on the social history, medical history and clinical examination). 'Red' patients were those identified as having a significant feature in their medical and/or social history, and/or active disease; 'amber' patients had a significant feature in their medical and/or social history and/or controlled disease; and 'green' patients were those with no relevant medical or social history and no evidence of recent disease. The proportion of red, amber and green patients in the three practices involved are shown in Figure 1. Practice 2, situated in the more socio-economically disadvantaged area, had the highest proportion of red patients while Practice 3 had the lowest. Notably, Practice 2 also had a relatively high proportion of green patients compared to Practice 1, perhaps reflecting a catchment area which was in close proximity to the university as well as being located in a low socio-economic area. This data gave commissioners information about the patient need and thus required resources, at a practice level, for the first time. The data could be used to inform negotiation of contract values in different areas:

Figure 1
figure 1

Proportion of high (red), medium (amber) and low risk/need patients in Practices 1, 2 and 3

Full size image

'If you put the middle group of patients, being say, the amber group [where there is some care but there is some stabilisation of disease] ... those people who have high levels of disease will have much more frequent visits and more frequent interventions and their cost of care may be double if not treble, someone who has very good oral health and is in the green category and they only need to see the dental team perhaps once in 18 months. So the cost of the pathway of say a green care pathway child would be in the order by a quarter or a third of what we might expect to see in an amber pathway.' (Consultant in Dental Public Health)

There was also some evidence from the pilot practices that higher needs patients required more resource in terms of provision of unscheduled care, not just in the provision of preventive care. In Practice 2, data showed that although there was a failed appointment rate of 9.9% overall (2,963 out of 29,972 appointments in the six months from March 2009), 69% of the failed appointments were by red patients, 21% of failed appointments were by amber patients and 10% of failed appointments were by green patients.

Outcome measures

The preponderance of process rather than outcome indicators did not signify that outcome indicators were viewed as less important as a KPI than process measures. On the contrary, they were viewed as being important, but there was some difficulty in identifying robust indicators which were measurable. One outcome indicator which was found to be measureable was the proportion of patients, when re-assessed a year later, who had made an improvement (reduced risk) in their red/amber/green categorisation. Table 2 shows data from Practice 1, based on clinical assessments. It shows that when re-assessed one year later, of the 178 who were red at their original assessment, 56 (31%) had shifted to an amber category (no active disease), and another four were categorised as 'green' on re-assessment. Other data from this practice showed that between April 2009 and March 2010, 713 red patients were scheduled to be re-assessed but only 178 (25%) returned for re-assessment when scheduled one year later.

Table 2 Change in risk banding on review (Practice 1)
Full size table

Practitioners suggested that a transition from a high to a lower risk category would take some time (longer than a year) to become apparent, given the broad measures being used:

'If they have got BPE scores of reds or if they have got gross caries, it takes a bit of time for them to change their diets although we would like to see some changes after 12 months and then I think after 24 months would be the time when we would really start to see a big difference – especially now X [the therapist] has been putting in her notes “Patient's oral hygiene improved” and when we are doing our reviews we mention that to the patients. So although they are still classified as reds maybe they are just on borderline now so maybe in another 12 months time we might be seeing them moving more towards amber.' (Practitioner)

In identifying outcome measures, the Steering Group took the approach that these should be relatively simple rather than highly specific in order to avoid making data collection and analysis onerous. A downside of this was that specificity was compromised (for example, BPE being a sextant score might not reflect site-specific improvements in periodontal status). This required that there be an understanding that contract negotiations would use these measures as an indicator of improvements in health, and as a basis for discussions, rather than a strict and stringent measure of performance.

Process measures

Data gathered from surgery IT systems demonstrated that high numbers of preventive procedures were being undertaken, as set out by clinical protocols used in the pilot. Central data, however, allowed comparisons to be made between pilot practices and other practices in the area. Data comparing the proportions of procedures in the three practices with averages from other practices in their areas (Figs 2 and 3) show a higher proportion of fluoride varnish procedures in the pilot practices (Practices 1 and 2).

Figure 2: Proportion of types of procedures undertaken in Practice 1 compared to the average of other practices in the same area.
figure 2

Note: proportions take into account varying size of practices with differences in numbers of procedures. Value labels denote numbers of procedures

Full size image
Figure 3: Proportion of types of procedures undertaken in Practices 2 and 3 compared to the average of other practices in the same area.
figure 3

Note: proportions take into account varying size of practices with differences in numbers of procedures. Value labels denote numbers of procedures

Full size image

Discussion

Quality of care is that portion of a patient's outcome over which healthcare providers, whether individuals or organisations, have control.11 A key concept which under-pins many quality of care measures is the Donabedian delineation into structures (the relatively stable traits of providers, their tools and resources and physical work setting, for example staffing ratios); processes of care (activities which providers undertake for their patients); and outcomes (changes in the patients' health status).12 Structure is viewed as having an impact on quality, but is recommended as a more distal measure.11 Process measures are found to make sense to clinicians since they are under the providers' control, although these have a downside in tending to lead to perverse behaviour which may not be to the benefit of the patient, or the best use of resources.13 Outcome measures are therefore becoming more favoured, but also are open to some criticism. Outcome measures may be affected by factors other than the quality of care, including case-mix: a severely ill patient is more likely to do poorly than a less severely ill patient, regardless of the quality of care. Adverse risk selection (providers selecting patients likely to benefit) can occur as a result. Furthermore, unexplained variation in outcomes may exist, where some patients do better than others, for no apparent reason, without being related to the performance of the provider.11

Outcome measures are also reliant on evidence being available that certain procedures inevitably bring about beneficial outcomes for patients. Unfortunately compiling this type of evidence can be complex, and thus not readily available.14 As attention moves to new contract forms in primary dental care based on a 'blended' contract with a variety of indicators, including quality measures, it will be necessary to address some of these contradictions and ascertain what type of performance measures are appropriate indicators in this setting.

The proportion of patients in the practice falling into high, medium and low risk categories was found to be one of the most salient and useful measures piloted. Having this information allows case mix to be taken into account when judging the performance of providers. Commissioners can use this indicator when interpreting other information, such as those concerned with activity. It can also be used to help judge whether any adverse risk selection of patients may be being employed by providers.

Since the role of the commissioner is two-fold: ensuring procured services deliver both clinically effective care as well as value for money; it is not surprising that their considerations of 'quality' were found to have wider connotations than for practitioners. This type of contrast reflects the paradigm outlined by Campbell et al.15 that although quality of care is at its most meaningful when applied to the individual user of services, care for individuals must be placed in a wider context of the provision of healthcare for populations, and notions of efficiency are therefore also relevant to quality considerations. While various strategies (such as selective contracting based on quality) have been used by purchasers to influence quality of care, evidence is rather weak concerning whether such strategies undermine clinicians' professional and personal motivation to provide a high quality service for their patients, and result in reductions in quality in areas not the focus of targets.16 Certainly these potential downsides of contracting for quality need to be considered when introducing this type of commissioning approach to primary dental care. Performance management of quality by commissioning epitomises a 'top down' approach to quality improvement and this runs contrary to the essence of clinical governance, which is conceived very much as a 'bottom up' mechanism to inspire and enthuse those providing care within a culture of a no-blame learning environment.17 Practitioners may adhere more to dynamic models of quality improvement, where no absolutes of ultimate quality are required, than the relativistic approaches invoked by the use of quality KPIs and benchmarks.11

These different perspectives can be seen as two different, although linked, and potentially synergistic pathways to quality improvement.18 In any population group evaluated (which in this case would be care provided by dental practitioners), a distribution in performance levels exists. The shape of the distribution may vary from population to population, and from one performance dimension to another; it may be bell-shaped, skewed or multi-modal, but most practitioners will be clustered around the mean. One strategy to achieve quality improvement is to employ selection: for example, based on knowledge about performance, commissioners reward new contracts, and renew contracts with high performing providers. However, selection does not by itself change the basic distribution of performance. The second strategy, improvement through changes in care, aims to shift the distribution of performance of all practitioners. Berwick et al.18 suggest this is done by providing both process and outcome information to providers, who have the power to change organisational and individual processes to achieve improvement in quality. A key argument supporting the use of the second strategy, which takes a more collaborative approach between commissioners and providers, is that a strategy based purely on selection tends to widen inequalities in healthcare, while a strategy based on shifting the performance of all providers narrows healthcare inequity.19

The pilot scheme identified one health outcome indicator (transfer of high need/risk patients to medium need/risk pathways) which was feasible to measure, acceptable to practitioners, and which also met with the requirements of a valid outcome measure in that the indicator discriminated between different subsets of patients and was responsive to real change over time.19 However, interpretation of success in achieving targets set in relation to this indicator still needs to take account of attributability and that the time focus may need to be longer than a year.

Shanks and Frater20 draw a clear distinction between health outcomes (an effect manifest as a change in health status) and healthcare outcomes (a result attributable and responsive to healthcare where a causal connection with healthcare is established). This recognises that the factors which determine the progress of patients are multi-factorial, and while clinicians may contribute to changes in the health status of patients, changes may also occur, for example, because of changes in social and economic circumstances. However, it can be argued that measurement of outcome as a change in health status is valuable, even before we know precisely the contribution of healthcare to the proposed changes.20 Shanks and Frater20 use the analogy of establishing that thunder is a product of lightning, and while the causal link may be difficult to establish, observing that the two phenomena usually occur together, and that both are more likely to occur in warm weather, and frequently occur with heavy rain, is still useful. Thus, collecting information on patient's health states can be still be useful even if attributability is unclear. Aggregated health outcome data, for example, may be used in establishing standards for comparison between dental practices, improving prognostic accuracy and generating causal hypotheses for testing.20 The key to resolving the ethical argument as to whether judgements based on healthcare outcome indicators are just19 probably lies in whether such information is used to unfairly penalise providers failing to perform, or whether it is used in an non-accusatory way, in a mutual (engaging both commissioner and clinician) search for quality improvement.

There is also a further perspective on healthcare outcomes that has so far been ignored, and which has not, as yet, been addressed in the pilot scheme: that of the patient. Substantial differences have been reported in the assessment of outcome as perceived by clinicians, patients and their relatives.21 Nevertheless, it has been argued that using patients' perceptions of outcomes is the most fair and ethical approach, since it puts medical care in the context of people's expectations and because the ultimate aim and ethos of medical care is to do what is in the best interests of the patient.19 Unfortunately, while the importance of patient-reported outcome measures is increasingly being recognised,2 patient generated outcome measures (PROMs) which truly reflect patients' definitions of the nature as well as the strength of effects are rare,19 and particularly so in the context of dentistry. Methodological issues exist, as well as practical problems (data need to be gathered at baseline as well as on follow-up). This leads some to query whether the whole exercise merits the investment of resources, with estimates putting a cost of £6.50 per patient on such data.22 Thus, a general word of caution is given to those contemplating embarking on gathering PROMs: that researchers and managers should give careful thought to understanding what the particular healthcare problem is that PROM data is thought to help illuminate; and to build into any evaluation whether value for money is being delivered by using such methods.22

Conclusion

There is a clear move away from monitoring clinical activity and towards outcome measurement as a means of evaluating whether contractors are supplying appropriate services to a sufficient quality standard. To some extent dentistry has lagged behind developments in medicine, with outcome measurement relatively rare in the dental sector. This paper describes the piloting of a range of performance measures in primary dental care and demonstrates that the search for outcome measures of high utility and practicality is difficult. Issues such as attributability and the time required for change in health status to become apparent means that there is a danger that such information could be misinterpreted. The findings, however, suggest that there are some measures which might be used, particularly to describe practice populations, as a tool in the mutual search for quality by both commissioners and providers.