Supporting the implementation of new healthcare technologies by investigating generalisability of pilot studies using area-level statistics

Implementation of new technologies into national health care systems requires careful capacity planning. This is sometimes informed by data from pilot studies that implement the technology on a small scale in selected areas. A critical consideration when using implementation pilot studies for capacity planning in the wider system is generalisability. We studied the feasibility of using publicly available national statistics to determine the degree to which results from a pilot might generalise for non-pilot areas, using the English human papillomavirus (HPV) cervical screening pilot as an exemplar.

From a publicly available source on population indicators in England (“Public Health Profiles”), we selected seven area-level indicators associated with cervical cancer incidence, to produce a framework for post-hoc pilot generalisability analysis. We supplemented these data by those from publicly available English Office for National Statistics modules. We compared pilot to non-pilot areas, and pilot regimens (pilot areas using the previous standard of care (cytology) vs. the new screening test (HPV)). For typical process indicators that inform real-world capacity planning in cancer screening, we used standardisation to re-weight the values directly observed in the pilot, to better reflect the wider population. A non-parametric quantile bootstrap was used to calculate 95% confidence intervals (CI) for differences in area-weighted means for indicators.

The range of area-level statistics in pilot areas covered most of the spectrum observed in the wider population. Pilot areas were on average more deprived than non-pilot areas (average index of multiple deprivation 24.8 vs. 21.3; difference: 3.4, 95% CI: 0.2–6.6). Participants in HPV pilot areas were less deprived than those in cytology pilot areas, matching area-level statistics. Differences in average values of the other six indicators were less pronounced. The observed screening process indicators showed minimal change after standardisation for deprivation.

National statistical sources can be helpful in establishing the degree to which the types of areas outside pilot studies are represented, and the extent to which they match selected characteristics of the rest of the health care system ex-post. Our analysis lends support to extrapolation of process indicators from the HPV screening pilot across England.