Evaluating Disease Management Program Effectiveness

Abstract

Disease management (DM) program evaluations are somewhat limited in scope because of typically small sample sizes comprising important subsets of the treated population. Identifying subsets of the data that have differing results from the aggregate of the whole program can lend insight into where, when, and how the program achieves its results. Additionally, there is a very limited set of classical tools available for the smaller sample sizes typically encountered in DM. Without readily available standard error and confidence interval (CI) calculations, the analyst may be fooled by specious details.

A method called the ‘bootstrap’ is introduced as a suitable technique for allowing DM program evaluators to use a broader array of quantities of interest and to extend inferences to the population based on results achieved in the program. The bootstrap uses the power of modern computers to generate many random samples from a given data set, allowing the use of repeated samples’ statistic (e.g. mean, proportion, and median). Using a congestive heart failure (CHF) program as an example, the bootstrap technique is used to extend a DM program evaluation beyond questions addressed using classical statistical inference: (i) how much of a median cost decrease can be expected as a result of the program?; (ii) did the program impact the highest and lowest costing members equally; and (iii) how much of a decrease in the proportion of patients experiencing a hospitalization can be expected as a result of the program?

The potential advantages of the bootstrap technique in DM program evaluation were clearly illustrated using this small CHF program example. A more robust understanding of program impact is possible when more tools and methods are available to the evaluator. This is particularly the case in DM, which is inherently biased in case-mix (e.g. strive to enroll sickest first), often has skewed distributions or outliers, and may suffer from small sample sizes.

The bootstrap technique creates distributions that allow for a more accurate method of drawing statistical inferences of a population. Moreover, since classical statistical inference techniques were designed specifically for parametric statistics (i.e. assuming a normal distribution), the bootstrap can be used for measures that have no convenient statistical formulae. Additionally, CIs can be defined around this statistic, making it a viable option for evaluating DM program effectiveness.