Bridging the Gap: A Hybrid Model to Link Efficacy and Effectiveness Research in Substance Abuse Treatment

A major issue facing virtually all areas of clinical psychiatry is the gap between research and practice and the associated call for clinical research to place a greater emphasis on effectiveness research and the evaluation of clinical utility of treatments (1,2,3). Disparities between research and practice are particularly apparent in drug abuse treatment (4). For example, despite overwhelming empirical support for the efficacy and cost-effectiveness of methadone maintenance (5,6,7), access to this form of treatment remains highly restricted in many areas of the United States (8,9,10). In locations in which methadone maintenance is available, this treatment is often administered in inadequate doses or with insufficient counseling and ancillary services, both of which make it much less effective (11,12). Similarly, behavioral treatments for which there is strong empirical support have rarely been implemented in clinical settings (13,14). Conversely, many treatments are widely used in clinical practice that have not undergone any evaluation of efficacy—for example, 12-step methods and auricular acupuncture (15,16).

Need for new research strategies

A recent landmark report by the Institute of Medicine (4) called for a greater emphasis on effectiveness research as a strategy for bridging the gap between research and practice. However, although methods and strategies for efficacy evaluation through randomized controlled trials have been well defined for both behavioral and pharmacologic therapies (17,18,19), the nature of what constitutes sound effectiveness research is much less clear. Traditional strategies—for example, large-scale demonstration projects—have too frequently had a negligible impact on the clinical community, policy makers, or third-party payers. This lack of impact may be due to several factors, such as failure to target critical questions about the treatment's utility in standard clinical practice, to methodologic flaws that have undermined internal validity, or to failure to address critical questions relevant to providers and policy makers.

The stage model of behavioral therapies research, developed by the National Institute on Drug Abuse (17), is one such attempt to conceptualize the transition from initial development of a new treatment intervention to ultimate community utilization. This model is innovative in the arena of behavioral therapies research, because it articulates three progressive stages that roughly parallel those for the development of pharmacologic therapies, as shown in Table 1 (18). Stage I consists of initial development and pilot or feasibility testing of new and untested treatments. Stage II consists principally of randomized controlled clinical trials to evaluate efficacy of treatments that have shown promise or efficacy in initial studies. Stage III, which corresponds to phase IV research for pharmacologic trials, is intended to address issues of transportability of treatments whose efficacy has been demonstrated in at least two stage II trials. A limitation of this model is the currently underdeveloped stage III, which is the focus of this article.

To illustrate the need to address practice-related issues in efficacy research, we present two examples of pharmacologic and behavioral treatments that have demonstrated efficacy and that have largely failed to be adopted in clinical practice. We then propose a hybrid strategy for effectiveness research that retains critical features of efficacy trials but that extends these features to address issues of interest to clinicians and policy makers.

Behavioral treatments

Cocaine dependence became a major public health problem in the 1980s and was met with a multitude of evaluations of novel behavioral and pharmacologic approaches. Despite many trials that have evaluated a wide range of pharmacologic agents, none has consistently been found to be effective in treating general groups of cocaine users (12). Thus, perhaps the most exciting findings on the effectiveness of treatment for cocaine dependence have been the reports (20,21) on contingency management.

However, the initial demonstrations of the impressive effects of contingency management (21) were conducted in a comparatively homogeneous population of Caucasian, middle-class cocaine users, which raised concerns about generalizability to more diverse populations. However, numerous clinical trials have since demonstrated the efficacy of voucher-based contingency management in a broad range of settings and populations. These include methadone maintenance settings for cocaine-dependent substance users in large urban areas (22), homeless substance users with significant levels of psychopathology (23), and freebase cocaine users (24).

Contingency management has also been shown to be flexible in that several other behaviors can also be targeted—for example, medication compliance (25,26). The consistency and strength of the empirical data supporting contingency management have been impressive; few negative results of trials have been reported, and the estimated effect size of .25 in methadone maintenance settings (27) is notable. Nevertheless, ten years after the initial report on their efficacy, contingency management approaches are virtually nonexistent outside research settings.

Several reasons have been cited for the failure of contingency management approaches to be adopted in community settings. First, the cost of this approach—in the original Higgins system, patients may receive more than $1,000 in vouchers—has been seen as unrealistic. However, lower-cost contingency management approaches, with an average cost of $200 per patient, have recently been demonstrated to be effective in community settings (28,29).

Second, ideological issues have limited the acceptance of contingency management procedures outside research settings. These approaches have been perceived as tantamount to "paying patients to stay clean." Moreover, it is not clear that heretofore limited efforts to disseminate contingency management have emphasized the basic behavioral principles underlying this approach, such as the value of positive reinforcement in changing behavior.

Third, several aspects of contingency management may be difficult for clinicians to implement without programmatic or payer support. For example, verification of abstinence requires much more frequent urine screening than is typical in many clinical programs (30). Successful implementation of contingency management also requires rapid feedback of urine test results, which is often unavailable in community-based settings.

Pharmacotherapies

In contrast with behavioral therapies, pharmacotherapies typically command vigorous, sophisticated treatment dissemination efforts funded by private pharmaceutical firms that stand to profit directly from widespread use. Despite such considerations, only two agonist agents—methadone for opioid dependence and nicotine replacement treatments for tobacco dependence—can be characterized as playing a major role in routine substance abuse treatment. Other agents that have been approved by the Food and Drug Administration are used among less than 5 percent of the target population of substance abusers who are in treatment (31,32,33). Factors that impede the transfer of new pharmacotherapies from research settings to routine clinical use are exemplified by the largely unsuccessful U.S. marketing efforts for naltrexone treatment for alcohol dependence in the two years after naltrexone received FDA approval in late 1994.

Many indicators suggested the promise of naltrexone as a treatment for alcohol dependence. In two well-designed randomized clinical trials, naltrexone demonstrated strong superiority over placebo for preventing relapse and maintaining abstinence in recently detoxified alcoholics in outpatient treatment (34,35,36). A third open-label study of more than 500 patients had similar success rates and affirmed safety and tolerability of naltrexone for alcoholics with and without comorbid psychopathology (37).

Alcohol dependence is the most frequently diagnosed mental disorder in large-scale community surveys (38,39,40), affecting 4 to 6 percent of adults at any given time. Naltrexone has only one pharmacological competitor—disulfiram—which received FDA approval more than 40 years ago and which has been infrequently used because of low patient acceptance and limited efficacy results (41). Nevertheless, and despite an extensive marketing campaign, naltrexone failed to become a mainstream treatment for alcohol dependence. Its manufacturer, DuPont Pharma, ceased U.S. marketing efforts after two years of prescription rates that fell far short of initial projections.

Rigorously designed clinical trials indicated naltrexone's strengths but failed to address what turned out to be major impediments to its dissemination. One major factor is the decentralization and diversity of alcoholism treatment and the limited role of physicians in specialized substance abuse programs. Hence traditional educational efforts that targeted physicians failed to reach the nonmedical counselors and clinicians who were responsible for day-to-day treatment. In addition, the highly influential Alcoholics Anonymous philosophy encourages patients to avoid reliance on medications to maintain sobriety (42). Thus, even if physicians were enthusiastic about naltrexone, lack of support by counseling staff tended to prevent referrals and to undermine compliance.

Second, naltrexone cannot be considered a stand-alone treatment, because its efficacy has been tested predominantly when the agent was combined with specialized behavioral treatment programs (34,35). No data are available on naltrexone's effectiveness in combination with more generally available treatments, such as counseling, self-help groups, and medically oriented clinical management. This requirement for nontraditional behavioral treatment has raised doubts about naltrexone's use in the context of standard treatment programs. Similarly, naltrexone's efficacy has been evaluated only among patients who have already achieved seven to 30 days of abstinence from alcohol, raising questions about its potential for the many patients who have difficulty achieving sustained abstinence.

Third, the use of naltrexone entails some risks and requires medical monitoring (43). The relatively low risk of hepatotoxicity requires that liver function blood tests be performed before treatment and periodically throughout treatment. Highly elevated liver function levels, a frequent finding in alcoholism, is a relative contraindication for naltrexone treatment. In addition, the use of naltrexone entails complicated considerations in the treatment of pain. Because naltrexone is a powerful opioid antagonist, it can cause withdrawal symptoms among patients who are currently using opioid medications and precludes routine use of opioid analgesics, such as codeine or morphine, if patients develop a painful condition of sudden onset—for example, renal stones or trauma. These medical complications tended to limit naltrexone prescribing to physicians with a special interest in substance abuse treatment and tended to put off generalist psychiatrists and primary care physicians.

Finally, the fact that there are only two FDA-approved medications for alcoholism has had unanticipated drawbacks. Although naltrexone has only one weak competitor—disulfiram—correspondingly few patients and clinicians are accustomed to thinking about pharmacotherapy as a major part of alcoholism treatment after detoxification. Finally, many patients and clinicians have expressed concerns about the cost of naltrexone at $5 to $6 a day, naltrexone remains unlisted in many prescription plans.

Limitations of existing efficacy models

Efficacy trials typically involve tightly controlled settings and more narrowly defined, homogeneous samples than those seen in clinical practice, which raises questions about their generalizability (44,45). The higher costs of treatment delivery and training relative to the potential benefits of novel approaches have not been addressed. The feasibility and acceptability of the approaches to clinicians, payers, and patients are rarely systematically evaluated in efficacy trials.

A hybrid model

The challenges of fostering wider use of empirically validated treatments in clinical settings revolve around a number of critical issues: generalizability (Will this treatment work with different practitioners, patients, and settings?), implementation (What kinds of training and what kinds of trainers are necessary for what kinds of clinicians to learn a new technique?), cost-effectiveness (Compared with the costs of learning and implementing this treatment, what are the savings, particularly in comparison to existing methods?), and marketing (How acceptable is a new treatment to both clinicians, patients, and payers outside research settings?)

Efficacy and effectiveness research have often been regarded as being at opposite ends of the internal-external validity continuum (46). Efficacy trials are frequently criticized as having limited generalizability to real-world issues and problems, whereas effectiveness studies that place a high premium on ecological validity often lack design features required to definitively answer basic issues regarding treatment effectiveness or to rule out alternative explanations of findings. Below we propose a "hybrid" model that shares critical components of efficacy research but also includes features intended to reduce barriers to implementation of these treatments in clinical settings (Table 2).

Elements of efficacy trials to be retained

In a hybrid model, it is critical that scientific rigor be preserved through the use of design features that protect crucial aspects of internal validity. These features include random assignment of patients to treatments, blind delivery of treatments, blind assessment of outcomes, intention-to-treat analyses, use of objective outcome measures, definition and monitoring of treatments delivered, specialized training of providers in delivering study treatments, and evaluation of the integrity of study treatments (for example, assessment of compliance in medication trials and evaluation of treatment fidelity in behavioral trials).

Although these features add to the cost and complexity of clinical trials, the disadvantages of omitting them are numerous and consequential. For example, lack of randomization to treatment conditions means that a study loses the ability to address fundamental questions, such as whether the experimental treatment was more effective than standard treatment and whether the level of change associated with the experimental treatment was meaningful. Most important, however, is that studies that do not randomly assign participants to a condition lose the ability to rule out numerous alternative explanations for findings, such as maturation, history, and measurement effects. Borkovec and colleagues (47) have noted that failure to protect these vital aspects of internal validity renders generalizability issues moot.

Moreover, for most empirically validated treatments for substance abuse, only a handful of supporting clinical trials may exist. Thus, even as effectiveness studies are undertaken, it will be essential to continue to conduct randomized controlled comparisons, because there are few true reference conditions in substance abuse treatment against which to evaluate the efficacy of novel approaches. For example, in the treatment of opioid dependence, methadone maintenance most resembles a standard "reference" treatment. However, largely due to variability in the clinical context in which it is delivered, outcomes for methadone maintenance are highly variable (11,48).

In addition, control conditions in efficacy trials are generally designed to evaluate highly focused research questions—for example, whether a given medication is more effective than a placebo that controls for expectations of improvement or whether a given behavioral therapy is more effective than a minimal discussion condition. Thus control conditions that are typically used in efficacy research rarely resemble treatment as delivered in clinical practice and thus do not address issues of interest to clinicians and policy makers—for example, whether adding a novel treatment enhances outcomes compared with treatment as usual.

Effectiveness elements to be retained

To expand the range of issues addressed in efficacy research, we propose a hybrid model in which the above design elements are retained while additional effectiveness components are added.

Enhanced diversity in patients and settings. Few single-site trials are designed or sufficiently powered to allow meaningful analysis of sources of variability in outcome due to variables such as ethnicity, gender, severity, or psychiatric comorbidity. Diversity may be enhanced by using a number of strategies, such as conducting clinical trials in community-based programs and multiple geographic settings, using less restrictive inclusion and exclusion criteria, reducing barriers to participation (for example, not restricting protocols to English-speaking participants or not requiring participants to undergo very lengthy assessment batteries), and allowing some variation in treatment implementation across sites.

Such strategies will greatly enhance our knowledge about the robustness of various treatments. For example, it will be extremely useful to know if treatment A is effective among persons with both alcohol and cocaine abuse but not those with both depression and cocaine abuse. Similarly, it is of great importance to know whether an innovative treatment is effective when implemented by a broad range of clinicians, as opposed to just those with advanced degrees.

Enhancing diversity in study populations typically requires multiple single-site replication studies or larger multisite trials. However, Klein and Smith (49) have noted that even community-based replications or multisite trials may have limited generalizability. That is, simply because a study is conducted in a community-based setting does not mean that its results are generalizable to all other community settings.

Attention to training issues. If clinicians are to implement empirically validated treatments, effective training is essential. Although pharmaceutical companies widely disseminate information about new pharmacologic approaches to physicians, many community-based treatment programs have no or few affiliated medical personnel to implement these treatments (50).

Similarly, although methods for training clinicians in manual-guided therapies for clinical efficacy trials are well established (51,52,53), such methods have largely been accepted on the basis of face validity (54,55,56). It is not known whether standard methods of training therapists will be feasible or effective when applied to real-world clinicians. Training in clinical efficacy trials is usually geared toward highly trained and experienced clinicians. Thus trainers may assume basic familiarity with underlying principles of the treatment. However, clinicians who work in community-based drug abuse settings have varied educational backgrounds (4). Thus standard training methods for clinical trials—which encompass brief review of a treatment manual, watching a few videotaped vignettes, and participating in a few role-playing exercises—may be insufficient for many drug abuse counselors.

If we are to learn what kind of training is needed by various types of clinicians to effectively implement scientifically validated treatments, empirical studies of training methods are needed. Focused evaluations of training methods may be integrated into large stage III hybrid studies in which, for example, the pilot or startup phase might be seen as providing a vehicle for systematically evaluating the effectiveness of various training methods with a variety of types of clinicians and settings. That is, clinicians could be randomly assigned to different training methods, such as low-intensity versus high-intensity training, and their adherence and competence in implementing treatments could be evaluated.

Evaluation of cost-effectiveness. Almost all novel therapies come with some added cost, either the cost of the treatment itself or training costs. Because the added cost of empirically validated treatments has typically not been a component of efficacy studies, researchers have had little basis on which to convince program leaders or policy makers that the efforts and costs entailed in introducing new treatments are justified.

The value of integrating cost-effectiveness analyses into clinical trials—even those that occur early during the course of a treatment's evaluation—is suggested by several recent studies that have demonstrated cost savings and benefits associated with adding enhancements to standard approaches to substance abuse treatment (57,58,59,60,61). When cost savings for innovative treatments are demonstrated, providers and third-party payers are likely to be much more interested in supporting the adoption of these treatments into standard care.

Assessment of patient and provider satisfaction. In recent years, increasing emphasis has been placed on client preference and patient satisfaction as indicators of a treatment's utility and value (62,63,64,65). Indicators of patient satisfaction are important in determining whether a new approach would add value to a program by making it more attractive to patients. Inclusion of satisfaction measures that address patients' global judgments of satisfaction and improvement, supplemented by more specific feedback on their reactions to individual components of innovative treatments—for example, content, duration, intensity, and therapist (66)—will be invaluable in evaluating the success of novel treatments in community settings. Greater satisfaction of clinicians with the nature of the interventions they provide to their patients may be important in reducing staff turnover in community programs, in which the rate of turnover is as high as 15 to 50 percent annually (13,67).

Acknowledgments

Preparation of this article was supported by grants U10-DA-13038, P50-DA0924, and K05-DA00457 from the National Institute on Drug Abuse.

The authors are affiliated with the department of psychiatry at Yale University School of Medicine and with the Department of Veterans Affairs VISN 1 Mental Illness, Research, and Clinical Center of the VA Connecticut Healthcare System in West Haven, Connecticut. Send correspondence to Dr. Carrol at the Division of Substance Abuse, VA Connecticut Healthcare System (151D), West Haven, Connecticut 06516 (e-mail, [email protected]).

Enlarge table

Enlarge table