Top

Health Services and Outcomes Research Methodology

Published in:

01-06-2012

Assessing the sensitivity of treatment effect estimates to differential follow-up rates: implications for translational research

Authors: Beth Ann Griffin, Daniel F. McCaffrey, Rajeev Ramchand, Sarah B. Hunter, Marika Suttorp Booth

Published in: Health Services and Outcomes Research Methodology | Issue 2-3/2012

Abstract

We develop a new tool for assessing the sensitivity of findings on treatment effectiveness to differential follow-up rates in the two treatment conditions being compared. The method censors the group with the higher response rate to create a synthetic respondent group that is then compared with the observed cases in the other condition to estimate a treatment effect. Censoring is done under various assumptions about the strength of the relationship between follow-up and outcomes to determine how informative differential dropout can alter inferences relative to estimates from models that assume the data are Missing at Random. The method provides an intuitive measure for understanding the strength of the association between outcomes and dropout that would be required to alter inferences about treatment effects. Our approach is motivated by translational research in which treatments found to be effective under experimental conditions are tested in standard treatment settings. In such applications, follow-up rates in the experimental setting are likely to be substantially higher than in the standard setting, especially when observational data are used in the evaluation. We test the method on a case study evaluation of the effectiveness of an evidence-supported adolescent substance abuse treatment program (Motivational Enhancement Therapy/Cognitive Behavioral Therapy-5) delivered by community-based treatment providers relative to its performance in a controlled research trial. In this case study, follow-up rates in the community-based settings were extremely low (54 %) compared to the experimental setting (95 %) giving raise to concerns about non-ignorable drop-out.

Available only for authorised users

Alford, G.S., Koehler, R.A., Leonard, J.: Alcoholics anonymous-narcotics anonymous model inpatient treatment of chemically dependent adolescents: a 2-year outcome study. J. Stud. Alcohol 52, 118–126 (1991)PubMed

American Society of Addiction Medicine: ASAM Patient Placement Criteria for the Treatment of Substance-Related Disorders, 2nd revised edn. Lippincott Williams & Wilkins, Chevy Chase (2001)

Barkham, M., Stiles, W.B., Connell, J., Twigg, E., Leach, C., Lucock, M., Mellor-Clark, J., Bower, P., King, M., Shapiro, D.A., Hardy, G.E., Greenberg, L., Angus, L.: Effects of psychological therapies in randomized trials and practice-based studies. Br. J. Clin. Psychol. 47, 397–415 (2008)PubMedCrossRef

Bertelsen, M., Jeppesen, P., Petersen, L., Thorup, A., Øhlenschlaeger, J., le Quach, P., Christensen, T.Ø., Krarup, G., Jørgensen, P., Nordentoft, M.: Five-year follow-up of a randomized multicenter trial of intensive early intervention vs standard treatment for patients with a first episode of psychotic illness: the OPUS trial. Arch. Gen. Psychiatry 65(7), 762–771 (2008)PubMedCrossRef

Blanco, C., Olfson, M., Goodwin, R.D., Ogburn, E., Liebowitz, M.R., Nunes, E.V., Hasin, D.S.: Generalizability of clinical trial results for major depression to community samples: results from the National epidemiologic survey on alcohol and related conditions. J. Clin. Psychiatry 69, 1276–1280 (2008a)PubMedCrossRef

Blanco, C., Olfson, M., Okuda, M., Nunes, E.V., Liu, S.M., Hasin, D.S.: Generalizability of clinical trials for alcohol dependence to community samples. Drug Alcohol Depend. 98, 123–128 (2008b)PubMedCrossRef

Brown, S.A., Gleghorn, A., Schuckit, M.A., Myers, M.G., Mott, M.A.: Conduct disorder among adolescent alcohol and drug abusers. J. Stud. Alcohol 57, 314–324 (1996)PubMed

Brown, S.A., Tapert, S.F., Tate, S.R., Abrantes, A.M.: The role of alcohol in adolescent relapse and outcome. J. Psychoact. Drugs 32, 107–115 (2000)CrossRef

Cady, M.E., Winters, K.C., Jordan, D.A., Solberg, K.B., Stinchfield, R.D.: Motivation to change as a predictor of treatment outcome for adolescent substance abusers. J. Child Adolesc. Subst. Abuse 5, 73–91 (1996). doi:10.1300/J029v05n01_04 CrossRef

Christensen, H., Griffiths, K.M., Farrer, L.: Adherence in internet interventions for anxiety and depression. J. Med. Internet Res. 11(2), e13 (2009)PubMedCrossRef

Cochran, W.G.: The effectiveness of adjustment by subclassification in removing bias in observational studies. Biometrics 24(2), 295–313 (1968)PubMedCrossRef

Cohen, J.: A power primer. Psychol. Bull. 112, 155–159 (1992)PubMedCrossRef

Copas, J.B., Eguchi, S.: Local sensitivity approximations for selectivity bias. J. R. Stat. Soc. Ser. B 63, 871–895 (2001)CrossRef

Copas, J.B., Li, H.G.: Inference for non-random samples (with discussion). J. R. Stat. Soc. Ser. B 59, 55–95 (1997)CrossRef

Daniels, M.J., Hogan, J.W.: Reparameterizing the pattern mixture model for sensitivity analyses under informative dropout. Biometrics 65, 1241–1248 (2000)CrossRef

Dennis, M.L.: Global Appraisal of Individual Needs (GAIN): administration guide for the GAIN and related measures, Version: 1299. Chestnut Health Systems, Bloomington (1999)

Dennis, M.L., Godley, S.H., Diamond, G., Tims, F.M., Babor, T., Donaldson, J., Liddle, H., Titus, J.C., Kaminer, Y., Webb, C., Hamilton, N., Funk, R.: The Cannabis Youth Treatment, CYT Study: Main findings from two randomized trials. J. Subst. Abuse Treat. 27(3), 197–213 (2004)PubMedCrossRef

Dennis, M.L., Titus, J.C., Diamond, G., Donaldson, J., Godley, S.H., Tims, F.M., Webb, C., Kaminer, Y., Babor, T., Roebuck, M.C., Godley, M.D., Hamilton, N., Liddle, H., Scott, C.K., C. Y. T. Steering Committee: The Cannabis Youth Treatment (CYT) experiment: rationale, study design and analysis plans. Addiction 97(Suppl 1), 16–34 (2002)PubMedCrossRef

Desmond, D.P., Maddux, J.F., Johnson, T.H., Confer, B.A.: Obtaining follow-up interviews for treatment evaluation. J. Subst. Abuse Treat. 12(2), 95–102 (1995)PubMedCrossRef

Diggle, P., Kenward, M.G.: Informative drop-out in longitudinal data analysis (with discussion). Appl. Stat. 43, 49–73 (1994)CrossRef

Dodge, R., Sindelar, J., Sinha, R.: The role of depression symptoms in predicting drug abstinence in outpatient substance abuse treatment. J. Subst. Abuse Treat. 28, 189–196 (2005)PubMedCrossRef

Frangakis, C.E., Rubin, D.B.: Principal stratification in causal inference. Biometrics 58, 21–29 (2002)PubMedCrossRef

Friedman, A.S., Glickman, N.W.: Program characteristics for successful treatment of adolescent drug abuse. J. Nerv. Ment. Dis. 174(11), 669–679 (1986)PubMedCrossRef

Friedman, A.S., Glickman, N.W., Morrissey, M.R.: Prediction to successful treatment outcome by client characteristics and retention in treatment in adolescent drug treatment programs: a large-scale cross validation study. J. Drug Educ. 16, 149–165 (1986)PubMedCrossRef

Fuller, W.A., McLoughlin, M.M., Baker, H.D.: Regression weighting in the presence of nonresponse with application to the 1987–1988 Nationwide Food Consumption Survey. Surv. Methodol. 20, 75–85 (1994)

Glynn, R.J., Laird, N., Rubin, D.: Selection modeling vs. mixture modeling with nonignorable response. In: Wainer, H. (ed.) Drawing Inferences from Self-Selected Samples. Springer, New York (1986)

Graubard, B.I., Korn, E.L.: Predictive margins with survey data. Biometrics 55(2), 652–659 (1999)PubMedCrossRef

Greenlees, J.S., Reece, W.S., Zieschang, K.D.: Imputation of missing values when the probability of response depends on the variable being imputed. J. Am. Stat. Assoc. 77(378), 251–261 (1982)CrossRef

Gruber, V.A., Delucchi, K.L., Kielstein, A., Batki, S.L.: A randomized trial of 6-month methadone maintenance with standard or minimal counseling versus 21-day methadone detoxification. Drug Alcohol Depend. 94(1–3), 199–206 (2008)PubMedCrossRef

Harder, V.S., Stuart, E.A., Anthony, J.C.: Propensity score techniques and the assessment of measured covariate balance to test causal associations in psychological research. Psychol. Methods 15(3), 234–249 (2010)PubMedCrossRef

Hedeker, D., Gibbons, R.D.: Longitudinal Data Analysis. Wiley, Hoboken (2006)

Hedeker, D., Mermelstein, R.J., Demirtas, H.: Analysis of binary outcomes with missing data: missing = smoking, last observation carried forward, and a little multiple imputation. Addiction 102, 1564–1573 (2007)PubMedCrossRef

Hirano, K., Imbens, G.W., Ridder, G.: Efficient estimation of average treatment effects using the estimated propensity score. Econometrica 71, 1161–1189 (2003)CrossRef

Holland, P.W.: Statistics and causal inference (with discussion). J. Am. Stat. Assoc. 81, 945–960 (1986)CrossRef

Hunter, S., Ramchand, R., Griffin, B.A., Suttorp, M.J., McCaffrey, D.F., Morral, A.R.: The effectiveness of community-based delivery of an evidence-based treatment for adolescent substance use. J. Subst. Abuse Treat. (in press)

Jenson, J.M., Wells, E.A., Plotnick, R.D., Hawkins, J.D., Catalano, R.F.: The effects of skills and intentions to use drugs on post-treatment drug use of adolescents. Am. J. Drug Alcohol Abuse 19, 1–18 (1993)PubMedCrossRef

Kang, J.D.Y., Schafer, J.L.: Demystifying double robustness: a comparison of alternative strategies for estimating a population mean from incomplete data. Stat. Sci. 22(4), 523–539 (2007)CrossRef

Kennedy, B.P., Minami, M.: The Beech Hill Hospital/outward bound adolescent chemical dependency treatment program. J. Subst. Abuse Treat. 10, 395–406 (1993)PubMedCrossRef

Lee, D.S.: Training, wages, and sample selection: Estimating sharp bounds on treatment effects. NBER Working Paper 11721. National Bureau of Economic Research (2005)

Lee, D.S.: Training, wages, and sample selection: estimating sharp bounds on treatment effects. Rev. Econ. Stud. 76(3), 1071–1102 (2009)CrossRef

Little, R.J.A., Rubin, D.B.: Statistical Analysis with Missing Data, 2nd edn. Wiley, New York (2002)

Little, R.J.A., Wang, Y.: Pattern-mixture models for multivariate incomplete data with covariates. Biometrics 52, 98–111 (1996)PubMedCrossRef

MacKinnon, A., Griffiths, K.M., Christensen, H.: Comparative randomised trial of online cognitive-behavioural therapy and an information website for depression: 12-month outcomes. BJP 192, 130–134 (2008). doi:10.1192/bjp.bp.106.032078 CrossRef

McCaffrey, D.F., Ridgeway, G., Morral, A.R.: Propensity score estimation with boosted regression for evaluating causal effects in observational studies. Psychol. Methods 9, 403–425 (2004)PubMedCrossRef

McGuigan, K.A., Ellickson, P.L., Hays, R.D., Bell, R.M.: Adjusting for attrition in school-based samples: bias, precision, and cost trade-offs of three methods. Eval. Rev. 21, 554–567 (1997)CrossRef

Miller, W.R., Sorensen, J., Selzer, J., Brigham, G.: Disseminating evidence-based practices in substance abuse treatment: a review with suggestions. J. Subst. Abuse Treat. 31(1), 25–39 (2006)PubMedCrossRef

Miller, W.R., Zweben, J., Johnson, W.R.: Evidence-based treatment: why, what, where, when, and how? J. Subst. Abuse Treat. 29, 267–276 (2005)PubMedCrossRef

Morral, A.R., McCaffrey, D.F., Ridgeway, G.: Effectiveness of community-based treatment for substance-abusing adolescents: 12-month outcomes of youths entering Phoenix Academy or alternative probation dispositions. Psychol. Addict. Behav. 18, 257–268 (2004)PubMedCrossRef

Myers, M.G., Brown, S.A., Mott, M.A.: Preadolescent conduct disorder behaviors predict relapse and progression of addiction for adolescent alcohol and drug abusers. Alcohol Clin. Exp. Res. 19, 1528–1536 (1995)PubMedCrossRef

Neugebauer, R., Van der Laan, M.J.: Why prefer double robust estimates? J. Stat. Plan. Inference 129(1–2), 405–426 (2005)CrossRef

Okuda, M., Hasin, D.S., Olfson, M., Khan, S.S., Nunes, E.V., Montoya, I., Liu, S.M., Grant, B.F., Blanco, C.: Generalizability of clinical trials for cannabis dependence to community samples. Drug Alcohol Depend. 111(1–2), 177–181 (2010)PubMedCrossRef

Ramchand, R., Griffin, B.A., Harris, K.M., Morral, A.R.: Using a cross-study design to assess the efficacy of MET/CBT-5 in treating adolescents with cannabis-related disorders. J. Stud. Alcohol Drugs 72(3), 380–389 (2011)PubMed

Richter, S.S., Brown, S.A., Mott, M.A.: The impact of social support and self-esteem on adolescent substance abuse treatment outcome. J. Subst. Abuse 3, 371–385 (1991)PubMedCrossRef

Ridgeway, G., McCaffrey, D., Morral, A., Burgette, L., Griffin, B.A.: Toolkit for weighting and analysis of nonequivalent groups: a tutorial for the twang package. R package (2012)

Robins, J.M., Rotnitzky, A., Zhao, L.P.: Analysis of semiparametric regression models for repeated outcomes in the presence of missing data. J. Am. Stat. Assoc. 90, 106–121 (1995)CrossRef

Rosenbaum, P.R., Rubin, D.B.: The central role of the propensity score in observational studies for causal effects. Biometrika 70, 41–55 (1983)CrossRef

Rosenbaum, P.R., Rubin, D.B.: Estimating the effects caused by treatments—comment. J. Am. Stat. Assoc. 79, 26–28 (1984)

Rosenbaum, P.R., Rubin, D.B.: The bias due to incomplete matching. Biometrics 41, 103–116 (1985)PubMedCrossRef

Rubin, D.B.: Estimating causal effects of treatments in randomized and nonrandomized studies. J. Educ. Psychol. 66, 688–701 (1976a)CrossRef

Rubin, D.B.: Inference and missing data. Biometrika 63, 581–592 (1976b)CrossRef

Rush, T.V.: Predicting treatment outcome for juvenile and young adult clients in the Pennsylvania substance abuse system. In: Beschner, G.M., Friedman, A.S. (eds.) Youth drug abuse: problems, issues, and treatment, pp. 629–656. Lexington Books, Lexington (1979)

Sampl, S., Kadden, R.: Motivational enhancement therapy and cognitive behavioral therapy for adolescent cannabis users: 5 sessions. In: CYT Cannabis Youth Treatment Series, vol. 1. Center for Substance Abuse Treatment, Substance Abuse and Mental Health Services Administration, Rockville, MD (2001)

Scharfstein, D.O., Rotnitzky, A., Robins, J.M.: Adjusting for nonignorable dropouts using semiparametric nonresponse models. J. Am. Stat. Assoc. 94, 1096–1120 (1999)CrossRef

Schluchter, M.D.: Methods for the analysis of informatively censored longitudinal data. Stat. Med. 11, 1861–1870 (1992)PubMedCrossRef

Scott, C.K.: A replicable model for achieving over 90% follow-up rates in longitudinal studies of substance abusers. Drug Alcohol Depend. 7, 21–36 (2004)CrossRef

Shoemaker, R.H., Sherry, P.: Posttreatment factors influencing outcome of adolescent chemical dependency treatment. J. Child Adolesc. Subst. Abuse 2, 89–106 (1991)CrossRef

Troxel, A.B.: A comparative analysis of quality of life data from a Southwest Oncology Group randomized trial of advanced colorectal cancer. Stat. Med. 17, 767–779 (1998)PubMedCrossRef

Troxel, A.B., Harrington, D.P., Lipsitz, S.R.: Analysis of longitudinal measurements with non-ignorable non-monotone missing values. Appl. Stat. 47, 425–438 (1998)

Troxel, A.B., Ma, G., Heitjan, D.F.: An index of local sensitivity to nonignorability. Stat. Sin. 14, 1221–1237 (2004)

Verbeke, G., Molenberghs, G., Thijs, H., Lesaffre, E., Kenward, M.G.: Sensitivity analysis for nonrandom dropout: a local influence approach. Biometrics 57, 7–14 (2001)PubMedCrossRef

Walton, M.A., Ramanathan, C.S., Reischil, T.M.: Tracking substance abusers in longitudinal research: understanding follow-up contact difficulty. Am. J. Community Psychol. 26, 233–253 (1998)PubMedCrossRef

Weisz, J., Donenberg, G., Han, S., Weiss, B.: Bridging the gap between laboratory and clinic in child and adolescent psychotherapy. J. Consult. Clin. Psychol. 63(5), 688–701 (1995)PubMedCrossRef

Witbrodt, J., Bond, J., Kaskutas, L.A., Weisner, C., Jaeger, G., Pating, D., Moore, C.: Day hospital and residential addiction treatment: randomized and nonrandomized managed care clients. J. Consult. Clin. Psychol. 75(6), 947–959 (2007)PubMedCrossRef

Wooldridge, J.M.: Econometric Analysis of Cross Section and Panel Data, 1st edn. The MIT Press, Cambridge (2001)

Wooldridge, J.M.: Econometric Analysis of Cross Section and Panel Data. MIT Press, Cambridge (2002)

Wu, M.C., Carroll, R.J.: Estimation and comparison of changes in the presence of informative right censoring by modeling the censoring process. Biometrics 44, 175–188 (1988)CrossRef

Xie, H., Qian, Y.: Measuring the impact of nonignorability in panel data with non-monotone nonresponse. J. Appl. Econ. 27(1), 129–159 (2012)CrossRef

Title: Assessing the sensitivity of treatment effect estimates to differential follow-up rates: implications for translational research
Authors: Beth Ann Griffin
Daniel F. McCaffrey
Rajeev Ramchand
Sarah B. Hunter
Marika Suttorp Booth
Publication date: 01-06-2012
Publisher: Springer US
Published in: Health Services and Outcomes Research Methodology / Issue 2-3/2012
Print ISSN: 1387-3741
Electronic ISSN: 1572-9400
DOI: https://doi.org/10.1007/s10742-012-0089-7

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Assessing the sensitivity of treatment effect estimates to differential follow-up rates: implications for translational research

Abstract

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Abstract

Please log in to get access to this content

Other articles of this Issue 2-3/2012

Introduction to the special issue: Recap of ninth International Conference on Health Policy Statistics

Assessing the privacy of randomized vector-valued queries to a database using the area under the receiver operating characteristic curve

Mixed-effects regression modeling of real-time momentary pain assessments in osteoarthritis (OA) patients

Joint modeling of longitudinal outcomes and survival using latent growth modeling approach in a mesothelioma trial

Clinically relevant graphical predictions from Bayesian joint longitudinal-survival models

Using AIC in multiple linear regression framework with multiply imputed data