Skip to main content
Key Specialties
Cardiology Diabetology Endocrinology Gastroenterology Geriatrics and Gerontology Gynecology and Obstetrics Hematology Infectious Disease Internal Medicine Nephrology Neurology Oncology Pediatrics Respiratory Medicine Rheumatology Surgery
Congress Coverage
2024 ACC Congress 2023 ESMO Congress 2023 EASD Congress 2023 ERS Congress 2023 ESC Congress
Clinical Collections
Advances in Alzheimer’s

At a glance: The STEP trials

A round-up of the STEP phase 3 clinical trials evaluating semaglutide for weight loss in people with overweight or obesity.
ADVANCED SEARCH
Log in
Top
BMC Medical Research Methodology
Published in: BMC Medical Research Methodology 1/2019

Open Access 01-12-2019 | Addiction | Research article

Preventing bias from selective non-response in population-based survey studies: findings from a Monte Carlo simulation study

Authors: Kristin Gustavson, Espen Røysamb, Ingrid Borren

Published in: BMC Medical Research Methodology | Issue 1/2019

Login to get access

Abstract

Background

Health researchers often use survey studies to examine associations between risk factors at one time point and health outcomes later in life. Previous studies have shown that missing not at random (MNAR) may produce biased estimates in such studies. Medical researchers typically do not employ statistical methods for treating MNAR. Hence, there is a need to increase knowledge about how to prevent occurrence of such bias in the first place.

Methods

Monte Carlo simulations were used to examine the degree to which selective non-response leads to biased estimates of associations between risk factors and health outcomes when persons with the highest levels of health problems are under-represented or totally missing from the sample. This was examined under different response rates and different degrees of dependency between non-response and study variables.

Results

Response rate per se had little effect on bias. When extreme values on the health outcome were completely missing, rather than under-represented, results were heavily biased even at a 70% response rate. In most situations, 50–100% of this bias could be prevented by including some persons with extreme scores on the health outcome in the sample, even when these persons were under-represented. When some extreme scores were present, estimates of associations were unbiased in several situations, only mildly biased in other situations, and became biased only when non-response was related to both risk factor and health outcome to substantial degrees.

Conclusions

The potential for preventing bias by including some extreme scorers in the sample is high (50–100% in many scenarios). Estimates may then be relatively unbiased in many situations, also at low response rates. Hence, researchers should prioritize to spend their resources on recruiting and retaining at least some individuals with extreme levels of health problems, rather than to obtain very high response rates from people who typically respond to survey studies. This may contribute to preventing bias due to selective non-response in longitudinal studies of risk factors and health outcomes.
Literature
1.
Gustavson K, von Soest T, Karevold E, Roysamb E. Attrition and generalizability in longitudinal studies: findings from a 15-year population-based study and a Monte Carlo simulation study. BMC Public Health. 2012;12:918.PubMedPubMedCentralCrossRef
2.
Graham JW. Missing data analysis: making it work in the real world. Annu Rev Psychol. 2009;60:549–76.PubMedCrossRef
3.
Fitzmaurice GM, Laird NM, Ware JH. Applied longitudinal analysis. New Jersey: Wiley-interscience; 2004.
4.
Little TD, Rubin DB. Statistical analysis with missing data. 2nd ed. New York: Wiley; 2002.CrossRef
5.
Allison PD. Missing data. Thousand Oaks, CA: Sage; 2001. p. 07–136.
6.
7.
Ibrahim JG, Chen MH, Lipsitz SR, Herring AH. Missing-data methods for generalized linear models: a comparative review. J Am Stat Assoc. 2005;100(469):332–46.CrossRef
8.
Kalaylioglu Z. Performances of Bayesian model selection criteria for generalized linear models with non-ignorably missing covariates. J Stat Comput Sim. 2014;84(8):1670–91.CrossRef
9.
Lipsitz SR, Ibrahim JG, Chen MH, Peterson H. Non-ignorable missing covariates in generalized linear models. Stat Med. 1999;18(17–18):2435–48.PubMedCrossRef
10.
Ibrahim JG, Lipsitz SR. Parameter estimation from incomplete data in binomial regression when the missing data mechanism is nonignorable. Biometrics. 1996;52(3):1071–8.PubMedCrossRef
11.
Ibrahim JG, Lipsitz SR, Chen M-H. Missing covariates in generalized linear models when the missing data mechanism is non-ignorable. J R Statist Soc B. 1999;61:173–90.CrossRef
12.
13.
Galimard JE, Chevret S, Curis E, Resche-Rigon M. Heckman imputation models for binary or continuous MNAR outcomes and MAR predictors. BMC Med Res Methodol. 2018;18(1):90.PubMedPubMedCentralCrossRef
14.
Galimard JE, Chevret S, Protopopescu C, Resche-Rigon M. A multiple imputation approach for MNAR mechanisms compatible with Heckman's model. Stat Med. 2016;35(17):2907–20.PubMedCrossRef
15.
Marshall A, Altman DG, Royston P, Holder RL. Comparison of techniques for handling missing covariate data within prognostic modelling studies: a simulation study. BMC Med Res Methodol. 2010;10:7.PubMedPubMedCentralCrossRef
16.
17.
Little TD, Jorgensen TD, Lang KM, Moore EW. On the joys of missing data. J Pediatr Psychol. 2014;39(2):151–62.PubMedCrossRef
18.
Greene W. A stochastic frontier model with correction for sample selection. J Prod Anal. 2010;34(1):15–24.CrossRef
19.
Lewin A, Brondeel R, Benmarhnia T, Thomas F, Chaix B. Attrition Bias related to missing outcome data: a longitudinal simulation study. Epidemiology. 2018;29(1):87–95.PubMedCrossRef
20.
Kristman V, Manno M, Cote P. Loss to follow-up in cohort studies: how much is too much? Eur J Epidemiol. 2004;19(8):751–60.PubMedCrossRef
21.
Cornish RP, Tilling K, Boyd A, Davies A, Macleod J. Using linked educational attainment data to reduce bias due to missing outcome data in estimates of the association between the duration of breastfeeding and IQ at 15 years. Int J Epidemiol. 2015;44(3):937–45.PubMedPubMedCentralCrossRef
22.
Gustavson K, Borren I. Bias in the study of prediction of change: a Monte Carlo simulation study of the effects of selective attrition and inappropriate modeling of regression toward the mean. BMC Med Res Methodol. 2014;14:133.PubMedPubMedCentralCrossRef
23.
Sullivan TR, Salter AB, Ryan P, Lee KJ. Bias and precision of the “multiple imputation, then deletion” method for dealing with missing outcome data. Am J Epidemiol. 2015;182(6):528–34.PubMedCrossRef
24.
Hoeymans N, Feskens EJ, Van Den Bos GA, Kromhout D. Non-response bias in a study of cardiovascular diseases, functional status and self-rated health among elderly men. Age Ageing. 1998;27(1):35–40.PubMedCrossRef
25.
Tambs K, Ronning T, Prescott CA, Kendler KS, Reichborn-Kjennerud T, Torgersen S, Harris JR. The Norwegian Institute of Public Health Twin Study of mental health: examining recruitment and attrition Bias. Twin Res Hum Genet. 2009;12(2):158–68.PubMedPubMedCentralCrossRef
26.
Thygesen LC, Johansen C, Keiding N, Giovannucci E, Gronbaek M. Effects of sample attrition in a longitudinal study of the association between alcohol intake and all-cause mortality. Addiction. 2008;103(7):1149–59.PubMedCrossRef
27.
Torvik FA, Rognmo K, Tambs K. Alcohol use and mental distress as predictors of non-response in a general population health survey: the HUNT study. Soc Psychiatry Psychiatr Epidemiol. 2012;47(5):805–16.PubMedCrossRef
28.
Van Loon AJM, Tijhuis M, Picavet HSJ, Surtees PG, Ormel J. Survey non-response in the Netherlands: effects on prevalence estimates and associations. Ann Epidemiol. 2003;13(2):105–10.PubMedCrossRef
29.
Nilsen RM, Vollset SE, Gjessing HK, Skjaerven R, Melve KK, Schreuder P, Alsaker ER, Haug K, Daltveit AK, Magnus P. Self-selection and bias in a large prospective pregnancy cohort in Norway. Paediatr Perinat Epidemiol. 2009;23(6):597–608.PubMedCrossRef
30.
Cuijpers P. Meta-analyses in mental health research. A practical guide. Amsterdam: Vrije Universitet Amsterdam; 2016.
31.
Axelsson GT, Putman RK, Araki T, Sigurdsson S, Gudmundsson EF, Eiriksdottir G, Aspelund T, Miller ER, Launer LJ, Harris TB, et al. Interstitial lung abnormalities and self-reported health and functional status. Thorax. 2018;73(9):884–6.PubMedPubMedCentralCrossRef
32.
Thompson R, Flaherty EG, English DJ, Litrownik AJ, Dubowitz H, Kotch JB, Runyan DK. Trajectories of adverse childhood experiences and self-reported health at age 18. Acad Pediatr. 2015;15(5):503–9.PubMedCrossRef
33.
Hagen KB, Aas T, Kvaloy JT, Eriksen HR, Soiland H, Lind R. Fatigue, anxiety and depression overrule the role of oncological treatment in predicting self-reported health complaints in women with breast cancer compared to healthy controls. Breast. 2016;28:100–6.PubMedCrossRef
34.
Liddell TM, Kruschke JK. Analyzing ordinal data with metric models: what could possibly go wrong? J Exp Soc Psychol. 2018;79:328–48.CrossRef
35.
Muthén LK, Muthén BO. Mplus User’s guide, 8th edn. Los Angeles: Muthén & Muthén; 1998-2017.
36.
Akande O, Li F, Reiter J. An empirical comparison of multiple imputation methods for categorical data. Am Stat. 2017;71(2):162–70.CrossRef
37.
van der Palm DW, van der Ark LA, Vermunt JK. A comparison of incomplete-data methods for categorical data. Stat Methods Med Res. 2016;25(2):754–74.PubMedCrossRef
38.
Moylan S, Gustavson K, Overland S, Karevold EB, Jacka FN, Pasco JA, Berk M. The impact of maternal smoking during pregnancy on depressive and anxiety behaviors in children: the Norwegian mother and child cohort study. BMC Med. 2015;13:24.PubMedPubMedCentralCrossRef
39.
Beebe TJ, Rey E, Ziegenfuss JY, Jenkins S, Lackore K, Talley NJ, Locke RG 3rd. Shortening a survey and using alternative forms of prenotification: impact on response rate and quality. BMC Med Res Methodol. 2010;10:50.PubMedPubMedCentralCrossRef
40.
Lungenhausen M, Lange S, Maier C, Schaub C, Trampisch HJ, Endres HG. Randomised controlled comparison of the health survey short form (SF-12) and the graded chronic pain scale (GCPS) in telephone interviews versus self-administered questionnaires. Are the results equivalent? BMC Med Res Methodol. 2007;7:50.PubMedPubMedCentralCrossRef
41.
42.
Hammarstrom A, Westerlund H, Kirves K, Nygren K, Virtanen P, Hagglof B. Addressing challenges of validity and internal consistency of mental health measures in a 27- year longitudinal cohort study - the northern Swedish cohort study. BMC Med Res Methodol. 2016;16:4.PubMedPubMedCentralCrossRef
43.
44.
Fraser A, Macdonald-Wallis C, Tilling K, Boyd A, Golding J, Davey Smith G, Henderson J, Macleod J, Molloy L, Ness A, et al. Cohort profile: the Avon longitudinal study of parents and children: ALSPAC mothers cohort. Int J Epidemiol. 2013;42(1):97–110.PubMedCrossRef
45.
Reichborn-Kjennerud T, Czajkowski N, Ystrom E, Orstavik R, Aggen SH, Tambs K, Torgersen S, Neale MC, Roysamb E, Krueger RF, et al. A longitudinal twin study of borderline and antisocial personality disorder traits in early to middle adulthood. Psychol Med. 2015;14:1–11.
46.
Eilertsen EM, Gjerde LC, Reichborn-Kjennerud T, Orstavik RE, Knudsen GP, Stoltenberg C, Czajkowski N, Roysamb E, Kendler KS, Ystrom E. Maternal alcohol use during pregnancy and offspring attention-deficit hyperactivity disorder (ADHD): a prospective sibling control study. Int J Epidemiol. 2017;46(5):1633–40.PubMedPubMedCentralCrossRef
47.
48.
R Core Team. R: a language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing; 2017.
49.
Nilsen W, Karevold E, Roysamb E, Gustayson K, Mathiesen KS. Social skills and depressive symptoms across adolescence: social support as a mediator in girls versus boys. J Adolesc. 2013;36(1):11–20.PubMedCrossRef
50.
Cummings SM, Savitz LA, Konrad TR. Reported response rates to mailed physician questionnaires. Health Serv Res. 2001;35(6):1347–55.PubMedPubMedCentral
51.
van Buuren S, Groothuis-Oudshoorn K. Mice: multivariate imputation by chained equations in R. J Stat Softw. 2011;45(3):1–67.CrossRef
52.
Muthén LK, Muthén BO: Regression analysis, exploratory factor analysis, confirmatory factor analysis, and structural equation modeling for categorical, censored, and count outcomes. http://​www.​statmodel.​com. 2009.
53.
Howe LD, Tilling K, Galobardes B, Lawlor DA. Loss to follow-up in cohort studies Bias in estimates of socioeconomic inequalities. Epidemiology. 2013;24(1):1–9.PubMedPubMedCentralCrossRef
54.
Greene N, Greenland S, Olsen J, Nohr EA. Estimating bias from loss to follow-up in the Danish National Birth Cohort. Epidemiology. 2011;22(6):815–22.PubMed
Metadata
Title
Preventing bias from selective non-response in population-based survey studies: findings from a Monte Carlo simulation study
Authors
Kristin Gustavson
Espen Røysamb
Ingrid Borren
Publication date
01-12-2019
Publisher
BioMed Central
Published in
BMC Medical Research Methodology / Issue 1/2019
Electronic ISSN: 1471-2288
DOI
https://doi.org/10.1186/s12874-019-0757-1

Other articles of this Issue 1/2019

BMC Medical Research Methodology 1/2019 Go to the issue

Research article

A comparison of methods to estimate the survivor average causal effect in the presence of missing data: a simulation study

Research article

Understanding health management and safety decisions using signal processing and machine learning

Debate

Self-reported and routinely collected electronic healthcare resource-use data for trial-based economic evaluations: the current state of play in England and considerations for the future

Research article

Health technology assessment (HTA) of optoelectronic biosensors for oncology by analytic hierarchy process (AHP) and Likert scale

Research article

Strategies for eliciting and synthesizing evidence for guidelines in rare diseases

Research article

A community-based intervention to increase participation in cervical cancer screening among immigrants in Norway