Skip to main content
Key Specialties
Cardiology Diabetology Endocrinology Gastroenterology Geriatrics and Gerontology Gynecology and Obstetrics Hematology Infectious Disease Internal Medicine Nephrology Neurology Oncology Pediatrics Respiratory Medicine Rheumatology Surgery
Congress Coverage
2024 ACC Congress 2023 ESMO Congress 2023 EASD Congress 2023 ERS Congress 2023 ESC Congress
Clinical Collections
Advances in Alzheimer’s

At a glance: The STEP trials

A round-up of the STEP phase 3 clinical trials evaluating semaglutide for weight loss in people with overweight or obesity.
ADVANCED SEARCH
Log in
Top
BMC Medical Research Methodology
Published in: BMC Medical Research Methodology 1/2019

Open Access 01-12-2019 | Public Health | Research article

Effectiveness of incentives and follow-up on increasing survey response rates and participation in field studies

Authors: Michael G. Smith, Maryam Witte, Sarah Rocha, Mathias Basner

Published in: BMC Medical Research Methodology | Issue 1/2019

Login to get access

Abstract

Background

Questionnaires are valuable data collection instruments in public health research, and can serve to pre-screen respondents for suitability in future studies. Survey non-response leads to reduced effective sample sizes and can decrease representativeness of the study population, so high response rates are needed to minimize the risk of bias. Here we present results on the success of different postal questionnaire strategies at effecting response, and the effectiveness of these strategies at recruiting participants for a field study on the effects of aircraft noise on sleep.

Methods

In total, we mailed 17 rounds of 240 questionnaires (total n = 4080) to randomly selected households around Atlanta International Airport. Different mailing rounds were varied in the length of the questionnaire (11, 26 or 55 questions), survey incentive (gift card or $2 cash), number of follow-up waves (0, 2 or 3), incentive for participating in a 5-night in-home sleep study ($100, $150 or $200), and address personalization.

Results

We received completed questionnaires from 407 respondents (response rate 11.4%). Personalizing the address, enclosing a $2 cash incentive with the initial questionnaire mailing and repeated follow-up mailings were effective at increasing response rate. Despite the increased expense of these approaches in terms of each household mailed, the higher response rates meant that they were more cost-effective overall for obtaining an equivalent number of responses. Interest in participating in the field study decreased with age, but was unaffected by the mailing strategies or cash incentives for field study participation. The likelihood that a respondent would participate in the field study was unaffected by survey incentive, survey length, number of follow-up waves, field study incentive, age or sex.

Conclusions

Pre-issued cash incentives and sending follow-up waves could maximize the representativeness and numbers of people from which to recruit, and may be an effective strategy for improving recruitment into field studies.
Appendix
Available only for authorised users
Literature
1.
Dillman DA, Smyth JD, Christian LM. Internet, phone, mail, and mixed-mode surveys : the tailored design method, 4th edition. Edn. Hoboken: Wiley; 2014.
2.
Teddlie C, Yu F. Mixed methods sampling a typology with examples. J Mixed Methods Res. 2007;1(1):77–100.CrossRef
3.
Singer E. Introduction - nonresponse bias in household surveys. Public Opin Q. 2006;70(5):637–45.CrossRef
4.
Baruch Y. Response rate in academic studies - a comparative analysis. Hum Relat. 1999;52(4):421–38.
5.
Groves RM. Nonresponse rates and nonresponse bias in household surveys. Public Opin Q. 2006;70(5):646–75.CrossRef
6.
Johnson TP, Wislar JS. Response rates and nonresponse errors in surveys. Jama. 2012;307(17):1805–6.PubMedCrossRef
7.
Groves RM, Peytcheva E. The impact of nonresponse rates on nonresponse bias - a meta-analysis. Public Opin Q. 2008;72(2):167–89.CrossRef
8.
9.
Cook C, Heath F, Thompson RL. A meta-analysis of response rates in web- or internet-based surveys. Educ Psychol Meas. 2000;60(6):821–36.CrossRef
10.
American Association for Public Opinion Research. Standard definitions: final dispositions of case codes and outcome rates for surveys. 9th ed. Lenexa: AAPOR; 2016.
11.
Czajka JL, Beyler A. Declining response rates in federal surveys: trends and implications. Final report - volume 1. In: Mathematica policy research; 2016.
12.
Armstrong JS, Overton TS. Estimating nonresponse Bias in mail surveys. J Mark Res. 1977;14(3):396–402.CrossRef
13.
Edwards PJ, Roberts I, Clarke MJ, DiGuiseppi C, Wentz R, Kwan I, Cooper R, Felix LM, Pratap S. Methods to increase response to postal and electronic questionnaires. Cochrane Database Syst Rev 2009;(3).
14.
Nakash RA, Hutton JL, Jorstad-Stein EC, Gates S, Lamb SE. Maximising response to postal questionnaires--a systematic review of randomised trials in health research. BMC Med Res Methodol. 2006;6:5.PubMedPubMedCentralCrossRef
15.
Knoll M, Soller L, Ben-Shoshan M, Harrington D, Fragapane J, Joseph L, La Vieille S, St-Pierre Y, Wilson K, Elliott S, et al. The use of incentives in vulnerable populations for a telephone survey: a randomized controlled trial. BMC Res Notes. 2012;5:572.PubMedPubMedCentralCrossRef
16.
Basner M, Dinges DF. Sleep duration in the United States 2003–2016: first signs of success in the fight against sleep deficiency? Sleep 2018; 41(4).
17.
18.
Watson NF, Badr MS, Belenky G, Bliwise DL, Buxton OM, Buysse D, Dinges DF, Gangwisch J, Grandner MA, Kushida C, et al. Joint consensus statement of the American Academy of sleep medicine and Sleep Research Society on the recommended amount of sleep for a healthy adult: methodology and discussion. Sleep. 2015;38(8):1161–83.PubMedPubMedCentralCrossRef
19.
World Health Organization. Burden of disease from environmental noise. Quantification of healthy life years lost in Europe. In: WHO Regional Office for Europe. Copenhagen: World Health Organization; 2011: 101.
20.
Miedema HM, Vos H. Associations between self-reported sleep disturbance and environmental noise based on reanalyses of pooled data from 24 studies. Behav Sleep Med. 2007;5(1):1–20.PubMedCrossRef
21.
Iber C, Ancoli-Israel S, Chesson A, Quan SF. The AASM manual for the scoring of sleep and associated events; rules, terminology and technical specifications. 1st ed. Westchester: American Academy of Sleep Medicine; 2007.
22.
Winser MA, McBean AL, Montgomery-Downs HE. Minimum duration of actigraphy-defined nocturnal awakenings necessary for morning recall. Sleep Med. 2013;14(7):688–91.PubMedCrossRef
23.
24.
McGuire S, Witte M, Kallarackal A, Basner M. Pilot study examining the effects of aircraft noise on sleep in communities near Philadelphia international airport. Sleep. 2017;40:A60–1.CrossRef
25.
26.
Buysse DJ, Reynolds CF 3rd, Monk TH, Berman SR, Kupfer DJ. The Pittsburgh sleep quality index: a new instrument for psychiatric practice and research. Psychiatry Res. 1989;28(2):193–213.PubMedCrossRef
27.
Weinstein ND. Individual differences in reactions to noise: a longitudinal study in a college dormitory. J Appl Psychol. 1978;63(4):458–66.PubMedCrossRef
28.
Fields JM, De Jong RG, Gjestland T, Flindell IH, Job RFS, Kurra S, Lercher P, Vallet M, Yano T, Guski R, et al. Standardized general-purpose noise reaction questions for community noise surveys: research and a recommendation. J Sound Vib. 2001;242(4):641–79.CrossRef
29.
Scott P, Edwards P. Personally addressed hand-signed letters increase questionnaire response: a meta-analysis of randomised controlled trials. BMC Health Serv Res. 2006;6:11.CrossRef
30.
Sahlqvist S, Song Y, Bull F, Adams E, Preston J, Ogilvie D, Consortium I. Effect of questionnaire length, personalisation and reminder type on response rate to a complex postal survey: randomised controlled trial. BMC Med Res Methodol. 2011;11:62.PubMedPubMedCentralCrossRef
31.
Möller-Leimkühler AM, Möller H-J, Maier W, Gaebel W, Falkai P. EPA guidance on improving the image of psychiatry. Eur Arch Psychiatry Clin Neurosci. 2016;266(2):139–54.PubMedCrossRef
32.
Halpern SD, Ubel PA, Berlin JA, Asch DA. Randomized trial of $5 versus $10 monetary incentives, envelope size, and candy to increase physician response rates to mailed questionnaires. Med Care. 2002;40(9):834–9.PubMedCrossRef
33.
Guski R, Schreckenberg D, Schuemer R. WHO environmental noise guidelines for the European Region: a systematic review on environmental noise and annoyance. Int J Environ Res Public Health. 2017;14(12):E1539.PubMedCrossRef
34.
Sinclair M, O'Toole J, Malawaraarachchi M, Leder K. Comparison of response rates and cost-effectiveness for a community-based survey: postal, internet and telephone modes with generic or personalised recruitment approaches. BMC Med Res Methodol. 2012;12:132.PubMedPubMedCentralCrossRef
35.
Singer E, Van Hoewyk J, Neugebauer RJ. Attitudes and behavior - the impact of privacy and confidentiality concerns on participation in the 2000 census. Public Opin Q. 2003;67(3):368–84.CrossRef
36.
Bowling A. Mode of questionnaire administration can have serious effects on data quality. J Public Health-Uk. 2005;27(3):281–91.CrossRef
37.
Groves RM, Singer E, Corning A. Leverage-saliency theory of survey participation - description and an illustration. Public Opin Q. 2000;64(3):299–308.PubMedCrossRef
38.
Ryan C. Language use in the United States: 2011. In.: U.S. Department of Commerce. Economics and Statistics Administration. U.S. Census Bureau. 2013.
39.
Brick JM, Montaquila JM, Han DF, Williams D. Improving response rates for Spanish speakers in two-phase mail surveys. Public Opin Q. 2012;76(4):721–32.CrossRef
40.
Guo YM, Kopec JA, Cibere J, Li LC, Goldsmith CH. Population survey features and response rates: a randomized experiment. Am J Public Health. 2016;106(8):1422–6.PubMedPubMedCentralCrossRef
41.
Ryan JM, Corry JR, Attewell R, Smithson MJ. A comparison of an electronic version of the SF-36 general health questionnaire to the standard paper version. Qual Life Res. 2002;11(1):19–26.PubMedCrossRef
42.
Patrick ME, Couper MP, Laetz VB, Schulenberg JE, O'Malley PM, Johnston LD, Miech RA. A sequential mixed-mode experiment in the U.S. National Monitoring the future study. J Surv Stat Methodol. 2018;6(1):72–97.PubMedCrossRef
43.
Medway RL, Fulton J. When more gets you less: a meta-analysis of the effect of concurrent web options on mail survey response rates. Public Opin Q. 2012;76(4):733–46.CrossRef
44.
McMaster HS, LeardMann CA, Speigle S, Dillman DA, Tea MCFS. An experimental comparison of web-push vs. paper-only survey procedures for conducting an in-depth health survey of military spouses. BMC Med Res Methodol. 2017;17:73.PubMedPubMedCentralCrossRef
45.
Halbesleben JRB, Whitman MV. Evaluating survey quality in health services research: a decision framework for assessing nonresponse Bias. Health Serv Res. 2013;48(3):913–30.PubMedCrossRef
46.
47.
Funkhouser E, Vellala K, Baltuck C, Cacciato R, Durand E, McEdward D, Sowell E, Theisen SE, Gilbert GH, Grp NDPC. Survey methods to optimize response rate in the National Dental Practice-Based Research Network. Eval Health Prof. 2017;40(3):332–58.PubMedCrossRef
48.
Lin IF, Schaeffer NC. Using survey participants to estimate the impact of nonparticipation. Public Opin Q. 1995;59(2):236–58.CrossRef
49.
Singer E, Ye C. The use and effects of incentives in surveys. Ann Am Acad Polit Soc Sci. 2013;645(1):112–41.CrossRef
50.
Curtin R, Singer E, Presser S. Incentives in random digit dial telephone surveys: a replication and extension. J Off Stat. 2007;23(1):91–105.
51.
Church AH. Estimating the effect of incentives on mail survey response rates - a Metaanalysis. Public Opin Q. 1993;57(1):62–79.CrossRef
52.
Edwards P, Roberts I, Clarke M, DiGuiseppi C, Pratap S, Wentz R, Kwan I. Increasing response rates to postal questionnaires: systematic review. Br Med J. 2002;324(7347):1183–5.CrossRef
53.
Mercer A, Caporaso A, Cantor D, Townsend R. How much gets you how much? Monetary incentives and response rates in household surveys. Public Opin Q. 2015;79(1):105–29.CrossRef
54.
Han DF, Montaquila JM, Brick JM. An evaluation of incentive experiments in a two-phase address-based sample mail survey. Surv Res Methods-Ger. 2013;7(3):207–18.
55.
Weil J, Mendoza AN, McGavin E. Recruiting older adults as participants in applied social research: applying and evaluating approaches from clinical studies. Educ Gerontol. 2017;43(12):662–73.
56.
Sharp EC, Pelletier LG, Levesque C. The double-edged sword of rewards for participation in psychology experiments. Can J Behav Sci. 2006;38(3):269–77.CrossRef
57.
Mayfield A, Amaya A, Carris K. A matter of time: the value and optimal timing of follow-up questionnaire mailings in a multimode survey. In: 68th Annual Conference of the American Association for Public Opinion Research Boston, Boston: American Statistical Association; 2013.
58.
Voigt LF, Koepsell TD, Daling JR. Characteristics of telephone survey respondents according to willingness to participate. Am J Epidemiol. 2003;157(1):66–73.PubMedCrossRef
59.
Basner M, McGuire S. WHO environmental noise guidelines for the European Region: a systematic review on environmental noise and effects on sleep. Int J Environ Res Public Health. 2018;15(3):E519.PubMedCrossRef
60.
Saleh A, Bista K. Examining factors impacting online survey response rates in educational research: perceptions of graduate students. J Multi Discip Eval. 2017;13(29):63–74.
Metadata
Title
Effectiveness of incentives and follow-up on increasing survey response rates and participation in field studies
Authors
Michael G. Smith
Maryam Witte
Sarah Rocha
Mathias Basner
Publication date
01-12-2019
Publisher
BioMed Central
Keyword
Public Health
Published in
BMC Medical Research Methodology / Issue 1/2019
Electronic ISSN: 1471-2288
DOI
https://doi.org/10.1186/s12874-019-0868-8

Other articles of this Issue 1/2019

BMC Medical Research Methodology 1/2019 Go to the issue

Research article

Tools used to assess the quality of peer review reports: a methodological systematic review

Research article

Spurious interaction as a result of categorization

Research article

Estimating the loss of lifetime function using flexible parametric relative survival models

Research article

Preventing bias from selective non-response in population-based survey studies: findings from a Monte Carlo simulation study

Research article

Chronic Headache Education and Self-management Study (CHESS) – a mixed method feasibility study to inform the design of a randomised controlled trial

Research article

Using marginal standardisation to estimate relative risk without dichotomising continuous outcomes