Skip to main content
Top
Published in: BMC Medical Research Methodology 1/2012

Open Access 01-12-2012 | Research article

Comparison of response rates and cost-effectiveness for a community-based survey: postal, internet and telephone modes with generic or personalised recruitment approaches

Authors: Martha Sinclair, Joanne O’Toole, Manori Malawaraarachchi, Karin Leder

Published in: BMC Medical Research Methodology | Issue 1/2012

Login to get access

Abstract

Background

Epidemiological research often requires collection of data from a representative sample of the community or recruitment of specific groups through broad community approaches. The population coverage of traditional survey methods such as mail-outs to residential addresses, and telephone contact via public directories or random-digit-dialing is declining and survey response rates are falling. There is a need to explore new sampling frames and consider multiple response modes including those offered by changes in telecommunications and internet technology.

Methods

We evaluated response rates and cost-effectiveness for three modes of survey administration (postal invitation/postal survey, postal invitation/internet survey and postal invitation/telephone survey) and two styles of contact approach (personalised and generic) in a community survey of greywater use. Potential respondents were contacted only once, with no follow up of non-responders.

Results

The telephone survey produced the highest adjusted response rate (30.2%), followed by the personalised postal survey (10.5%), generic postal survey (7.5%) and then the internet survey (4.7% for the personalised approach and 2.2% for the generic approach). There were some differences in household characteristics and greywater use rates between respondents to different survey modes, and between respondents to personalised and generic approaches. These may be attributable to the differing levels of motivations needed for a response, and varying levels of interest in the survey topic among greywater users and non-users. The generic postal survey had the lowest costs per valid survey received (Australian $22.93), followed by the personalised postal survey ($24.75).

Conclusions

Our findings suggest that postal surveys currently remain the most economic option for population-based studies, with similar costs for personalised and generic approaches. Internet surveys may be effective for specialised groups where email lists are available for initial contact, but barriers other than household internet access still exist for community-based surveys. Given the increasing recruitment challenges facing community-based studies, there is an imperative to gather contemporary comparative data on different survey modes and recruitment approaches in order to determine their strengths, limitations and costs. Researchers also need to document and report on the potential biases in the target and respondent populations and how this may affect the data collected.
Literature
3.
go back to reference Guterbock TM, Diop A, Ellis JM, Holmes JL, Le KT: Who needs RDD? Combining directory listings with cell phone exchanges for an alternative telephone sampling frame. Soc Sci Res. 2011, 40 (3): 860-872. 10.1016/j.ssresearch.2011.01.001.CrossRef Guterbock TM, Diop A, Ellis JM, Holmes JL, Le KT: Who needs RDD? Combining directory listings with cell phone exchanges for an alternative telephone sampling frame. Soc Sci Res. 2011, 40 (3): 860-872. 10.1016/j.ssresearch.2011.01.001.CrossRef
4.
go back to reference Galea S, Tracy M: Participation rates in epidemiologic studies. Ann Epidemiol. 2007, 17: 643-653. 10.1016/j.annepidem.2007.03.013.CrossRefPubMed Galea S, Tracy M: Participation rates in epidemiologic studies. Ann Epidemiol. 2007, 17: 643-653. 10.1016/j.annepidem.2007.03.013.CrossRefPubMed
5.
go back to reference Curtin R, Presser S, Singer E: Changes in Telephone Survey Nonresponse over the Past Quarter Century. Public Opin Q. 2005, 69 (1): 87-98. 10.1093/poq/nfi002.CrossRef Curtin R, Presser S, Singer E: Changes in Telephone Survey Nonresponse over the Past Quarter Century. Public Opin Q. 2005, 69 (1): 87-98. 10.1093/poq/nfi002.CrossRef
6.
go back to reference Ekman A, Litton J-E: New times, new needs; e-epidemiology. Eur J Epidemiol. 2007, 22: 285-292. 10.1007/s10654-007-9119-0.CrossRefPubMed Ekman A, Litton J-E: New times, new needs; e-epidemiology. Eur J Epidemiol. 2007, 22: 285-292. 10.1007/s10654-007-9119-0.CrossRefPubMed
7.
go back to reference Van Gelder MMHJ, Bretveld RW, Roeleveld N: Web-based questionnaires: the future in epidemiology?. Am J Epidemiol. 2010, 172 (11): 1292-1298. 10.1093/aje/kwq291.CrossRefPubMed Van Gelder MMHJ, Bretveld RW, Roeleveld N: Web-based questionnaires: the future in epidemiology?. Am J Epidemiol. 2010, 172 (11): 1292-1298. 10.1093/aje/kwq291.CrossRefPubMed
8.
go back to reference Link MW, Mokdad AH: Alternative modes for health surveillance surveys: an experiment with web, mail and telephone. Epidemiology. 2005, 16 (5): 701-704. 10.1097/01.ede.0000172138.67080.7f.CrossRefPubMed Link MW, Mokdad AH: Alternative modes for health surveillance surveys: an experiment with web, mail and telephone. Epidemiology. 2005, 16 (5): 701-704. 10.1097/01.ede.0000172138.67080.7f.CrossRefPubMed
9.
go back to reference Källmén H, Sinadinovic K, Berman AH, Wennberg P: Risky drinking of alcohol in Sweden: a randomized population survey comparing web- and paper-based self-reports. NAT Nordic Stud Alcohol and Drugs. 2011, 28 (2): 123-130. 10.2478/v10199-011-0013-4. Källmén H, Sinadinovic K, Berman AH, Wennberg P: Risky drinking of alcohol in Sweden: a randomized population survey comparing web- and paper-based self-reports. NAT Nordic Stud Alcohol and Drugs. 2011, 28 (2): 123-130. 10.2478/v10199-011-0013-4.
10.
go back to reference Eriksen L, Grønbæk M, Helge JW, Tolstrup JS, Curtis T: The Danish health examination survey 2007–2008 (DANHES 2007–2008). Scand J Public Health. 2011, 39 (2): 203-211. 10.1177/1403494810393557.CrossRefPubMed Eriksen L, Grønbæk M, Helge JW, Tolstrup JS, Curtis T: The Danish health examination survey 2007–2008 (DANHES 2007–2008). Scand J Public Health. 2011, 39 (2): 203-211. 10.1177/1403494810393557.CrossRefPubMed
11.
go back to reference Manfreda KL, Bosnjak M, Berzelak J, Haas I, Vehovar V: Web surveys versus other survey modes. A meta-analysis comparing response rates. Int J Market Res. 2008, 50 (1): 79-104. Manfreda KL, Bosnjak M, Berzelak J, Haas I, Vehovar V: Web surveys versus other survey modes. A meta-analysis comparing response rates. Int J Market Res. 2008, 50 (1): 79-104.
12.
go back to reference Shih TH, Xitao F: Comparing response rates from web and mail surveys: A meta-analysis. Field Methods. 2008, 20 (3): 249-271. 10.1177/1525822X08317085.CrossRef Shih TH, Xitao F: Comparing response rates from web and mail surveys: A meta-analysis. Field Methods. 2008, 20 (3): 249-271. 10.1177/1525822X08317085.CrossRef
13.
go back to reference Balter KA, Balter O, Fondell E, Lagerros YT: Web-based and mailed questionnaires: A comparison of response rates and compliance. Epidemiology. 2005, 16 (4): 577-579. 10.1097/01.ede.0000164553.16591.4b.CrossRefPubMed Balter KA, Balter O, Fondell E, Lagerros YT: Web-based and mailed questionnaires: A comparison of response rates and compliance. Epidemiology. 2005, 16 (4): 577-579. 10.1097/01.ede.0000164553.16591.4b.CrossRefPubMed
14.
go back to reference Ekman A, Dickman PW, sa Klint A, Weiderpass E, Litton JE: Feasibility of using web-based questionnaires in large population-based epidemiological studies. Eur J Epidemiol. 2006, 21: 103-111. 10.1007/s10654-005-6030-4.CrossRefPubMed Ekman A, Dickman PW, sa Klint A, Weiderpass E, Litton JE: Feasibility of using web-based questionnaires in large population-based epidemiological studies. Eur J Epidemiol. 2006, 21: 103-111. 10.1007/s10654-005-6030-4.CrossRefPubMed
15.
go back to reference Couper MP: The future of modes of data collection. Public Opin Q. 2011, 75 (5 SPEC): 889-908.CrossRef Couper MP: The future of modes of data collection. Public Opin Q. 2011, 75 (5 SPEC): 889-908.CrossRef
16.
go back to reference Blyth B: Mixed mode: The only ‘fitness’ regime?. Int J Market Res. 2008, 50 (2): 241-266. Blyth B: Mixed mode: The only ‘fitness’ regime?. Int J Market Res. 2008, 50 (2): 241-266.
17.
go back to reference Brogger J, Nystad W, Cappelen I, Bakke P: No increase in response rate by adding a web response option to a postal population survey: a randomized trial. J Med Internet Res. 2007, 9 (5): e40-10.2196/jmir.9.5.e40.CrossRefPubMedPubMedCentral Brogger J, Nystad W, Cappelen I, Bakke P: No increase in response rate by adding a web response option to a postal population survey: a randomized trial. J Med Internet Res. 2007, 9 (5): e40-10.2196/jmir.9.5.e40.CrossRefPubMedPubMedCentral
18.
go back to reference Turunen M, Paanala A, Villman J, Nevalainen A, Haverinen-Shaughnessy U: Evaluating housing quality, health and safety using an Internet-based data collection and response system: a cross-sectional study. Environ Health. 2010, 9: 69-10.1186/1476-069X-9-69.CrossRefPubMedPubMedCentral Turunen M, Paanala A, Villman J, Nevalainen A, Haverinen-Shaughnessy U: Evaluating housing quality, health and safety using an Internet-based data collection and response system: a cross-sectional study. Environ Health. 2010, 9: 69-10.1186/1476-069X-9-69.CrossRefPubMedPubMedCentral
19.
go back to reference Messer BL, Dillman DA: Surveying the general public over the internet using address-based sampling and mail contact procedures. Public Opin Q. 2011, 75 (3): 429-457. 10.1093/poq/nfr021.CrossRef Messer BL, Dillman DA: Surveying the general public over the internet using address-based sampling and mail contact procedures. Public Opin Q. 2011, 75 (3): 429-457. 10.1093/poq/nfr021.CrossRef
20.
go back to reference Dillman DA, Phelps G, Tortora R, Swift K, Kohrell J, Berck J, Messer BL: Response rate and measurement differences in mixed-mode surveys using mail, telephone, interactive voice response (IVR) and the Internet. Soc Sci Res. 2009, 38 (1): 1-18. 10.1016/j.ssresearch.2008.03.007.CrossRef Dillman DA, Phelps G, Tortora R, Swift K, Kohrell J, Berck J, Messer BL: Response rate and measurement differences in mixed-mode surveys using mail, telephone, interactive voice response (IVR) and the Internet. Soc Sci Res. 2009, 38 (1): 1-18. 10.1016/j.ssresearch.2008.03.007.CrossRef
21.
go back to reference Fan W, Yan Z: Factors affecting response rates of the web survey: a systematic review. Comput Hum Behav. 2010, 26: 132-139. 10.1016/j.chb.2009.10.015.CrossRef Fan W, Yan Z: Factors affecting response rates of the web survey: a systematic review. Comput Hum Behav. 2010, 26: 132-139. 10.1016/j.chb.2009.10.015.CrossRef
22.
go back to reference Edwards PJ, Roberts I, Clarke MJ, DiGuiseppi C, Wentz R, Kwan I, Cooper R, Felix LM, Pratap S: Methods to increase response to postal and electronic questionnaires. Cochrane Database Syst Rev. 2009, Issue 3: Art. No.:MR000008-10.1002/14651858.MR000008.pub4. Edwards PJ, Roberts I, Clarke MJ, DiGuiseppi C, Wentz R, Kwan I, Cooper R, Felix LM, Pratap S: Methods to increase response to postal and electronic questionnaires. Cochrane Database Syst Rev. 2009, Issue 3: Art. No.:MR000008-10.1002/14651858.MR000008.pub4.
23.
go back to reference Dillman DA, Lesser V, Mason R, Carlson J, Willits F, Robertson R, Burke B: Personalization of mail surveys for general public and populations with a group identity: results from nine studies. Rural Sociol. 2007, 72 (4): 632-646. 10.1526/003601107782638693.CrossRef Dillman DA, Lesser V, Mason R, Carlson J, Willits F, Robertson R, Burke B: Personalization of mail surveys for general public and populations with a group identity: results from nine studies. Rural Sociol. 2007, 72 (4): 632-646. 10.1526/003601107782638693.CrossRef
24.
go back to reference Heerwegh D: Effects of personal salutations in e-mail invitations to participate in a web survey. Public Opin Q. 2005, 69 (4): 588-598. 10.1093/poq/nfi053.CrossRef Heerwegh D: Effects of personal salutations in e-mail invitations to participate in a web survey. Public Opin Q. 2005, 69 (4): 588-598. 10.1093/poq/nfi053.CrossRef
25.
go back to reference Muñoz-Leiva F, Sánchez-Fernández J, Montoro-Ríos F, Ibáñez-Zapata JA: Improving the response rate and quality in Web-based surveys through the personalization and frequency of reminder mailings. Qual Quant. 2010, 44 (5): 1037-1052. 10.1007/s11135-009-9256-5.CrossRef Muñoz-Leiva F, Sánchez-Fernández J, Montoro-Ríos F, Ibáñez-Zapata JA: Improving the response rate and quality in Web-based surveys through the personalization and frequency of reminder mailings. Qual Quant. 2010, 44 (5): 1037-1052. 10.1007/s11135-009-9256-5.CrossRef
27.
go back to reference Gajic A, Cameron D, Hurley J: The cost-effectiveness of cash versus lottery incentives for a web-based, stated-preference community survey. Eur J Health Econ. 2011, 1-11. 10.1007/s10198-011-0332-0. Gajic A, Cameron D, Hurley J: The cost-effectiveness of cash versus lottery incentives for a web-based, stated-preference community survey. Eur J Health Econ. 2011, 1-11. 10.1007/s10198-011-0332-0.
28.
go back to reference Göritz AS: Incentives in web studies: methodological issues and a review. Int J Internet Sci. 2006, 1 (1): 58-70. Göritz AS: Incentives in web studies: methodological issues and a review. Int J Internet Sci. 2006, 1 (1): 58-70.
31.
go back to reference De Leeuw E, Callegaro M, Hox J, Korendijk E, Lensvelt-Mulders G: The influence of advance letters on response in telephone surveys. A meta-analysis. Public Opin Q. 2007, 71 (3): 413-443. 10.1093/poq/nfm014.CrossRef De Leeuw E, Callegaro M, Hox J, Korendijk E, Lensvelt-Mulders G: The influence of advance letters on response in telephone surveys. A meta-analysis. Public Opin Q. 2007, 71 (3): 413-443. 10.1093/poq/nfm014.CrossRef
32.
go back to reference McCluskey S, Topping AE: Increasing response rates to lifestyle surveys: a pragmatic evidence review. Perspect Public Health. 2011, 131 (2): 89-94. 10.1177/1757913910389423.CrossRefPubMed McCluskey S, Topping AE: Increasing response rates to lifestyle surveys: a pragmatic evidence review. Perspect Public Health. 2011, 131 (2): 89-94. 10.1177/1757913910389423.CrossRefPubMed
34.
go back to reference Mokdad AH: The behavioral risk factors surveillance system: Past, present, and future. Annu Rev Public Health. 2009, 30: 43-54. 10.1146/annurev.publhealth.031308.100226.CrossRefPubMed Mokdad AH: The behavioral risk factors surveillance system: Past, present, and future. Annu Rev Public Health. 2009, 30: 43-54. 10.1146/annurev.publhealth.031308.100226.CrossRefPubMed
35.
go back to reference Fenner Y, Garland SM, Moore EE, Jayasinghe Y, Fletcher A, Tabrizi SN, Gunasekaran B, Wark JD: Web-based recruiting for health research using a social networking site: an exploratory study. J Med Internet Res. 2012, 14 (1): e20-10.2196/jmir.1978.CrossRefPubMedPubMedCentral Fenner Y, Garland SM, Moore EE, Jayasinghe Y, Fletcher A, Tabrizi SN, Gunasekaran B, Wark JD: Web-based recruiting for health research using a social networking site: an exploratory study. J Med Internet Res. 2012, 14 (1): e20-10.2196/jmir.1978.CrossRefPubMedPubMedCentral
36.
go back to reference Ramo DE, Hall SM, Prochaska JJ: Reaching young adult smokers through the Internet: Comparison of three recruitment mechanisms. Nicotine Tob Res. 2010, 12 (7): 768-775. 10.1093/ntr/ntq086.CrossRefPubMedPubMedCentral Ramo DE, Hall SM, Prochaska JJ: Reaching young adult smokers through the Internet: Comparison of three recruitment mechanisms. Nicotine Tob Res. 2010, 12 (7): 768-775. 10.1093/ntr/ntq086.CrossRefPubMedPubMedCentral
37.
go back to reference Groves RM: Nonresponse rates and nonresponse bias in household surveys. Public Opin Q. 2006, 70 (5): 646-675. 10.1093/poq/nfl033.CrossRef Groves RM: Nonresponse rates and nonresponse bias in household surveys. Public Opin Q. 2006, 70 (5): 646-675. 10.1093/poq/nfl033.CrossRef
Metadata
Title
Comparison of response rates and cost-effectiveness for a community-based survey: postal, internet and telephone modes with generic or personalised recruitment approaches
Authors
Martha Sinclair
Joanne O’Toole
Manori Malawaraarachchi
Karin Leder
Publication date
01-12-2012
Publisher
BioMed Central
Published in
BMC Medical Research Methodology / Issue 1/2012
Electronic ISSN: 1471-2288
DOI
https://doi.org/10.1186/1471-2288-12-132

Other articles of this Issue 1/2012

BMC Medical Research Methodology 1/2012 Go to the issue