Top

Health Services and Outcomes Research Methodology

28-11-2023

Evaluation of survey delivery methods in a national study of Veteran’s healthcare preferences

Authors: Natalie Disher, Jennifer Scott, Anna Tyzik, Sara Golden, Georgia Baker, Denise M. Hynes, Christopher G. Slatore

Published in: Health Services and Outcomes Research Methodology

Abstract

Researchers often use survey methods to elicit patient perspectives on their healthcare. Survey results are often subject to response bias and missing data. As part of an observational study of Veterans’ healthcare preferences, we conducted a national survey of Veterans receiving care in the Veterans Affairs healthcare system. We describe two recruitment strategies offering concurrent online, paper, and phone survey options to different sets of participants and examine response and missingness patterns between recruitment methods and modalities. In Strategy 1 we sent recruitment letters presenting options to complete our survey online or on paper. If patients indicated that they wanted to complete a paper survey, we mailed them a paper survey. In Strategy 2 we sent recruitment letters with paper surveys and included the option to complete the survey online. We compared response rates, characteristics, and missingness for the strategies and survey modalities. We sent 4399 initial letters using Strategy 1 and 8148 initial letters using Strategy 2 with response rates 7.7% and 13.2%, respectively; 70.6% of respondents completed paper surveys. Across both strategies, paper survey respondents were older and had lower educational attainment. There were significantly more paper surveys missing greater than 2% of items than online surveys (OR 6.3, 95% CI [4.8, 8.1]). Our findings suggest tradeoffs associated with survey modality and recruitment strategies. Mixed-modality recruitment may increase response rates and decrease missing data and response bias. Researchers should consider their target population when choosing survey modalities given differing characteristics between paper and survey respondents.

Available only for authorised users

American Association for Public Opinion Research: Standard Definitions. AAPOR, Alexandria (2016)

Brown, E.M., Olson, L.T., Farrelly, M.C., Nonnemaker, J.M., Battles, H., Hampton, J.: Comparing response rates, costs, and tobacco-related outcomes across phone, mail, and online surveys. Surv. Pract. 11, 1–18 (2018). https://doi.org/10.29115/SP-2018-0029CrossRef

Callas, P.W., Solomon, L.J., Hughes, J.R., Livingston, A.E.: The influence of response mode on study results: offering cigarette smokers a choice of postal or online completion of a survey. J. Med. Internet Res. 12, e46 (2010). https://doi.org/10.2196/jmir.1414CrossRefPubMedPubMedCentral

Christensen, A.I., Ekholm, O., Kristensen, P.L., Larsen, F.B., Vinding, A.L., Glümer, C., Juel, K.: The effect of multiple reminders on response patterns in a Danish health survey. Eur. J. Public Health 25, 156–161 (2015). https://doi.org/10.1093/eurpub/cku057CrossRefPubMed

Degner, L.F., Sloan, J.A., Venkatesh, P.: The control preferences scale. Can. J. Nurs. Res. Rev. Can. Rech. Sci. Infirm. 29, 21–43 (1997)

Dillman, D.A.: The promise and challenge of pushing respondents to the Web in mixed-mode surveys. Surv. Methodol. 43, 3–30 (2017)

Dolnicar, S., Laesser, C., Matus, K.: Online versus paper: format effects in tourism surveys. J. Travel Res. 47, 295–316 (2009). https://doi.org/10.1177/0047287508326506CrossRef

Ebert, J.F., Huibers, L., Christensen, B., Christensen, M.B.: Paper- or web-based questionnaire invitations as a method for data collection: cross-sectional comparative study of differences in response rate, completeness of data, and financial cost. J. Med. Internet Res. 20, e24 (2018). https://doi.org/10.2196/jmir.8353CrossRefPubMedPubMedCentral

Fowler, F.J., Cosenza, C., Cripps, L.A., Edgman-Levitan, S., Cleary, P.D.: The effect of administration mode on CAHPS survey response rates and results: a comparison of mail and web-based approaches. Health Serv. Res. 54, 714–721 (2019). https://doi.org/10.1111/1475-6773.13109CrossRefPubMedPubMedCentral

Greenlaw, C., Brown-Welty, S.: A comparison of web-based and paper-based survey methods: testing assumptions of survey mode and response cost. Eval. Rev. 33, 464–480 (2009). https://doi.org/10.1177/0193841X09340214CrossRefPubMed

Guo, Y., Kopec, J.A., Cibere, J., Li, L.C., Goldsmith, C.H.: Population survey features and response rates: a randomized experiment. Am. J. Public Health 106, 1422–1426 (2016). https://doi.org/10.2105/AJPH.2016.303198CrossRefPubMedPubMedCentral

Harris, P.A., Taylor, R., Thielke, R., Payne, J., Gonzalez, N., Conde, J.G.: Research electronic data capture (REDCap)—a metadata-driven methodology and workflow process for providing translational research informatics support. J. Biomed. Inform. 42, 377–381 (2009). https://doi.org/10.1016/j.jbi.2008.08.010CrossRefPubMed

Janssen, E.M., Hauber, A.B., Bridges, J.F.P.: Conducting a Discrete-choice experiment study following recommendations for good research practices: an application for eliciting patient preferences for diabetes treatments. Value Health 21, 59–68 (2018). https://doi.org/10.1016/j.jval.2017.07.001CrossRefPubMed

Karahalios, A., Baglietto, L., Carlin, J.B., English, D.R., Simpson, J.A.: A review of the reporting and handling of missing data in cohort studies with repeated assessment of exposure measures. BMC Med. Res. Methodol. 12, 96 (2012). https://doi.org/10.1186/1471-2288-12-96CrossRefPubMedPubMedCentral

Kelfve, S., Kivi, M., Johansson, B., Lindwall, M.: Going web or staying paper? The use of web-surveys among older people. BMC Med. Res. Methodol. 20, 252 (2020). https://doi.org/10.1186/s12874-020-01138-0CrossRefPubMedPubMedCentral

Kongsved, S.M., Basnov, M., Holm-Christensen, K., Hjollund, N.H.: Response rate and completeness of questionnaires: a randomized study of internet versus paper-and-pencil versions. J. Med. Internet Res. 9, e25 (2007). https://doi.org/10.2196/jmir.9.3.e25CrossRefPubMedPubMedCentral

Link, M.W., Mokdad, A.H.: Alternative modes for health surveillance surveys: an experiment with web, mail, and telephone. Epidemiology 16, 701–704 (2005). https://doi.org/10.1097/01.ede.0000172138.67080.7fCrossRefPubMed

McMaster, H.S., LeardMann, C.A., Speigle, S., Dillman, D.A.: An experimental comparison of web-push vs. paper-only survey procedures for conducting an in-depth health survey of military spouses. BMC Med. Res. Methodol. 17, 73 (2017). https://doi.org/10.1186/s12874-017-0337-1CrossRefPubMedPubMedCentral

Mlikotic, R., Parker, B., Rajapakshe, R.: Assessing the effects of participant preference and demographics in the usage of web-based survey questionnaires by women attending screening mammography in British Columbia. J. Med. Internet Res. 18, e70 (2016). https://doi.org/10.2196/jmir.5068CrossRefPubMedPubMedCentral

Patrick, M.E., Couper, M.P., Parks, M.J., Laetz, V., Schulenberg, J.E.: Comparison of a web-push survey research protocol with a mailed paper and pencil protocol in the monitoring the future panel survey. Addiction 116, 191–199 (2021). https://doi.org/10.1111/add.15158CrossRefPubMed

Pyne, J.M., Kelly, P.A., Fischer, E.P., Miller, C.J., Wright, P., Zamora, K., Koenig, C.J., Stanley, R., Seal, K., Burgess, J.F., Fortney, J.C.: Development of the perceived access inventory: a patient-centered measure of access to mental health care. Psychol. Serv. 17, 13–24 (2020). https://doi.org/10.1037/ser0000235CrossRefPubMed

RAND Corporation: A product of the CMS alliance to modernize healthcare federally funded research and development center centers for medicare and medicaid services (CMS) (2015)

Rose, A., Peters, N., Shea, J.A., Armstrong, K.: Development and testing of the health care system distrust scale. J. Gen. Internal Med. 19, 57–63 (2004). https://doi.org/10.1111/j.1525-1497.2004.21146.xCrossRef

Rübsamen, N., Akmatov, M.K., Castell, S., Karch, A., Mikolajczyk, R.T.: Comparison of response patterns in different survey designs: a longitudinal panel with mixed-mode and online-only design. Emerg. Themes Epidemiol. 14, 4 (2017). https://doi.org/10.1186/s12982-017-0058-2CrossRefPubMedPubMedCentral

Scherer, L.D., Caverly, T.J., Burke, J., Zikmund-Fisher, B.J., Kullgren, J.T., Steinley, D., McCarthy, D.M., Roney, M., Fagerlin, A.: Development of the medical maximizer-minimizer scale. Health Psychol. 35, 1276–1287 (2016). https://doi.org/10.1037/hea0000417CrossRefPubMed

Sjetne, I.S., Iversen, H.H., Holmboe, O., Helgeland, J.: Response probabilities and response-mode preferences in a self-administered survey. BMC. Res. Notes 12, 289 (2019). https://doi.org/10.1186/s13104-019-4328-7CrossRefPubMedPubMedCentral

US Department of Veterans Affairs: Survey of Healthcare Experiences of Patients (2015)

VA Information Resource Center (VIReC): VHA Corporate Data Warehouse (CDW) (2022a)

VA Information Resource Center (VIReC): Community Care for Veterans (2022b)

Zuidgeest, M., Hendriks, M., Koopman, L., Spreeuwenberg, P., Rademakers, J.: A comparison of a postal survey and mixed-mode survey using a questionnaire on patients’ experiences with breast care. J. Med. Internet Res. 13, e68 (2011). https://doi.org/10.2196/jmir.1241CrossRefPubMedPubMedCentral

Title: Evaluation of survey delivery methods in a national study of Veteran’s healthcare preferences
Authors: Natalie Disher
Jennifer Scott
Anna Tyzik
Sara Golden
Georgia Baker
Denise M. Hynes
Christopher G. Slatore
Publication date: 28-11-2023
Publisher: Springer US
Published in: Health Services and Outcomes Research Methodology
Print ISSN: 1387-3741
Electronic ISSN: 1572-9400
DOI: https://doi.org/10.1007/s10742-023-00320-3

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Evaluation of survey delivery methods in a national study of Veteran’s healthcare preferences

Abstract

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Abstract

Please log in to get access to this content