Abstract
Data from a longitudinal occupational health survey of professional fire fighters were used to explore the potential impact of two types of methodological bias: sample selection and reactivity. No significant differences on demographic variables were observed between the group who first responded after a within-study change in survey administration format (Delayed) and respondents who had completed surveys since the study's inception (Initial). However, statistically significant differences in the study's 26 outcome measures provided some evidence that between-group differences did exist and that an “administration format” type of response bias was also potentially present. The effect sizes associated with the 37 observed significant differences ranged from small to medium. These results provide a context for a reexamination of standard techniques for the identification and interpretation of survey research biases. Methods are suggested to strengthen tests for selection bias and to minimize the impact of response biases.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
REFERENCES
American Psychological Association. (1974). Standards for educational and psychological tests and manuals. Washington D.C.: APA Press.
Beaton, R., Burr R., Nakagawa-Kogan, H., Osborne, O., & Thompson, E. (1978). Empirical inconsistencies of stress response indices: Some preliminary findings. Communicating Nursing Research, 11, 73-74.
Beaton, R., Egan, K., Nakagawa-Kogan, H., & Morrison, K. (1991). Self-reported symptoms of stress with temporomandibular disorders: Comparisons to healthy men and women. Journal of Prosthetic Dentistry, 65, 289-293.
Beaton, R., & Murphy, S. (1993). Sources of occupational stress among firefighters and paramedics and correlations with job-related outcomes. Prehospital and Disaster Medicine, 8, 140-150.
Beaton, R., Murphy, S., Johnson, L. C., Pike, K., & Corneil, W. (1998). Exposure to duty related incident stressors in urban firefighters and paramedics. Journal of Traumatic Stress, 11, 821-828.
Beaton, R., Murphy, S., Pike, K., & Jarret, M. (1995). Stress symptoms in firefighters and paramedics. In S. Sauter & S. Murphy (Eds.), Organizational risk factors for job stress (pp. 227-245). Washington, D.C.: APA Press.
Beaton, R., Nakagawa-Kogan, H., Hendershot, S., & Betrus, P. (1985). Psychological benefits of multimodal EMG biofeedback therapy for patients with musculoskeletal pain. Proceedings of the 16th Annual Meeting of the Biofeedback Society of America (pp. 14-17). New Orleans, LA: Biofeedback Society of America.
Berk, M., Mueller, C., & Throm, S. (1996). Can survey data be used to estimate physician practice costs. Evaluation and the Health Professions, 19, 14-29.
Berk, R. (1983). An introduction to sample selection bias in sociological data. American Sociological Review, 48, 386-398.
Block, J. (1965). The challenge of response sets: Unconfounding meaning, acquiescence, and social desirability in the MMPI. New York: Appleton-Century-Crofts.
Block, J. (1990). More remarks on social desirability. American Psychologist, 45, 1076-1077.
Braver, S., & Bay, R. C. (1992). Assessing and compensating for self-selection bias (non-representativeness) of the family research sample. Journal of Marriage and the Family, 54, 925-939.
Brown, M. (1994). What price response? Journal of the Market Research Society, 36(3), 227-244.
Campbell, D., & Stanley, J. (1966). Experimental and quasi-experimental designs for research. Chicago: Rand-McNally.
Carstensen, L., & Core, J. (1983). Social desirability and the measurement of psychologic well-being in elderly persons. Journal of Gerontology, 38, 713-715.
Catania, J. (1999). A framework for conceptualizing reporting bias and its antecedents in interviews assessing human sexuality. Journal of Sex Research, 35(1), 25-38.
Cochran, W., & Cox, G. (1957). Experimental designs (2nd ed.). New York: John Wiley & Sons.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum.
Collins, L. (1996). Is reliability obsolete? A commentary on "are simple gain scores obsolete?" Applied Psychological Measurement, 20(3), 289-292.
Dutka, S., & Frankel, L. (1997). Measuring response error. Journal of Advertising Research. January/February, 33-39.
Edwards, A. (1953). The relationship between the judged desirability of a trait and the probability that the trait will be endorsed. Journal of Applied Psychology, 37, 90-93.
Edwards, A. (1957). The social desirability variable in personality assessment and research. New York: Dryden.
Edwards, A. (1970). The measurement of personality traits by scales and inventories. New York: Holt, Rinehart & Winston.
Edwards, A. (1990). Construct validity and social desirability. American Psychologist, 45, 207-209.
Ellsworth, R. (1979). Does follow-up loss reflect poor outcome? Evaluation and the Health Professions, 2, 419-437.
Fern, E., & Monroe, K. (1996). Effect size estimates: Issues and problems in interpretation. Journal of Consumer Research, 23, 89-105.
Gibson, D. Hudes, E., & Donovan, D. (1999). Estimating and correcting for response bias in self-reported HIV risk behavior. Journal of Sex Research, 36(1), 96-101.
Gist, R., & Woodall, S. (1995). Occupational stress in contemporary fire service. In P. Orris, J. Melius, & R. Duffey (eds.) Occupational medicine: Firefighters' safety and health, 10(4), 763-788. Philadelphia: Hanley & Belfus.
Glass, G. (1976). Primary, secondary, and meta-analysis of research. Educational Researcher, 5, 3-8.
Humphreys, L. (1996). Linear dependence of gain scores on their components imposes constraints on their use and interpretation: A commentary on "are simple gain scores obsolete?" Applied Psychological Measurement, 20(3), 293-294.
John, U., Rumpf, H., & Hapke, U. (1999) Estimating prevalence of alcohol abuse and dependence in one general hospital: An approach to reduce sample selection bias. Alcohol and Alcoholism, 34(5), 786-794.
Keating, K. (1989). Self-selection: Are we beating a dead horse? Population and Program Planning, 12(2), 137-142.
McCraw, R., & Costa, P., Jr. (1983). Social desirability scales: More substance than style. Journal of Consulting and Clinical Psychology, 51, 881-888.
Mellenbergh, G. (1999). A note on simple gain score precision. Applied Psychological Measurement, 23(1), 87-89.
Nakagawa-Kogan, H., & Betrus, P. (1984). Self-management: A nursing mode of therapeutic influence. Advances in Nursing Science, 6, 55-73.
Neal, M., Carder, P., & Morgan, D. (1996). Use of public records to compare respondents and nonrespondents in a study of recent widows. Research on Aging, 18(2), 219-242.
Newcomer, R., Clay, T., Luxenberg, J., & Miller, R. (1999). Misclassification and selection bias when identifying Alzheimer's disease solely from Medicare claims records. Journal of the American Geriatrics Society, 47(2), 215-219.
Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory (3rd ed.). New York: McGraw-Hill.
Raykov, T. (1999). Are simple change scores obsolete? An approach to studying correlates and predictors of change. Applied Psychological Measurement, 23(2), 120-126.
Rosenthal, R. (1994). Parametric measures of effect size. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 231-244). New York: Russell Sage Foundation.
Rush, M., Phillips, J., & Panek, P. (1978). Subject recruitment bias: The paid volunteer subject. Perceptual & Motor Skills, 47(2), 443-449.
Spector, P., Fox, S., & Van Katwyk, P. (1999). The role of negative affectivity in employee reactions to job characteristics: Bias effect or substantive effect? Journal of Occupation and Organizational Psychology, 72(2), 205-218.
Stolzenberg, R., & Relles, D. (1997). Tools for intuition about sample selection bias and its correction. American Sociological Review, 62, 494-507.
Thompson, E., & Leckie, M. (1991). Therapeutic manual for stress response management. Unpublished manuscript, University of Washington, Seattle, WA.
Turner, H. (1999). Participation bias in AIDS related telephone surveys: Results from the National AIDS behavioral survey (NABS) non-response study. Journal of Sex Research, 35(1), 52-58.
Valdez, A., & Kaplan, C. (1999). Reducing selection bias in the use of focus groups to investigate hidden populations: The case of Mexican-American gang members from south Texas. Drugs and Society, 14(1-2), 209-224.
Walsh, J. A. (1990). Comment on social desirability. American Psychologist, 45, 289-290.
Watkins, D., & Cheung, S. (1995). Culture, gender, and response bias: An analysis of responses to the self-description questionnaire. Journal of Cross Cultural Psychology, 26(5), 490-504.
Watson, D., Clark, L., & Tellegen, A. (1988). Development and validation of brief measures of positive and negative affect: The PANAS scales. Journal of Personality and Social Psychology, 55, 128-139.
Wiederman, M. (1999). Volunteer bias in sexuality research using college student participants. Journal of Sex Research, 36(1), 59-66.
Williams, R., & Zimmerman, D. (1996). Are simple gain scores obsolete? Applied Psychological Measurement, 20(1), 59-69.
Wilson, M., Holman, P., & Hammock, A. (1966). A comprehensive review of the effects of worksite health promotion on health related outcomes. American Journal of Health Promotion, 10, 429-452.
Winship, C., & Mare, R. (1992). Models for sample selection bias. Annual Review of Sociology, 18, 327-350.
Rights and permissions
About this article
Cite this article
Johnson, L.C., Beaton, R., Murphy, S. et al. Sampling Bias and Other Methodological Threats to the Validity of Health Survey Research. International Journal of Stress Management 7, 247–267 (2000). https://doi.org/10.1023/A:1009589812697
Issue Date:
DOI: https://doi.org/10.1023/A:1009589812697