Skip to main content
Key Specialties
Cardiology Diabetology Endocrinology Gastroenterology Geriatrics and Gerontology Gynecology and Obstetrics Hematology Infectious Disease Internal Medicine Nephrology Neurology Oncology Pediatrics Respiratory Medicine Rheumatology Surgery
Congress Coverage
2024 ACC Congress 2023 ESMO Congress 2023 EASD Congress 2023 ERS Congress 2023 ESC Congress 2023 EHA Congress 2023 ASCO Annual Meeting
Clinical Collections
Advances in Alzheimer’s

At a glance: The ONWARDS insulin icodec trials

A round-up of the ONWARDS phase 3 clinical trials evaluating the novel weekly insulin icodec in people with diabetes.
ADVANCED SEARCH
Log in
Top
Quality of Life Research
Published in: Quality of Life Research 9/2019

Open Access 01-09-2019 | Commentary

Methods for questionnaire design: a taxonomy linking procedures to test goals

Authors: Paul Oosterveld, Harrie C. M. Vorst, Niels Smits

Published in: Quality of Life Research | Issue 9/2019

Login to get access

Abstract

Background

In the clinical field, the use of questionnaires is ubiquitous, and many different methods for constructing them are available. The reason for using a specific method is usually lacking, and a generally accepted classification of methods is not yet available. To guide test developers and users, this article presents a taxonomy for methods of questionnaire design which links the methods to the goal of a test.

Methods

The taxonomy assumes that construction methods are directed towards psychometric aspects. Four stages of test construction are distinguished to describe methods: concept analysis, item production, scale construction, and evaluation; the scale construction stage is used for identifying methods. It distinguishes six different methods: the rational method utilizes expert judgments to ensure face validity. The prototypical method uses prototypicality judgments to ensure process validity. In the internal method, item sets are selected that optimize homogeneity. The external method optimizes criterion validity by selecting items that best predict an external criterion. Under the construct method theoretical considerations are used to optimize construct validity. The facet method is aimed at optimizing content validity through a complete representation of the concept domain.

Conclusion

The taxonomy is comprehensive, constitutes a useful tool for describing procedures used in questionnaire design, and allows for setting up a test construction plan in which the priorities among psychometric aspects are made explicit.
Footnotes
1
Some scholars (e.g., [62, Chap. 6]) have warned against using principal components analysis for modeling questionnaire data as it would amount to a formative model, which does not allow for testing the hypothesis that the item responses are induced by an underlying common factor.
 
Literature
1.
Murphy, K. R., & Davidshofer, C. O. (2005). Psychological testing: Principles and applications. Upper Saddle River, NJ: Pearson Education.
2.
3.
4.
5.
6.
Anastasi, A. (1988). Psychological testing. New York: Macmillan.
7.
Cronbach, L. J. (1990). Essentials of psychological testing (5th ed.). New York: Harper Collins Publishers.
8.
Gregory, R. J. (2013). Psychological testing: History, principles, and applications. Boston, MA: Allyn & Bacon.
9.
Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory. New York: McGraw-Hill.
10.
Spector, P. E. (1992). Summated rating scale construction. Newbury Parks: Sage.CrossRef
11.
Irwing, P., Booth, T., & Hughes, D. J. (2018). The Wiley handbook of psychometric testing: A multidisciplinary reference on survey, scale and test development. Hoboken, NJ: Wiley.CrossRef
12.
American Psychological Association (2014). American Educational Research Association, and National Council on Measurement in Education.Standards for educational and psychological tests. Washington, DC: American Psychological Association.
13.
Fayers, P. M., & Machin, D. (2015). Quality of life: The assessment, analysis and reporting of patient-reported outcomes. New York: Wiley.CrossRef
14.
Johnson, C., Aaronson, N., Blazeby, J. M., Bottomley, A., Fayers, P., Koller, M., et al. (2011). Guidelines for developing questionnaire modules (4th ed.). Brussels: EORTC Quality of Life Group.
15.
Food and Drug Administration. (2009). Patient-reported outcome measures: Use in medical product development to support labeling claims. Guidance for industry, US Department of Health and Human Services.
16.
Hathaway, S. R., & McKinley, J. C. (1967). Minnesota multiphasic personality inventory revised manual. New York: The Psychological Corporation.
17.
National Institute of Neurological Disorders and Stroke. (2015). User manual for the quality of life in neurological disorders (Neuro-QOL) measures, version 2.0, March 2015. Technical report, National Institute of Neurological Disorders and Stroke (NINDS).
18.
Ware, J. E., Kosinski, M., Dewey, J. E., & Gandek, B. (2000). SF-36 health survey: Manual and interpretation guide. Lincoln, RI: Quality Metric Inc.
19.
Carlson, J. F., Geisinger, K. F., & Jonson, J. L. (2017). The twentieth mental measurement yearbook., Buros Center for Testing Lincoln, RI: University of Nebraska press.
20.
Oosterveld, P., & Vorst, H. C. M. (1996). Methoden van vragenlijstconstructie (methods of questionnaire construction). Nederlands Tijdschrift voor de Psychologie, 51, 11–27.
21.
Oosterveld, P., & Vorst, H.C.M. (1998). A taxonomy for questionnaire construction methods for quality of life assessment. Paper presented at the meeting of 5th annual conference of the International Society for Quality of Life Research. Baltimore, MD, USA.
22.
Burisch, M. (1978). Construction strategies for multiscale personality inventories. Applied Psychological Measurement, 2, 97–111.CrossRef
23.
Burisch, M. (1984). Approaches to personality inventory construction: A comparison of merits. American Psychologist, 39, 214–277.CrossRef
24.
Burisch, M. (1986). Methods of personality inventory development: A comparative analysis. In A. Angleiter & J. S. Wiggins (Eds.), Personality assessment via questionnaires: Current issues in theory and measurement (pp. 109–122). Berlin: Springer.CrossRef
25.
Hase, H. D., & Goldberg, L. R. (1967). Comparative validity of different strategies of constructing personality inventory scales. Psychological Bulletin, 67, 231–248.CrossRefPubMed
26.
Kelly, E. L. (1967). Assessment of human characteristics. Belmont, CA: Brooks Cole.
27.
Simms, L. J. (2008). Classical and modern methods of psychological scale construction. Social and Personality Psychology Compass, 2(1), 414–443.CrossRef
28.
29.
Wells, G. A., Russell, A. S., Haraoui, B., Bissonnette, R., & Ware, C. F. (2011). Validity of quality of life measurement tools-from generic to disease-specific. The Journal of Rheumatology Supplement, 88, 2–6.CrossRefPubMed
30.
Wiggins, J. S. (1973). Personality and prediction: principles of personality assessment. Reading, MA: Addison-Wesley.
31.
Wilde, G. J. S. (1977). Trait description and measurement by personality questionnaires. In R. B. Catell & R. M. Dreger (Eds.), Handbook of modern personality theory (pp. 69–103). Washington, DC: Hemisphere.
32.
Broughton, R. (1984). A prototype strategy for construction of personality scales. Journal of Personality and Social Psychology, 47, 1334–1346.CrossRef
33.
Hermans, H. M. (1969). The validity of different strategies of scale construction in predicting academic achievement. Educational and Psychological Measurement, 29, 877–883.CrossRef
34.
Broughton, R. (1990). The prototype concept in personality assessment. Canadian Psychology, 31, 26–37.CrossRef
35.
Oosterveld, P. (1996). Questionnaire design methods. PhD thesis, University of Amsterdam, Berkhout Nijmegen, NL.
36.
Crocker, L. M., & Algina, J. (1986). Introduction to classical and modern test theory. Holt, Rinehart and Winston, Orlando, FL.
37.
Irwing, P., & Hughes, D. J. (2018). Test development. In Irwing, P., Booth, T., Hughes, D. J. editors, The Wiley handbook of psychometric testing: A multidisciplinary reference on survey, scale and test development, pp. 1–47. Hoboken, NJ: Wiley.
38.
Cook, T. D., & Campbell, D. T. (1979). Quasi-experimentation: Design & analysis issues for field settings. Boston, MA: Houghton Mifflin.
39.
Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin.
40.
DuBois, P. H. (1970). A history of psychological testing. Boston, MA: Allyn & Bacon.
41.
Jackson, D. N. (1973). Structural personality assessment. In B. B. Wolman (Ed.), Handbook of general psychology (pp. 775–792). Englewood Cliffs, NJ: Prentice Hall.
42.
Francis, S. E., & Chorpita, B. F. (2010). Development and evaluation of the parental beliefs about anxiety questionnaire. Journal of Psychopathology and Behavioral Assessment, 32(1), 138–149.CrossRef
43.
Agorastos, A., Nash, W. P., Nunnink, S., Yurgil, K. A., Goldsmith, A., Litz, B. T., et al. (2013). The peritraumatic behavior questionnaire: Development and initial validation of a new measure for combat-related peritraumatic reactions. BMC Psychiatry, 13(1), 9.CrossRefPubMedPubMedCentral
44.
Vogt, D. S., King, D. W., & King, L. A. (2004). Focus groups in psychological assessment: Enhancing content validity by consulting members of the target population. Psychological Assessment, 16(3), 231.CrossRefPubMed
45.
46.
Buss, D. M., & Craik, K. H. (1983). The act frequency approach to personality. Psychological Review, 90, 105–126.CrossRef
47.
Buss, D. M., & Craik, K. H. (1985). Why not measure that trait? Alternative criteria for identifying important dispositions. Journal of Personality and Social Psychology, 48, 934–946.CrossRef
48.
Rosch, E. H. (1973). On the internal structure of perceptual and semantic categories. In T. E. Moore (Ed.), Cognitive development and the acquisition of language (pp. 111–144). New York: Academic Press.CrossRef
49.
Rosch, E. H. (1978). Principles of categorization. In E. H. Rosch & D. B. Lloyd (Eds.), Cognition and categorization (pp. 27–48). Hillsdale, NJ: Erlbaum.
50.
Kuncel, R. B., & Kuncel, N. R. (1995). Response-process models: Toward an integration of cognitive-processsing models, psychometric models, latent-trait theory, and self schemas. In P. E. Shrout & S. T. Fiske (Eds.), Personality research, methods, and theory: A festschrift honoring Donald W. Fiske (pp. 181–200). New York: Psychology Press.
51.
Fiske, D. W. (1991). Macropsychology and micropsychology: Natural categories and natural kinds. In R. E. Snow & D. E. Wiley (Eds.), Improving inquiry in social science: A volume in honor of Lee J. Cronbach (pp. 61–74). Hillsdale, NJ: Lawrence Erlbaum.
52.
De Jong, P. F. (1988). An application of the prototype scale construction strategy to the assessment of student motivation. Journal of Personality, 56, 487–508.CrossRef
53.
54.
55.
Amelang, M., Herboth, G., & Oefner, I. (1991). A prototype strategy for the construction of a creativity scale. European Journal of Personality, 5, 261–285.CrossRef
56.
Cattell, R. B. , Saunders, D. R., & Stice, G. (1957). Handbook for the sixteen personality factor questionnaire, the ‘16 P. F. test’ forms A, B, and C. Champaign, IL: IPAT.
57.
Comrey, A. L. (1988). Factor-analytic methods of scale development in personality and clinical psychology. Journal of Consulting and Clinical Psychology, 56, 754–761.CrossRefPubMed
58.
Thurstone, L. L. (1931). Multiple factor analysis. Psychological Review, 38, 406–427.CrossRef
59.
Costa, P. T., & McCrea, R. R. (1985). The NEO personality Inventory manual. Odessa, FL: Psychological Assessment Resources.
60.
Reeve, B. B., Hays, R. D., Bjorner, J. B., Cook, K. F., Crane, P. K., Teresi, J. A., et al. (2007). Psychometric evaluation and calibration of health-related quality of life item banks: Plans for the patient-reported outcomes measurement information system (PROMIS). Medical Care, 45, S22–31.CrossRefPubMed
61.
Briggs, S. R., & Cheek, J. M. (1986). The role of factor analysis in the development and evaluation of personality scales. Journal of Personality, 54, 106–148.CrossRef
62.
Markus, K. A., & Borsboom, D. (2013). Frontiers of test validity theory: Measurement, causation, and meaning. New York: Routledge.
63.
Comrey, A. L. (1961). Factored homogeneous item dimensions in personality research. Journal of Clinical Psychology, 34, 283–301.
64.
Comrey, A. L. (1978). Common methodological problems in factor analytic studies. Journal of Consulting and Clinical Psychology, 46, 648–659.CrossRef
65.
Lorr, M., & More, W. W. (1980). Four dimensions of assertiveness. Multivariate Behavioral Research, 15, 127–138.CrossRef
66.
Gorsuch, R. L. (1997). Exploratory factor analysis: Its rol in item analysis. Journal of Personality Assessment, 68(3), 532–560.CrossRefPubMed
67.
68.
Bollen, K. (1989). Structural equations with latent variables. New York: Wiley.CrossRef
69.
Browne, M. W. (2000). Cross-validation methods. Journal of Mathematical Psychology, 44(1), 108–132.CrossRefPubMed
70.
MacCallum, R. C., Roznowski, M., & Necowitz, L. B. (1992). Model modifications in covariance structure analysis: The problem of capitalization on chance. Psychological Bulletin, 111(3), 490–504.CrossRefPubMed
71.
72.
Hughes, D. J. (2018). Psychometric validity: Establishing the accuracy and appropriateness of psychometric measures. In P. Irwing, T. Booth, & D. J. Hughes (Eds.), The Wiley handbook of psychometric testing: A multidisciplinary reference on survey, scale and test development (pp. 751–779). Hoboken, NJ: Wiley.CrossRef
73.
McDonald, R. P. (1999). Test theory: A unified treatment. Mahwah, NJ: Lawrence Erlbaum.
74.
Meehl, P. E. (1945). The dynamics of “structured” personality tests. Journal of Clinical Psychology, 1(4), 296–303. Reprinted in D. N. Jackson and S. Messick (Eds.), (1967). Problems in human assessment (pp. 517–522).
75.
76.
Cronbach, L. J., & Gleser, G. C. (1965). Psychological tests and personnel decisions (2nd ed.). Urbana, IL: University of Illinois Press.
77.
Hand, D. J. (1987). Screening vs prevalence estimation. Journal of the Royal Statistical Society: Series C (Applied Statistics), 36(1), 1–7.
78.
Gough, H. G. (1969). California psychological inventory, revised manual. Palo Alto, CA: Consulting Psychologists.
79.
Kroenke, K., & Spitzer, R. L. (2002). The PHQ-9: A new depression diagnostic and severity measure. Psychiatric Annals, 32(9), 509–515.CrossRef
80.
Spitzer, R. L., Kroenke, K., Williams, J. B. W., & Löwe, B. (2006). A brief measure for assessing generalized anxiety disorder: The GAD-7. Archives of Internal Medicine, 166(10), 1092–1097.CrossRefPubMed
81.
Edwards, A. L. (1970). The measurement of personality traits by scales and inventories. New York: Holt, Rinehart, and Winston.
82.
Guttman, L. (1941). An outline of the statistical theory of prediction. Supplementary study B-1. In Subcommittee on Prediction of Social Adjustment, editor, The prediction of personal adjustment, pp. 253–318. New York: Social Science Research Council.
83.
Lord, F. M., & Novick, M. R. (1968). Statistical Theories of Mental Test Scores. Reading, MA: Addison-Wesley.
84.
85.
Travers, R. M. W. (1951). Rational hypotheses in the construction of tests. Educational and Psychological Measurement, 11(1), 128–137.CrossRef
86.
Fayers, P. M., & Hand, D. J. (1997). Factor analysis, causal indicators and quality of life. Quality of Life Research, 6(2), 139–150.CrossRefPubMed
87.
Edwards, J. R. (2011). The fallacy of formative measurement. Organizational Research Methods, 14(2), 370–388.CrossRef
88.
89.
Jackson, D. N. (1971). The dynamics of structured personality tests. Psychological Review, 78(3), 229.CrossRef
90.
Tellegen, A., Ben-Porath, Y. S., Sellbom, M., Arbisi, P. A., McNulty, J. L., & Graham, J. R. (2006). Further evidence on the validity of the MMPI-2 Restructured Clinical (RC) scales: Addressing questions raised by Rogers, Sewell, Harrison, and Jordan and Nichols. Journal of Personality Assessment, 87(2), 148–171.CrossRefPubMed
91.
Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52, 281–302.CrossRefPubMed
92.
Loevinger, J. (1957). Objective tests as instruments of psychological theory. Psychological Reports, 3(3), 635–694.CrossRef
93.
Trochim, W., Donnelly, J. P., & Arora, K. (2015). Research methods: The essential knowledge base. Boston, MA: Cengage Learning.
94.
Newton, P., & Shaw, S. (2014). Validity in educational & psychological assessment. London: Sage.CrossRef
95.
Jackson, D. N. (1984). Personality research form manual (3rd ed.). Port Huron, MI: Research Psychologists Press.
96.
Ettema, T. P., Droës, R. M., de Lange, J., Mellenbergh, G. J., & Ribbe, M. W. (2007). QUALIDEM: Development and evaluation of a dementia specific quality of life instrument: Scalability, reliability and internal structure. International Journal of Geriatric Psychiatry, 22, 549–556.CrossRefPubMed
97.
Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (pp. 13–103). Washington, DC: American Council on Education and National Council on Measurement in Education.
98.
Ettema, T. P., Droës, R. M., de Lange, J., Mellenbergh, G. J., & Ribbe, M. W. (2006). QUALIDEM: Development and evaluation of a dementia specific quality of life instrument: Validation. International Journal of Geriatric Psychiatry, 22, 424–430.CrossRef
99.
Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81–105.CrossRefPubMed
100.
Ziegler, M., Booth, T., & Bensch, D. (2013). Getting entangled in the nomological net: Thoughts on validity and conceptual overlap. European Journal of Psychological Assessment, 29(3), 157–161.CrossRef
101.
102.
Mellenbergh, G. J. (2011). A conceptual introduction to psychometrics: Development, analysis and application of psychological and educational tests. The Hague: Eleven International Publishing.
103.
104.
Guttman, L. (1954). An outline of some new methodology for social research. Public Opinion Quarterly, 18, 395–404.CrossRef
105.
Guttman, L. (1965). Introduction to facet design and analysis. In Proceedings of the 15th international congress of psychology, Amsterdam.
106.
107.
Mellenbergh, G. J., Kelderman, H., Stijlen, J. G., & Zondag, E. (1979). Linear models for the analysis and construction of instruments in a facet design. Psychological Bulletin, 86, 766–776.CrossRef
108.
Stouthard, M. E. A., Mellenbergh, G. J., & Hoogstraten, J. (1993). Assessment of dental anxiety: A facet approach. Anxiety, stress, and coping, 6, 89–105.CrossRef
109.
Landsheer, J. A., & Boeije, H. R. (2008). In search of content validity: Facet analysis as a qualitative method to improve questionnaire design. An application in health research. Quality & Quantity, 44, 59.CrossRef
110.
Borg, I. (1979). Some basic concepts of facet theory. In J. Lingoes, E. E. Roskam, & I. Borg (Eds.), Geometric representation of relational data. Ann Arbor, MI: Mathesis.
111.
Levy, S. (1985). Lawful roles of facets in social theories. In D. Canter (Ed.), Facet theory: Approaches to social research (pp. 59–96). New York: Springer.CrossRef
112.
Levy, S. (1990). The mapping sentence in cumulative theory construction: Well-being as an example. In J. J. Hox & J. De Jong-Gierveld (Eds.), Operationalization and research strategy (pp. 155–178). Amsterdam: Swets & Zeitlinger.
113.
Borg, I., & Shye, S. (1995). Facet theory: Form and content. Thousand Oaks, CA: Sage.
114.
Guttman, L. (1982). Facet theory, smallest space analysis, and factor analysis. Perceptual and Motor Skills, 54(2), 491–493.CrossRef
115.
Shye, S. (1998). Modern facet theory: Content design and measurement in behavioral research. European Journal of Psychological Assessment, 14(2), 160–171.CrossRef
116.
McDonald, R. P. (2003). Behavior domains in theory and in practice [measurement for the social sciences: Classical insights into modern approaches]. Alberta Journal of Educational Research, 49(3), 212.
117.
Maruyama, G., & Ryan, C. S. (2014). Research methods in social relations. Oxford: Wiley.
Metadata
Title
Methods for questionnaire design: a taxonomy linking procedures to test goals
Authors
Paul Oosterveld
Harrie C. M. Vorst
Niels Smits
Publication date
01-09-2019
Publisher
Springer International Publishing
Published in
Quality of Life Research / Issue 9/2019
Print ISSN: 0962-9343
Electronic ISSN: 1573-2649
DOI
https://doi.org/10.1007/s11136-019-02209-6

Other articles of this Issue 9/2019

Quality of Life Research 9/2019 Go to the issue

OriginalPaper

Socioeconomic inequalities in oral health-related quality of life in adolescents: a cohort study

OriginalPaper

Cross-cultural selection and validation of instruments to assess patient-reported outcomes in children and adolescents with achondroplasia

OriginalPaper

Response shift effects in patients’ assessments of their quality of life after cardiac rehabilitation

OriginalPaper

Association of the built environments and health-related quality of life in community-dwelling older adults: a cross-sectional study

OriginalPaper

Health utility and health-related quality of life of Japanese prostate cancer patients according to progression status measured using EQ-5D-5L and FACT-P

Review

Quality of life assessment scales in polio survivors: a scoping review