Skip to main content
Key Specialties
Cardiology Diabetology Endocrinology Gastroenterology Geriatrics and Gerontology Gynecology and Obstetrics Hematology Infectious Disease Internal Medicine Nephrology Neurology Oncology Pediatrics Respiratory Medicine Rheumatology Surgery
Congress Coverage
2024 ACC Congress 2023 ESMO Congress 2023 EASD Congress 2023 ERS Congress 2023 ESC Congress
Clinical Collections
Advances in Alzheimer’s

At a glance: The STEP trials

A round-up of the STEP phase 3 clinical trials evaluating semaglutide for weight loss in people with overweight or obesity.
ADVANCED SEARCH
Log in
Top
The European Journal of Health Economics
Published in: The European Journal of Health Economics 2/2011

01-04-2011 | Original Paper

Analysis of the validity of the vignette approach to correct for heterogeneity in reporting health system responsiveness

Authors: Nigel Rice, Silvana Robone, Peter Smith

Published in: The European Journal of Health Economics | Issue 2/2011

Login to get access

Abstract

Despite the growing popularity of the vignette methodology to deal with self-reported, categorical data, the formal evaluation of the validity of this methodology is still a topic of research. Some critical assumptions need to hold in order for this method to be valid. In this paper we analyse the assumption of “vignette equivalence” using data on health system responsiveness contained within the World Health Survey. We perform several tests to check the assumption of vignette equivalence. First, we use a test based on the global ordering of the vignettes. A minimal condition for the assumption of vignette equivalence to hold is that individual responses are consistent with the global ordering of vignettes. Secondly, using the hierarchical ordered probit model (HOPIT) model on the pool of countries, we undertake sensitivity analyses, stratifying countries according to the Inglehart–Welzel scale and the Human Development Index. The results of this analysis are robust, suggesting that the vignette equivalence assumption is not contradicted. Thirdly, we model the reporting behaviour of the respondents through a two-step regression procedure to evaluate whether the vignettes construct is perceived by respondents in different ways. Overall, across the analyses the results do not contradict the assumption of vignette equivalence and accordingly lend support to the use of the vignette methodology when analysing self-reported data and health system responsiveness.
Appendix
Available only for authorised users
Footnotes
1
Other studies focus on the assumption of response consistency when trying to assess the validity of the anchoring vignettes methodology, i.e. [6, 15, 17].
 
2
The long-form questionnaire uses two questions items per domain, while the short-form questionnaire uses only one. We use the eight items that are common to the long- and short-form questionnaire.
 
3
This map has been utilised to assess the validity of the vignette equivalence assumption also by Kristensen and Johansson [11].
 
4
“Self Secular” = Austria, Belgium, Denmark, Germany, Spain, Finland, France, Great Britain, Greece, Israel, Italy, Luxemburg, Netherlands, Slovenia, Sweden. “Self-Traditional” = Brazil, Dominican Republic, Ecuador, Guatemala, Ireland, Portugal, Uruguay. “Survival-Traditional” = United Arab Emirates, Burkina Faso, Bangladesh, Chad, Cote d’Ivoire, Congo, Comoros, Ethiopia, Ghana, India, Kenya, Lao, Sri Lanka, Malaysia, Mauritania, Malay, Morocco, Myanmar, Mauritius, Malawi, Namibia, Nepal, Pakistan, Philippines, Senegal, Swaziland, Tunisia, South Africa, Zambia, Zimbabwe. “Survival Secular” = Bosnia, China, Croatia, Czech Republic, Georgia, Hungary, Kazakhstan, Latvia, Russia, Slovakia, Ukraine, Vietnam.
 
5
For an example of consistent vignette ordering, consider Murray et al. [19], Fig. 30.3.
 
6
The average is computed assigning the same weight to each country within a group.
 
7
As an example “running a marathon” could be viewed as a multidimensional construct. Some individuals may view running a marathon as evidence of a high level of mobility and some as a result of exceptional talent. Others might consider it as an attribute related to health, whist others might as an attribute related to sport [19].
 
8
Perfect agreement of the rankings leads to a coefficient of 1, perfect disagreement −1, and independence 0.
 
9
We do not include in the analysis individuals who gave the same evaluation of all the vignettes (i.e. they judge all the vignettes as excellent responsiveness). Indeed, for these individuals it is not possible to compute the Spearman rank order correlation coefficient between their ranking and the global ordering ranking. However, we perform a robustness check including in the sample the observations about respondents who gave the same evaluation of all the vignettes. Referring to the domain “Confidentiality”, we perform the robustness check by just moving one vignette of one rank, in a consistent way with the global ordering. The results obtained including these observations are extremely similar to those not including them.
 
10
The average SROCCs have been computed assuming equal weight for each individual.
 
11
We exclude only Australia, Norway and Turkey since data on “Dignity” are not available for these countries.
 
12
See pp. 463–464 of Kapteyn et al. [5] for a formal description of the model estimated by the authors.
 
13
This set of vignettes is coded as Set A in the WHS. We are unable to perform our analysis on a pool of all the vignettes contained in the responsiveness module, since each set is evaluated by a different group of respondents.
 
14
The first vignette of the set (q7501) is assumed to be the base category.
 
15
The strategy adopted by STATA (the software we utilize for the empirical estimates) for identification in the ordered probit model is to set the constant term to zero. Therefore, we assume the coefficient of the base reference vignette-dummy to be equal to zero.
 
16
Australia, Turkey and Guatemala are excluded from the analysis since data on vignettes are not reported for all the domains considered.
 
17
The coefficient of variation of the number of alternative orderings is 14.35, while for the number of SROCCS that occur with a frequency greater than 1% it is 0.91.
 
18
For each domain, we have computed the median SROCC on the basis of tables analogous to Table 4.
 
19
We are not aware of any study that explicitly defines a threshold of acceptability for the rank order correlation coefficient above which we can assume that vignette equivalence holds. However, according to Murray et al. [19], a rank order correlation coefficient greater than 0.9 strongly corroborates the assumption of vignette equivalence.
 
20
Only the result related to the first cut point in the reporting bias equations is reported in Tables 8 and 9. Results related to the other cut points are available on request.
 
21
The results of the first and second step regression are available on request.
 
22
The results are not affected by the distribution of the gender of individuals across vignettes, since both women and men are represented in vignettes describing high and low levels of responsiveness.
 
Literature
1.
Murray, C., Frenk, J.: A framework for assessing the performance of health systems. Bull. World Health Org. 78, 717–731 (2000)
2.
Valentine, N., De Silva, A., Kawabata, K., Darby, C., Murray, C.J.L., Evans, D.: Health system responsiveness: concepts, domains and operationalization. In: Murray, C.J.L., Evans, D.B. (eds.) Health systems performance assessment: debates, methods and empiricism, pp. 573–596. World Health Organization, Geneva (2003)
3.
Salomon, J., Tandon, A., Murray, C.J.L., World Health Survey Pilot Study Collaborating Group: Comparability of self-rated health: cross sectional multi-country survey using anchoring vignettes. Brit. Med. J. 328(258), 258–261 (2004)CrossRef
4.
Bago d’Uva, T., van Doorlsaer, E., Lindeboom, M., O’Donnell, O.: Does reporting heterogeneity bias the measurement of health disparities? Health Econ. 17(3), 351–375 (2008)CrossRef
5.
Kapteyn, A., Salomon, J., van Soest, A.: Vignettes and self-reports of work disability in the US and the Netherlands. AER 97(1), 461–473 (2007)
6.
Van Soest, A., Delaney, A., Harmon, C., Kapteyn, A., Smith, J.P.: Validating the use of vignettes for subjective threshold scales. Discussion Paper, Tilburg University (2007)
7.
King, G., Murray, C.J.L., Salomon, J., Tandon, A.: Enhancing the validity and cross-cultural comparability of measurement in survey research. Am. Polit. Sci. Rev. 98(1), 184–191 (2004)CrossRef
8.
Rice, N., Robone, S., Smith, P.C.: Analysis of the validity of the vignette approach to correct for heterogeneity in reporting health system responsiveness. HEDG Working Paper, University of York (2009)
9.
Rice, N., Robone, S., Smith, P.C.: International comparison of public sector performance: the use of anchoring vignettes to adjust self-reported data. Evaluation 16(1), 81–101 (2010)
10.
Sirven, N., Santos-Eggimann, B., Spagnoli, J.: Comparability of health care responsiveness in Europe using anchoring vignettes from SHARE. IRDES working paper DT15 (2008)
11.
Kristensen, N., Johansson, E.: New evidence on cross-country differences in job satisfaction using anchoring vignettes. Labour Econ. 15, 96–117 (2008)CrossRef
12.
Hsee, C.K., Tang, J.N.: Sun and water: on a modulus-based measurement of happiness. Emotion 7, 213–218 (2007)CrossRef
13.
Javaras, K.N., Ripley, B.D.: An “unfolding” latent variable model for likert attitude data: drawing inferences adjusted for response style. JASA 102(478), 454–463 (2007)
14.
Grzymala-Busse, A.: Rebuilding Levithan: party competition and state exploitation in post-communist democracies. Cambridge University Press, New York (2007)CrossRef
15.
Bago d’Uva, T., Lindeboom, M., O’Donnell, O., van Doorslaer E.: Slipping anchor? Testing the vignettes approach to identification and correction of reporting heterogeneity. HEDG Working Paper, University of York (2009)
16.
Hopkins, D., King, G.: Improving anchoring vignettes: designing surveys to correct interpersonal incomparability. Public Opin. Q. (2010) (in press)
17.
Gupta N., Kristensen, N., Pozzoli, D.: External validation of the use of vignettes in cross-country health studies. IZA Discussion Paper (2008)
18.
Wand, J.: Credible comparisons using interpersonally incomparable data: ranking self-evaluations relative to anchoring vignettes or other common survey questions. Mimeo (2007)
19.
Murray, C.J.L., Ozaltin, E., Tandon, A.J., Salomon, J.: Empirical evaluation of the anchoring vignettes approach in health surveys. In: Murray, C.J.L., Evans, D.B. (eds.) Health systems performance assessment: debates, methods and empiricism, pp. 369–399. World Health Organization, Geneva (2003)
20.
King, G., Wand, J.: Comparing incomparable survey responses: new tools for anchoring vignettes. Polit. Anal. 15(1, Winter), 46–66 (2007)
21.
Kapteyn, A., Salomon J., van Soest, A.: Are Americans really less happy with their incomes? Rand Working Paper (2008)
22.
Üstün, T.B., Chatterji, S., Mechbal, A., Murray, C.: The world health surveys. In: Murray, C.J.L., Evans, D.B. (eds.) Health systems performance assessment: debates, methods and empiricism, pp. 762–796. World Health Organization, Geneva (2003)
23.
Valentine, N., Prasat, A., Rice, N., Robone, S., Chatterji, S.: Health systems responsiveness—a measure of the acceptability of health care processes and systems. In: Mossialos, E., Smith, P., Leatherman, S. (eds.) Performance measurement for health system improvement: experiences, challenges and prospects, pp. 256–305. WHO European Regional Office, London (2009)
24.
Ferguson, B.D., Tandon, A., Gakidou, E., Murray, C.J.L.: Estimating permanent income using indicator variables. In: Murray, C.J.L., Evans, D.B. (eds.) Health systems performance assessment: debates, methods and empiricism, pp. 748–760. World Health Organization, Geneva (2003)
25.
United Nation Development Programme: Human Development Report. New York (2006)
26.
27.
Tandon, A., Murray, C.J.L., Salomon, J.A., King, G.: Statistical models for enhancing cross-population comparability. In: Murray, C.J.L., Evans, D.B. (eds.) Health systems performance assessment: debates, methods and empiricism, pp. 727–746. World Health Organization, Geneva (2003)
28.
Terza, J.V.: Ordinal probit: a generalization. Commun. Stat. 14(1), 1–11 (1985)
29.
Lewis, J.B., Linzer, D.A.: Estimating regression models in which the dependent variable is based on estimates. Polit. Anal. 13, 345–364 (2005)CrossRef
30.
Jones, A.M., Rice, N., D’Uva, T.B., Balia, S.: Applied health economics. Routledge, New York (2007)
31.
Efron, B.: The Jackknife, the Bootstrap and other resampling plans. Society for Industrial and Applied Mathematics, Philadelphia (1982)
Metadata
Title
Analysis of the validity of the vignette approach to correct for heterogeneity in reporting health system responsiveness
Authors
Nigel Rice
Silvana Robone
Peter Smith
Publication date
01-04-2011
Publisher
Springer-Verlag
Published in
The European Journal of Health Economics / Issue 2/2011
Print ISSN: 1618-7598
Electronic ISSN: 1618-7601
DOI
https://doi.org/10.1007/s10198-010-0235-5

Other articles of this Issue 2/2011

The European Journal of Health Economics 2/2011 Go to the issue

Original Paper

Trust on doctor, social capital and medical care use of the elderly

Original Paper

Severity as an independent determinant of the social value of a health service

Original Paper

Technical efficiency in primary health care: does quality matter?

Editorial

Relative effectiveness and the European pharmaceutical market

Original Paper

The effect of alcoholic beverage excise tax on alcohol-attributable injury mortalities

Original Paper

Economic policies for healthier food intake: the impact on different household categories