Preference Testing

Abstract

Preference testing refers to consumer tests in which the consumer is given a choice and asked to indicate their most liked product, usually from a pair. Although these tests appear straightforward and simple, several complications are encountered in the methods, notably how to treat replicated data and how to analyze data that include a “no-preference” option as a response. Additional methods are discussed including ranking more than two products, choosing both the best and worst from a group, and rating the degree of preference.

The number of judges that are involved in a study may be such that rather unimportant differences may receive undue attention. It is quite possible to produce statistically significant differences in preference for product which have little practical value by simple increasing the number o judges that are utilized.

—H. G. Schutz (1971).

13.1 Appendix 1: Worked Example of the Ferris k-Visit Repeated Preference Test Including the No-Preference Option

A consumer test with 900 respondents is completed with the following results.

Questions: (1) Is there a significantly higher preference for product A or product B? (2) Is the preference for the winning product higher than 45%? (Example from Ferris (1958) and Bi (2006), pp. 72–76).

Table 6

(N = 900)

Here are the basic equations we need:

• N _y = N _ao + N _oa + N _bo + N _ob

• N _x = N _ab + N _ba

• M = N–N _aa–N _bb

$$p = \frac{{M - \sqrt {\left[ {M^2 - (N_{{\textrm{oo}}} + N_y /2)(2N_x + N_y )} \right]} }}{{2N_{{\textrm{oo}}} + N_y }}$$

So for this data set:

M = 100 (all those not showing AA or BB behavior, i.e., a consistent choice for one of the two products)

• N _x = 8 + 14 = 22

• N _y = 14 + 12 + 17 + 11 = 54

• p = 0.257

Now we need the equations for the best estimates of each segment/proportion:

$$\pi _{\textrm{A}} = \frac{{\left[ {N_{{\textrm{aa}}} (1 - p^2 )} \right] - \left[ {(N - N_{{\textrm{bb}}} )p^2 } \right]}}{{N(1 - 2p^2 )}}$$

$$\pi _{\textrm{B}} = \frac{{\left[ {N_{{\textrm{bb}}} (1 - p^2 )} \right] - \left[ {(N - N_{{\textrm{aa}}} )p^2 } \right]}}{{N(1 - 2p^2 )}}$$

and

$$\pi _{\textrm{o}} = 1 - \pi _{\textrm{A}} - \pi _{\textrm{B}}$$

Now we can get our segment size estimates:

• π _A = 0.497 or 49.7% true preference for product A.

• π _B = 0. 370 or 37% true preference for product B.

• π _o = 0.133 or 13.3% no real preference.

Next, we need the variability and covariance estimates for the Z-tests:

$${\textrm{Var}}(\pi _{\textrm{A}} ) = \frac{{\pi _{\textrm{A}} (1 - \pi _{\textrm{A}} ) + {{(3\pi _{\textrm{o}} p^2 )} \mathord{\left/ {\vphantom {{(3\pi _{\textrm{o}} p^2 )} 2}} \right. \kern-\nulldelimiterspace} 2}}}{N}$$

$${\textrm{Var}}(\pi _{\textrm{B}} ) = \frac{{\pi _{\textrm{B}} (1 - \pi _{\textrm{B}} ) + {{(3\pi _{\textrm{o}} p^2 )} \mathord{\left/ {\vphantom {{(3\pi _{\textrm{o}} p^2 )} 2}} \right. \kern-\nulldelimiterspace} 2}}}{N}$$

$${\textrm{COV}}(\pi _{\textrm{A}} ,\pi _{\textrm{B}} ) = \frac{{{{(\pi _{\textrm{o}} p^2 } \mathord{\left/ {\vphantom {{(\pi _{\textrm{o}} p^2 } {2) - (\pi _{\textrm{A}} \pi _{\textrm{B}} )}}} \right. \kern-\nulldelimiterspace} {2) - (\pi _{\textrm{A}} \pi _{\textrm{B}} )}}}}{N}$$

• Var(π _A) = 0.000296 (so π _A is 47.9% ± 1.7%, 0.017 = $\sqrt{0.000296}$)

• Var(π _B) = 0.000297

• COV(π_A, π_B) = –0.000198

Now for the hypothesis tests:

$$Z = \frac{{\pi _{\textrm{A}} - \pi _{\textrm{B}} }}{{\sqrt {{\textrm{Var}}(\pi _{\textrm{A}} ) + {\textrm{Var}}(\pi _{\textrm{B}} ) - 2{\textrm{Cov}}\pi _{\textrm{A}} \pi _{\textrm{B}} } }}$$

Note that this is a little different from the simple binomial test for paired preference. In the simple case we test the larger of the two proportions against a null value of 0.5. In this case we actually test for a difference of the two proportions, since we do not expect a 50/50 split any more with the no-preference option.

So the Z for test of A versus B gives Z = 4.067, an obvious win for product A.

Finally, a test against a minimum required proportion or benchmark:

$$Z = \frac{{\pi _{\textrm{A}} - 0.45}}{{\sqrt {{\textrm{Var}}(\pi _{\textrm{A}} )} }}$$

Z-test for A versus benchmark of 0.45 (45%).

13.2 Appendix 2: The “Placebo” Preference Test

In this method a pair of identical samples are given on one of two preference test trials (Alfaro-Rodriguez et al., 2007; Kim et al., 2008). These physically identical samples are not expected to differ, hence the parallel to a placebo, or a sham medical intervention with no expected therapeutic value. In theory, this could provide a baseline or control condition, against which performance in the preference test (with a no-preference option) could be measured. However, the amount of information gained from this design is relatively small and once again the analysis becomes more complicated. For these reasons, the sensory professional should consider the potential cost, additional analysis, and interpretations that are necessary. A recommended analysis is given at the end of this section.

Issues and complications. The use of a no-preference option was proposed to be a solution to the problem of a 50/50 preference test result, which could result from two stable segments of consumers who have a (perhaps strong) preference for each of the versions, respectively. Hence the idea was to offer a no-preference option with the reasoning that if there were no preferences (rather than stable segments) respondents should opt for the no-preference response. However, persons given identical samples will avoid the no-preference option 70–80% of the time, as discussed earlier in this chapter. So an answer concerning the question of stable segments cannot be obtained by this approach. Evidence for stable segments could be found by replicated testing or by converging evidence from different kinds of tests and/or questions.

Possible analyses. It might be tempting to just eliminate those persons expressing a preference for one of the identical pair members, on the grounds that such respondents are biased. However, this could eliminate 70–80% of the consumers. It is generally not advisable to pre-select consumers on any other grounds than their product consumption, and this approach eliminates individual who are in fact a portion of the representative population we are trying to generalize the results to. Such persons may not be “biased” in any dysfunctional way. Even identical samples may seem different from moment to moment. Furthermore, individuals are clearly responding to the demands of the task (you are expecting them to state a preference in a preference test!).

If historical data are available concerning the frequencies of response to identical pairs, a chi-square test can be performed as shown below. Note that this analysis cannot be done for a test in which the same subjects provide the “placebo” judgments because the chi-square test assumes independent samples.

Placebo analysis #1. Using historical data for expected frequencies.

Cells in the top row (A1, NP1, and B1) are expected frequencies (expected proportion × N judges). Cells in row 2 (A2, NP2, and B2) are the obtained data, frequencies of response in the preference comparison for the actual (different) test samples.

Table 7

Placebo analysis #2. The same consumers participate in the placebo trial and the test pair.

If the same people are used for the “placebo” trial and the normal preference comparison trial, an option is to recast the data into another 2 × 3 table, with the rows now representing whether the individual expressed a preference (or not) on the placebo trial. The columns remain Prefer A, no preference, Prefer B. A chi-square test will now tell you whether the proportions of preference changed comparing those who elected the no-preference option for the placebo pair versus those who expressed a “false preference” for the identical pair. This does not provide evidence for the existence of any stable segments nor will it tell you if there is a significant preference from this analysis alone.

If there is no significant chi square, you can feel justified in combining the two rows. If not, you may then analyze each row separately. The correct analyses are as stated in the no-preference Section 13.4: eliminating no-preference judgments, apportioning them, doing a test of d-prime values, or the confidence interval approach if the assumptions are met.

Each judge is classified into one of the six cells, A1, A2, B1, B2, NP1, or NP2. The first row contains the data from people who reported no preference on the placebo pair. The second row contains the data from people who expressed a preference with the placebo pair. A chi-square test will now show whether the two rows have different proportions. If not, the rows may be combined. If they are different, separate analyses may be performed on each row, using the methods of analysis of the no-preference option discussed earlier in the chapter.

Table 8

13.3 Appendix 3: Worked Example of Multinomial Approach to Analyzing Data with the No-Preference Option

This approach yields multinomial distribution confidence intervals for “no-preference” option, Data should be from a large test, N > 100, and the no-preference option was used rarely (<20%) (Quesenberry and Hurst, 1964, p. 193, Eq. (2.9)).

Upper and lower confidence interval boundaries are given by

$${\textrm{CI}} = \frac{{\chi ^2 + 2X \pm \sqrt {\chi ^2 \left[ \displaystyle{\frac{{\chi ^2 + 4X(N - X)}}{N}} \right]} }}{{2\left( {N + \chi ^2 } \right)}}$$

where

• $\chi _{{\textrm{critical}}}^{\textrm{2}} = 5.99$ for α at 5% and 2 df,

• X = number of observed preference votes for one sample,

• N = sample size.

Example: For: N = 162, X ₁ = 83, X ₂ = 65, and no preference = 14.

First find the confidence interval for product X ₁ (choices = 83/162)

$$\begin{aligned}{\textrm{CI}}&= \frac{{5.99 + 2(83) \pm \sqrt {5.99\left[ {\frac{{5.99 + 4(83)(162 - 83)}}{{162}}} \right]} }}{{2\left( {162 + 5.99} \right)}}\\& = \frac{{171.99 \pm 31.15}}{{335.98}}\end{aligned}$$

which gives an interval from 0.42 to 0.60 for product X ₁.

Then find the confidence interval for product X ₂ (choices = 65/162)

$$\begin{aligned}{\textrm{CI}}&= \frac{{5.99 + 2(65) \pm \sqrt {5.99\left[ {\frac{{5.99 + 4(65)(162 - 65)}}{{162}}} \right]} }}{{2\left( {162 + 5.99} \right)}}\\& = \frac{{135.99 \pm 30.39}}{{335.98}}\end{aligned}$$

which gives an interval from 0.31 to 0.50 for product X ₂. The lower bound of the higher proportion (0.42) overlaps with the upper bound of the lower proportion (0.50). We therefore conclude that there is not enough evidence for any difference in the preference proportions.