Abstract
Using multiple choice tasks per respondent in discrete choice experiment studies increases the amount of available information. However, respondents’ learning and fatigue may lead to changes in observed utility function preference (taste) parameters, as well as the variance in its error term (scale); they need to be controlled to avoid potential bias. A sizable body of empirical research offers mixed evidence in terms of whether these ordering effects are observed. We point to a significant component in explaining these differences; we show how accounting for unobservable preference and scale heterogeneity can influence the magnitude of observed ordering effects. (JEL Q23, Q51)
I. Introduction
Stated preference methods, and discrete choice experiments (DCEs) in particular, are often the only way to collect demand data for public or quasi-public goods when markets for them do not exist, and are frequently used to estimate their value to society (Carson and Czajkowski 2014). These techniques involve presenting detailed objective information to a randomly selected sample of the population and subsequently eliciting respondents' stated preferences. One of the characteristics of a DCE study is that respondents are presented with more than one choice task. This may have significant implications, as concerns for the stability of preference structures become more pronounced than in other stated preference methods. The accuracy of the choices observed in a DCE study, as well as the underlying decision strategies, may change as a respondent proceeds through multiple choice tasks. These phenomena are generally known as ordering effects, and their manifestations are usually referred to in the literature as (1) institutional learning, (2) value learning, (3) fatigue or boredom, and (4) choice set order dependence (Day et al. 2012). These effects are not mutually exclusive, and therefore, many studies investigated them jointly.
Institutional learning relates to the fact that most respondents in a DCE have never participated in a stated preference survey before.1 It is generally expected that institutional learning leads to an increase in the accuracy of responses; as respondents progress through choice tasks, their responses are likely to become more accurate, at least until they complete their “burn-in” number of choice tasks.2 By more accurate (more deterministic) choices, we mean decreased variance in the utility function's error term, that is, a higher utility function scale. This is what Swait and Adamowicz (2001a) call a “smaller noise to signal ratio.”
Value learning can work in a similar way; as respondents complete more choice tasks, they may discover or form their preferences, learn which attributes are the most important, and determine what trade-offs they are willing to make. Eventually, this simplifies choice tasks for them and is thus expected to reduce variance in the error term. It is also possible that value learning can result in changes in respondents' preferences (tastes) associated with the choice attributes between choice tasks. It is unclear how the effects of value and institutional learning could be explained separately; however, it is usually expected that institutional learning takes place during the first few choice tasks (Bateman et al. 2008a; Ladenburg and Olsen 2008).
Fatigue or boredom can be expected to work in the opposite direction, increasing the variance in the utility function's error term. Therefore, it has been postulated that because the effects of learning decrease, while the effects of fatigue increase with the number of choice tasks, there may be a U-shaped relationship between the unobservable component of the utility function and the number of choice tasks.
Finally, ordering effects may be a result of path-dependency. Should anchoring, framing, or strategic behavior be operative, respondents' choices could become path-dependent. Theoretical analysis demonstrates that these effects could influence decision rules in multiple ways (Carson and Groves 2007; Day et al. 2012).3
Empirical studies, discussed in detail in the next section, provide contradictory conclusions regarding the presence of learning and fatigue effects. However, these studies differ significantly with respect to the methodology used to observe ordering effects, in particular with respect to the number of choice tasks and whether unobservable preference and scale heterogeneity were accounted for. Our study aims to fill this gap.
In what follows, we demonstrate that controlling for differences in methodology, in particular allowing for unobservable preference or scale heterogeneity, plays a significant role in whether ordering effects are observed. We argue that these differences allow us to explain some of the contradictory evidence presented by earlier studies. In addition, we propose the most flexible (thus far) parametric methods to control for ordering effects, and we investigate the possible implications of the bias resulting from failing to account for them.
Although it is rather common to test different model approaches and investigate their impact on the parameter estimates and potential policy conclusions, our motivation here is not necessarily to focus on parameter estimates, implicit prices, and the bias resulting from not taking the scale dynamics into account (even though we investigate these issues too). Instead, we show that utilizing a more flexible model specification (in terms of allowing for the unobservable preference and/ or scale heterogeneity) leads to more apparent scale parameter dynamics. We believe this is a significant factor that could be used to (at least partly) explain why the evidence of fatigue or learning, presented in many empirical studies, is so mixed: these studies often use very different model specifications in terms of accounting for the unobservable preference and scale heterogeneity, and hence the effects may be more or less pronounced.
II. Ordering Effects in The Literature
Appendix A provides a list of empirical studies devoted to investigating ordering effects. The existing evidence is mixed. While some studies report preference or scale changes resulting from institutional or value learning, others do not find statistically significant effects or observe fatigue on the part of respondents. We note, however, that the studies differ substantially in the number of choice tasks used for the analysis, from as few as 4 to as many as 96 or 120 choice tasks per respondent (Brazell and Louviere 1997). A closer examination of these studies reveals tremendous variability not only with respect to the number of choice tasks but also in the design, context, and methods used to account for preference and scale differences.
There are many reasons that existing studies could observe different results in terms of whether and how ordering effects manifest themselves. These include differences in the context of a study (e.g., familiar vs. unfamiliar goods, Oppewal et al. 2010; Day et al. 2012), overall choice complexity (Swait and Adamowicz 2001a; Caussade et al. 2005; Chung, Boyer, and Han 2011), cumulative cognitive burden (Swait and Adamowicz 2001b), and administration mode (e.g., mail vs. web survey, Savage and Waldman 2008).
Anchoring, framing, or acting strategically may also lead to changes in respondents' choices. Therefore, it is necessary to control for these sources of ordering effects if one wishes to isolate the effects of learning and fatigue. Many studies did not effectively take this into account. For instance, employing a design in which a sequence of choice tasks presented at the beginning is repeated at the end of the series of discrete choice experiments allows one to observe possible deviations in choice patterns; researchers are then able to observe changes in observed marginal utilities when comparing implicit prices between the subsets of choices or differences in the variance of the error term estimated for choices made at different times.4 However, utilizing such a design makes it impossible to control for path-dependent ordering effects caused by anchoring, framing, or acting strategically, and these effects might be taken as evidence of learning or fatigue, or mask their influence.
One way to minimize the effects of anchoring, framing, or acting strategically is to use a counterbalanced design, that is, present each respondent with a different ordering of choice tasks, such that the potential effects of, for example, starting point bias is canceled out for the sample. Counterbalancing a DCE design plays an important role in retrieving the underlying dynamics of ordering effects (Keppel and Wickens 2004). Liechty, Fong, and DeSarbo (2005) show theoretically that without counterbalancing, static models fail to capture average preference changes and that dynamic models can provide relatively poor estimates. Some examples of counterbalancing include cyclical design, in which choice tasks are rotated for different respondents (i.e., choice sets 1, 2, 3, …, T for the first respondent, 2, 3,…, T, 1 for the second respondent, and so on), and the Latin square design, in which the cycling is reversed (Street and Street 1987). Another possibility is to randomize the ordering of choice tasks for each respondent, in which case the sequence not only begins at a different point but is also shuffled, in other words, choice sets do not appear in the same order.
We note that the empirical studies listed in Appendix A employed a wide range of econometric models that, as we will show, can influence the observed result. Most studies repeated a choice task (or a sequence of choice tasks) and observed differences in implicit prices or scale, included interactions of choice attributes or scale with the number of choice tasks, and included the number of choice tasks as a covariate of scale or as an explanatory variable of latent class membership (latent class model). The most common means of controlling for scale differences were as follows: the procedure proposed by Ben-Akiva and Morikawa (1990), in which observations from separate (groups of) choice tasks are used simultaneously to maximize a joint likelihood function; the Bradley and Daly (1992) one-step estimation approach of Ben-Akiva and Morikawa, which can be implemented using a nested logit (the logit-based scaling approach); the Swait-Louviere procedure (Swait and Louviere 1993), a sequential scaling approach; and the heteroskedastic multinomial logit model with choice task–specific covariates of scale (Hensher, Louviere, and Swait 1998; Dellaert, Brazell, and Louviere 1999; Swait and Adamowicz 2001a).
More importantly, these studies differ substantially in their treatments of unobservable preference or scale heterogeneity. In recent years, substantial research efforts have been devoted to the issue of consumers' unobservable preference heterogeneity (e.g., McFadden and Train 2000). Another research stream aimed at accounting for consumers' heterogeneity has focused on modeling unobservable scale differences (e.g., Louviere et al. 2002). In what follows, we demonstrate that allowing for unobservable preference or scale heterogeneity may significantly influence the strength of observed learning and fatigue effects. Therefore, using a different model specification is an important factor that helps one to explain the mixed evidence presented by empirical studies investigating ordering effects.
III. Methods for Modeling Ordering Effects
In this section, we review the methods for accounting for unobservable preference and scale heterogeneity in discrete choice models. We subsequently apply these methods to investigate how different model specifications may influence ordering effects.
Heteroskedastic Multinomial Logit Model
The modeling of discrete choice data is based on random utility theory developed most notably by McFadden (1986). It assumes that the utility associated with any state (choice) can be divided into a sum of contributions that can be observed by a researcher and a component that cannot, and hence, it is assumed to be random. This formulation of the utility function and choice-specific alternatives leads to a multinomial logit (MNL) model that allows the observed choices of an individual to be used to compare their utility levels associated with the choice alternatives.
Formalizing, let individual i choose among J alternatives, each characterized by a vector of observed attributes xij. The utility associated with alternative j is given by
[1]
where β is a parameter vector of the marginal utilities of the attributes. Introducing the error term assumes that utility levels are random variables, as it is otherwise impossible to explain why apparently identical individuals (in terms of observable attributes) may select different options.
Random utility theory is transformed into different classes of choice models by making different assumptions about the random term. To ensure that the random component is able to represent the necessary amount of randomness in the respondents' choices, its variance needs to be sufficiently large, or as the utility function is ordinal, the assumptions regarding the variance in the random term may be expressed by scaling the utility function in the following way:
[2]
where the random component of the utility function is conveniently assumed to be independently and identically distributed (iid) across individuals and alternatives: extreme value type 1 distribution. The scale coefficient σ and β cannot both be identified. The MNL model is derived, with the following closed-form expression of the probability of choosing alternative j from a set of J available alternatives:
[3]
The heteroskedastic MNL (H-MNL) model allows the scale for some observations to systematically differ from the others. The utility specification of the H-MNL model with scale covariates entering linearly (Dellaert, Brazell, and Louviere 1999) is as follows:
[4]
while by assuming an exponential formulation for the multiplicative scale (Swait and Adamowicz 2001a), it is possible to drop the 1, and there is no need to assume the scale is strictly positive:
[5]
In both cases, the “effective” scale is a function of ki, a vector of respondent-or choice-specific and observable variables. The scale is still normalized, but with respect to the reference group, and may differ for selected observations (e.g., choice tasks occurring later in the sequence).
Scale Heterogeneity Model
The MNL model implausibly assumes that the random term is iid for all choices, that is, the scale coefficient is the same for every respondent, choice task, and alternative. This results in the assumption that each respondent makes his choices with the same degree of randomness. One way to relax this assumption is to allow the (unobservable) scale coefficient to be individual specific while making it a random variable following a particular (usually log-normal) distribution. The new utility specification becomes
[6]
where σi ~ LN(1,τ) or with σi = exp(
+ τε0iwith ε0i ~ N(0,1). Note that the scale coefficient is now respondent specific. Because it is still convenient to normalize the scale to 1, we want Eσi = exp(σ + τ2/2). This may be achieved by assuming 
= − τ2/2. Therefore, the scale is no longer fixed; instead, it is assumed to follow a log-normal distribution, with the new parameter τ reflecting the level of scale heterogeneity in the sample. The resulting model is a scale heterogeneity (S-MNL) model (Fiebig et al. 2010).
To use this model to observe scale changes between choice tasks, we introduced additional explanatory variables of scale k, such that σi = exp(
+ τε0i + θ′kit. These may be, for example, choice task–specific variables, provided that one of the choice tasks is used as a reference level.
Heteroskedastic Random Parameters Logit Model
Another implausible assumption of the MNL model is that all respondents have the same preferences (and hence the same coefficients in their utility functions β). The state-of-practice methods of relaxing these assumptions, that is, allowing for some level of (unobservable) preference heterogeneity and possibly correlations between the alternatives and choice tasks, include the random parameters (RPL) model (McFadden and Train 2000; Hensher and Greene 2003).
In the RPL model, the utility function becomes
[7]
Note that the parameters of utility functions are now respondent specific. It is assumed that they follow distributions specified by a modeler: βi ~ f(b+Δ′zi,Σi), with means b and variance-covariance matrix Σ. In addition, it is possible to make the means and variances of the distributions a function of observable respondent or choice-specific characteristics z.
In this paper, we refer to the heteroskedastic RPL (H-RPL) model as an RPL model in which scale is allowed to systematically differ for some observations. This is a natural extension of an H-MNL model. Its variant has been used before to control for possible scale differences when combining revealed and stated preference data (Brownstone, Bunch, and Train 2000; Small, Winston, and Yan 2005), although, to our knowledge, it has never been used to account for within-sample scale heterogeneity before.
The utility specification in the H-RPL model becomes
[8]
where the scale is a function of observable explanatory variables k and, in our case, is choice task specific, but generally, any observable variables, as long as the scale may be normalized for some reference level observations.
Generalized Multinomial Logit Model
A method that makes it possible to simultaneously control for both respondents' preference and scale heterogeneity is the generalized multinomial logit (G-MNL) model (Fiebig et al. 2010). In this model, the utility function takes the following form:
[9]
Similar to the RPL model, the coefficients in the utility function are individual specific (b represents the population means of the parameters, while υ represents individual-specific deviations from these means). Unlike in the RPL, however, the scale coefficient is also individual specific. The new coefficient γ∈ [0,1]5 controls how the variance of residual taste heterogeneity varies with scale. If γ = 0, the individual coefficients become βi = σib+υi, while if γ = 1, they are βi = σi(b + υi). These are the two extreme cases of scaling (or not scaling) residual taste heterogeneity in the G-MNL model (type I and type II, respectively); however, all intermediate solutions are possible.
In the estimation, the individual scale is normalized in the same way as in the S-MNL model. In this paper, we use a new, more flexible variant of the G-MNL model by making the individual scale parameter a function of observable (choice task–specific) characteristics k:
[10]
This decision enables us to investigate learning and fatigue effects while allowing for both preference and scale heterogeneity across respondents.
IV. Empirical Study
Our empirical study was conducted in the context of environmental protection: management changes in the protection of Polish forests. We were interested in the attributes of the Polish forests that are the most significant for the general public in terms of recreation and biodiversity conservation. Through extensive qualitative studies, we discovered that the forest attributes that Poles would most like to see changed were the following: (1) the protection of the most ecologically valuable forests, (2) less litter in forests, and (3) an increasing the amount of recreational infrastructure. These were the attributes that we used for the hypothetical scenario of our DCE study.
Of the 90,000 km2 of Polish forests, approximately 3% are the most ecologically valuable in terms of having many of the characteristics of natural forests, such as the age and structure of trees, the presence of natural environmental processes, large amounts of dead wood, rare species of fauna and flora, and high biodiversity in general (see Appendix B, Figure B1 for an illustration). Approximately 50% of these forests are currently properly protected, usually in the form of national parks and nature reserves. The remaining forests are under significant human pressure and are often treated as conventionally managed forests. Appendix C provides a map of the locations and areas with the most ecologically valuable forests in Poland. Therefore, the first attribute in our CE scenario was the area change in ecologically valuable forests that could be protected. The possible levels of this attribute were the following:
Status quo: passive protection of 50% of the most ecologically valuable forests (1.5% of all forests)
Partial improvement: passive protection of 75% of the most ecologically valuable forests (2.25% of all forests, 50% increase)
Substantial improvement: passive protection of 100% of the most ecologically valuable forests (3% of all forests, 100% increase)
The second attribute used in the final study was the amount of litter that was present in the forest. This litter could be left in forests by tourists or be illegal trash dumping sites (see Appendix B, Figure B2 for an illustration). Litter obviously decreases the recreational value of a forest, may leak dangerous substances, and constitutes a hazard for animal life and health. In our hypothetical scenario, proposals to reduce the amount of litter by 50% or by 90% were presented, through tougher law enforcement and increasing forest cleaning services. The available levels of this attribute were the following:
Status quo: no change in the amount of litter in the forests
Partial improvement: decrease the amount of litter in the forests by half (50% reduction)
Substantial improvement: litter found in the forests only occasionally (90% reduction)
Qualitative pretesting also revealed that sufficient tourist infrastructure is important in determining the recreational value of forests. This could include local roads allowing easier access to a forest, parking spaces, paths and trails for tourists, organized resting areas (e.g., picnic sites), or toilets. Our scenario proposed and described two levels of increased infrastructure quantity and quality. It was explained that such infrastructure would be built only where necessary and only in a way that does not influence the environment. In short, these were the following:
Status quo: no change in tourist infrastructure
Partial improvement: appropriate tourist infrastructure in 50% more forests (50% increase)
Substantial improvement: appropriate tourist infrastructure available in double the current forests (100% increase)
The last attribute was monetary: additional annual cost per household, in the form of increased income taxes.
The final survey was conducted on a representative sample of 1,001 Poles. We hired a professional polling agency that collected the questionnaires using high-quality, face-to-face, computer-assisted surveying techniques. A multistage sampling strategy was employed in which communities were randomly selected to represent different community types, within each of the selected communities a starting point address was randomly selected and followed by choosing a set of addresses using the random route method, and random selection of an adult household member was employed.6 The 26 choice sets utilized in our study were prepared using Bayesian d-efficient design optimized for the RPL model (Sándor and Wedel 2001; Ferrini and Scarpa 2007; Bliemer, Rose, and Hess 2008; Scarpa and Rose 2008). To obtain initial estimates (priors) and verify the qualitative properties of the questionnaire itself, we conducted a pilot study on a sample of 50 respondents.
Each respondent was presented with 26 choice tasks, each consisting of four alternatives. Each alternative was described with the four attributes specified above. An example of a choice card shown to respondents is presented in Figure 1.
Example of a Choice Card (Translation)
A significant contribution of our empirical work results from our design being counterbalanced. This allowed us to minimize ordering effects occurring as a result of starting point bias, framing, or acting strategically. We achieved this by randomizing the order of the 26 choice tasks presented to each respondent.7 The randomization of the choice set order for each respondent did not result in an equal number of times each choice set was used as a particular choice task for the entire sample, however. Since there were 1,001 respondents in the sample and each respondent was presented with 26 choice tasks, the optimal number of times each choice set should be presented as a particular choice task was 38.5. Due to randomization, however, some choice sets were over-or underrepresented in some choice tasks. The average deviations from the optimal number of times each choice set should be used as a particular choice task were in the range of 6 to 10.
The variance of the error term of the utility function might depend on how “easy” or “difficult” a particular choice set was. If a particularly easy (or difficult) choice set was presented as a given choice task more often than the other choice sets, this could influence the observed choice task–specific scale. In order to control the influence of some choice sets being over-or underrepresented as particular choice tasks, in what follows we used weighted log-likelihood functions. The weights were inversely proportional to the number of times each choice set was presented as a particular choice task. Overall, this treatment allows us to focus on the effects of learning and fatigue, while the potential effect of the starting point (anchoring) is canceled out.
V. Results
Preference (Taste) Dynamics
We begin the analysis by testing whether there are systematic changes in respondents' preferences associated with the choice attributes. To do so, we estimate a separate MNL model for each choice task. This is made possible by the fact that the order of choice sets was randomized for each respondent, and so different respondents could be presented with different choice sets as their first choice task. Estimating 26 separate models allows utility function coefficients and scales to be fully choice task–specific. The fit measures for each of the choice task–specific models are presented in Table 1.
Fit Measwes for the Choice Task-Specific Models
Although in the following analysis we applied other, more sophisticated models, we argue that the MNL model is appropriate for the task of analyzing the dynamics of implicit prices based on choice task–specific models. There are two main reasons for this. First, estimating choice task–specific models that allow for scale heterogeneity (S-MNL), preference heterogeneity (RPL), or both (G-MNL) proves difficult, as having only a single observation for each respondent makes it difficult to distinguish between random heterogeneity and the iid extreme value term in the model (Ruud 1996; Revelt and Train 1998; Fosgerau and Nielsen 2010; Hess and Train 2011). Second, the standard errors associated with the implicit prices derived from the MNL model are the smallest and, hence, allow for the most conservative comparison criteria. The models allowing for heterogeneity with respect to attribute coefficients result in much wider confidence intervals for the implicit prices, as they result from not only the standard errors associated with the coefficients but also from the standard deviations of the empirical distributions of random coefficients (as well as the standard errors associated with their means and standard deviations). Therefore, using confidence intervals derived from a simple MNL model seems the most conservative means to analyze significant differences between choice task–specific implicit prices.
Since the parameters of each choice task– specific model are confounded with a different scale, it is not possible to directly compare their values across the models. Instead, we calculated implicit prices for each of the choice attributes and each choice task; therefore, model-specific scales are canceled out, and we are able to observe the potential dynamics of implicit prices. There results are presented in Figure 2. The seven panels of Figure 2 present the dynamics of choice task– specific implicit prices associated with the choice attributes (IP), along with 95% confidence intervals and the implicit prices derived from MNL models estimated for all choice tasks (IP-all). The qualitative attributes were dummy coded with the status quo as the reference level; hence, the presented implicit prices are as follows:
Dynamics of Choice Task–Specific Implicit Prices
NAT1, NAT2: partial (50%) and substantial (100%) improvement in the area of passively protected ecologically valuable forests
TRA1, TRA2: partial (50%) and substantial (90%) reduction of litter in the forests
INF1, INF2: partial (50%) and substantial (100%) increase in the forests containing tourist infrastructure
SQ : the alternative specific constant associated with the status quo alternative
The analysis of implicit price dynamics reveals that there is a considerable amount of variation between choice tasks. However, no systematic patterns are visible, and the implicit prices are not significantly different between choice tasks, as indicated by the overlapping confidence intervals. This finding is supported by a similar analysis based on the results of other models mentioned above; taking preference and scale heterogeneity into account results in substantially wider confidence intervals of implicit prices and even less significant differences between choice task– specific implicit prices.8
In conclusion, we do not find significant preference changes between choice tasks.
This result is in line with many earlier empirical studies that do not observe statistically significant differences in preferences (tastes) between choice tasks (see Appendix A for details). We note, however, that the mixed evidence provided by some of these studies is related to problems with identifying intrarespondent taste heterogeneity, as discussed by Hess and Train (2011). In particular, the low statistical power of the tests may result in the taste dynamics rarely being statistically significant.
Based on the findings reported in this section, in what follows, we assumed that individual preference (taste) parameters are constant throughout the choice experiment and focused on analyzing scale dynamics and potential bias resulting from failing to account for them.
Scale Dynamics
To investigate scale dynamics under different methodological assumptions, we estimated H-MNL, S-MNL, H-RPL, and G-MNL models (the last two with and without allowing for correlations between random parameters)9 in which dummy coded choice task numbers (CS2–CS26) are included as explanatory variables of scale (the first choice task was used as a reference level). All random parameters associated with the choice attributes were assumed to be normally distributed. We accounted for the fact that each respondent faced 26 choice tasks by allowing for individual-specific random coefficients and scale, where applicable. All models were estimated using 26,026 choice observations and 1,000 random draws.
The results are presented in Table 2. Panel 1 presents the coefficients associated with choice attributes or their means for models that assume they are random. In addition to the variable names used earlier, we provide estimates for FEE, the cost coefficient. In Panel 2, we present the standard deviations of normally distributed parameters. Panel 3 contains the coefficients of parameters associated with scale, that is, τ and γ (where applicable), and 25 choice task–specific covariates of scale. The estimates of the elements of the Cholesky matrices were omitted for brevity but are available from the authors upon request.
Scale Dynamics under Different Model Specifications
We begin the analysis by noting that all explanatory variables are revealed to be significant determinants of the choices and have the expected signs. The statistical significance of the coefficients associated with the standard deviations of the random parameters distributions indicates that they are significantly different from zero; hence, the variables should be modeled as random. This is strong evidence of unobservable preference heterogeneity. Additionally, in the S-MNL and both G-MNL models, the τ coefficient representing the dispersion of individual scale coefficients is significantly different from 0 that indicates significant (unobservable) heterogeneity in individual scale coefficients. Finally, as expected, models that allow for unobservable preference and scale heterogeneity generally perform better than models that do not; the same holds for the models that allow for correlations between random parameters compared to those that do not.
To facilitate the examination of scale dynamics, Figure 3 presents the choice task–specific scale parameters of each model, along with their 95% confidence intervals. A general pattern becomes apparent: irrespective of the model used, the scale appears to increase until approximately the eighth choice task. This corresponds to respondents' learning in that their choices become more deterministic. In the subsequent choice tasks the scale exhibits some degree of variability; however, it does not seem to further increase or decrease significantly. Therefore, we conclude that we did not observe fatigue effects that would lead to increased randomness in our respondents' choices.
Scale Dynamics and Associated 95% Confidence Intervals under Different Model Specifications
Second, we note that greater sophistication of the model employed, in terms of allowing for unobservable preference and scale heterogeneity, led to greater increases in the choice task–specific scale parameter. The magnitude of this increase grows from the H-MNL model, through allowing for scale heterogeneity alone (S-MNL), and preference heterogeneity only (H-RPL-d), to allowing for both (H-RPL, GMNL-d, GMNL).10 Figure 4 presents choice task–specific scale coefficients derived using different modeling approaches allowing for convenient comparison.
Scale Dynamics under Different Model Specifications: Summary
The increase of the scale coefficients with the number of choice tasks presented to the respondent is an indication of respondents' choices becoming more deterministic. The absolute value of the scale coefficient is difficult to interpret; it reflects the relation of deterministic part of the utility function to the random (error) component. Since the scale for the first choice task is normalized to 1, the increase occurring in the subsequent choice tasks indicates how much lower the variance of the error term becomes in relation to the deterministic part of the utility function. For example, in the case of the H-MNL model the scale does not become significantly different from 1 until the fifth choice task, and in the end of the choice sequence it becomes close to 1.2. This means that under this model specification the respondents' choices in the last choice task were roughly 20% more deterministic than in the first one. For comparison, in the case of the G-MNL model, the scale coefficient becomes significantly different from 1 already in the second choice task; the value of the scale coefficient estimated for the last choice task was approximately 2.0, indicating that on average, the respondents' choices in this task were twice as consistent as in the first.
Finally, we note that the differences in the observed scale dynamics between the alternative model specifications (i.e., depending on whether the unobservable preference and scale heterogeneity is allowed or not allowed) could lead to different conclusions in terms of whether statistically significant scale dynamics are observed. For example, using the H-MNL model with the experimental design of only the first eight choice tasks would lead us to reject the hypothesis of statistically significant scale increase; this is not the case, however, for the more sophisticated models.
Therefore, depending on which model specification was used in the analysis, in terms of allowing for unobservable preference and scale heterogeneity, different conclusions could be drawn in terms of the existence and magnitude of the ordering effects. We argue that this could be a potential factor in explaining why the evidence of scale dynamics, presented in many empirical studies, is so mixed, especially that many of the studies that do not find significant scale dynamics (e.g., Hess, Hensher, and Daly 2012) used an approach equivalent to our H-MNL model.
Bias Resulting from Failing to Account for Ordering Effects
Allowing for scale differences between choice tasks can substantially improve model fit, in addition to improvements resulting from allowing for unobservable preference and scale heterogeneity. We illustrate this with the results presented in Table 3. Every model used in our analysis was estimated with (panel 1) and without (panel 2) allowing for choice task–specific scales. In the case of all our model specifications, allowing for scale dynamics by introducing 25 choice task–specific dummy variables as covariates of scale significantly improved model performance, as represented by a statistically significant increase in model fit.
The Influence of Accounting for Scale Dynamics on Model Performance
Failing to account for differences in error term variance (scale) between choice tasks can result in biased estimates in a way that is similar to failing to account for the panel structure of a dataset (Ortúzar and Willumsen 2001). One might think that because implicit prices are derived from the ratio of two parameters, scale cancels out, and implicit prices are invariant to scale; however, this is not the case. Because the entire utility function (including scale) is used in the estimation and derived probabilities are included in the maximized log-likelihood function, incorrectly accounting for scale will influence the parameter estimates, just as if the observations for different choice tasks were weighted in an arbitrary way.
To investigate the influence of this bias on implicit prices, we used all of the models discussed above to derive median willingness to pay for marginal changes in choice attributes. These results are presented in Table 4. The results reveal that while there are substantial differences between the implicit prices derived from different model specifications, failing to account for choice task–specific scale does not lead to significant differences in implicit prices; they are well within the confidence intervals. However, this result may be specific to our dataset. Because we did not
Implicit Prices under Different Model Specifications (in PLN)
observe significant preference (taste) dynamics between choice tasks, not accounting for scale differences and thus effectively using an arbitrary weighting of data from different choice tasks does not lead to changes in the calculated implicit prices.
VI. Discussion and Conclusions
In this study, we investigated the dynamics of a utility function's parameters and its random variance component (scale) that may arise in repeated choice tasks of discrete choice experiments. Our study demonstrates that allowing for unobservable preference and scale heterogeneity substantially increases the magnitude of scale changes. We show how overly restrictive model specifications may lead to drawing incorrect conclusions in terms of whether scale dynamics are observed. We argue that this is an important factor that allows us to explain some of the contradictory evidence presented in existing empirical studies.
Our analysis utilized state-of-the-art methods to control for unobservable preference and scale heterogeneity. We proposed extensions of these methods to allow the scale to be choice task specific. We find that employing a more sophisticated model, in terms of allowing for unobservable preference and scale heterogeneity, resulted in greater observed increases in the choice task–specific scale parameter. Using the H-MNL model as a reference, we demonstrate that the magnitude of this increase grows from allowing for unobservable scale heterogeneity alone (S-MNL) through allowing for unobservable preference heterogeneity (H-RPL-d) to allowing for both (H-RPL, G-MNL-d, G-MNL).
Empirically, we do not find evidence of statistically significant preference (taste) dynamics. However, we observe significant dynamics of scale; the scale appears to increase until approximately the eighth choice task. Therefore, as our respondents' choices became more deterministic in the number of completed choice tasks, we can conclude that we found evidence of learning. Conversely, there are no signs of scale decrease, at least for the 26 choice tasks used in our study. This finding might be interpreted as a lack of evidence for fatigue.
Our results do not support the inverted U-shaped relationship between scale and the number of choice tasks. In our case, the choice task–specific scale was increasing at a decreasing rate, until it became relatively stable after approximately the eighth to twelfth choice task (with some variability). We note, however, that virtually all studies that reported the inverted U-shaped relationship between scale and the number of choice tasks assumed a quadratic relationship rather than allowing the scale to be choice task specific. As a result, their findings that scale reaches its maximum at 10 (Caussade et al. 2005) or 6 (Chung, Boyer, and Han 2011) choice tasks may be a result of assuming the relationship's functional form, combined with the studies employing an insufficient number of choice tasks.
We acknowledge that investigating the presence of fatigue is not straightforward, however. Swait and Adamowicz (2001b) note that due to the high level of complexity perceived in the first few choice-situations, or cumulative cognitive burden, respondents may simplify their choice strategies and adopt noncompensatory decision rules. As a result, one would observe increases (not decreases) in scale as choices become more deterministic. Our dataset does not allow us to verify this hypothesis. However, such a change in decision strategies would likely lead to changes in the estimated choice task–specific preference (taste) parameters. Because in our case there was no evidence of such preference changes, we presume that such a change in decision strategies did not take place.
The economic theory of value maximization assumes that consumers have constant utility functions that are revealed when elicited (von Neumann and Morgenstern 1944). However, there are at least three streams of research that indicate that this might not be the case, both in terms of the parameters of utility functions and their scale.
Following Simon's (1955) questioning of complete rationality in human behavior, Heiner (1983) incorporated the notion of information processing limitations in the consumer's ability to make rational choices. These limitations could cause the respondents' decisions to change as they become more familiar with the “institution” of DCE choice tasks (institutional learning; Braga and Starmer 2005). Respondents may also be expected to make errors, possibly increasingly as they become fatigued or bored in later choice tasks. Finally, if the choice tasks are complex, respondents may engage in simplification strategies and change the choice strategies they use as fatigue or boredom sets in.
The second stream of research indicating problems with consumer utility functions stability comes from research on human decision making. There is some evidence that preferences may be constructed, rather than revealed (Payne, Bettman, and Johnson 1992; Slovic 1995). This evidence may mean that consumers could be learning about their true preferences throughout the course of a DCE study (value learning, Plott 1999) or that after a period of “burn-in” choice-situations, individuals evolve a systematic approach to evaluating alternatives (Luce and Tukey 1964), in either case resulting in changes in their underlying decision rules from one choice task to another. Another manifestation of constructed preferences would be that choices may be influenced by a number of external stimuli, such as changes in the task environment (Payne, Bettman, and Johnson 1993), which is a yet another manifestation of the well-known framing effect (Kahneman and Tversky 2000). Finally, the constructed preference hypothesis may also cause path-dependence in a respondent's choices. For instance, constructed preferences may depend on the attribute levels observed in the initial or previous choice tasks, possibly with decreasing strength. These constructed preferences may also be viewed as a form of the anchoring effect (Kahneman, Slovic, and Tversky 1982). Even if an individual's choices are internally coherent, they may still be anchored to some reference point, such as the first choice task (coherent arbitrariness; Ariely, Loewenstein, and Prelec 2003).
Third, in addition to behavioral reservations, there may be theoretical explanations for why choice outcomes may change over the course of a DCE study. Carson and Groves (2007) demonstrate that the conditions under which the single-bounded, referendum format elicitation questions used in contingent valuation studies are incentive compatible (a take-it-or-leave it question format, respondents' perceptions of the consequences of their responses, the possibility of introducing compelling payments at the stated price) may not hold for DCE studies, in particular those with repeated choice tasks. Because respondents may be aware that they will be presented with multiple choice tasks in advance and because as they work through the tasks they can exploit information about previous choice tasks and decisions, it remains unknown whether the mechanism is incentive incompatible and, hence, if the same decision mechanism underlies the choices in all choice tasks. Evidence of such lag effects is provided by, for example, Holmes and Boyle (2005).
In the case of our study, using a counterbalanced design allows us to rule out the systematic effects of many of the choice set–specific effects discussed above, such as anchoring, framing, or acting strategically. Because the order in which the 26 choice sets in our design were presented to different respondents was random, even if effects such as starting point bias or anchoring to a previous choice task attribute levels were present, our approach causes these effects to cancel out across respondents, allowing us to focus on the effects of learning and fatigue in different choice tasks.
From a statistical point of view, it has long been recognized that treating repeated choice data in the same way as cross-sectional data may not be appropriate and may lead to, among other effects, biased standard errors for the estimated parameters (Guilkey and Murphy 1993; Ortuúzar and Willumsen 2001). Theoretically, the same holds if choice task–specific scale differences are not accounted for. We investigated the magnitude of this bias on implicit prices under different model specifications. In the case of our study, while accounting for choice task–specific scale proved to significantly improve model fit, we did not observe any statistically significant differences in implicit prices if this was not accounted for. However, we note that this effect may be specific to our study, as we did not observe significant changes in preference (taste) dynamics between choice tasks.
We acknowledge that our empirical results relate to a single dataset. We used data from a large-scale, high-quality, nationally representative study conducted using face-to-face, computer-assisted surveying techniques. At least in the case of our study, the scale stabilized after approximately eight choice tasks and remained at this level until the last, twenty-sixth, choice task. However, this result may depend on many study-specific factors, such as context of the study, the number of alternatives, and the number of attributes. There is also some evidence that other surveying techniques, such as internet-based surveys, might result in weaker learning effects and much stronger fatigue effects (Savage and Waldman 2008).
In conclusion, our study adds to the research on ordering effects. Our analysis reveals that the model specification, in particular allowing for unobservable preference and scale heterogeneity, may influence the magnitude of observed learning and fatigue effects. We demonstrate how overly restrictive specifications of a model, as well as failing to include enough choice tasks, may lead to drawing incorrect conclusions in terms of whether scale dynamics are observed. This finding may prove to be an important factor in explaining some of the contradictory evidence presented in existing empirical studies. Finally, we show that accounting for choice task–specific scale dynamics can significantly improve model fit, and we urge other researchers to take this finding into account in their applications.
Acknowledgments
The authors wish to thank Richard Carson, whose valuable comments facilitated this study. We also thank two anonymous reviewers for their insightful comments and suggestions that helped to improve this manuscript. The support of the Polish Ministry of Science and Higher Education and the Foundation for Polish Science is gratefully acknowledged.
Appendix A
Review of Empirical Studies Investigating Ordering Effects
Appendix B
An Illustration of an Economic Forest and a Close-to-Natural Forest
An Illustration of Litter in the Forests
An Illustration of Tourist Infrastructure
Appendix C
The Most Ecologically Valuable Forests in Poland
Footnotes
The authors are, respectively, assistant professor, Faculty of Economic Sciences, University of Warsaw, Warsaw, Poland; assistant professor, Faculty of Economic Sciences, University of Warsaw, Warsaw, Poland; and professor, Department of Economics, Stern School of Business, New York University, New York.
↵1 A number of studies show that respondent consistency depends on the complexity of a choice task, such as the number of attributes, their levels, ranges, and correlations, and the number of alternatives (e.g., Palma, Myers, and Papageorgiou 1994; Dellaert, Brazell, and Louviere 1999; Swait and Adamowicz 2001a; Caussade et al. 2005; Hensher 2006a, 2006b; Day and Pinto Prades 2010). This observation together with the hypothesis of information processing limitations supports institutional learning.
↵2 However, as Swait and Adamowicz (2001b) note, if due to a perception of high complexity in the choice tasks or cumulative cognitive burden, respondents may simplify their choice strategies, adopting noncompensatory decision rules; one can also expect changes in the estimated marginal utilities of the attributes.
↵3 Day et al. (2012) distinguish between starting point effects, acting strategically with full recall of the presented choice tasks and with imperfect recall, weighted toward more recent choice tasks. However, this distinction is not relevant for our analysis.
↵4 Alternative treatments focused on observing the amount of time the respondents took to complete a choice task (Haaijer, Kamakura, and Wedel 2000; Rose and Black 2006).
↵5 To assure that γ∈[0,1], it is usually modeled as γ = (exp(γ∗))/(1+ exp(γ*)), and γ∗ is estimated.
↵6 As noted by one of the anonymous reviewers, since our survey mode used computer assisted personal interviews, there might be a potential interviewer effect. Unfortunately, our data did not allow for the investigation of this. In addition, we did not receive information on the nonresponse rate.
↵7 In addition, we randomized the order of the three non– status quo alternatives for each choice task and each respondent. Admittedly, the position of the status quo alternative in the choice task was not randomized, as in the pretesting of the questionnaire we found that the respondents found it more natural to see the status quo alternative first. Meyerhoff and Liebe (2009) discuss potential status quo biases in DCE studies.
↵8 A careful inspection of the results presented in Figure 2 reveals that implicit prices for some choice tasks exhibit consistent peaks, and the confidence intervals become wider (e.g., choice task 4 or 13). A possible explanation for this could be that if the estimate of the cost coefficient for a particular choice task was closer to zero than in other choice tasks, it could result in higher implicit prices of all attributes calculated for this choice task.
↵9 In what follows, we used suffix “-d” for RPL and G-MNL models in which only elements on the diagonal of the Cholesky matrix are estimated (no correlations are allowed), and no suffix otherwise.
↵10 Hess and Rose (2012) note that the RPL model with all taste coefficients modelled as random and with no constraints on the correlation structure already accounts for some degree of scale heterogeneity. This is in line with our findings: the choice task specific scale parameters are very similar between the H-RPL, GMNL-d and GMNL model, as all these models account for both preference and scale heterogeneity.
References
In this issue
Jump to section
Related Articles
Cited By...
- Learning and Fatigue in Real Discrete Choice Experiments
- Do We Choose Differently after a Discussion? Results from a Deliberative Valuation Study in Ireland
- Do We Choose Differently after a Discussion? Results from a Deliberative Valuation Study in Ireland
- Scale Heterogeneity and Its Implications for Discrete Choice Analysis
- Testing Preference Formation in Learning Design Contingent Valuation Using Advance Information and Repetitive Treatments