What are the consequences of consequentiality?

Abstract

We investigate the extent to which dichotomous choice referenda responses are shaped by whether the individual believes the survey itself will ultimately impact policy. Using survey data from the Iowa Lakes Project, we test this supposition. Specifically, we employ a Bayesian treatment effect model in which the degree of perceived consequentiality, measured as an ordinal response, is permitted to have a structural impact on willingness to pay (WTP) for a hypothetical environmental improvement. We test whether the estimated WTP distributions are the same for each value of the ordinal response.

In our survey data, a subsample of individuals were randomly assigned supporting information suggesting that their responses to the questionnaires were important and will have an impact on policy decisions. In conjunction with a Bayesian posterior simulator, we use this source of exogenous variation to identify the structural impacts of consequentiality perceptions on willingness to pay, while controlling for the potential of confounding on unobservables. We find evidence consistent with a “knife-edge” result, namely that the willingness to pay distributions are equal among those believing the survey to be at least minimally consequential, and different for those believing that the survey is irrelevant for policy purposes.

Introduction

Despite lingering concerns within the economics literature, stated preferences methods for valuing non-market commodities are frequently used both by policymakers and in the judicial system to assess changes to environmental goods and services. Best practices, starting with the NOAA panel recommendations, have evolved over time aimed at minimizing potential biases in survey responses. These guidelines emphasize the careful design and pre-testing of the survey instrument so as “$\dots$ to induce respondents to take the [survey] question[s] seriously” [1, p. 4606]. In this context, it is surprising that relatively little attention has been given to assessing whether respondents actually believe that the survey itself will potentially influence future policy. If they do not believe this to be the case, they have little incentive to respond truthfully and their answers may be largely noise. Indeed, Carson and Groves (CG) [2] argue that individuals can be expected to answer the standard dichotomous choice referendum question truthfully if two conditions hold. First, the respondent must believe that the results of the survey might influence an outcome they care about, a condition we refer to as policy consequentiality. Second, the respondent must perceive that there is some probability that they will have to pay, or payment consequentiality. Jointly, these two conditions (strong consequentiality) yield a knife-edge implication; i.e., as long as the respondent believes that their answers will be consequential in both senses with any positive probability, their dominant strategy is to answer truthfully. If accurate, this result has clear and important implications regarding how a researcher should handle survey responses: all data arising from respondents who believe the survey is at least minimally consequential can be assumed to provide truthful answers to survey questions.¹

In this paper, we focus our attention on the role that policy consequentiality has on the responses to a dichotomous choice referendum question and the efficacy of an information treatment designed to increase the perceived degree of consequentiality.²${}^{,}$³ Specifically, we use respondents’ perceptions of consequentiality elicited in the 2005 Iowa Lakes Survey to determine whether individuals have different perceptions concerning the degree of consequentiality of the valuation exercise and whether these perceptions affects respondents’ willingness to pay (WTP).⁴ In the survey, respondents were asked whether they would vote in favor of a referendum to improve water quality at a lake where bid values were varied across the sample. Respondents were also asked to answer, on a scale from 1 to 5, how likely it was that the survey results would influence decisions in the state concerning water quality programs. Thus, a measure of the degree to which respondents perceived the survey as policy consequential was directly elicited. Based on the CG arguments, respondents who do not believe that the survey is consequential could be omitted from the sample for estimation purposes. Additionally, the distributions of WTP from respondents with differing views concerning the degree of consequentiality could be tested for equality.

To extract accurate estimates of the impact of consequentiality perceptions on WTP, it is important to recognize that respondents who indicate a high degree of consequentiality may do so because they also place a high value in the proposed water quality improvement project. In other words, there is a potential endogeneity, or unobserved confounding problem. To address this concern, a split sample treatment was administered in the survey. Specifically, half of the sample was provided with a highlighted article from the Iowa Conservationist—the magazine of the Iowa Department of Natural Resources (IDNR), the state agency with primary responsibility for water quality control—indicating that IDNR was already using results from the survey in their policy decisions and planned to continue to do so. Our assumption, which is borne out empirically, is that the presentation of this information will positively affect the respondents’ perceived degree of consequentiality. This exogenous treatment aids us in estimating the “causal” impacts of consequentiality perceptions on WTP, as we will describe below.

Making use of the information treatment in the survey, we can then explore the impact that perceived policy consequentiality has on willingness to pay within the framework of a standard triangular treatment-response model. We proceed using a Bayesian approach and derive and employ a new algorithm that improves upon standard estimation methods, and can be applied by other practitioners seeking to fit models with a similar structure. Specifically, since our consequentiality responses are ordinal, our model must contend with the estimation of cutpoint values, and it is well-documented in the literature that standard Gibbs sampling schemes in such models can suffer from very poor mixing, particularly in moderately large data sets, thus producing imprecise and potentially inaccurate posterior inference (e.g., [3], [4]).⁵ Our proposed posterior simulator offers significant improvements by sampling the cutpoints, latent willingness to pay and latent consequentiality variables in a single step rather than sampling each component from its corresponding complete posterior conditional distribution.

Finally, before proceeding, it is important to clarify what implications can be drawn from the analysis below. Our survey gathers information as to whether respondents believe the results of the survey “$\dots$ will affect decisions about water quality in Iowa Lakes.” This is, at a minimum, a measure of the perceived policy conquentiality of the survey. As such, our econometric analysis identifies the extent to which responses to a dichotomous choice CV question differ as these perceptions differ. Drawing additional implications become difficult. If respondents also assume that the statement carries with it an implied obligation to pay, then we have a measure of strong consequentiality and Carson and Grove's theoretical results would suggest that all respondents with a perceived consequentiality of 2 or greater would respond truthfully. This is consistent with our findings below that WTP is only different for those individuals who perceive the survey to be completely inconsequential. However, as the survey does not directly ask individuals whether they perceive the survey carries with it a potential obligation to pay, the results are open to interpretation. As one reviewer suggests, if no one believes that they will have to pay, then those perceiving the survey to be consequential have an incentive to respond strategically, indicating a higher willingness to pay assuming that they will never have to pay. We leave it to the reader to draw their own conclusions.