Abstract
Recent empirical and theoretical research stresses it is important for survey respondents to believe that survey votes are consequential, meaning their votes can potentially influence whether a proposed policy is undertaken. We test the effect of a randomly assigned referendum tax on consequentiality, using a survey about water conservation in western North Carolina. We find that consequentiality is endogenous to hypothetical referendum responses. Specifically, as the assigned tax amount increases, respondents are less likely to find the survey consequential. As in related studies, respondents who self-report they perceive the survey to be consequential have a higher willingness to pay. (JEL Q25, Q51)
I. INTRODUCTION
Hypothetical bias occurs when there is a divergence between behavioral intentions elicited in a survey setting and actual behavior. Debate continues around the accuracy of the contingent valuation method (CVM), with hypothetical bias being one of the major issues. In 2012, the Journal of Economic Perspectives featured a symposium on the CVM. Kling, Phaneuf, and Zhao (2012) provided an overview of the method and its development, concluding that when well designed, the CVM can provide important policy insights. Carson (2012) agreed, arguing that the CVM is “a practical alternative when prices aren’t available.” In stark contrast, Hausman’s (2012) opinion on CVM went from “dubious to hopeless” in the ability of the CVM to accurately measure value. One of Hausman’s (2012) issues with the CVM is “hypothetical response bias that leads contingent valuation to overstatements of value.” Several meta-analyses compare value estimates from hypothetical and real choices. List and Gallet (2001), Little and Berrens (2004), and Murphy et al. (2005) find that values based on hypothetical choices are about 1.35 to 3 times higher than those based on real choices in experimental settings. Carson, Groves, and List (2014), however, suggest that the hypothetical bias critique is overstated, considering the inconsequential experimental settings where most of the tests of hypothetical bias have been attempted.
Carson and Groves (2007, 2011) argue that field survey respondents are likely to perceive the elicitation to be “consequential” if they believe their responses potentially influence policy and the policy agent cares about the subsequent survey outcomes. With consequentiality and an appropriately designed elicitation mechanism, it is theoretically possible for a survey elicitation to be incentive compatible in the sense that respondents have incentives to truthfully reveal their preferences. There is mounting empirical evidence from laboratory and field experiments that consequential questions are not prone to hypothetical bias (Landry and List 2007; Vossler and Evans 2009; Vossler and Poe 2011; Vossler, Doyon, and Rondeau 2012; Carson, Groves, and List 2014).1
Theoretical and empirical research has found a “knife-edge” result; laboratory and field experiment behavior, when the probability of a real outcome is nonzero, is similar to behavior when the probability of a real outcome is one. When the probability of a real outcome is zero in the hypothetical setting, hypothetical and real behaviors diverge. This suggests that hypothetical behavior will be similar to real behavior if there is a positive chance that the hypothetical behavior will have real consequences.
While the lack of consequentiality helps explain why laboratory and field experiments exhibit hypothetical bias for private goods or voluntary contributions, it is not necessarily an explanation for hypothetical bias in referendum contingent valuation surveys. First, consequential CVM surveys are expected to be more accurate, but there are no predictions on the direction of bias for inconsequential surveys. Inconsequential stated preferences may be understated if respondents answer with “protest no’s” or overstated if respondents “yea say.” In two CVM applications, Herriges et al. (2010) and Vossler and Watson (2013) ask a follow-up question to determine if respondents consider the survey to be consequential. They find that respondents who believe the survey results are inconsequential are less likely to support the policy.
Interis and Petrolia (2014) further explore the effects of consequentiality in binary and multiple discrete choice experiment questions. Interis and Petrolia (2014) do not find the knife-edge result with a binary discrete choice experiment but do with a multiple discrete choice experiment question. Willingness to pay is greatest for respondents who believe it is very likely that policy makers will take survey results into consideration, and lowest when respondents think that this is unlikely. In addition, they find that respondents who believed the survey was inconsequential were less sensitive to scope effects. Vossler, Doyon, and Rondeau (2012) also find that willingness to pay is a function of the level of consequentiality perceptions. Vossler and Watson (2013) conduct sensitivity analysis of their results by incorporating consequentiality in the empirical willingness-to-pay model. A dummy variable indicating respondents who find the survey to be inconsequential has a negative effect on willingness to pay. Deleting respondents who find the survey to be inconsequential increases the theoretical validity of the willingness-to-pay model. Both Interis and Petrolia (2014) and Vossler and Watson (2013) find evidence to support an important implication of Carson and Groves’s (2007) results: respondents who perceive the survey to be inconsequential may not care about the outcome of the survey, so they have little reason to invest in well thought out responses.
In this paper we contribute to the literature in several important ways. First, we provide another test for the effects of perceived consequentiality on willingness to pay and differences across covariates under different levels of perceived consequentiality. Vossler and Watson (2013) consider the determinants of consequentiality. Herriges et al. (2010), Interis and Petrolia (2013), and Vossler and Watson (2013) consider whether consequentiality perceptions are endogenous. We test for the determinants of consequentiality and consider endogeneity by estimating a joint bivariate probit model of consequentiality and willingness to pay.
No study to date has considered the endogeneity of the randomly assigned tax on consequentiality in a hypothetical referendum. This is a potentially important omission, as stated cost has been found to affect respondent perceptions. For instance, Groothuis and Miller (1997) find that the tax amount influenced people’s perception of risk, and Carson and Groves (2007) find that implausibly high or low amounts can reduce the scenario’s credibility. We include the stated cost as a determinant in the consequentiality model. We find that individuals who received a high stated cost perceive that the referendum is less likely to pass, rendering the referendum question inconsequential. In addition, we speculate that respondents might believe that policy makers, if costs are high, might be less supportive of the policy. This would then weaken the influence of the survey and/or mean that the bar is raised (e.g., overwhelming citizen support is needed for implementation). We also suggest that the endogeneity of consequentiality has implications for the measurement of willingness to pay.
II. THEORETICAL MODEL
The literature does not discuss a theoretical link between the randomly assigned stated cost (e.g., tax) that the respondent is asked to pay and the consequentiality of the survey. We suggest that higher tax values can lead subjects to perceive the survey to be less consequential, and our empirical results support this conjecture. While our goal in this paper is not to provide a formal proof of the conditions under which this result is expected, we provide a simple model to illustrate how the tax value may affect respondents’ beliefs about consequentiality.
Carson, Groves, and List (2014), hereafter CGL, suggest “a testable implication of [Carson and Grove’s (2007)] framework is that the fraction of people who favor a policy action in the population of interest should be invariant to the probability that the survey will influence the decision to provide the public good under the specified terms as long as that probability is positive” (p. 173). A number of ballot rules can be considered for the referendum as long as the probability a project is pursued is increasing in the number of votes in favor of the project and the probability of pursuing the project given any votes in favor is greater than the probability when no one votes in favor.
In addition to CGL, Cummings and Taylor (1998), Landry and List (2007), and Vossler and Evans (2009) have varied the probability of the vote being binding and examined the resulting distribution of yes votes.2 The stochastic treatments in these experimental designs can be modeled as a two-stage lottery where subjects first vote on whether or not to provide (and pay for) the good and then find out if their votes will be binding. The certainty treatments in these experiments are equivalent to a real, incentive compatible referendum wherein the results are binding as long as the appropriate plurality is met. In the stochastic treatments, subjects knew the probability the vote would be binding; this can be compared to an advisory referendum in which voters know the likelihood the good will be provided, subject to the plurality being met.
Modeling the voting process as a two-stage lottery can provide some insights into the importance of the tax size on perceptions of consequentiality. The compound lottery involves (1) the probability the referendum passes (either with a majority or a specified plurality) and (2) the probability the referendum will be binding given that it passes. We call (1) P(majority), or P(m), and the conditional probability in (2) p(binding I majority), or P(b | m). This notation is convenient, but we note that “majority” can be replaced with “plurality” or even “threshold.” Moreover, if the referendum is advisory, “binding” can be replaced with “influential” without loss of generality. The model can also allow for the fact that the ballot rules are such that the likelihood the vote is binding is increasing in the number of those who vote in favor.
Assuming the decision-maker has the power to coerce payment if the referendum is binding, then (1) and (2) form the joint probability P(m ⋂ b) = P(m)P(b | m). In a “real” binding referendum P(b | m) = 1, so P(m ⋂ b) = P(m). In other words, the probability the referendum both passes and is binding is just the probability of a majority vote in favor.
Let superscript R denote the real referendum and S denote the survey referendum. In the real referendum P(m ⋂ b) = P(m)R. In the survey referendum P(m ⋂ b) = P(m)S P(b | m)S. CGL suggest that the percentage voting in favor should be invariant to the probability of the vote being binding, as long as that probability is positive. Using our notation this implies that for P(b | m)S > 0, !
Thus, the subjective probability of the survey referendum passing equals the subjective probability of the real referendum passing only when the probability of the survey referendum being binding (if passed) is certain. That is,
Alternatively, if the (subjective or real) probability of the referendum is binding is less than 1 it must be true that the subjective probability of a majority is higher in the survey referendum. That is,
The somewhat counterintuitive explanation here is that the lower the conditional probability the survey will be binding, the higher the subjective probability it will pass relative to a real referendum. This is in fact what Cummings and Taylor (1998) find in their experiment. On the other hand, CGL find support for their hypothesis that the percentage of yes votes is statistically the same in the stochastically binding and deterministically binding treatments.3
The compound lottery example also illustrates the link between the stated cost or tax and the respondents’ beliefs about the consequentiality of the survey. For example, a well-behaved willingness-to-pay function will be declining in the size of the stated cost. Hence, the probability of a yes vote decreases as the stated cost increases. As the probability of yes votes decreases, the subjective probability of a vote threshold being met must also decrease. Therefore, the probability of influencing policy decreases, and the probability of believing the referendum is consequential could also decrease. This is suggestive of an inverse relationship between the value of the stated cost and beliefs about the consequentiality of the referendum. This effect is a direct result of the impact of the stated cost’s magnitude on voters’ probability of voting in favor of the good or policy.
In our study, although we randomly vary the stated cost across respondents, there is no language in the survey that indicates that costs differ across households. Even in situations where such a belief over uniform payment may not hold (e.g., see Vossler and Watson 2013), our results should still follow. Vossler and Watson (2013) set the payment vehicle as a property tax increase that would be levied on the respondent’s taxable home value. One could interpret changes in the tax rate (1% vs. 1.5%, etc.) as equivalent to changes in the stated cost. A separate question is how knowledge of or beliefs about potentially differentiated payments affect the probability of a yes vote and hence the consequentiality of the survey. That question is outside the scope of this paper and a topic for future research.
III. WATER CONSERVATION SURVEY
We use data from a survey on water conservation measures in the mountains of western North Carolina. The survey of 51 questions, including demographics, was mailed in May 2013 to a random sample of 3,000 Watauga and Ashe County residents. It consisted of a primary mailing, a post card reminder, and a second mailing to all nonrespondents of the first mailing. In the end, 2,413 useable addresses and 591 responses were obtained, for a useable response rate of 25%.
Table 1 contains a summary of the demographic variables. The average age of respondents was 61 years and average income was $56,000. In the two counties of our sample, 24% of respondents have a high school degree or less, 18% have some college but no degree, 10% have an associate’s degree, 24% have a bachelor’s degree, and 24% have a graduate or professional degree. Comparing our sample to U.S. Census data from the counties, we find that about 23% of Watauga County residents (over age 20) and 35% of Ashe County residents (over age 20) are 60 or older; 38% of Watauga County and 19% of Ashe County residents have a college degree; average household income in Watauga County is about $52,000 and is about $47,000 in Ashe County. Therefore, our survey respondents tend to be older, slightly more educated, and have higher income than the general population in this area. In addition, we find that 50% report having ancestors who lived in this region. Regarding water source, 52% report having their own well, 12% their own spring, and 19% a shared well, and 17% are on a municipal water supply. The willingness-to-pay question for water conservation measures is a single binary choice framework (Carson and Louviere 2011). The variable for is a qualitative variable equal to one if the respondents answered for to the following referendum question:
Suppose that to implement water conservation measures county residents would pay a one-time payment of $A per household in higher county taxes. The money would be used to provide rebates to residents for the purchase of low-flow toilets or rain barrels to help save water at home. The money would also be used to revegetate creek banks and install permeable pavement where feasible. These measures reduce runoff from storms and help with recharging the groundwater supply. The goal of the program is to provide more water security in the county and to ensure a more stable water supply that can ease stress during droughts. Suppose that this proposal to approve the tax and provide conservation measures will be on the next election ballot. Remember, if the proposal passes you would make a one-time payment of $A in higher taxes and you would have $A less to spend on other things. Also remember that if the referendum passes the conservation measures would be implemented and more water would be available in your county during times of drought.
The tax amount variable $A took on the values of $5, $20, $40, $80, or $150.4 We asked respondents how they would vote on this proposal with three alternatives: for, against, or don’t know. We find that the frequency of respondents who would vote for falls from 60% at $5 to 30% at $150. About 18% of respondents answered don’t know over all values of $A. One problem that arises when coding single binary choice CVM questions is what should be done with don’t know responses. We code all don’t know responses as voting against the policy. This is supported empirically by Carson, Hanemann, et al. (1998), Groothuis and Whitehead (2002), and Caudill and Groothuis (2005).5
To test for the influence of consequentiality we use a follow up question to our contingent valuation question that is similar to that of Vossler and Watson (2013): “To what extent do you believe that the indicated votes on the above proposal from you and other survey participants will be taken in to consideration by county policy makers?” Possible responses ranged from 1 to 5, where respondents who stated 1 believe policy makers will definitely not take the information into account and 5 means respondents believe policy makers will definitely take the information into account.
In Table 2, we report the percentage of respondents who indicated they would vote for the proposal by both the bid level and the degree of consequentiality. The first column contains responses for those who think the referendum is inconsequential, and the degree of consequentiality increases to the right. We test for equal frequencies of for votes across the tax amounts and find that for C = 1 and C =2, as the bid increases the percentages of for votes are not statistically different. As the bid increases the percentage of for votes falls when C =3,C = 4, and C = 5 at the 95% level of confidence. Considering the total votes, the percentage of for votes increases with perceived consequentiality until the final option. Of the individuals who answered 1 to the consequentiality question, only 17% voted for in the referendum. Of those who answered 2 on the consequentiality statement, we find 43% answered for, and of those who answered 3, we find 46% voted for. The proportion of for votes rises to 73% for respondents who answered for to the consequentiality question, and falls back to 46% for votes among respondents who selected 5 (“definitely taken into account”) to the consequentiality question.
Mechanism theory suggests that a survey referendum can be incentive compatible under certain assumptions, including consequentiality (Carson and Groves 2007; Vossler, Doyon, and Rondeau 2012). The theory of consequentiality suggests that respondents who believe the incentive-compatible CVM question might influence policy will answer the questions as if they face real payments. Following the law of demand, the most basic empirical evidence that a referendum question is incentive compatible is that the probability of voting for the policy decreases as the tax amount increases. Using a chi-squared test, we find that for individuals who answered either 1 or 2 to the consequentiality question, the tax variable does not affect the proportion of for responses, suggesting these individuals do not find the CVM question incentive compatible. Past research suggests that a knifeedge result occurs when a survey is considered at least somewhat consequential relative to when the survey is completely inconsequential (Herriges et al. 2010; Vossler and Watson 2013). We find that a structural shift in elicited preferences occurs as the consequentiality question response goes from 2 to 3. Our interpretation differs from the interpretation of some of the past research in that we also consider incentive compatibility, and using this criterion we find the threshold for valid CVM responses is at a greater level of perceived consequentiality than that implied by the knife-edge result.
IV. WILLINGNESS TO PAY AND CONSEQUENTIALITY
To test the influence of consequentiality on willingness to pay, we provide several probit model specifications: where for is equal to 1 if a respondent said he would vote in favor of the referendum, tax is the randomly assigned tax amount, α0 is a constant, α1 is the coefficient on the log of the tax variable, and X is a vector of explanatory variables with corresponding coefficient vector δ. In the first column of Table 3 we report the results of the basic model, which includes no control for consequentiality. In this specification, we find that gender, education, and the respondent’s water source all influence the probability of voting for the water conservation policy. We also find that a respondent having had an ancestor in the region lowered the likelihood of voting for the water conservation policy. This is consistent with the results of Cockerill and Groothuis (2014). The coefficient on the tax amount is negative and significant. We estimate the median willingness to pay for conservation measures using the censored probit approach6 and find that the median is $11.96 (standard error = 3.26).7
In the second column, we control for consequentiality using a dummy variable, as do Vossler and Watson (2013). Following the results from Table 2, where we test for response validity at each level of consequentiality, if a respondent answered 3, 4, or 5 to the conse-quentiality question it was coded as one (C > 2 = 1) and if she answered 1 or 2 it was coded zero (C ≤ 2 = 0). We find that the sign and significance of all explanatory variables stay the same with a slight decrease in the magnitude on the ancestor coefficient, while the coefficient on the consequentiality dummy variable is positive and statistically significant. This suggests that when respondents find the CVM question consequential, they are more likely to vote for the policy. When respondents find the survey consequential, the median WTP estimate is $22.80 (standard error = 6.60). When respondents find the survey inconsequential, the willingness-to-pay estimate is $3.60 (standard error = 1.96).
To further test the influence of consequentiality, we split the sample into two groups based on the consequentiality dummy variable. In Table 3, column 3 we report the subsample of respondents who find the CVM question consequential and exhibit valid responses. We find that the coefficient on the tax amount is negative and statistically significant, and the magnitude of the coefficient increased relative to the basic model. This suggests that the tax amount has more influence on the likelihood of voting for the conservation measures. The willingness-to-pay estimate increases to $27.20 (standard error = 5.98). In the last column of Table 3 we report the results for the respondents who found the survey relatively inconsequential and find that the coefficient on the log of the tax amount is not statistically significant, as in Table 2. Overall, we consider these models as insightful but naïve because they do not treat the perceived consequentiality of the question as endogenous as theory suggests. The coefficient on the consequentiality dummy variable may be biased.
To correct for the potential for endogeneity bias, we estimate bivariate probit models on the likelihood of voting for the water conservation measure and the likelihood of respondents finding the survey consequential. We include the consequentiality dummy variable in the voting equation and an instrumental variable to identify the bivariate probit (Wilde 2000). Our instrument is a dummy variable equal to 1 if the respondent agreed or strongly agreed with the statement: “Local public officials (city/county) should have the final authority to make decisions about how our water supply is managed.”8 The negative coefficient on the instrument variable in the consequentiality model shows that respondents who believe that decisions should be made at the local level do not believe that the survey is consequential. One possible explanation for this result is that although respondents feel decisions should be made at the local level, they are not confident that decisions actually are or will be made at that level, making the survey inconsequential. Another potential explanation is that in situations where there is high trust in public officials, there is less perceived need to engage in policy or decision-making processes (Sabatier et al. 2005). If respondents prefer local control because they trust local officials, perhaps this extends to trusting those officials to take appropriate action regardless of the survey results. The instrumental variable is not a statistically significant determinant of the referendum votes.
To test for the influence of consequentiality, we estimate two bivariate probit models: the first does not include the log of the tax amount variable in the consequentiality equation, while the second does. The bivariate probit model provides a unique way to test for the impact of consequentiality on referendum votes. First, the coefficient on the tax amount captures the influence of the tax on the perception that the survey results will have an impact on policy. Our theory suggests that as the tax amount rises, the perceived level of consequentiality should fall (because the subjective probability of the referendum passing declines). Second, the coefficient on the consequentiality dummy variable in the referendum vote equation will capture the direct effect of the perception of consequentiality on the likelihood of voting for the referendum. Third, the rho (ρ) coefficient measures the correlation between error terms of the two equations.
Table 4 shows in both bivariate probits the coefficient on the consequential dummy variable is positive and significant, suggesting that the observed effect of a survey being consequential increases the likelihood of a for vote.9 The coefficient on the consequentiality dummy variable is three times greater than the coefficient in the similar model in Table 3.10 The rho coefficient is negative and significant in both specifications. This result suggests that there are some unobservable characteristics that decrease the likelihood of voting for the water conservation policy and increase the perception that the survey is consequential.
To test for the influence of the tax payment on consequentiality in the second model, we include the log of the tax amount in the consequentiality specification. Consistent with our theoretical predictions and Table 2, we find that high tax amounts increase the probability of inconsequential survey results. The other variables that determine the level of consequentiality are if a respondent had an ancestor in the county, had attended some college, or shared a well.
V. CONCLUSIONS
Consequentiality exists when the respondent believes that the results of the survey will influence the policy and when respondents perceive that they will be affected by the payment vehicle. Our analysis finds that consequentiality is endogenous to the hypothetical referendum question. Using a survey about water conservation measures, we find that as the tax amount increases, respondents are less likely to believe that the survey is consequential. Respondents might believe governments are less likely to implement policies with higher taxes. Survey researchers thus need to take care when framing the referendum because high tax payments may lower the credibility of the survey, making individuals less likely to find the survey consequential. In addition, as in previous research, we find that respondents who self-report that they perceive the survey to be consequential are more likely to be willing to pay positive amounts for the policy.
These results are important for improving stated preference research. Our work provides guidance for including consequentiality questions in any CVM study. Future studies that assess the effects of consequentiality should consider its determinants and whether including it as a determinant of willingness to pay is appropriate. Alternative wording of perceived consequentiality questions should also be explored, as well as differentiating between policy consequentiality and payment consequentiality. Our results also have implications for bid design in referendum surveys, as the tax amount was found to influence conse-quentiality. CVM researchers should work to find the balance between high tax amounts that better identify the willingness-to-pay distribution and those that negatively impact consequentiality. Follow-up debriefing questions could help identify these problem areas.
These results are also important for improving policy making. Specific to water conservation, our work can contribute to improved policy making, as it provides a higher level of confidence in public survey results about conservation management. As human populations continue to grow and climate change drives increased pressure on water resources, better understanding of public perceptions about water issues and management preferences will be valuable. This may be especially true in humid areas that have historically not faced water concerns and hence have not had explicit public discussion or debate about water management options. The CVM can be used to help water managers identify options that are most palatable to their constituents and thereby reduce conflict as they make potentially controversial decisions.
Acknowledgments
We would like to acknowledge Appalachian State University’s University Research Council for funding the study “Public Attitudes about Water Management in Western North Carolina,” as well as the Walker College of Business Dean Club Grant for additional funding. We would also like to thank anonymous referees for useful suggestions.
Footnotes
The authors are, respectively, professor, Department of Economics; associate professor, Department of Economics; professor, Department of Economics; and associate professor, Interdisciplinary Studies, Appalachian State University, Boone, North Carolina.
↵1 Consequentiality may also improve results when the hypothetical question is not incentive compatible. Bulte et al. (2005), using an implicit donation payment vehicle, find that a hypothetical question with a consequential script generates lower willingness-to-pay estimates than the hypothetical question without a consequential script.
↵2 Cummings and Taylor’s probability values are p = {0, 0.25, 0.5, 0.75, 1}. CGL use p = {0, 0.2, 0.5, 0.8, 1}. Landry and List (2007) also have a consequentiality treatment in which p = {0, 0.5, 1}. Their results are similar to CGL’s, and the good being valued (sports memorabilia) is similar.
↵3 CGL note in footnote 25 that the alternative hypothesis using a one-tail test is that the probabilistic treatments would have more yes votes. That is what this two-stage lottery model predicts and is what Cummings and Taylor (1998) find.
↵4 A pretest survey was developed in 2012 and administered in the Town of Boone, the largest town in the broader study area, and this generated 129 responses that were used to revise several of the survey questions and survey structure. Additionally, a group of 12 students at Appalachian State University served as a focus group that took the survey and provided feedback.
↵5 We find that the results do not change when we exclude don’t know responses from the data.
↵6 The censored probit approach involves dividing the sum of the coefficients evaluated at the means of the independent variables by the coefficient on the tax amount (Cameron and James 1987; Haab and McConnell 2002). Willingness to pay is the exponential of this sum when the natural log of the tax is used.
↵7 Mean willingness-to-pay estimates are very similar to the medians in the rest of this section. The median willingness-to-pay estimates are more straightforward to estimate with the log-linear functional form, so we present them in the paper.
↵8 We find that this dummy variable coefficient is statistically significantly in the consequentiality probit but insignificant in the willingness-to-pay probit.
↵9 We find that the results do not change in sign depending upon how we group consequentiality levels but are insignificant when including only individuals who answered 1 as inconsequential. In our analysis we use the cutoff at 2 because the results at C = 2 are not theoretically valid, as shown in Table 2 (although, given the overall level of votes, 46% for, the willingness to pay is likely about the same).
↵10 The willingness-to-pay estimate with the consequentiality dummy variable set equal to 1 is unrealistically high and statistically insignificant in this model. Our advice would be to use the willingness-to-pay estimate from the split sample model in Table 3 for policy analysis.