Skip to main content
Log in

The Use (and Abuse) of Meta-Analysis in Environmental and Natural Resource Economics: An Assessment

  • Published:
Environmental and Resource Economics Aims and scope Submit manuscript

Abstract

Motivated by the 2006 report of a Work Group appointed by the Environmental Protection Agency (EPA), this paper examines the present state of meta-analysis in environmental economics and offers recommendations for its future use. To this end we summarize and assess 140 meta-analyses from 125 published and unpublished studies, covering 17 topical categories in environmental and resource economics. First, we provide several generic meta-analysis models as reference points and discuss major estimation issues. Five econometric issues are identified as part of a complete analysis: (1) sample selection criteria; (2) basic data summary; (3) primary data heterogeneity; (4) heteroskedasticity; and (5) non-independence of multiple observations from primary studies. Second, a tabular summary is presented for the 140 meta-analyses with respect to estimation methods. Third, a narrative summary is presented for 19 meta-analyses, including the three value-of-statistical-life studies examined by the EPA Work Group and one analysis from each of 16 other categories. Fourth, we offer a set of “best practice” guidelines for future meta-analyses in this and other areas of economics. Last, the paper comments on the use of meta-analytic methods for benefit transfers of environmental values.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Aigner DJ (1973) Regression with a binary independent variable subject to errors of observation. J Econom 1: 49–60

    Article  Google Scholar 

  • Asenso-Boadi F, Peters TJ, Coast J (2008) Exploring differences in empirical time preference rates for health: an application of meta-regression. Health Econ 17: 235–248

    Article  Google Scholar 

  • Baltagi BH (2005) Econometric analysis of panel data, 3rd edn. Wiley, New York

    Google Scholar 

  • Banzhaf HS, Smith VK (2007) Meta-analysis in model implementation: choice sets and the valuation of air quality improvements. J Appl Econom 22: 1013–1031

    Article  Google Scholar 

  • Bateman IJ, Jones AP (2003) Contrasting conventional with multi-level modeling approaches to meta-analysis: expectation consistency in UK woodland recreation values. Land Econ 79: 235–258

    Article  Google Scholar 

  • Bellavance F, Dionne G, Lebeau M (2007) The value of a statistical life: a meta-analysis with a mixed effects regression model. Working paper 06–12, Ecole des HEC Montreal

  • Bergstrom JC, Taylor LO (2006) Using meta-analysis for benefits transfer: theory and practice. Ecol Econ 60: 351–360

    Article  Google Scholar 

  • Berk R (2007) Statistical inference and meta-analysis. J Exp Criminol 3: 247–270

    Article  Google Scholar 

  • Bickel R (2007) Multilevel analysis for applied research. Guilford Press, New York

    Google Scholar 

  • Bijmolt THA, Pieters RGM (2001) Meta-analysis in marketing when studies contain multiple measurements. Mark Lett 12: 157–169

    Article  Google Scholar 

  • Bom PRD, Ligthart JE (2008) How productive is public capital? a meta-analysis. Unpublished paper, Tilburg University

  • Brander LM, Florax RJGM, Vermaat JE (2006) The empirics of wetland valuation: a comprehensive summary and a meta-analysis of the literature. Environ Resource Econ 33: 223–250

    Article  Google Scholar 

  • Breusch TS, Mizon GE, Schmidt P (1989) Efficient estimation using panel data. Econometrica 57: 695–700

    Article  Google Scholar 

  • Brons M, Nijkamp P, Pels E, Rietveld P (2008) A meta-analysis of the price elasticity of gasoline demand: a SUR approach. Energy Econ 30: 2015–2022

    Article  Google Scholar 

  • Brouwer R, Langford IH, Bateman IJ, Turner RK (1999) A meta-analysis of wetland contingent valuation studies. Reg Environ Chang 1: 47–57

    Article  Google Scholar 

  • Brown M, Forsythe A (1992) Robust tests for the equality of variances. J Am Stat Assoc 69: 364–367

    Article  Google Scholar 

  • Cameron AC, Trivedi PK (2005) Microeconometrics: methods and applications. Cambridge University Press, Cambridge

    Google Scholar 

  • Cavlovic TA, Baker KH, Berrens RP, Gawande K (2000) A meta-analysis of environmental Kuznets curve studies. Agric Resour Econ Rev 29: 32–42

    Google Scholar 

  • Christensen P (2003) Topics in meta-analysis: a literature survey. Institute of Transport Economics, Oslo

    Google Scholar 

  • Cooper H, Hedges LV (1994) The handbook of research synthesis. Russell Sage Foundation, New York

    Google Scholar 

  • Dalhuisen JM, Florax RJGM, de Groot HLF, Nijkamp P (2003) Price and income elasticities of residential water demand: a meta-analysis. Land Econ 79: 292–308

    Article  Google Scholar 

  • Daniel VE, Florax RJGM, Rietveld P (2007) Flooding risk and housing values: an economic assessment of environmental hazard. Working paper 07–02, Purdue University

  • Day B (1999) A meta-analysis of wage-risk estimates of the value of statistical life. In: EC, Benefits transfer and the economic valuation of environmental damages, European Commission, Brussels, pp 1–29

  • de Blaeij A, Florax RJGM, Rietveld P, Verhoef E (2003) The value of statistical life in road safety: a meta-analysis. Accid Anal Prev 35: 973–986

    Article  Google Scholar 

  • Debrezion G, Pels E, Rietveld P (2007) The impact of railway stations on residential and commercial property value: a meta-analysis. J Real Estate Fin Econ 35: 161–80

    Article  Google Scholar 

  • Desvousges WH, Johnson FR, Banzhaf HS (1998) Environmental policy analysis with limited information: principles and applications of the transfer method. Elgar, Cheltenham

    Google Scholar 

  • Elvik R (2005) Can we trust the results of meta-analyses. Transp Res Rec 1909: 221–29

    Article  Google Scholar 

  • Espey M (1996) Explaining the variation in elasticity estimates of gasoline demand in the United States. Energy J 17: 49–60

    Google Scholar 

  • Espey JA, Espey M (2004) Turning on the lights: a meta-analysis of residential electricity demand elasticities. J Agric Appl Econ 36: 65–81

    Google Scholar 

  • Feld LP, Heckemeyer JH (2008) FDI and taxation: a meta-study. Unpublished paper, University of Heidelberg and ZEW Mannheim

  • Florax RJGM (2002a) Accounting for dependence among study results in meta-analysis: methodology and applications to the valuation and use of natural resources. Research memorandum 2002–2005, Free University of Amsterdam

  • Florax RJGM (2002b) Methodological pitfalls in meta-analysis: publication bias. In: Florax RJGM, Nijkamp P, Willis KG (eds) Comparative environmental assessment. Elgar, Cheltenham, pp 177–207

    Google Scholar 

  • Florax RJGM, Nijkamp P, Willis KG (2002) Comparative environmental assessment. Elgar, Cheltenham

    Google Scholar 

  • Florax RJGM, Travisi CM, Nijkamp P (2005) A meta-analysis of the willingness to pay for reductions in pesticide risk exposure. Eur Rev Agric Econ 32: 441–467

    Article  Google Scholar 

  • Gelman A, Hill J (2007) Data analysis using regression and multilevel/hierarchical models. Cambridge University Press, Cambridge

    Google Scholar 

  • Ghermandi A, van den Bergh JCJM, Brander LM, de Groot HLF, Nunes PALD (2007) Exploring diversity: a meta-analysis of wetland conservation and creation. Working paper, Free University of Amsterdam

  • Glass GV (1976) Primary, secondary, and meta-analysis of research. Educ Res 10: 3–8

    Google Scholar 

  • Goldfarb RS, Stekler HO (2002) Meta-analysis. J Econ Perspect 16: 225–226

    Google Scholar 

  • Greene WH (2008) Econometric analysis, 6th edn. Pearson Prentice Hall, Upper Saddle River

    Google Scholar 

  • Hausman JA, Taylor WE (1981) Panel data and unobservable individual effects. Econometrica 49: 1377–1398

    Article  Google Scholar 

  • Hedges LV (1992) Meta-analysis. J Educ Stat 17: 279–296

    Article  Google Scholar 

  • Hedges LV (1994) Fixed effects models. In: Cooper H, Hedges LV (eds) The handbook of research synthesis. Russell Sage Foundation, New York, pp 285–299

    Google Scholar 

  • Hedges LV, Olkin I (1985) Statistical methods for meta-analysis. Academic Press, New York

    Google Scholar 

  • Higgins JPT, Thompson SG (2002) Quantifying heterogeneity in a meta-analysis. Stat Med 21: 1539–1558

    Article  Google Scholar 

  • Hoehn JP (2006) Methods to address selection effects in the meta regression and transfer of ecosystem values. Ecol Econ 60: 389–398

    Article  Google Scholar 

  • Horowitz JK, McConnell KE (2002) A review of WTA/WTP studies. J Environ Econ Manag 44: 426–447

    Article  Google Scholar 

  • Hox JJ, de Leeuw ED (2003) Multilevel models for meta-analysis. In: Reise SP, Duan N (eds) Multilevel modeling: methodological advances and applications. Lawrence Erlbaum, Mahwah, pp 90–111

    Google Scholar 

  • Hunter JE, Schmidt FL (2004) Methods of meta-analysis: correcting error and bias in research findings. Sage, Thousand Oaks

    Google Scholar 

  • Jarrell SB, Stanley TD (1999) A meta-analysis of the union wage gap. Ind Labor Relat Rev 44: 54–67

    Article  Google Scholar 

  • Jeppesen T, List JA, Folmer H (2002) Environmental regulations and new plant location decisions: evidence from a meta-analysis. J Reg Sci 42: 19–49

    Article  Google Scholar 

  • Johnson FR, Fries EE, Banzhaf HS (1997) Valuing morbidity: an integration of the willingness-to-pay and health-status index literatures. J Health Econ 16: 641–665

    Article  CAS  Google Scholar 

  • Johnston RJ, Besedin EY, Wardwell RF (2003) Modeling relationships between use and nonuse value for surface water quality: a meta-analysis. Water Resour Res 39: 1363–1371

    Article  Google Scholar 

  • Johnston RJ, Besedin EY, Iovanna R, Miller CJ, Wardwell RF, Ranson MH (2005) Systematic variation in willingness to pay for aquatic resource improvements and implications for benefit transfer: a meta-analysis. Can J Agric Econ 53: 221–248

    Article  Google Scholar 

  • Johnston RJ, Ranson MH, Besedin EY, Helm EC (2006) What determines willingness to pay per fish? a meta-analysis of recreational fishing values. Mar Resour Econ 21: 1–32

    Google Scholar 

  • Kennedy PE (1981) Estimation with correctly interpreted dummy variables in semilogarithmic equations. Am Econ Rev 71: 802

    Google Scholar 

  • Kennedy PE (2002) Sinning in the basement: what are the rules? the ten commandments of applied econometrics. J Econ Surv 16: 569–589

    Article  Google Scholar 

  • Kennedy PE (2008) A guide to econometrics, 6th edn. Blackwell, Malden

    Google Scholar 

  • Kiel KA, Williams M (2007) The impact of Superfund sites on local property values: are all sites the same.J Urban Econ 61: 170–192

    Article  Google Scholar 

  • Kochi I, Hubbell B, Kramer R (2006) An empirical Bayes approach to combining and comparing estimates of the value of a statistical life for environmental policy analysis. Environ Resource Econ 34: 385–406

    Article  Google Scholar 

  • Koetse MJ, Florax RJGM, de Groot HLF (2005) Correcting for primary study misspecification in meta-analysis. Tinbergen Institute Discussion Paper TI 2005-029/3, Free University of Amsterdam

  • Koetse MJ, Florax RJGM, de Groot HLF (2007) The impact of effect size heterogeneity on meta-analysis: a Monte Carlo experiment. Tinbergen Institute Discussion Paper TI 2007-052/3, Free University of Amsterdam

  • Koetse MJ, de Groot HLF, Florax RJGM (2008) Capital-energy substitution and shifts in factor demand: a meta-analysis. Energy Econ 30: 2236–2257

    Article  Google Scholar 

  • Lindhjem H, Navrud S (2008) How reliable is meta-analysis for international benefit transfers?. Ecol Econ 66: 425–435

    Article  Google Scholar 

  • Lipsey MW, Wilson DB (2001) Practical meta-analysis. Sage, Thousands Oaks

    Google Scholar 

  • Loomis JB, White DS (1996) Economic benefits of rare and endangered species: summary and meta-analysis. Ecol Econ 18: 197–206

    Article  Google Scholar 

  • Messer KD, Schultze WD, Hackett KF, Cameron TA, McClelland GH (2006) Can stigma explain large property value losses? psychology and economics of superfund. Environ Resource Econ 33: 299–324

    Article  Google Scholar 

  • Moeltner K, Boyle KJ, Paterson RW (2007) Meta-analysis and benefit transfer for resource valuation–addressing classical challenges with Bayesian modeling. J Environ Econ Manag 53: 250–269

    Article  Google Scholar 

  • Moulton BR (1990) An illustration of a pitfall in estimating the effect of aggregate variables on microeconomic units. Rev Econ Stat 72: 334–338

    Article  Google Scholar 

  • Mrozek JR, Taylor LO (2002) What determines the value of life? a meta-analysis. J Policy Anal Manag 21: 253–270

    Article  Google Scholar 

  • Mulatu A, Florax RJGM, Withagen CA (2003) Environmental regulation and competitiveness: an exploratory meta-analysis. In: Böhringer C, Löschel A (eds) Empirical modeling of the economy and the environment. Physica Verlag, Berlin, pp 23–54

    Google Scholar 

  • Murphy JJ, Allen PG, Stevens TH, Weatherhead D (2005) A meta-analysis of hypothetical bias in stated preference valuation. Environ Resource Econ 30: 313–325

    Article  Google Scholar 

  • Nelson JP (2004) Meta-analysis of airport noise and hedonic property values: problems and prospects. J Transp Econ Policy 38: 1–28

    Google Scholar 

  • Oltmer K, Nijkamp P, Florax R, Brouwer F (2000) A meta-analysis of environmental impacts of agri-environmental policies in the European Union. Tinbergen Institute Discussion Paper TI 2000-058/3, Free University of Amsterdam

  • Patuelli R, Nijkamp P, Pels E (2005) Environmental tax reform and the double dividend: a meta-analytical performance assessment. Ecol Econ 55: 564–583

    Article  Google Scholar 

  • Poe GL, Boyle KJ, Bergstrom JC (2001) A preliminary meta analysis of contingent values for ground water quality revisited. In: Bergstrom JC, Boyle KJ, Poe GL (eds) The economic value of water quality. Elgar, Cheltenham, pp 137–162

    Google Scholar 

  • Quantitative Micro Software (2007) EViews 6.0 User’s Guide, Part II. QMS, Irvine

  • Rabe-Hesketh S, Skrondal A (2008) Multilevel and longitudinal modeling using Stata, 2nd edn. Stata Press, College Station

    Google Scholar 

  • Raudenbush SW (1994) Random effects models. In: Cooper H, Hedges LV (eds) The handbook of research synthesis. Russell Sage Foundation, New York, pp 301–321

    Google Scholar 

  • Raudenbush SW, Bryk AS (2002) Hierarchical linear models: applications and data analysis methods, 2nd edn. Sage, Thousand Oaks

    Google Scholar 

  • Ringquist EJ (2005) Assessing evidence of environmental inequities: a meta-analysis. J Policy Anal Manag 24: 223–247

    Article  Google Scholar 

  • Roberts CJ, Stanley TD (2005) Meta-regression analysis: issues of publication bias in economics. Blackwell Publishing, Oxford

    Google Scholar 

  • Rosenberger RS, Johnston RJ (2008) Selection effects in meta-analysis and benefit transfer: avoiding unintended consequences. Land Econ (in press)

  • Rosenberger RS, Loomis JB (2000a) Using meta-analysis for benefit transfer: in-sample convergent validity tests of an outdoor recreation database. Water Resour Res 36: 1097–1107

    Article  Google Scholar 

  • Rosenberger RS, Loomis JB (2000b) Panel stratification in meta-analysis of economic studies: an investigation of its effects in the recreation valuation literature. J Agric Appl Econ 32: 459–470

    Google Scholar 

  • Rosenberger RS, Stanley TD (2007) Publication bias in the recreation use value literature: a preliminary investigation. Working paper, Oregon State University

  • Rothstein HR, Sutton AJ, Borenstein M (2005) Publication bias in meta-analysis: prevention, assessment and adjustments. Wiley, New York

    Book  Google Scholar 

  • Santos JML (1998) The economic valuation of landscape changes: theory and policies for land use and conservation. Elgar, Cheltenham

    Google Scholar 

  • Saxonhouse GR (1976) Estimated parameters as dependent variables. Am Econ Rev 66: 178–183

    Google Scholar 

  • Schulze R (2004) Meta-analysis: a comparison of approaches. Hogrefe & Huber, Cambridge

    Google Scholar 

  • Smith VK, Huang J-C (1993) Hedonic models and air pollution: twenty-five years and counting. Environ Resource Econ 3: 381–94

    Article  Google Scholar 

  • Smith VK, Huang J-C (1995) Can markets value air quality? a meta-analysis of hedonic property value models. J Polit Econ 103: 209–227

    Article  Google Scholar 

  • Smith VK, Kaoru Y (1990a) What have we learned since Hotelling’s letter? a meta-analysis. Econ Lett 32: 267–272

    Article  Google Scholar 

  • Smith VK, Kaoru Y (1990b) Signals or noise? Explaining the variation in recreation benefit estimates. Am J Agric Econ 72: 419–433

    Article  Google Scholar 

  • Smith VK, Osborne LL (1996) Do contingent valuation estimates pass a ‘scope’ test? a meta-analysis. J Environ Econ Manag 31: 287–301

    Article  Google Scholar 

  • Smith VK, Pattanayak SK (2002) Is meta-analysis a Noah’s Ark for non-market valuation?. Environ Resource Econ 22: 271–296

    Article  Google Scholar 

  • Stanley TD (2001) Wheat from chaff: meta-analysis as quantitative literature review. J Econ Perspect 15:131–150

    Article  Google Scholar 

  • Stanley TD (2005) Beyond publication bias. J Econ Surv 19: 309–345

    Google Scholar 

  • Stanley TD (2008) Meta-regression methods for detecting and estimating empirical effects in the presence of publication selection. Oxf Bull Econ Stat 70: 103–127

    Google Scholar 

  • Stanley TD, Jarrell SB (1989) Meta-regression analysis: a quantitative method of literature surveys. J Econ Surv 3: 161–170

    Article  Google Scholar 

  • StataCorp (2007) Stata statistical software, release 10, base reference manual, vol 3. Stata Press, College Station

  • Sutton AJ, Abrams KR, Jones DR, Sheldon TA, Song F (2000) Methods for meta-analysis in medical research. Wiley, Chichester

    Google Scholar 

  • U.S. Environmental Protection Agency (2000) Guidelines for preparing economic analyses, EPA 240-R-00-003. EPA, Washington, DC

  • U.S. Environmental Protection Agency (2006) Report of the EPA work group on VSL meta-analysis, report NCEE-0494. National Center for Environmental Economics, EPA, Washington, DC. http://yosemite.epa.gov/ee/epa/eerm.nsf/vwRepNumLookup/EE-0494?OpenDocument

  • U.S. Environmental Protection Agency (2007) SAB advisory on EPA’s issues in valuing mortality risk reduction, EPA-SAB-08-001. National Center for Environmental Economics, EPA, Washington, DC

  • van den Bergh JCJM, Button KJ, Nijkamp P, Pepping GC (1997) Meta-analysis in environmental economics. Kluwer, Dordrecht

    Google Scholar 

  • Van Garderen KJ, Shah C (2002) Exact interpretation of dummy variables in semilogarithmic equations with estimation uncertainty. Econom J 5: 149–159

    Article  Google Scholar 

  • Van Houtven G, Powers J, Pattanayak SK (2007) Valuing water quality improvements in the United States using meta-analysis: is the glass half-full or half-empty for national policy analysis. Resour Energy Econ 29: 206–228

    Article  Google Scholar 

  • van Kooten GC, Eagle AJ, Manley J, Smolak T (2004) How costly are carbon offsets? a meta-analysis of carbon forest sinks. Environ Sci Policy 7: 239–251

    Article  Google Scholar 

  • van Kooten GC, Laaksonen-Craig S, Wang Y (2007) Costs of creating carbon offsets via forestry activities: a meta- regression analysis. Working paper 2007-2003, University of Victoria

  • Viscusi WK, Aldy JE (2003) The value of a statistical life: a critical review of market estimates throughout the world. J Risk Uncertain 27: 5–76

    Article  Google Scholar 

  • Walsh RG, Johnson DM, McKean JR (1989) Issues in nonmarket valuation and policy applications: a retrospective glance. West J Agric Econ 14: 178–188

    Google Scholar 

  • Walsh RG, Johnson DM, McKean JR (1990) Nonmarket values from two decades of research on recreation demand. In: Link A, Smith VK (eds) Advances in applied microeconomics. JAI Press, Greenwich, pp 167–193

    Google Scholar 

  • Weitzman ML, Kruse DL (1990) Profit sharing and productivity. In: Blinder A (eds) Paying for productivity: a look at the evidence. Brookings Institution, Washington, DC, pp 95–140

    Google Scholar 

  • Wood JA (2008) Methodology for dealing with duplicate study effects in a meta-analysis. Organ Res Methods 11: 79–95

    Article  Google Scholar 

  • Wooldridge JM (2003) Cluster-sample methods in applied econometrics. Am Econ Rev Pap Proc 93: 133–138

    Article  Google Scholar 

  • Zamparini L, Reggiani A (2007) Meta-analysis and the value of travel time savings: a transatlantic perspective in passenger transport. Netw Spat Econ 7: 377–96

    Article  Google Scholar 

  • Ziliak ST, McCloskey DN (2008) The cult of statistical significance. University of Michigan Press, Ann Arbor

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Jon P. Nelson.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Nelson, J.P., Kennedy, P.E. The Use (and Abuse) of Meta-Analysis in Environmental and Natural Resource Economics: An Assessment. Environ Resource Econ 42, 345–377 (2009). https://doi.org/10.1007/s10640-008-9253-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10640-008-9253-5

Keywords

JEL Classification

Navigation