How reliable are meta-analyses for international benefit transfers?

Abstract

Meta-analysis has increasingly been used to synthesise the environmental valuation literature, but only a few test the use of these analyses for benefit transfer. These are typically based on national studies only. However, meta-analyses of valuation studies across countries are a potentially powerful tool for benefit transfer, especially for environmental goods where the domestic literature is scarce. We test the reliability of such international meta-analytic transfers, and find that even under conditions of homogeneity in valuation methods, cultural and institutional conditions across countries, and a meta-analysis with large explanatory power, the transfer errors could still be large. Further, international meta-analytic transfers do not on average perform better than simple value transfers averaging over domestic studies. Thus, we question whether the use of meta-analysis for practical benefit transfer achieves reliability gains justifying the increased effort. However, more meta-analytic benefit transfer tests should be performed for other environmental goods and other countries before discarding international meta-analysis as a tool for benefit transfer.

Introduction

Meta-analysis (MA) is now common in environmental economics and non-market valuation. Since Smith and Kaoru's (1990) seminal study of recreational benefits, MA has been conducted for a wide range of environmental goods, from wetlands (Woodward and Wui, 2001) to visibility (Smith and Osborne, 1996). Common to all of these studies is the focus on research synthesis and hypothesis testing, rather than on the more interesting policy question of how MA can be used to improve benefit transfer (BT) practices (Smith and Pattanayak, 2002). Meta-analytic benefit transfer (MA-BT) to unstudied sites (“policy sites”) has only been cursory treated in the literature, typically a few pages add-ons at the end of lengthy MA papers, although authors emphasise its potential importance for future research and applications, for example in cost–benefit analysis (see the special issue on BT in Ecological Economics, 2006, Van Houtven et al., 2007). While there is some knowledge of how unit value and value function-based BT from single studies perform (Rosenberger and Phipps, 2007), Bergstrom and Taylor (2006, pp. 359) point out that “before widespread application of MA-BT models, there is a need for additional MA-BT validity tests across different types of natural resources and environmental commodities.” Only a few studies have, to our knowledge, investigated the validity and reliability of MA-BT (Santos, 1998, Rosenberger and Loomis, 2000, Shrestha and Loomis, 2001, Shrestha and Loomis, 2003, Santos, 2007, Shrestha et al., 2007). Four of the studies, however, are based on the same large dataset of use values for different recreational activities in the USA, and are unable to cover the breadth of issues involved in more typical MA-BT exercises, i.e. limited datasets, complex goods with significant non-use values, different level of methodological heterogeneity and mix of international studies to mention a few. Santos (2007) is the only study attempting a comprehensive comparison of two versions of a domestic MA-BT with simple BT techniques often used in practice. Further, all the above studies can be said to under-appreciate the potential impacts on the MA-BT performance of model specifications, values of methodological variables (Johnston et al., 2006) and other choices the meta-analyst needs to make (Hoehn, 2006)¹.

This paper aims to investigate the validity and reliability of international MA-BT of non-timber benefits based on a recently published MA of contingent valuation (CV) studies in Norway, Sweden and Finland (Lindhjem, 2007). Compared to previous research on MA-BT, our paper adds several new and interesting dimensions: (i) a more systematic and diverse testing of different MA-BT models, including comparisons with simple BT techniques, (ii) the good we investigate is complex and has substantial non-use values related to biodiversity (rather than mainly use values), (iii) data from three countries, which are similar culturally, economically, institutionally (e.g. people's right to walk in private forests), and in the way the good is perceived and used, and (iv) data are generally more homogenous methodologically since only CV studies are included. We investigate the transfer error (TE) of four different meta-regression model specifications, and use the best two models to compare MA-BT with simple unit value transfer techniques. A key question is whether MA-BT achieves reliability gains justifying the increased effort. As pointed out by Navrud and Ready (2007a, pp. 288): “Simple approaches should not be cast aside until we are confident that more complex approaches do perform better”.