Evaluating Mobility Data for Recreation Monitoring

1. Introduction

Commercially available mobility data (MD) present an opportunity for researchers interested in understanding outdoor recreational behavior.¹ In recent years, the proliferation of smartphone users and applications has generated vast amounts of MD and an emerging industry of vendors who aggregate and sell these products for research and commercial applications. There are over a billion acres of federal lands and waters designated for recreation uses (Leggett et al. 2017), not to mention state, local, and other properties, and many resource managers require estimates of recreational visitation to effectively manage the lands and waters as well as visitor experiences. For this reason, MD provides an opportunity to measure and characterize recreational visitation, potentially replacing or enhancing traditional data-collection methods (Tsai et al. 2023; Merrill et al. 2025).

Researchers have explored this opportunity in a variety of public land and water applications, resulting in a growing body of work that relies on MD as a proxy for visitation (e.g., Csomós, Borza, and Farkas 2023), to explore visitor demographics (e.g., Zhang et al. 2024), estimate recreation value (e.g., Gonyo, Burkart, and Regan 2024), and reveal visitor preferences (e.g., Song et al. 2022), among other applications. Indeed, many of the other articles in this special issue use MD to address a variety of questions related to visitor estimation, demand estimation equations, zero market-share adjustments, and so on. However, MD are rarely validated against traditional measures of visitation that are understood to be reliable (Hanson et al. 2025), and research that does conduct these evaluations has found that the relationships between these traditional data and MD vary across time, space, and mobile provider (Monz et al. 2018; Filazzola et al. 2022; Liang et al. 2022; Tsai et al. 2023; Parkinson et al. 2025; Winder et al. 2025). Further, MD are rarely validated using within-dataset comparisons over time and space, which this article presents.

Multiple causes likely explain the poor track record of MD in these evaluations. To begin with, data quality can vary considerably across the different MD vendors (Winder et al. 2025). The data-generating process and complex processing decisions made by MD vendors are considered trade secrets. This lack of transparency means a researcher cannot replicate a mobility dataset or truly understand the quality of the product. Further, these datasets have evolved considerably due to changes in the proprietary methods, data vendor companies and their business models, consumer protection laws, and smartphone user behavior, as well as other factors (e.g., the extent of fraudulent data). Our experience using these data suggests that these changes do not demonstrate consistent improvement in data quality.

All sampling schemes have some degree of measurement error, and in the best-case scenario we expect all the typical complications (see Heckman 1979) in our statistical inferences based on MD. This is also generally true for the reference data we use for comparison. Specific components of our conceptual model (Section 4) have connections to classical (e.g., spatial imprecision of estimates) and nonclassical (e.g., willingness to share location data is correlated with demographics) measurement error, but overall, we focus on ideas related to sampling error specifically, not measurement error broadly. Sampling error and issues of representativeness are direct consequences of the numerous and opaque upstream processing steps in delivering an MD product. Consider the example of heterogeneity in willingness to share location data. None of the vendors we spoke with offer individual-level demographic data, even though these are likely an important determinant of the sampling rate. For instance, Struminskaya et al. (2021) highlight the importance of demographics in willingness to share GPS location data with researchers. This challenge also has a clear connection to nonresponse bias in the survey literature. Dutwin and Buskirk (2017) show that nonresponse rate varies by demographics and this relationship varies across technologies, a relationship that is underexplored in the MD literature.

This article focuses on the topic of critically evaluating mobility data for the purpose of recreation monitoring. Although our case studies and discussion are primarily oriented toward the task of accurately calculating visitation, the conclusions extend directly to recreational demand models. For example, key parameters in recreational demand analysis (price elasticities, substitution patterns, travel cost) depend on accurately measuring the underlying behavior of recreators. Our study highlights how the use of MD introduces additional sources of classical and nonclassical measurement error in calculating these parameters.

Using case studies with datasets from four vendors, we demonstrate a number of approaches for interrogating the quality of an MD source.² We also propose common language for these problems so that researchers may communicate more effectively. Given the evaluation of MD from our case studies, thearticle describes a conceptual model of the potential sources of error and structural variability in a visitation model derived from MD. We argue these characteristics of MD are likely to be a common problem across vendors and merit consideration for virtually all recreation monitoring applications.

2. Datasets by Case Study

Datasets from four different MD vendors are applied to three case studies, as summarized in Appendix Table A1. Each vendor provided data in a slightly different format,³ but each can be reprocessed into unique daily visitors to a custom defined area of interest (AOI) (Wood et al. 2013), the common unit of analysis in this study and in many MD applications.

Data from provider 1 (used only in the first case study) are the most heavily processed of the four, offering AOI-level daily “count” data alongside an “estimate” count, modeled by the vendor to reflect true (population-level) visitation. Our case study features only the “estimate” variable for reasons described later. The provider 2 data are more disaggregated, with individual pings combined into algorithmically defined “activities” from a single device. As a result, we often observe multiple activities from the same device in a site-day, which are further combined to form a count of unique daily visitors. One advantage to this level of aggregation is that it allows researchers to plot activities on a map and examine the spatial distribution. Provider 3 has the same data format as provider 2. The provider 4 dataset is the most disaggregated of the four, with the unit of observation being a single “ping” of location and time stamp recorded by a phone. This level of disaggregation can be helpful for additional customization in how activities or unique daily visitors are calculated, but it also requires additional geographical data-processing steps.

Our analyses cannot make exact evaluative comparisons among the four vendors. These datasets were pulled together from different research groups and obtained at different points in time. The earlier datasets from providers 1 and 2 are final products that cannot be requeried over additional points in space, as the subscriptions have lapsed. Providers 3 and 4 offered a limited back catalog that did not reach back far enough to make a full comparison. For the periods where we have overlap in areas of spatial analysis (e.g., Figures 2 and 6), the results do not suggest a recommendation of one over the other. This is a productive goal for future analysis, but it is outside the scope of this article.

We use three reference data sources corresponding to the case studies. In the first, we use Port Chilkoot cruise ship dock data from Haines, Alaska. The data were obtained from the 2023 Haines Borough Tourism Department’s seasonal report and describe yearly landings of ships, passengers, and crew from the 2019 and 2022 seasons. In the second case study, we use daily counts of paid vehicles from 2019 to 2022 at several California state park beaches where parking is closely related to total site recreation and managers use reliable and consistent counting methods. In the third case study, we use Major League Baseball (MLB) attendance data for 2019–2024 obtained from baseball-reference.com, representing the number of tickets sold for a given day (with occasional double-headers omitted from analysis for simplicity).

3. Case Studies

We present case study results using the mobility and reference data sources described in Section 2. Collectively, these datasets cover four prominent MD vendors and include data from 2019–2024 in different contexts. Through these case studies, we identify problematic data issues with MD and make relative statements about dataset reliability. We avoid making universal claims about whether a poor result in any particular evaluation invalidates a dataset’s reliability because there may be research applications for which the data issues are acceptable and MD still suitable. These are heuristic starting points for such an evaluation. The reliability of a mobility dataset appears to vary considerably according to settings, time frame, and even when it was queried.⁴ The case studies provide some potentially useful approaches for evaluating a mobility dataset in future applications.

The case studies compare MD with reliable reference datasets and make within-dataset comparisons of MD over time and space. Our first two case studies are motivated by specific research questions in Alaska and Southern California, as described in the sections that follow, and focus on evaluating datasets for those specific purposes. The third case study presents a comparison of mobility datasets to MLB stadium attendance data simply for evaluating MD rather than answering any research questions about MLB attendance. A key question across these case studies is whether the patterns in the data are stable across years and exhibit expected responses to exogenous variables, namely, weather and closed sites.

California Beaches: Data from Provider 2, Provider 4, and Local Agencies

In the second case study, we explore the suitability of MD for assessing recreation use impacts from an October 2, 2021, oil spill near Huntington Beach, California. Around 25,000 gallons of oil were spilled from an offshore pipeline. Several beach sites were fully or partially closed in the days after the spill. A key research question is how many beach trips were diverted from the affected coastline, both at closed and unclosed sites. Furthermore, were there lingering impacts to visitation after closures were lifted due to oil and tarballs coming ashore for several more weeks in Orange and San Diego Counties?

Before using MD to estimate the effects of the spill on recreation, we consider the time-series properties of the data, as was done in the Alaska case study. We start with a within-dataset comparison of the provider 2 MD over time, which was obtained from 2019 to 2022. Specifically, we evaluate how provider 2 counts correlate with explanatory variables like weather, day of the week, and holidays at two points in time: 2019 and 2022. We would expect the relationships between the MD and exogenous weather variables to be relatively stable across time periods. Based on established patterns, we expect beach use to be higher on warmer days, weekends, and holidays, holding all else equal.

We use weather data for John Wayne International Airport (a representative location in Orange County, California) from NOAA’s climate data online portal.⁶ We use provider 2’s daily count data for four beach sites that were closed as a result of the October 2021 spill: Bolsa Chica State Beach, Huntington State Beach, Crystal Cove State Park, and Laguna Beach. We estimate a standard Poisson count model of the provider 2 counts as a function of the maximum daily temperature, weekend/holiday, site dummies for each beach site, and a constant. The count models were estimated separately for 2019 and 2022 so the results could be compared across years.

Appendix Table A3 shows the results of these regressions. In 2019, the weather and weekend/holiday variables behave as expected, with use increasing on warmer days and weekends/holidays. In 2022, the regression shows no significant relationship between temperature and use. This result is inconsistent with established understanding of recreational use patterns because one of the most important factors determining beach use is weather, specifically temperature. This raises concerns about the stability of this dataset over time. This type of simple validation can help future researchers evaluate their mobility datasets.⁷

Our next evaluation of MD for this context compares it with a high-quality reference dataset. As described in Section 2, we obtained visitation data collected by California State Parks. Reliable reference data of beach visitation based on paid vehicle receipts are available for Bolsa Chica State Beach and Huntington State Beach in Orange County.⁸ The vehicle counts at these sites are conducted with a robust and consistent methodology from staffed entrance kiosks. State parks periodically use site-specific studies to develop people-per-vehicle multipliers, which are constant over the time period of our analysis. Therefore, we simply use the paid vehicle counts in our analysis. Appendix Figures A3–A6 show that the AOIs used in MD queries were drawn to omit the parking areas, in addition to some ambiguous pieces of the general area, in all cases. For Bolsa Chica State Beach in particular, the vast majority of visitation is expected to be paid vehicle entrance; walk-in visitation is unlikely to vary much or constitute a large share of the total. We present the daily provider 2 MD compared with the daily paid vehicle counts at Bolsa Chica State Beach. The comparison at Huntington State Beach (not depicted) provides similar results.

Figure 1 shows this comparison for each day from 2019 through 2022, with separate graphs for reference data, MD, and their ratio. We use seven-day moving averages of the counts to smooth out daily variability. In the first graph, the state park data demonstrate fairly stable seasonal patterns which hold across years. Visitation is highest in mid- to late summer and is lowest in mid-winter. The MD show a broadly similar seasonal pattern in 2019–2021, but the pattern does not manifest in 2022.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

Reference Data and Provider 2: Counts and Ratio for Bolsa Chica State Beach, California

Note: We use paid vehicle entrants as a proxy measure for individual visits. The ratio in the third graph shows the number of paid vehicles (not unique visitors) per mobility data observation. The ratio is calculated relative to the seven-day moving average of both measures.

We observe a spike in MD observations in the middle graph in April and May 2022, also seen in the drop in the third graph from over 250 to below 10 in only a few days. After this spike, the following summer observations (e.g., July 2022) are significantly lower than the previous three years’ summers, where visitation is typically highest. Further, July is the summer low point in 2022, whereas it is the peak of summer observations in the other three years. This shows some clear ways the MD are diverging from the reliable reference data.

The third graph shows the calculated ratio of paid vehicles per unique MD count, which allows us to visualize how the relationship between the data sources evolves over time. It is possible that the relationship between the reference data and true visitation changes over time as well. Many of the same evaluative approaches we discuss here also apply to reference data, though the conceptual model will generally be different. These observations support the following recommendations for initial evaluation steps for MD: (1) identify patterns within years, compare patterns across years, and attempt to explain observed variation; and (2) compare any existing a priori expectations about true recreation patterns (e.g., known seasonal use patterns, construction closures, opening of new amenities) to MD and reference data observations.

An example of step 1, applied to the reference data in Figure 1, shows consistent patterns across years. For the MD, we see significant divergence between the data in 2022 and the previous years. Recommended evaluation step 2 could make use of a priori expectations of typical seasonal patterns where the winter is the lowest use part of the year and summer is the busiest season. This is met by both datasets fairly well, except for 2022 in the MD again. In both datasets, we expect to see the onset of the COVID-19 pandemic (March 2020) and the oil spill (October 2021). In the reference data, a data gap corresponds to the first few months of COVID-19, as expected given the policy at the time. In the MD, a dip appears to be followed by an increase in summer visitation, which is also consistent with expectations about outdoor recreation in 2020. In both datasets, we see some basic visual evidence of a drop (relative to previous years) in visitation in early October 2021, corresponding to the oil spill.

In summary, we find both datasets in this evaluation to be potentially useful for answering questions about recreation, especially in earlier years. The reference data appear to be consistent across years and conform to expectations (with respect to seasonality, COVID-19, and the oil spill). However, the divergence from historical patterns and expectations remains a puzzle. These issues, combined with the April/May 2022 spike, cast doubt on the precision of MD for an event study of the 2021 oil spill, given its temporal proximity to the 2022 data anomaly. Identification of a spill effect requires consistency in the data before and after the spill.

Another perspective on MD performance in this context, shown in Appendix Figure A2, uses data from an additional vendor: provider 4. We compare the mobility datasets from providers 2 and 4 at Bolsa Chica State Beach for 2019–2024, although the available datasets only overlap in 2021 and 2022. For the period of overlap, the datasets from different vendors paint a different picture of use patterns, which raises questions about the reliability of both datasets. The analysis identified problems observed with using provider 2’s data as a proxy for visitation, and Appendix Figure A2 shows clear problems in provider 4’s MD as well. Interestingly, the problems with provider 4’s data do not appear to correlate with the problems in provider 2’s data. Provider 4 data in 2021 and 2022 show a lack of seasonality. Provider 4 in 2023 begins to show a seasonal pattern, but as we show in Section 3, provider 4’s data fail to track day-to-day MLB attendance data.

Our final evaluation of MD for this California analysis looks at the data around the time of the October 2021 spill for sites that were closed and known to have very little or zero use: Bolsa Chica State Beach, Huntington State Beach, Crystal Cove State Park, and Laguna Beach. Closure information was compiled from publicly available news sources documenting the presence of closures. Nonrecreators (e.g., cleanup crews or site management staff) may have represented some limited use of beaches during the closures, but this would pale in comparison to normal use levels. Figure 2 shows provider 2 counts during the closures and surrounding weeks at these four sites. The dark gray shading represents full closure (i.e., sand and water); the light gray shading is partial closure (i.e., water only).

• Download figure
• Open in new tab
• Download powerpoint

Figure 2

Provider 2 Counts During Closure Periods at Four Beach Sites in California

Note: See Appendix Figures A3–A6 for an image of the shapefiles used to generate these counts. The dark gray zone represents a full site closure; light gray is a partial (e.g., water only) closure.

Provider 2 counts do not always demonstrate the expected pattern with respect to closures. The MD counts at Bolsa Chica decline during the closures, relative to preclosure counts. Bolsa Chica is generally the most isolated from external sources of MD (see Appendix Figure A3) and thus can be held to the highest standard with respect to the expected effect of a closure on visitation. The MD counts at Huntington State Beach and Crystal Cove may show reductions in use, but MD observations are still greater than zero. Weekday visitation in particular is in the same range as many nonclosure period observations. Laguna is the site with the most notable divergence from expectations, with visitation patterns appearing to be unaffected by the closure. This could be explained by its proximity to a high-quality nonbeach park (which was not closed when the beach and water access closed) on the other side of the boardwalk (see Appendix Figure A6). Provider 2’s dataset does not apparently indicate a decline in visitation, yet these beaches were fenced off during the closures and not available for use.

The datasets with the most pronounced closure effects are generally the most isolated from irrelevant sources of MD. This highlights the importance of spatial error in MD analyses. The ability of MD to detect known, significant drops in visitation on a short time frame is a reasonable expectation that proves difficult to achieve. Duff, Giguere, and Murray (2026, this issue) examine this objective in detail for a set of closed parks in Texas, finding significant variation in the strength of the closure effect across similar sites. In particular, the contribution of spatial error from surrounding roads, neighborhoods, and other sites of interest (such as the park next to Laguna Beach) can lead to significant measurement error. Our analysis of these four California parks gives further evidence of the importance of site characteristics in the context of MD analysis.

Evaluating MD Using MLB Stadium Attendance Data

In the final case study, we compare MD from multiple vendors to MLB stadium attendance data. MLB stadiums and attendance data offer several advantages for understanding how MD compare to actual attendance. These data provide numerous observations, as MLB teams host up to 81 regular-season games per year in their home stadium, and teams report attendance for every game. MLB attendance data was unavailable for the 2020 season due to COVID. While stadiums are similar in size and shape, there are a few idiosyncratic differences that motivated our choice of a common AOI definition. In the following analyses, we define the MD counts relative to a circle with a 125 m radius surrounding second base (approximately the centroid of most stadiums; see Appendix, under “Additional Material for Case Study 3,” and Appendix Figure A7 for discussion of this assumption). Using this setting as a backdrop for analysis, we demonstrate some notable patterns that warrant further investigation and comparison with other MD sources.

In nearly all MD research applications of which we are aware, the question of key importance is how many true visitors are to be inferred from an MD observation. We refer to this relationship as the “coverage rate,” the ratio between observed MD counts and true counts of visitation over any given timeframe. The inverse of a coverage rate is referred to as the MD expansion factor. Although other functional relationships between MD and reference datasets have been found to exist in application (Merrill et al. 2020; Tsai et al. 2023), we simplify the discussion with an assumption of linearity.

Figure 3 shows a typical pattern in the calculation of MD expansion factors on a log scale at an example MLB stadium. At this particular stadium (American Family Field in Milwaukee, Wisconsin), expansion factors in 2019 ranged from around 25–100, with a similar range (aside from a few outlier days) in 2021. The MD analyzed from 2022 appears to perform so poorly as to be unusable. We observe a wide fluctuation in expansion factors and a gradual trend toward a very high average expansion factor.

• Download figure
• Open in new tab
• Download powerpoint

Figure 3

Provider 2 and Major League Baseball Attendance: Daily Expansion Factors at American Family Field Baseball Stadium, Wisconsin

Note: This expansion factor measure is the inverse of the coverage rate measure. This is conceptually comparable to the third graph in Figure 1 but in terms of individual ticket instead of vehicles.

Figure 4 suggests a simpler explanation for what went wrong. It shows MD counts for game days and counts on nongame days. Of particular interest is the mobility dataset’s ability to detect spikes in visitation on days when they really occur.

• Download figure
• Open in new tab
• Download powerpoint

Figure 4

Provider 2 Counts on Game Days and Nongame Days at American Family Field Baseball Stadium, Wisconsin

Note: The figure shows the raw mobility data counts associated with dates on which Major League Baseball (MLB) games are scheduled. Instances of high counts on nongame days might be explained by non-MLB events using the stadium. Instances of low counts on game days (which become common in 2022) lack an obvious explanatory mechanism.

In 2019 and 2021, the MD counts on game days were systematically higher than counts on nongame days, suggesting that they are capturing the effect of MLB attendance. There are occasionally nongame days with high attendance that may be attributable to other events, such as concerts (e.g., early October 2021). Because we lack systematic data on these events, we assume that a few isolated occurrences are not evidence of a problem with MD.

However, 2022 demonstrates different patterns. Clustered groups of MLB games at a stadium (also known as homestands) show the correct number of visitation spikes in the provider 2 data in 2022 but not on the dates that correspond to actual attendance. Meanwhile, there are a corresponding number of game-day counts that are nearly zero. This is a misclassification pattern common across stadiums that suggests a back-end systematic problem with the time stamps of individual observations, rather than, for instance, the gradual accumulation of contribution from structural factors over time.

Beginning in July 2022, the MD fails to track any games at any of the stadiums, with a sharp decline to nearly zero for all observations. This example highlights the possibility that problems with mobility data could be caused by simple processing errors. The stadium data presented here provide a setting that is well targeted to identifying this particular problem: a location where there are events with visitation spikes that are orders of magnitude larger than the visitation on common nonevent days (e.g., stadium staff). If this problem were to manifest in a setting with smoother daily visitation (e.g., public parks), it would be difficult to know whether, for instance, a Monday’s visitation were swapped with the same Thursday’s visitation.

We can analyze MLB stadium-years as separate datasets to explore spatial and temporal patterns in data quality. In our dataset of regular-season MLB games, a year maps cleanly to a season of professional baseball, generally April–October. A simple approach is to calculate the bivariate correlation between MD and attendance counts for all game days in a stadium-year. This correlation statistic relates to the stability of the coverage rate in a stadium-year. Although MD with higher correlation to a reference dataset might not be holistically stronger than another, a strong correlation is at least an indication that changes in visitation are being reflected dynamically. As with all evaluation approaches in this section, the appropriateness of this comparison may depend on the specific application.

Appendix Table A4 shows the results of one such comparison across vendors and years. Each cell represents the mean stadium-level Pearson correlation coefficients between MD counts and MLB attendance counts for the 27 stadiums we observe in each corresponding year.⁹ The correlations were much stronger in 2019 and 2021 than in the following years, regardless of vendor. This evaluation approach allows the researcher to make rank-order comparisons across vendors and year. While the specific correlation-based metric described here should not be relied on too heavily, the pattern it suggests is clear. The quality of MD deteriorated across multiple vendors after 2021.

Figure 5 shows provider 3 and provider 4 counts, respectively, for American Family Field in 2022–2024. These datasets share a 2022 pattern that differs from that in provider 2’s dataset. For providers 3 and 4, in the early months of 2022 before the MLB season began, there was a relatively low, stable level of MD counts on nongame days. After the 2022 season began, MD counts on nongame days increased considerably, typically hovering around the same level as on game days. After the season ended, MD counts fell back down to low levels, except for a few very high-use days, a pattern that is common across all the stadiums (and thus unlikely to be explained by non-MLB events).

• Download figure
• Open in new tab
• Download powerpoint

Figure 5

Providers 3 and 4: Mobility Data Counts on Game Days and Nongame Days at American Family Field Baseball Stadium, Wisconsin

Note: The figure is a continuation of Figure 4 with two other vendors. For both datasets in 2022, systematically higher counts on nongame days coincide with the beginning of the season.

One potential explanation is temporal imprecision of recorded activities (intentional by the vendor or not), where the expected visitation spike on game days is diluted across other nearby days. Other datasets that we evaluated with this approach (provider 3 in 2023 and 2024, provider 3 in 2023) did not exhibit this specific pattern, but they did show a general lack of distinction between counts on game days and counts on nongame days.

The foregoing examples demonstrate notable unexplained problems with mobility data. These findings should encourage other researchers to apply evaluation approaches like these to other vendors so that results can be compared. Given the usefulness of considering how these data are derived and the processes that govern the relationships between MD and true recreation, we propose a conceptual model of the mobility data-generating process and discuss some likely reasons for the problems that we have demonstrated.

4. Conceptual Model: Data-Generating Process

Mobility datasets like the device location data used in this article have several characteristics that are common across vendors and can present technical challenges for researchers. Ex ante, it is difficult for a researcher to form realistic expectations about data quality based on a vendor’s marketing claims. Vendors have the incentive to understate the importance of these destabilizing factors, while overstating their ability to mitigate them through proprietary techniques. Ultimately, the researcher must evaluate the finished product and decide whether it is of sufficient quality to support the desired analysis. We explain some of the ways a mobility dataset can be unstable across vendors, space, and especially time. Here we focus on defining the issues and exploring how they relate to “known unknowns” in the data-generating process.

One might assume that MD reflects a simple random process where each person in an AOI has a fixed probability of being sampled and therefore captured in a dataset. Yet MD variability is a function of not only the number of people in the AOI but also the variability in the process by which those people are identified in the MD metric. We describe a conceptual model of how several features of MD contribute to the relationship between true recreation and recreation as measured by MD. In this model, mobility datasets are decomposed into three categories: false negatives (which are not observed in the final product), true recreation, and false positives. The relative contribution of each piece in this process is of key importance to researchers; in particular, how these contributions vary over time. Figure 6 illustrates the decomposition of recreators’ recruitment into the MD sample.

• Download figure
• Open in new tab
• Download powerpoint

Figure 6

Conceptual Model of Mobility Data Generating Process

Note: True recreators either show up in the final dataset or are filtered out (false negatives) for various reasons. Additional data (false positives) show up in the final dataset. The reasons shown in this diagram are likely nonexhaustive.

This stylized diagram is a decomposition of true recreation and a visual explanation of our conceptual model of the coverage rate. At the top left are true recreators, who may be excluded from the mobility data count if they do not satisfy all the depicted conditions. We define false positives as observations that appear in the researcher’s final mobility dataset but do not actually indicate a true recreation activity. We define false negatives as hypothetical observations of true recreation that do not appear in the final mobility dataset for several possible reasons. Each offramp box (e.g., “device not with user”) is a mediator, which reduces the number of remaining recreators to be captured in the MD count. Each mediator is potentially influenced by a combination of exogenous (structural factors) and endogenous (decisions made by the device user) variables that influence the data-generating process. Table 1 describes how each mediator is influenced by these variables, though this is not necessarily an exhaustive list.

Even if an MD observation from a true recreator satisfies all the mediator conditions for inclusion, data points can still be excluded during black-box processing steps undertaken by the MD vendor. For instance, vendors may decide to exclude an otherwise included data point if it is flagged as suspicious based on implausible location or time stamp patterns. Other data-processing steps can lead to false inclusion. For example, many commercial data providers “salt” datasets sent to customers by adding a small amount of synthetic data for anonymizing the data or identifying their product. All of the data processing done by vendors relies on different methods and assumptions, these methods differ by vendor and over time, and the methods are all proprietary and opaque to researchers buying the products (at least for every vendor that we are aware of, including the ones used for this study). These details are generally articulated to the researcher in broad terms (e.g., “we use location data from a variety of apps in categories X, Y, and Z”) and the full list is a trade secret.

To account for the false negatives and false positives in Table 1, some vendors offer measures of aggregate visitation that extrapolate a total number of visitors from observed counts of mobile devices. For example, provider 1 data are accompanied by a proprietary “estimate” count, which extrapolates a total number of visitors from observed counts of mobile devices. This type of measure has the potential advantage of accounting for some of the variables mentioned above that can only be observed by the vendor, but it has the disadvantage of near-total opacity from the researcher’s perspective.

Because coverage rate is downstream of all the factors described in this section, its stability is vulnerable. Some of these factors could be modeled using aggregate information to estimate the coverage rate (e.g., aggregate device ownership rates, timing of major OS changes, contribution of observable variables such as weather), but some potentially key factors are typically unobservable proprietary information (e.g., which apps contribute to a vendor’s supply). Sites of interest may have variable coverage rates because the mediators described above can vary according to visitor demographics as well. The temporal pattern of the contribution of these unobserved structural factors to the coverage rate is an open question. Some may have impacts that are smoothed over time (e.g., a contributing app slowly gains popularity), and others may have sharp impacts (e.g., an app suddenly discontinues a function that contributes location data). This dynamic is potentially important because it influences the comparability of MD estimates that are temporally proximate. Sharp changes in the coverage rate over time can invalidate comparisons even in the short run.

Although we do not have the metadata one might need to connect specific instances of MD instability to one or more root causes, the conceptual model provides a taxonomy of causes. We apply this taxonomy to some of our datasets in this article. Even without clear evidence of specific causal mechanisms, these potential causal categories can help provide context for the unusual or unexpected features of the data. In the California case study (Figure 1), we observe a sudden change in the relationship between MD and reference data in 2022. This change corresponded with a sudden supply shock from at least one important app supplying location data to multiple vendors (Life360, the same app that was likely affecting provider 1 data). This may have caused a sharp drop in the number of true recreators who used a relevant app (mediator 2) associated with the provider’s dataset. Meanwhile, Figure 2 and its discussion suggest a story about how MD visitation estimates at different beach sites are differentially impacted by spatial imprecision (mediators 4 and 5), leading to noisy estimates during a time when true recreation was low. In the stadium data example, Figure 5 shows a breakdown in the MD counts’ relationship with the game-day indicator variable. Although the timing of this problem is consistent with the same mediator 2 explanation as before, the pattern we observe with clustered game-day misclassifications implies an additional problem with the time stamps (mediators 4 and 5). Other causes in this conceptual model might be demonstrated on other mobility datasets by analysis with respect to external data sources (e.g., industry data, timing of operating system changes, timing of privacy laws) or internal data sources (e.g., vendor-specific metadata, as described in Section 5).

This conceptual model is our attempt to characterize some of the idiosyncratic complexities inherent to the MD-generating process. The goal is to highlight the sources of uncertainty that upstream processing decisions impose on the researcher’s understanding of their analysis. Next we suggest that mobility data vendors should provide researchers with supplementary datasets, metadata, and other insights to enhance understanding of the mechanisms described in (or missing from) Table 1.

5. Discussion

This article presents several empirical examples where MD fails to meet expected data quality standards. We illustrate examples where (1) important patterns (e.g., seasonality) are missing, muted, or degraded over time; (2) datasets from multiple vendors show incongruent results; (3) MD fails to track with relevant conditions (e.g., closures and game days); and (4) MD provides no apparently usable information at all. The study presents a conceptual model that describes structural features of the data-generating process that may explain these issues. Researchers must carefully consider their assumptions about the coverage rate on a case-by-case basis. We recommend the following:

• ■ Examine new mobility datasets for evidence of instability and data supply problems. This article shows several possible approaches for identifying problems if they exist. Demonstration of a positive outcome with respect to these approaches (e.g., 2019 and 2021 in Figure 4) can be presented as evidence of a strong mobility dataset.

• ■ Ask vendors to supply helpful metadata (see discussion below).

• ■ Choose analyses according to the strengths of the available dataset.

• ■ Report data weaknesses and caveats in publications.

A general diagnostic checklist for researchers with a new MD dataset is beyond the scope of this study and current understanding, given the wide and multidimensional range of applications, MD products, vendors, and settings. Nevertheless, Section 3 provides a generic starting point for evaluation of a mobility dataset. Regardless of the endpoint, all outdoor recreation data queries can and should be examined for consistency of patterns across time (e.g., seasonality) and can be compared against externally developed expectations (e.g., responsiveness to known disruptions).

In Section 3, we demonstrate a novel evaluation strategy applying MD to estimating MLB stadium attendance, which can be easily replicated with new datasets. Stadiums provide a rare set of nationwide reference data for MD validation, characterized by a sharply distinguished pattern of true visitation. Researchers should determine if their mobility dataset can precisely record peaks and troughs of visitation at the single-day level. Researchers should be creative in looking for additional datasets that can validate or ground-truth their MD dataset. Validation data are the key to expanding researcher’s understanding of the MD, its weaknesses, and appropriate uses.

MD may be appropriate for some recreation applications, but temporal instability will make certain approaches infeasible. Event studies aimed at detecting potential changes in recreation use are challenging to accomplish with currently available MD products. Temporal instability in MD, as this analysis demonstrates, can overwhelm the variation a researcher is trying to detect. Calibrating the MD to reference data may not solve the problem if the underlying relationship is changing over a short time scale. To rigorously model the dynamics of the coverage rate, researchers will need more information about the data-generating process than is currently possible. Simulating MD characteristic impacts on recreational demand models represents a productive direction for future research. Short of these more detailed approaches, estimates of changes in visitation will rely on strong assumptions about the stability of the MD coverage rate.

On the other hand, analysis aimed at understanding relative visitation among sites might be a good application for MD. This approach may be largely shielded from the problem of intertemporal instability because the contribution of the exogenous variables to the coverage rate may simply be canceled out in the ratio. Important considerations remain with this strategy. Researchers can define the spatial extent of their AOI in an unlimited number of ways, and the MD vendor’s count estimate is directly influenced by an ultimately arbitrary boundary drawing. Matching the geographic extent between what the reference data are representing and what is sampled by the MD is an important task for making comparisons and MD-based visitation inferences. Challenges with defining boundaries exist more generally in recreation monitoring, but they are especially complex with MD.

Many applications for recreation using MD are less concerned with intertemporal comparisons, and thus they are less concerned about temporal stability. For example, valuation models can exploit the home-origin information available from some MD providers. The patterns we describe here might affect a valuation model to the extent that they skew the distribution of parameters of interest. We do not have strong expectations about how the pieces described in our conceptual model affect the demographic representation of the resulting mobility dataset. Future work on this topic should target the secondary variables included in some MD products, such as home-origin locations and time stamps of observations.

One might hope that difference-in-difference approaches can be used to purge shared but unknown structural variability in MD. While difference-in-difference approaches might successfully account for the background trends in MD and observed covariates, we are typically interested in modeling or predicting the underlying quantities (e.g., true recreation) that MD is intended to measure and not in the MD counts per se. This study demonstrates that the relationship between observed MD and target recreators can change over time and space and be difficult to predict a priori. It would be optimistic to assume that an estimated difference in MD indicators at control and treatment sites that are each polluted with unknown variability would leave anything but a difference that is also polluted with unknown variability. This is yet another reason untangling the structural variability in the measurement process from the measurement target is a critical next step in understanding how to best use MD data.

We hope the ideas in this article will motivate vendors to better communicate key background parameters with the research community. Examples include indices over time describing the data supply chain—number of contributing mobile apps per month, distribution of data volume contributed by individual apps (e.g., a Herfindahl-Hirschman type index), average number of apps by which a device is observed, distribution of volume by type of app, and so on. This information could help researchers diagnose problematic patterns (e.g., the 2022 divergence between data sources shown in Figure 1), identify eras of relative stability, understand changes in the data-generating process, and optimize research designs accordingly.

Similarly, it would be useful to access indices over time describing the data sample, such as a constructed index of the number of unique devices observed in various levels of spatial aggregation.¹⁰ This would be useful for helping normalize the sample of visitors (using visitor origin data) at any given site and improve the researcher’s ability to adjust for changes in representativeness (e.g., Figures 4 and 5, changes in representativeness relative to other technical problems with time stamps). It would also be useful for identifying temporal shocks (common over space) in total data volume, which provides a useful control for contextualizing visitation changes in AOIs.

Finally, researchers need robust documentation of the timeline of versioning of processing changes and samples, and how those changes affect data before and after the change. Rather than deliver a product with opaque ad hoc adjustments, vendors should allow researchers access to the product upstream of these adjustments and provide the information required to assess them. This would allow researchers to better compare analyses and make a compelling case for robust results.

Additional studies are needed on the representativeness and temporal stability of MD datasets for popular applications, where strong assumptions are needed to create valid results, given the problems documented here.