Environ. Sci. Technol. 1999, 33, 1559-1565
Sample Representativeness: A Must for Reliable Regional Lake Condition Estimates S P E N C E R A . P E T E R S O N , * ,† N. SCOTT URQUHART,‡ AND EUGENE B. WELCH§ U.S. EPA National Health and Ecological Effects Research Laboratory, Western Ecology Division, Corvallis, Oregon 97333, Department of Statistics, Oregon State University, Corvallis, Oregon 97331, and Department of Civil and Environmental Engineering, University of Washington, Seattle, Washington 98195
Reliable environmental resource estimates are essential to informed regional scale decisions regarding protection, restoration, and enhancement of natural resources. Reliable estimates depend on objective and representative sampling. Probability-based sampling meets these requirements and provides accuracy estimates (confidence limits). Non-probability-based (judgment or convenience) sampling often is biased, thus less reliable (no accuracy estimates), and potentially misleading. We compare results from a probability- and a non-probability-based Secchi transparency sampling of lakes in the northeastern geographic region of the United States and its three primary ecoregions. Results from these samplings are compared on the basis of sample representativeness relative to the regional lake population and subsequent reliability of lake condition estimates. Statistically derived sampling indicates the northeast lake population median lake size to be about 9.5 (( 2.3) ha and the Secchi disk transparency (SDT) to be about 2.4 (( 0.4) m. On the basis of judgment sampling estimates, the median SDT for lakes in the same area would be 4.2 m. However, only about 15% of the regional lake population based on statistically designed sampling estimates has a SDT g4.2 m. Estimate unreliability of this magnitude can have profound effects on lake management decisions. Thus, regional extrapolation of nonprobability-based sampling results should be avoided.
Introduction Environmental monitoring and assessment employs a variety of terms having specific meaning to this field. To facilitate reading and understanding, we defined several of these terms, as used in this paper, in Table 1. Sampling to characterize environmental resource conditions typically has been based on judgment or convenience sampling of individual or groups of lakes. It has become increasingly apparent that the condition of these lakes is strongly influenced by the environmental characteristics of the landscapes within which they reside (2-5). These regional landscape characteristics also affect the response * Corresponding author phone: (541) 754-4457; fax: (541) 7544716; e-mail:
[email protected]. † U.S. EPA National Health and Ecological Effects Research Laboratory. ‡ Oregon State University. § University of Washington. 10.1021/es980711l CCC: $18.00 Published on Web 04/08/1999
1999 American Chemical Society
of lake populations to changes relative to their structural stability (6), resiliency (7), and recovery rates (8). These lake responses act at both regional and continental scales (9, 10 ). Therefore, regional lake conditions are important, both as background and as a basis for determining if the condition of specific lakes or groups of lakes is typical with respect to the overall regional condition of lakes. Thus, it is important that population estimates of regional lake condition be established in a reliable manner if landscape level changes are to be observed and used meaningfully in management decisions. Unbiased representation of the resource in question is perhaps the most critical component in the process of establishing reliable estimates of regional lake conditions. A complete census (i.e., sample and test every lake in the population of interest) could be conducted. However, that is not practical given the large number of lakes in some areas; there are more than 11 000 lakes g1 ha in the northeastern United States (11). Another approach is to survey the resource of interest by collecting a statistically valid sample: (1) identifying the total population (e.g., all lakes in the northeastern United States g1 ha), (2) using a simple or systematic random sample selection process that ensures representativeness and includes the known probability (or equivalently the expansion factor) of selecting any member of the lake population, and (3) applying the expansion factors to the samples such that inferences can be drawn from those samples to the total population. This three-phase process permits reliable inference from the samples to the general lake population (12). These types of procedures commonly are referred to as probability-based surveys or sampling. Still another relatively common approach to regional environmental condition assessment relies on extrapolation from non-probability-based data not originally meant to be used for regional extrapolations or inferences (13-15). Regional extrapolations from these data are unreliable because the samples frequently are biased in one way or another and lack representativeness relative to the resource population being assessed. Concerns about sample representativeness probably contributed, at least in part, to the identification of environmental monitoring and ecology as one of the six major areas needing increased attention and resources by the National Forum on Science and Technology Goals (16). Until recently, there have been few opportunities to compare judgment-based environmental resource estimates to probability-based estimates. This paper explores quantifiable differences in a regional lake condition [Secchi disk transparency (SDT)] derived from probability- (Environmental Monitoring and Assessment Program [EMAP]) and non-probability-based (Great American Dip-In [Dip-In Lakes]) sampling surveys conducted in the northeastern region of the United States. The focus of the comparison is on the representativeness of samples to the general lake population and on subsequent reliability of regionally extrapolated/ inferred lake condition estimates based on these two sampling approaches. Other examples are discussed.
Approach and Methods Two data sets were compiled for the analysis: data obtained using the EMAP sampling design (17) and data from the Great American Secchi Dip-In (13). EMAP Lakes. The EMAP lake sampling design and selection process used the U.S. Geological Survey (USGS) 1:100000-scale map series in digital format (DLGs) (18) VOL. 33, NO. 10, 1999 / ENVIRONMENTAL SCIENCE & TECHNOLOGY
9
1559
TABLE 1. Definitions of Selected Environmental Monitoring and Assessment Termsa term
definition
attribute bias
Any property, quality, or characteristic of a sampling unit. In a sampling context, the difference between the conceptual weighted average value of an estimator over all possible samples and the true value of the quantity being estimated. An estimator is said to be unbiased if that difference is zero. The systematic or persistent distortion of a measurement process that deprives the result of representativeness (i.e., the expected sample measurement is different from the sample’s true value). characterization Determination of the attributes of resource units, populations, or sampling units. cumulative A means of representing the variation of some attribute by giving running totals of the resource with distribution attribute values less than or equal to a specified series of values. The cumulative distribution function (cdf) of some specified attribute of a population is the function F(x) that gives the proportion of the population with the value of the attribute less than or equal to x, for any choice of x. ecoregion A relatively homogeneous geographic area perceived by simultaneously analyzing a combination of causal and integrative factors including land surface form, soils, land uses, and potential natural vegetation. inclusion The probability of including a specific sampling unit within a sample. Its reciprocal is the weighting probability factor. judgment A form of nonprobability sample in which the sample is chosen according to the judgment of the sample sampler. population In statistics and sampling design, the total universe addressed in a sampling effort; an assemblage of units of a particular resource, or any subset of extensive resources, about which inferences are desired or made. probability A sample chosen in such a manner that the probabilities of including the selected units in the sample sample are known, and all population units have a positive probability of selection. This implies that the target population is represented by the sample and that the target population is explicitly defined. region Any explicitly defined geographic area. representativeness The degree to which data accurately and precisely represent the frequency distribution of a specific variable in the population. sample A set of sampling units (lakes) or sites that will be characterized. sampling unit An entity (lake) that is subject to selection and characterization under a sampling design. a
Modified from ref 1.
represented by the U.S. EPA Reach Files, Version 3 (RF3) (19) to represent the population of lakes to be sampled (target population). Each lake in the target population database was individually identified by geographic coordinates and a unique label. Our interest was in all northeastern lakes g1 ha, based on DLG identification. Larsen et al. (17) developed a protocol, meeting the criteria for probability sampling, by which a sample of lakes was drawn from the target population represented on the DLGs. As part of that protocol, GIS techniques were used to characterize lakes by their size and spatial location. Because the distribution of lake sizes in the northeast is very skewed toward small lakes, larger lakes were included in the sample with a higher probability than small lakes, and sampling density was generally increased in some areas of high interest (e.g., lakes in the Adirondacks and in parts of New Hampshire and Vermont, because of their sensitivity to acidification). The resulting spatial distribution of the sample (Figure 1) thus reflected the natural spatial density of the lakes, the increased density in some areas, and a mild, but known and quantifiable, spatial restriction on the selection process. In probability sampling, the inclusion probability for each sample unit (each lake) characterizes the proportion of the population that sample unit represents. Thus, sample unit inclusion probabilities determine the expansion or weighting factors (reciprocal of the inclusion probability) used in making population inferences from the sample. Lakes selected with relatively high probability represent relatively few lakes in the population. Therefore, they carry relatively low weight and influence the final inferences less than lakes selected with low probability. The weighting or expansion factors are used to infer or estimate population cumulative distributions and to establish their confidence limits. This approach minimizes various selection biases from the lake selection 1560
9
ENVIRONMENTAL SCIENCE & TECHNOLOGY / VOL. 33, NO. 10, 1999
process to provide a representative picture of conditions for all lakes across the region, while sampling only a small fraction (3.1%) of the lakes in the population. See Larsen et al. (1994) for further details on the lake selection process and the resulting computation of lake inclusion probabilities and their associated weighting factors. We applied the EMAP sampling design (17) to a select lake population throughout the northeastern United States from 1991 through 1994 (11). A lake was defined as any standing water body g1 ha in surface area and g1 m deep, with g1,000 m2 of open water. Approximately one-quarter of the 344 lakes selected for sampling were visited once each year during the sampling index period (July-August). The EMAP SDT estimates reported in this article result from the 4-year (1991-1994) cumulative sampling. SDT was determined with an 8-in. (20.24 cm), black and white Secchi disk according to EPA (20) procedures at the lake index site (deepest part of the lake). Any EMAP lake with SDT data missing, or with a value equal to maximum lake depth, was discarded for the purpose of these analyses. The final data set consisted of 312 EMAP lakes. SDT measurements were entered into SAS (21) data files for analysis. Quality assurance of measurements was prescribed by the EMAP Quality Assurance Project Plan (22). Great American Dip-In Lakes. Lakes in this data set are hereafter referred to as Dip-In lakes. To gather data for the Dip-In lakes, postage-paid questionnaires were mailed to approximately 5000 participants in various lake-monitoring programs throughout the United States in late May and early June of 1995 and 1996 (13). Volunteers were asked to measure SDT in their lakes between July 1 and July 9 using an 8-in. (20.24 cm), black and white Secchi disk. Dip-In sampling data were sent to Dr. Robert Carlson at Kent State University, Ohio, where they were entered into a
FIGURE 1. Location of 1991-1994 EMAP and 1995-1996 Dip-In lake sample sites across the three ecoregions of the northeastern United States. Microsoft Access database. Carlson supplied us with a copy of the Dip-In database for northeastern lakes. The data consisted of 422 records. Lakes were identified by name, county, and state. Lake size class (500 acres) was reported, rather than the specific areal size of each lake. Some records were incomplete. No lake area was recorded for 16 lakes. Location information was incomplete for 26 others. We contacted state lake coordinators in all of the northeastern states to obtain lake location and actual size for the Dip-In lakes. We identified the specific lake size and coordinates of Dip-In lakes by matching county and lake size class from the original Dip-In data to specific coordinates and lake sizes from the state list. We determined the specific size for all but 16 of the 422 Dip-In lakes. Of these 16 lakes, two were duplicates. Fourteen records did not exist in the databases sent from the states. Lakes without specific sizes and/or coordinates were discarded. Data from the remaining 404 Dip-In lakes were compared with data from the 312 EMAP lakes (Figure 1). The data were analyzed from two geographic perspectives. The first included the entire northeastern region of the United States. The second employed Omernik Level II ecoregions, based on elevation, general soil types, surficial and bedrock geology, potential natural vegetation, slope, climate, and physiography (2). Because of special interest in the Adiron-
dack lakes, we separated this area from Omernik’s Atlantic Highland ecoregion for reporting purposes. In addition, New Jersey lakes were aggregated into Omernik’s Mixed Wood Plain Ecoregion, on the basis of similarities in elevation, physiography, and other lesser features. From these modifications, renamed reporting regions were the Adirondacks (ADI), the New England Upland (NEU), and the Coastal Lowland and Plateau (CLP) (Figure 1). Statistical analyses of the two data sets were performed using SAS (21).
Results and Discussion Results reported here are based on the analysis of samples from 312 EMAP lakes (91% of those sampled by EMAP from 1991 to 1994) and 404 Dip-In lakes (96% of those sampled by the northeast Dip-In in 1995 and 1996). EMAP lakes dropped from our analysis had transparencies to the bottom. Seventy percent of them were small (
