Subscriber access provided by UNIV OF LOUISIANA
Characterization of Natural and Affected Environments
Assessing the accuracy of citizen scientist reported measurements for agrichemical contaminants Jonathan Ali, Brandon Noble, Ipsita Nandi, Alan Kolok, and Shannon L. Bartelt-Hunt Environ. Sci. Technol., Just Accepted Manuscript • DOI: 10.1021/acs.est.8b06707 • Publication Date (Web): 01 May 2019 Downloaded from http://pubs.acs.org on May 1, 2019
Just Accepted “Just Accepted” manuscripts have been peer-reviewed and accepted for publication. They are posted online prior to technical editing, formatting for publication and author proofing. The American Chemical Society provides “Just Accepted” as a service to the research community to expedite the dissemination of scientific material as soon as possible after acceptance. “Just Accepted” manuscripts appear in full in PDF format accompanied by an HTML abstract. “Just Accepted” manuscripts have been fully peer reviewed, but should not be considered the official version of record. They are citable by the Digital Object Identifier (DOI®). “Just Accepted” is an optional service offered to authors. Therefore, the “Just Accepted” Web site may not include all articles that will be published in the journal. After a manuscript is technically edited and formatted, it will be removed from the “Just Accepted” Web site and published as an ASAP article. Note that technical editing may introduce minor changes to the manuscript text and/or graphics which could affect content, and all legal disclaimers and ethical guidelines that apply to the journal pertain. ACS cannot be held responsible for errors or consequences arising from the use of information contained in these “Just Accepted” manuscripts.
is published by the American Chemical Society. 1155 Sixteenth Street N.W., Washington, DC 20036 Published by American Chemical Society. Copyright © American Chemical Society. However, no copyright claim is made to original U.S. Government works, or works produced by employees of any Commonwealth realm Crown government in the course of their duties.
Page 1 of 27
Environmental Science & Technology
1
Assessing the accuracy of citizen scientist reported measurements for agrichemical
2
contaminants
3 4
Authors: Jonathan M. Ali1, Brandon C. Noble1, Ipsita Nandi2, Alan S. Kolok3 and Shannon L.
5
Bartelt-Hunt1*
6 7
Affiliations:
8
1
Department of Civil Engineering, University of Nebraska Lincoln, Omaha, NE USA
9
2
Institute of Environment and Sustainable Development, Banaras Hindu University, Varanasi,
10
India
11
3
Idaho Water Resources Research Institute, University of Idaho, Moscow, ID USA
12 13
*Corresponding Author:
14
Department of Civil Engineering
15
College of Engineering, University of Nebraska–Lincoln
16
1110 South 67th Street, Omaha, NE 68182-0178
17
402-554-3868 |
[email protected] 1 ACS Paragon Plus Environment
Environmental Science & Technology
18
Abstract
19
Citizen science is a research tool capable of addressing major environmental challenges,
20
including contamination of water resources by agrichemicals, such as nutrients and pesticides.
21
The objectives of this study were 1) to identify the proportion of accurate observations by citizen
22
scientists using rapid assessment water quality tools, and 2) to characterize how a user’s prior
23
experience with water quality tools was associated with the accuracy of citizen scientists. To
24
achieve these objectives, we conducted group testing with over 136 citizen scientists and
25
compared their results from water quality testing of water samples to results obtained using
26
laboratory analytical methods. Following brief training, we observed that accuracy of reported
27
results varies based on the user’s experience level where experienced and expert users shared
28
consistent and reliable measurements. Where erroneous measures were reported, citizen
29
scientists tend to overestimate contaminant concentrations when using colorimetric water
30
quality tools. Additionally, we identified differences in accuracy related to the types of water
31
quality assessment tools used by citizen scientists from each experience group. This study
32
demonstrates the importance of evaluating participant background experience in designing
33
citizen science campaigns.
34
2 ACS Paragon Plus Environment
Page 2 of 27
Page 3 of 27
Environmental Science & Technology
35
1. Introduction
36
The detection and mitigation of agrichemical runoff into surface and groundwater resources is
37
an ongoing global challenge. Nutrients, pesticides and other agrichemicals enter surface water
38
annually after land application to agricultural soils, contributing to degradation of downstream
39
freshwater resources and coastal environments.1,2 Recent analysis of the U.S. EPA’s Safe
40
Drinking Water Information System database found that although surface water trends for
41
nitrate violations have recently declined, there has been an increasing trend in groundwater
42
contamination from sources including agricultural runoff, confined animal feeding operations and
43
improperly functioning septic systems.3 Agrichemical contamination of surface water and
44
groundwater present environmental and human health risks, thereby necessitating monitoring
45
efforts capable of tracking this pervasive and wide-spread problem.
46 47
Citizen science, also referred to as crowdsourced science, is a contemporary research method
48
recently highlighted as an important research technique by the National Academies.4 Unlike
49
traditional sampling strategies, citizen science can address several logistical and technical
50
impediments to large-scale water quality sampling such as prohibitive per sample costs, small
51
numbers of sample, and the logistical issues of collecting samples over large temporal and
52
spatial scales.5,6 Simple and affordable test kits provide the potential for increasing sample size
53
and participation by a broad audience of participants.7,8 Large sampling geographies that are
54
both time and cost prohibitive for a single laboratory to sample are not limiting to citizen science
55
campaigns that have monitored water quality across cities,7,9-11 shorelines,12-15 and watersheds
56
as large as the Mississippi River.16 Given the scale of most contemporary water quality
57
problems, especially as they relate to non-point source runoff, citizen science is an important, if
58
not essential, tool for large-scale data collection and monitoring efforts.
59
3 ACS Paragon Plus Environment
Environmental Science & Technology
60
Despite its many advantages, the accuracy of citizen science-collected data remains a scientific
61
concern. From the perspective of organizers of citizen science programs, potential sources of
62
data bias include lack of proper study design, non-standardized protocols, sustained
63
participation and measurement errors from tools of varying complexity.17-20 One aspect of this
64
concern that has yet to be thoroughly explored is the role of pre-existing STEM experience and
65
its influence on the accuracy of citizen scientists. As reviewed by Lewandowski and Specht,21
66
several studies have demonstrated that citizen scientist-collected data is comparable to that of
67
professionally-collected data. For example, citizen scientists in Toronto, Canada, reported
68
measurements of PO4 and NO3 from colorimetric chemical assays that were comparable to
69
historical surface water data collected from professional scientists.7 To date, there is little
70
evidence regarding limitations in accuracy between sub-groups of citizen scientists, such as
71
differences in age or STEM background.
72 73
If citizen science is to serve as a complementary data collection method to existing and
74
emerging water quality monitoring programs for nutrient and agricultural contaminants, the
75
accuracy of citizen science across different experience levels must be assessed. Thus, the
76
objectives of this study were to 1) identify the proportion of accurate observations by citizen
77
scientists using rapid assessment water quality tools, and 2) characterize how expertise was
78
associated with the proportion of accurate observations made by citizen scientists using these
79
tools. To achieve these objectives, we conducted test groups with over 136 citizen scientists
80
and compared their results from water quality testing of spiked laboratory solutions and field
81
collected samples to results obtained using laboratory analytical methods. The contaminants
82
evaluated in this study were two nutrients, nitrate and phosphate, and a single herbicide,
83
atrazine, which are common non-point source pollutants detected across the United States.
84
Given their nearly ubiquitous geographic distribution and availability of rapid assessment test
85
strips, these chemicals are well-suited for monitoring by citizen science programs. 4 ACS Paragon Plus Environment
Page 4 of 27
Page 5 of 27
Environmental Science & Technology
86 87
2. Materials and Methods
88
2.1 Citizen Scientist Recruitment and Study Design
89
To evaluate the accuracy of the citizen scientist-collected data, volunteers were recruited from
90
various student, community and professional groups in Omaha, NE. In this study, the
91
participants were recruited using strategies previously employed by the authors in prior citizen
92
science work conducted since 2011 and were not recruited as a convenience sample. Specific
93
recruitment strategies included hosting booths at professional conferences for voluntary
94
participation, holding voluntary drop-in events at the University of Nebraska-Omaha’s
95
Community Engagement Center, recruitment from among prior citizen science volunteers, and
96
advertising in Omaha area schools to work with science educators and students. Approximately
97
40 of the 86 STEM college students participated in the testing as part of a laboratory course that
98
they were taking for credit. These volunteers were grouped into three separate experience
99
classes: expert, experienced, and inexperienced (Supplemental Table S1). Inexperienced
100
testers were defined as testers with no significant laboratory experience or prior training with
101
water quality testing which included middle and high school students, as well as college
102
students without exposure to college level STEM coursework. Experienced testers were those
103
with some exposure to laboratory testing, through a college level course or other means,
104
whereas expert testers were those that had extensive prior experience with water quality tests
105
or other related laboratory testing methods. The citizen scientists included middle, high school
106
and college students as well as individuals recruited from two professional organizations and a
107
local company’s sustainability team.
108 109
Figure 1 outlines the study design used for the collection and analysis of the citizen scientist
110
measurements that were collected from a series of educational workshops from October 12
111
through November 17, 2016. Initially, measurements were collected from laboratory prepared 5 ACS Paragon Plus Environment
Environmental Science & Technology
112
solutions provided to inexperienced and experienced participants to screen for issues
113
associated with the training and instructional pamphlets provided along with the test strips.
114
Following adjustment to the instructions, described below, a larger pool of participants including
115
expert (n=37), experienced (n=43) and inexperienced (n=55) citizen scientists were provided
116
field-collected stream water samples for assessment of accuracy (Supplemental Table S1).
117 118
2.2. Nitrate, Phosphate and Atrazine Test Strips
119
Test group participants evaluated three water quality parameters: nitrate, phosphate (measured
120
in parts per million (ppm)) and atrazine as a presence/absence test. Measurement of nitrate and
121
phosphate were conducted utilizing colorimetric test strips manufactured by Hach, a similar test
122
strip platform used for a previously published citizen-science survey of nutrients.28 The color
123
scale for the test strip of each nutrient indicates discrete concentrations of contaminant after a
124
30-60 second development period (Figure 1). Measurement of atrazine was done using atrazine
125
test strips (Abraxis) which detects the presence of atrazine at or above 3 ppb following a 10-
126
minute incubation period. Each citizen scientist participant was provided with two test strips for
127
each chemical parameter where one test strip was used to test a laboratory-prepared water
128
sample and the other test strip was used to test a field-collected (natural) water sample. Citizen
129
scientists were provided with two samples in identical containers and were not informed at the
130
time of testing as to the origin of the samples.
131 132
2.3 Instructional Pamphlets
133
At the beginning of each test group, a short verbal introduction and demonstration of the test
134
strips was given along with instructions. The citizen scientists were each issued a set of written
135
instruction for the tests that they would be performing on the spiked samples (see Supplemental
136
Figure 1). The instructions had data entry locations for each of the citizen scientists to record
137
their test results. As the original data collection was done through the interpretation of 6 ACS Paragon Plus Environment
Page 6 of 27
Page 7 of 27
Environmental Science & Technology
138
handwritten data entries, some of the data were either disregarded or unusable. This was
139
caused due to illegible notation. Of the original data set of 136 testers, the percentage of
140
unusable data was 1.5%, 15% and 11.8% for nitrate, phosphate and atrazine tests, respectively,
141
as these results were either illegible or the citizens scientists were observed copying another’s
142
response.
143 144
Follow initial testing with laboratory prepared solutions, it was determined that a manufacturer
145
discrepancy in the position of the color-changing pad on the atrazine test strip lead to difficulties
146
in use interpretation of the atrazine test strip. This was addressed by a modification of the
147
instructional pamphlets (Supplemental Figure 2) where the interpretive diagram of the blue
148
indicator bars for positive and negative readings on the atrazine test strip were adjusted to
149
reflect the discrepancy of specific manufacturer lots. These modified instructions were utilized
150
for the remaining tests on field-collected water samples.
151 152
2.4 Laboratory Solution Testing
153
Laboratory prepared solutions for the initial tests were prepared so that participants received
154
differing combinations of low, medium and high concentrations (Figure 1) of nitrate, phosphate
155
and atrazine. These solutions were prepared by mixing concentrated stock solutions of each
156
compound with filtered tap water to produce three solution combinations for the testing groups
157
on October 12th and October 21st, 2016. All stock solutions were stored at or below 4 °C. Stock
158
solutions were diluted using Omaha tap water to achieve combinations of varying
159
concentrations for each of the three compounds. 1 L samples were collected for analytical
160
testing of nitrate, phosphate and atrazine concentrations in the diluted solutions by a contract
161
laboratory (Midwest Labs, Omaha, NE).
162 163
2.5 Field Water Testing 7 ACS Paragon Plus Environment
Environmental Science & Technology
164
Field samples were collected from Elmwood Creek, a stream that flows through central Omaha
165
and is surrounded by residential housing, a golf course, recreational park and small shopping
166
complex. Similar to the laboratory prepared solutions, samples of the Elmwood Creek water
167
were sent to Midwest Laboratories for measurement of phosphate, nitrate, nitrite and atrazine
168
concentrations (Supplemental Table S2).
169 170
2.6 Statistical Analysis
171
All data were analyzed using JMP 11 software (SAS, Cary, NC, USA). All reported
172
measurements collected from the expert, experienced and inexperienced participants were
173
compared with the actual concentrations of nitrate, phosphate and atrazine determined through
174
the analyses of the water samples by Midwest Labs. From this, the reported responses were
175
scored as either accurate, underestimated or overestimated. Accurate responses were either 1)
176
those where the reported response matched the actual concentration (e.g., participant reported
177
5 ppm of phosphate when the actual concentration was 5.41 ppm), or 2) those where the
178
reported response flanked an actual concentration that was not discretely recognized by the
179
specified test (e.g., participant reported 5 or 10 ppm of nitrate when the actual concentration
180
was 8.30 ppm). Underestimated responses were those that were below actual concentration or
181
below the available lower boundary for the appropriate flanking response option on the test
182
strip. Overestimated responses were those that were above actual concentration or above the
183
available upper boundary for the appropriate flanking response option on the test strip. For the
184
measurements of water samples from Elmwood Creek, Chi-square tests were applied to detect
185
differences in the proportion of each of these response types between user experience level for
186
each of the test strips. For all statistical tests, statistical significance was assumed at p
