Article pubs.acs.org/est
Use of Remotely Reporting Electronic Sensors for Assessing Use of Water Filters and Cookstoves in Rwanda Evan A. Thomas,*,† Christina K. Barstow,‡ Ghislaine Rosa,§ Fiona Majorin,§ and Thomas Clasen§,∥ †
Department of Mechanical Engineering, Portland State University, Portland, Oregon 97201, United States Department of Civil, Environmental and Architectural Engineering, University of Colorado at Boulder,Boulder, Colorado 80302, United States § Department of Infectious and Tropical Diseases, London School of Hygiene and Tropical Medicine, London WC1E 7HT, United Kingdom ∥ Department of Environmental Health, Rollins School of Public Health, Emory University, Atlanta, Georgia 30322, United States ‡
ABSTRACT: Remotely reporting electronic sensors offer the potential to reduce bias in monitoring use of environmental health interventions. In the context of a five-month randomized controlled trial of household water filters and improved cookstoves in rural Rwanda, we collected data from intervention households on product compliance using (i) monthly surveys and direct observations by community health workers and environmental health officers, and (ii) sensor-equipped filters and cookstoves deployed for about two weeks in each household. The adoption rate interpreted by the sensors varied from the household reporting: 90.5% of households reported primarily using the intervention stove, while the sensors interpreted 73.2% use, and 96.5% of households reported using the intervention filter regularly, while the sensors interpreted no more than 90.2%. The sensor-collected data estimated use to be lower than conventionally collected data both for water filters (approximately 36% less water volume per day) and cookstoves (approximately 40% fewer uses per week). An evaluation of intrahousehold consistency in use suggests that households are not using their filters or stoves on an exclusive basis, and may be both drinking untreated water at times and using other stoves (“stove-stacking”). These results provide additional evidence that surveys and direct observation may exaggerate compliance with household-based environmental interventions.
■
INTRODUCTION
Even if subjects do not overstate their water treatment practices, research from other studies has shown that people who report treating their water often fail to do so consistently or correctly, frequently practicing water treatment only when they have time, the required resources, or a perceived need for it.6−9 At the same time, epidemiological modeling has suggested that even occasional exposure to untreated water can vitiate the potential health benefits from safe drinking water.10,11 Electronic sensors offer the potential to reduce bias in monitoring use and performance of environmental health interventions, yielding more accurate assessments of program targets and potential health impacts. There are several related sensor development and applications efforts, including work conducted at the University of California at Berkeley on indoor air pollution instrumentation including a particle monitor,12 a stove temperature sensor that has been used to study stove use behavior,13 a hand-pump motion monitor with remote
Efforts to assess the impact of household-based environmental interventions such as water filters and cookstoves often rely on data collected through person-to-person surveys or subjective observations. However, this conventional approach has two major shortcomings. First, surveys often overestimate adoption rates due to courtesy bias (where the participant is attempting to please the surveyor) or recall bias (tendency to forget details in more distant past). Second, the presence or repeated visits of observers or enumerators can cause reactivityinfluencing the behavior they are measuring. Several studies have demonstrated that study participants consistently and significantly over-report “good” practices;1−4 this has been observed to be the case with reported handwashing, and may well be the case with household water treatment or cookstove use. A published study of an evaluation of a 3-year household water treatment and handwashing intervention found significant differences between self-reported water treatment use and confirmed use, providing some suggestive evidence for the over-reporting of water treatment.5 Results showed that self-reported household water treatment (HWT) use was 3.8−6.4 times higher than confirmed use. © 2013 American Chemical Society
Received: Revised: Accepted: Published: 13602
August 1, 2013 October 23, 2013 November 4, 2013 November 4, 2013 dx.doi.org/10.1021/es403412x | Environ. Sci. Technol. 2013, 47, 13602−13610
Environmental Science & Technology
Article
Table 1. Relevant Household Survey Parameters, Questions Asked, and Logic Applied parameter
question (asked in Kinyarwanda)
reported stove adoption
Primarily, which type of stove do you currently use?
reported stove uses per day reported filter use (a)
How many times per week are you using the EcoZoom stove? The last time you drank water, did you treat it?
reported filter use (b)
How did you treat the water?
reported water volume treated per day reported number of filter uses per day reported water volume treated per use
If you treat your water, how many liters of water per day are treated? How many times per day do you add water to the LifeStraw filter? Primarily, what size container is used to fill the LifeStraw filter?
options (not read out to respondent, shown only on phone to enumerator) 1. EcoZoom 2. 3-stone 3. Rondereza 4. Imbabura 5. Mud woodstove 6. Ceramic woodstove 7. Metal woodstove 8. Charcoal stove 9. Improved charcoal stove 10. Gasifier stove 11. Biogas stove 12. LPG 13. Kerosene 14. Electric 15. Other Times per week________
logic applied Total number of valid survey respondents answering “EcoZoom” over total number of valid surveys yields percentage of reported adoption for provided stove.
Applies a “0” for any non-EcoZoom stove users. Excludes outliers outside 1.5 × interquartile range. Divides value by 7 to yield reported uses per day.
1. Yes 2. No 1. Filtered with Lifestraw 2. Filtered with other filter 3. Boiled 4. PUR 5. Other Amount liters _________
If answer is “yes”, triggers next question. Otherwise, next question skipped automatically by phone.
Times per day________
Applies a “0” for any non LifeStraw users. Excludes outliers outside 1.5 × interquartile range.
1. 1 2. 2 3. 3 4. 5 5. 7 6. 10 7. 15 8. 20 9. 30 10. Other
Considers only reported LifeStraw users. Does not consider “Other” responses.
Total number of valid survey respondents answering “Lifestraw” over total number of valid surveys yields percentage of reported adoption for provided filter. Considers any nontreaters (answering “no” to previous question) as nonfilter users.
Applies a “0” for any non LifeStraw users. Excludes outliers outside 1.5 × interquartile range.
reporting developed at the University of Oxford,14 and a passive latrine use monitor for sanitation studies developed by the University of California at Berkeley and the London School of Hygiene and Tropical Medicine.15 In Rwanda, the Ministry of Health has partnered with a forprofit company, DelAgua Health, to design a program to distribute household water treatment and high efficiency cookstoves to approximately 600,000 households, about three million citizens covering all 30 districts of Rwanda. The technology distributions include a household-scale water treatment system that addresses microbiological contamination and a high-efficiency cookstove that addresses indoor air pollution, two of the leading risk factors for illness in children under 5 years of age in Rwanda.16 As part of a pilot program, researchers conducted a five-month randomized, controlled trial (RCT) to assess use of the devices and the impact of their intervention on drinking water quality and household air pollution (HAP). This RCT was designed to derive differential exposure to air and water quality between control and intervention groups, randomly assigned on a household level
in three villages out of the 15 villages that received the products in the same period. Usage of the devices was assessed in the RCT intervention population by using both conventional household surveys and observations by field enumerators, and sensor-equipped filters and cookstoves that relay usage data to the Internet over the cell phone network. This paper summarizes results regarding the use of intervention filters and stoves and compares conventional survey/observational data with those from instrumented devices within this RCT. We also demonstrate how the data from instrumented devices can be used to explore patterns of use, and especially consistency of use. This is the first known application of sensors to evaluate consistency in use for water filters within a randomized controlled trial. The broader RCT study is described in another forthcoming paper.17
■
MATERIALS AND METHODS Intervention. The intervention consisted of Vestergaard Frandsen LifeStraw Family 2.0 water filters and EcoZoom Dura
13603
dx.doi.org/10.1021/es403412x | Environ. Sci. Technol. 2013, 47, 13602−13610
Environmental Science & Technology
Article
improved cookstoves distributed at community meetings in each village. The water filter uses a ultrafiltration membrane that meets the World Health Organization “highly protective” rating for household water treatment. The filter is a tabletop unit with a 6-L input bucket on top where untreated water is introduced, and a safe water storage container below it with a tap to access the filtered water. The unit filters about 2 L per hour. The stove is a “rocket-stove” design intended to reduce fuel use and emissions.18 The stove has a 32-in. cast iron top with a “stick support” for introducing wood fuel, and carrying handles to allow mobility. At the time of distribution, householders received a group demonstration on the operation and maintenance of the hardware, together with posters with graphic and written instructions. Among other things, they were encouraged to always drink filtered water and to use the stove in lieu of traditional household stoves. Distributions were followed by monthly household-level education and follow-up visits by the Community Health Workers (CHWs) to reinforce messages, address problems, and answer questions. During these follow up visits, CHWs also conducted surveys and spot checks on use in addition to collecting other information to evaluate the program. The program is described in greater detail in Barstow et al.18 Surveys. After product distribution, CHWs commissioned by the Rwanda Ministry of Health and trained by DelAgua for this evaluation visited every household that received a stove and filter and completed a phone-based survey which captured baseline information about the household’s water and cooking practices, GPS coordinates of the household, household identifier information, and technology barcodes. During follow-up campaigns, CHWs and more highly trained supervising environmental health officers (EHOs) visited each household to conduct spot check observations and collect additional data on maintenance and use of the filters and stoves. All data from surveys and spot checks were entered directly into smartphone-based questionnaires and, similar to the sensor data, stored online in a secure SQL database. The survey questions used for the parameters of interest in this paper, along with the logic applied to the responses, are shown in Table 1. Instrumented Filters and Stoves. A set of 23 LifeStraw Family 2.0 water filters and 27 EcoZoom Dura cookstoves were instrumented with the SweetSense technology, as described in Thomas et al.19 (Figure 1). The sensors cost approximately US $500 per stove or filter to produce and install. Between October 2012 and March 2013, these 50 sensor-equipped stoves/filters were rotated through randomly selected households within the three RCT villages for approximately 2 weeks in each household, yielding a total of 118 observations covering 41% of the RCT households over the 5-month study. For each installation, trained enumerators carried the sensor-equipped stove and filter pair to a household and introduced the sensors to the head of household. Households were informed that the purpose of the sensors was to collect performance data on the stoves and filters. Households were not explicitly informed that the sensors collected water volume or use frequency data. If consent to participate in the study was given, the enumerator then used a “check in” smartphone survey that included manually entering the existing household stove and filter barcode, and scanning the sensor-equipped stove and filter. Then, the enumerator temporarily “locked out” the household stove/filter pair by locking them together to prevent use, and encouraged the household members to use the sensor-equipped
Figure 1. Sensors on household water filter.
units instead until a future visit. Most of the units were cellular (GSM) reporting only, while several had back-up SD cards that allowed later manual data retrieval (5 of the filters and 6 of the stoves). Over the course of the study, some sensor boards were repaired, replaced, or decommissioned, leading to a loss of some data. Data, including water pressure in the input bucket of the water filter and temperature of the combustion chamber of the cookstove, were periodically uploaded (nominally at midnight local time) over the cellular network directly to an online database and Web site at www.sweetdata.org as text files. About 2 weeks later, enumerators visited again to recover the sensor-equipped devices and unlock the householder’s own filter and stove. The remote reporting nature of the sensors allowed for data verification and review during the installation periods, as well as reduced the manual handling of the data. Uploading of Sensor Data. A C++ routine, running every 6 hours, interpreted the raw sensor text file data and stored the decoded data in an Amazon EC2-hosted MySQL database. Every MySQL data file included the unique ID of the sensor based on a MAC chip on board, the sensor type (filter or stove), the date and time of each sensor reading, and the date and time of every successful transmission; it also included battery level and received cellular signal strength. The data relayed did not include any household information or locations. Also on a 6-hour basis, algorithms written in R (www.r-project. org) analyzed the MySQL stored data and produced derived “events”. In the case of the water filter, these include each identified addition of water to the input bucket, the time/date, and amount. For the cookstove, the start and stop time of each cooking event is reported. These events are stored in a complementary MySQL table. The water filter sensor algorithm was validated as described in Thomas et al. 2013.19 The stove algorithm is identical except for variable coefficients adjusted based on observation to match the slope profile of the stove temperature signal. The stove algorithm was not validated with structured observations because of the inherently more straightforward signal. Interpretation of Sensor Data. An R script then imported these records and parsed the algorithm-derived event data for each sensor and grouped the data into household-level sets, based on the dates and times of each installation and correlating household filter and stove barcodes to the corresponding IDs of the sensors installed. Another R script imported the cleaned EHO survey data sets (as corrected for data entry errors by the 13604
dx.doi.org/10.1021/es403412x | Environ. Sci. Technol. 2013, 47, 13602−13610
Environmental Science & Technology
Article
study managers) for all 5 monthly rounds of follow-up and extracted survey-reported stove use (units of times per week) and water filter use (units of liters added to the filter per day), as well as the EHO-recorded household filter and stove barcodes. Finally, a statistical analysis was conducted in R that examined the following: 1. Comparison of average daily water treatment events and corresponding volumes, and daily cooking events across households during each survey round for both sensor and survey data. Surveys and sensors were compared for each round, and surveys and sensors were each analyzed for variance across the rounds. 2. Paired comparisons of survey- versus sensor-reported data on the household level. This comparison did not disaggregate based on survey round, i.e., though a single household may have been surveyed in November and had a sensor installed in February, this comparison matched only on household ID criteria. The EHO surveys were conducted only once in each household during the study. While the CHW surveys were monthly in each household, they did not ask pertinent usage questions. A linear regression was applied to evaluate correlation between reported and recorded data on a household level. 3. Comparison of the household pairings subset for both survey and sensor data to the aggregate data to evaluate if the subset samples are different from the total samples. 4. An analysis of variance (ANOVA) was conducted across survey and sensor rounds to evaluate if there was a significant change in reported or recorded behavior over the 5-month period. The ANOVA analyses were conducted independently for the survey and sensor data sets. 5. The number of people in the household was then applied to the sensor-recorded data to yield units of use per person per day, and intrahousehold behavior was examined for consistency in use behavior over the installation period in the household. 6. Sensor data were grouped for comparisons between roughly week 1 and week 2 of installation, or more accurately average use between the first day of installation and halfway to the last day of installation, and from halfway to sensor removal. Various assumptions were made in the statistical analysis that impacted the outputs. The most significant included the following: 1. For all sensor data, the total event data detected for the household installation period were averaged over the total number of successful data transmissions recorded, minus one day to account for the morning of installation and the evening of removal. This allowed consideration of data between households in terms of water volume or cooking events per day, regardless of the number of days of installation or of any lost periods of data. 2. An outlier analysis was conducted for both the summarized household-level data and the survey data that at first excluded any values for stove/filter use in excess of 1.5 times the interquartile range. In the case of the sensor data, the excluded households were than manually reviewed and several outliers reintroduced as likely valid samples as judged by what appeared to be actual filter/stove use. Other outliers were most likely
associated with the algorithm misinterpreting sensor data. 3. The data sets for both sensors and surveys were generally not normally distributed, likely due to clustered low-end reported and recorded behavior, and some valid outliers. Therefore, groups were compared using the Wilcox ranked sum test that is less sensitive to non-normal data than the t test. 4. A subset of sensor data was further manually excluded after review of the raw signal and algorithm-derived event detections. In most cases, these exclusions were because of obvious hardware problems associated with ungrounded analog to digital converters that led to inaccurate or unrecorded sensor data. Consistent Use Evaluation. We also compared survey- and sensor-collected data on consistency of use of both stoves and filters. For the stoves, intrahousehold consistency in use was evaluated by examining the proportion of uses with gaps between use of less than 12 h, suggesting use at least twice per day. In the RCT, 76.3% of households reported using their stove twice per day in the preintervention baseline. Evaluating consistent use with the household water filter necessarily considers the household size and composition. A WHO report referenced a minimum of 1 L per day for children under 10 and 1.4 L for adult males under normal conditions.20 In this analysis, household sizes were disaggregated by individuals less than 5 years old with a minimum consumption of 1 L per child per day, and older than 5 years with a minimum of 1.4 L applied, to yield a total targeted minimum daily water consumption per unique household. This volume may be an overestimate, as small children may be breastfeeding instead of drinking water. HH rec. min. water consumption(liters) = (no. < 5) · day ⎛ 1.4 L ⎞ ⎛1L ⎞ ⎟ ⎟ + (no. > 5) ·⎜ ⎜ ⎝ day ⎠ ⎝ day ⎠ (1)
Then, a maximum gap in hours between water filter uses is derived based on this minimum water consumption requirement for each household and the sensor-recorded mean water volume treated per usage event, to account for variability on the household level in water volume treated per use. This period may be an underestimate, as households may store filtered water in other containers, thereby lengthening the required treatment period. maximum gap between filter use(hours) = 1 HH rec. min. water consumption(liters) day
24 hours 1 day
·
(HH mean volume per use(L))
(2)
Not all household use events were considered. A subset of all uses was considered wherein the water volume added was greater than the mean water volume per use for that household minus one standard deviation. This attempts to account for variability within household and between households for typical fill volumes. 13605
dx.doi.org/10.1021/es403412x | Environ. Sci. Technol. 2013, 47, 13602−13610
Environmental Science & Technology
Article
Table 2. Stove Adoption and Daily Use Reported by Survey and Recorded by Sensor for All Available Data, Subset of Data in the Three RCT Villages, and among Sensor Data with Direct Household Pairings to Surveys stove adoptiona
stove use per dayb
Wilcox rank sum test
%
n
value
SD
n
p-value
all EHO surveys (15 villages) RCT EHO surveys (3 villages)
90.5 87.9
1634 226
1.27 1.53
0.73 0.80
1633 257
RCT sensor data (3 villages)
73.2
96
1.10
1.10
96
median diff.
95% CI
HH mean volume per use − HH SD volume per use
consumption HH use events separated by less than maximum gap = considered uses
(3)
Events are then identified that are separated by less than the
·100
household specific maximum use gap.
Ethics. The study was reviewed and approved by the Rwanda National Ethics Committee, University of Colorado Institutional Review Board, the Portland State University Institutional Review Board, and the London School of Hygiene and Tropical Medicine Ethics Committee. Households participated in the study only if they consented after receiving complete details regarding the purpose of the study and the use of data collected. Participants were given the opportunity to ask any questions before agreeing to participate.
HH use events separated by less than maximum gap = no.(considered use time − previous considered use or install time(hours)) < max gap
(5)
(4)
And finally these events are considered as a percentage of total considered household events. 13606
dx.doi.org/10.1021/es403412x | Environ. Sci. Technol. 2013, 47, 13602−13610
Environmental Science & Technology
■
Article
RESULTS
Of 118 total sensor installations for both stoves and filters, we obtained usable data sets for 97 (82%) stoves and 82 (69%) filters. This resulted in paired (but not necessarily synchronous) sets of survey and sensor data for 70 (72.2%) stoves and 63 filters (76.8%) from the same households. The mean installation period duration of valid data sets was 12.9 days (2− 15 day range) for the water filters and 9.8 days (0−15 day range) for the stoves. Stoves. Although 90.5% of households reported that they primarily used the intervention stove, the adoption as interpreted by the sensors was only 73.2%. The reported adoption for the intervention cookstove was 10−15% higher than that recorded by the sensor, based on the sensor recording at least two events during the installation period (suggesting at least one more use than the day of installation). The RCT households (3 villages) reported weekly stove use was significantly higher than the overall program (all 15, inclusive of the RCT), by about 1 use per week (p value 0.0018) (Table 2). On average, the sensors recorded approximately 0.3 fewer uses per household per day, or about 2.2 fewer uses per week than the self-reported data within the RCT. The sensor and RCT survey data were disaggregated on a survey round basis (Figure 2). ANOVA analysis was conducted for the sensor and survey rounds separately, and indicated that there is a significant difference in use of the stove between rounds for the survey reported data (p-value
