Structured Expert Judgment to Characterize ... - ACS Publications

Article pubs.acs.org/est

Structured Expert Judgment to Characterize Uncertainty between PM2.5 Exposure and Mortality in Chile Pamela C. Cisternas,*,†,‡ Nicolas C. Bronfman,†,‡ Raquel B. Jimenez,† Luis A. Cifuentes,‡,§ and Cristobal De La Maza∥ †

Engineering Sciences Department, Universidad Andres Bello, Santiago 8370146, Chile National Research Center for Integrated Natural Disaster Management CONICYT/FONDAP/15110017, Santiago, Chile § Industrial and Systems Engineering Department, Pontificia Universidad Católica de Chile, Santiago, Chile ∥ Chile Foundation, Santiago, Chile ‡

S Supporting Information *

ABSTRACT: To further the understanding and implementation of expert elicitation methods in the evaluation of public policies related to air pollution, the present study’s main goal was to explore the potential strengths and weaknesses of structured expert judgment (SEJ) methodology as a way to derive a C-R function for chronic PM2.5 exposure and premature mortality in Chile. Local experts were classified in two groups according to background and experience: physicians (Group 1) and engineers (Group 2). Experts were required to provide an estimate of the true percent change in nonaccidental mortality resulting from a permanent 1 μg/m3 reduction in PM2.5 annual average ambient concentration across the entire Chilean territory. Cooke’s Classical Model was used to combine the individual experts’ assessments. Experts’ mortality estimations varied markedly across groups: while experts in Group 1 delivered higher estimations than those reported in major international cohort studies, estimations from Group 2 were, to varying degrees, anchored to previous studies. Accordingly, combined distributions for each group and all experts were significantly different, due to the high sensitivity of the weighted distribution to experts’ performance in calibration variables. Results of this study suggest that, while the use of SEJ has great potential for estimating C-R functions for chronic exposure to PM2.5 and premature mortality and its major sources of uncertainty in countries where no studies are available, its successful implementation is conditioned by a number of factors, which are analyzed and discussed.

■

INTRODUCTION Air pollution is a major risk factor for public health, since public exposure to air pollution has been linked to numerous negative health impacts. Over the last two decades, time series and cohort studies have found evidence of a relationship between short- and long-term exposure to fine particulate matter (PM2.5) and increased incidence of adverse health impacts such as cancer, cardiovascular and respiratory diseases, and premature mortality.1−10 In these studies, the relationship between ambient concentration of air pollutants and the incidence of a specific health effect is described using concentration−response (C-R) functions. More specifically, results from cohort studies suggest significant relationships for long-term PM2.5 exposure and increased mortality from general and cardiovascular diseases. The most cited studies were developed in the 90s.11,12 More recent studies have been performed reanalyzing data from previous work2,13−18 and using new data.19−23 While, in general, these studies have reached some degree of consensus regarding the existence of this relationship, significant variability is observed in C−R © XXXX American Chemical Society

functions across studies (see, for example, results from Pope et al. (2002)2 (P50 = 0.6, CI = 0.16−1.1), versus results from Laden et al. (2006)16 (P50 = 1.6, CI = 0.7−2.6)). Additionally, there remains significant uncertainty about the true nature of the relationship between PM2.5 ambient concentration and increased incidence of health end points, provoking debate among experts in the field. The major sources of uncertainty are related to (i) the existence of a causal relationship between health impacts and PM2.5 exposure, (ii) if there is a relationship, what is the shape of the C−R function and does it present a threshold for effects, (iii) the time window between exposure and the first manifestation of effects, and (iv) the use of PM2.5 mass as a measure of its potential toxicity.24 Epidemiological evidence from long- and short-term studies provide the informational basis for assessing air pollution Received: January 28, 2014 Revised: June 27, 2014 Accepted: July 7, 2014

A

dx.doi.org/10.1021/es500037k | Environ. Sci. Technol. XXXX, XXX, XXX−XXX

Environmental Science & Technology

Article

calibration variables are defined in section 2.4), eight related to premature mortality from short- and long-term PM2.5 exposure, two to the toxicity of PM2.5 components, and two related to health effects from exposure to the Kuwait oil fires in 1991. Additionally, 12 calibration questions were included and were combined with expert estimates using the Classical Model (see the Classic Model section for a description). Both studies confirm that implementing expert judgment to quantify uncertainty in the PM2.5-mortality C−R function enables the combination of empirical results with expert judgments to achieve a comprehensive approach to the uncertainty inherent in assessing adverse health effects from PM2.5 exposure. Studies in Chile. Despite the high levels of air pollution observed in medium-size and large cities across Chile, only a few studies addressing public health impacts from exposure to atmospheric pollutants have been developed locally. Cifuentes et al.46 estimated a 4.2% increase in nonaccidental daily mortality attributable to PM2.5 ambient concentrations of 64 μg/m3in Santiago. Sanhueza et al.47 found strong relationships between PM10 exposure and daily cardiovascular and respiratory mortality in Temuco (southern Chile). Other studies have focused on increased rates of hospital48−51 and emergency admissions.52−54 So far, no study has been developed in Chile to estimate the increment in premature mortality attributable to long-term PM2.5 exposure. As a result, regulations and standards to protect public health from adverse effects of air pollution have been developed using results directly transferred from international studies. For example, in determining the primary environmental quality standard for PM2.5 in Chile, nonaccidental mortality attributed to chronic PM2.5 exposure was quantified using results from Pope et al.2 It is worth noting that the direct transference of results assumes that the mortality risk from PM2.5 is independent of the area of analysis, that is, independent of demographic characteristics, health status of the population, pollutant mixture and climatological features. As the understanding of how these factors modify the effects of PM2.5 exposure is rather limited, is reasonable to assume that the direct transference of results from international studies adds uncertainty to local estimations of health benefits from air quality improvements. Within this context, the development of local studies provides relevant information to better understand how PM2.5 exposure impacts local population. Including this information in the assessment of public health impacts from pollutant exposure will provide results that better reflect the local context, as it complements the international literature with more specific, local information for a more complete characterization of impacts and uncertainties in this process. Overview of the Present Study. The economic evaluation of public health risks and benefits from reductions in air pollution ambient concentrations using C−R functions is fundamental to evaluating public policies and normative instruments aimed at improving air quality. In Chile, air quality standards for PM2.5 have been evaluated using C−R functions directly transferred from international studies, as no study has been conducted in this country to estimate the relationship between premature mortality and long-term PM2.5 exposure. While directly transferring results adds uncertainty to this process, developing cohort studies involves high costs and long periods of time to get results, both fundamental limitations to the widespread implementation of this type of study. For these reasons, in countries where epidemiological cohort studies of

management strategies and normative instruments. For example, C−R functions are key inputs in assessing public health benefits from reductions in adverse health effects expected as a result of air quality improvements. Nevertheless, there are a limited number of cohort studies addressing longterm exposure to PM2.5, mainly because of the high economic costs involved in their development and the amount of time it takes to get conclusive results. This has led to the use of alternative methodologies for estimating C−R functions in countries where no cohort studies are available. These alternatives include directly transferring results from studies developed elsewhere and using expert judgment to adjust such results (both central estimates and uncertainty intervals) to better represent local impacts from exposure to this pollutant. In the last few decades, the structured expert judgment (SEJ)a tool used to assess uncertainty in complex problems where parameters are unknown or data is scarce−has been used to quantitatively characterize the state of knowledge and uncertainties about health effects from exposure to ambient pollutants. The main difference between expert judgment and SEJ is that the latter uses two measures of performance (calibration and information detailed in section 3 of the Supporting Information (SI)), by which a weight is assigned to each expert based on their performance on both measures. This allows to transmit the virtues of the “good experts” (those who have good performance in both measures) to the combined distribution (for more detail refer to refs 25 and 26). SEJ was originally developed to address the high uncertainty embedded in nuclear plant accident risk and consequence assessments in the U.S.27−35 In 2000, Cooke and Goossens36 developed a methodological guide for implementing SEJ for uncertainty assessments in quantitative models, thus opening the door to this tool’s application outside of the nuclear power community. Since then, expert judgment has been used in risk assessments for different areas, such as pollution in the food production chain,37,38 volcanic eruptions,39 inhalation of toxic chemicals40 and exposure to air pollution.41−43 In this study, we have implemented a SEJ in order to explore the potential strengths and weaknesses when used in the estimation and uncertainty assessment of local C−R functions for regulatory and policy assessment in Chile. Structured Expert Judgment and Its Application in PM2.5 Health Effects. In 2002, the U.S. National Research Council (NRC) suggested the U.S. Environmental Protection Agency (U.S. EPA) the use of expert judgment to determine the probability distributions of key sources of uncertainty in the information sources, process and key assumptions used to estimate health benefits from air pollution reductions.44 In 2006, after the successful implementation of a pilot study in 2004,43 the U.S. EPA developed a formal study incorporating expert judgment elicitation, in which a 12 expert panel (eight epidemiologists, three toxicologists and one physician) was implemented in order to address the major uncertainties surrounding the relationship between PM2.5 exposure and mortality. Additionally, experts made quantitative estimates of the impact of a 1 μg/m3 reduction in PM2.5 annual average on nonaccidental mortality in U.S. adults. However, it did not include a combination of expert estimates in a single probability distribution that reflected the experts’ degree of uncertainty.42,43 More recently, Tuomisto et al.45 implemented a SEJ, in which six European experts quantified uncertainty in the relationship between PM2.5 exposure and premature mortality. The protocol included 12 query variables (query variables and B



Article

relation to a background distribution (uniform or log-uniform), selected by the researchers overseeing the study. Such weights are assigned using Equal Weight criteria, where all experts have the same weight, or performance-based weight criteria, in which experts’ estimates are weighted according to their performance over the whole set of seed variables (Global Weight) or in each seed variable (Item Weight). Participants. Two Chilean experts in the field were contacted to identify potential participants by listing renowned local experts. In this way, a list of 16 potential participants was created based on the experts’ nominations, ISI journal publications, and other criteria, all of which have directed or participated in epidemiological studies related to air pollution exposure and adverse health effects in Chile. Seven of them agreed to participate in the study. Participation in the study was voluntary, and no economic or material compensation was offered. To maintain confidentiality, experts are designated throughout this document as “Expert A” through “Expert G”. In order to assess potential differences between experts from different backgrounds, the expert panel was divided into two groups according to field of expertise. Experts C and D comprised Group 1 based on their training as physicians and experience in the fields of environmental epidemiology and environmental and public health. Group 2 comprised experts A, B, E, F, and G, with experience in a variety of fields, including air pollution dispersion and exposure modeling, environmental epidemiology and cost-benefit analysis for evaluating environmental regulatory instruments. Also, experts in Group 2 have been actively involved in air pollution policies and regulatory decision-making processes. Seed Variables. Seven seed variables (S1−S7) were defined based on Tuomisto et al.45 Three variables were related to the number of days where daily average PM10 ambient concentration exceed 150 μg/m3 in at least one monitoring station in the Metropolitan Region (MR) in the years 2000 (S1), 2005 (S2), and 2010 (S3). Two variables were related to the effect of PM10 on all-cause mortality, estimated as the number of nonaccidental deaths in the week with the highest average PM10 concentration to the number of nonaccidental deaths in a typical week in years 2000 (S4) and 2005 (S6). The last two variables were related to the impact of PM10 on cardiovascular mortality, estimated as the ratio of the number of cardiovascular deaths in the week with the highest average PM10 concentration to the number of cardiovascular deaths in a typical week in years 2000 (S5) and 2005 (S7). A complete description of seed variables is provided in the SI. It is worth noting that in Chile, as well as in several other countries, high pollution concentration episodes are observed more frequently in the winter, influenced by atmospheric and meteorological conditions, such as lower temperatures, high atmospheric stability, and low ventilation. Such factors, in turn, strongly influence higher cardiovascular and respiratory morbidity and mortality rates observed in winter, compared to annual tendencies. Therefore, it is expected that experts’ estimates for variables S4, S5, S6, and S7 are greater than one. Experts delivered their valuations for each seed variable using 5th, 25th, 50th, 75th, and 95th percentiles. The information used to estimate the real values of seed variables was obtained from death certificates and historical records of PM10 ambient concentration from seven of the 11 air-quality monitoring sites in the MR. The real values are shown in Figure 1.

PM-mortality are not available, the use of SEJ has great potential to provide probabilistic characterizations of the effects of chronic exposure to fine particulate matter. To further the understanding and implementation of this instrument, the present study’s main goal is to explore the potential strengths and weaknesses of SEJ as a way to derive C−R functions for regulatory policy assessment and to characterize the uncertainty in these estimates.

■

MATERIALS AND METHODS Structured Expert Judgment: The Protocol. As defined by Cooke and Goossens,36 the SEJ protocol has three main stages: (i) preparation for elicitation, (ii) elicitation, and (iii) postelicitation. Preparation for Elicitation. First, the problem to be addressed, that is, the case study is defined. The main goal and uncertain parameters that experts must estimate are established. Once the case study is defined, query variables are defined to assess and quantify uncertain parameters directly related to the study’s goal. Then, calibration or seed variables are defined. These variables are used to quantify experts’ performance as subjective probability assessors and to enable performance-optimized combinations of expert distributions. Seed variables also provide valuable feedback to experts, helping them to gauge their subjective sense of uncertainty against quantitative measures of performance.34 The experts’ estimates for query and seed variables must be delivered in percentiles (i.e., P5, P25, P50, P75, and P95). Next, potential participants are identified, usually using peer nomination techniques. Then, a dry-run session is performed to verify that the elicitation document’s contents, structure and format are suitable. Prior to the elicitation, a training session must be carried out to ensure that the experts are familiar with the procedure. Elicitation. In the elicitation stage, experts must make estimates of seed and query variables using percentiles and clearly stating all assumptions and fundaments that support their probabilistic estimation. Post-Elicitation. For each query variable, experts’ estimates are combined into a single probability distribution, the decision maker distribution (DM), which quantifies the uncertainty inherent to the query variable. The DM is obtained using Cooke’s Classic Model, which is explained in the following section. Robustness analyses are performed to evaluate the significance of information loss in the combined distribution when experts or seed variables are omitted from the DM estimate. Individual estimates are sent back to each expert, with their respective scores for calibration, information and weight in the combined distribution. With this information, experts can modify their estimates. Finally, all relevant information and data are consolidated in a formal report to be presented to the project’s Decision Managers and the experts.36 Classic Model. The Classic Model25 allows researchers to estimate a combined distribution for each query variable by making a linear combination of experts’ estimates assigning weights according to two measures of performance in seed variables: calibration and information. A “good expert” is one with high calibration and information scores.31 The calibration score is the p-value of falsely rejecting the hypothesis that an expert’s probability statements are statistically accurate. The maximal value is 1, and the minimal is 0. Information refers to the degree to which the experts’ distribution is concentrated in C



Article

Figure 1. Expert panel estimates for seed variables S1−S7. Horizontal lines indicate the true value of each variable, unknown by experts. A complete description of seed variables is provided in SI.

Query Variables. The elicitation protocol included seven query variables Q1−Q7, one of which was the quantitative question (Q7). A complete description of query variables is provided in Table 1. Variables Q1 and Q2causality questionswere developed based on U.S. EPA 41 and

addressed uncertainty in the causality of the relationship between long- and short-term PM2.5 exposure and premature mortality in Chile. Variable Q3the question on transference of resultsaddressed the experts’ position regarding the validity of using results from epidemiologic studies developed D


a

Based on your interpretation of the evidence, do you want to incorporate a threshold in your characterization of the C−R relationship?

What is your estimate of the true percentage change in annual mortality from nonaccidental causes in the adult population of Chile (over 18 years) due to a permanent reduction of 1 μg/m3 in annual average PM2.5 concentrations (over the entire concentration range of 10−35 μg/m3)? In formulating its response, you must consider effects of reductions in PM2.5 concentrations on both the short and long-term mortality. To characterize the uncertainty in the C−R function, you must provide your estimate as percentiles (fifth, 25th, 50th, 75th, and 95th).

Q6

Q7

E

Note: Question Q4 refers to the shape of the C−R function. Questions Q5 and Q6 are related to threshold. Question Q7 is the quantitative question on mortality estimation in Chile.

Do you want to specify if the C−R function for PM2.5 and mortality differs across the specified concentration range? If your answer is yes, indicate the form of the functions, the approach to incorporate causality, and if it considers a concentration threshold.

Do you believe that a concentration threshold for annual nonaccidental mortality associated with population exposure to PM2.5 is detectable in any currently available study

Q5

Query Variables Related to Mortality Estimates for PM2.5 Exposure in Chile. Q4 Determine the shape of the C−R function for long-term exposure (you can make a sketch of the general shape of the function).

Query Variables Related to Transference of Results. Q3 Despite the growing number of studies about health effects of exposure to air pollution, the extrapolation of results obtained in one location to estimate impacts in another is very frequent. The assumption of the validity of transference of results states that the risk of mortality from exposure to PM2.5 is independent of the area of analysis. It is assumed that the demographic characteristics, pollutant mixture, the health status of the population and climatic features are transferable between study areas because the current understanding of how these factors modify the effects of exposure to PM2.5 is rather limited. Do you consider this as a reasonable assumption? Please specify your arguments.

What evidence do you considered to support or reject the possibility of a causal relationship between reductions in annual average exposure to PM2.5 (including long and short-term exposure reductions) and changes in mortality in Chile (at annual average 10−35 μg PM2.5/m3). Do you wish to make a distinction between short-term and long-term relationship? Briefly specify your arguments to make such distinction.

Q2

query variables related to causality between PM2.5 exposure and mortality in Chile.

In your opinion, is there a causal relationship between short/long-term exposure to PM2.5 and mortality? The answer must be delivered according to the U.S. EPA standard on “weight of evidence for the determination of causality”.24

Q1

Table 1. Query Variablesa

Environmental Science & Technology Article



Article

short-term exposure and likely to be causal for long-term, and experts A and E declared that there is a probable causal relationship in both short- and long-term exposure. The experts based their causality judgments for short-term exposure on studies conducted in Chile,46,50,56 and studies conducted in other countries,57.58 The experts cited international cohort studies,11,12,2,5960 and the WHO guidelines61 when making their causality judgments for long-term exposure and mortality. Transference of Results. Experts C, D, and F stated that they did not agree with the direct transference of results and the underlying assumption that mortality risk from PM2.5 exposure is independent of the demographic characteristics and, health status of the population, pollutant mixture and climatological features (Q3 in Table 1). In general, these experts argued that the assumption was not valid since it would violate the existence of four determinants: biological, socio-cultural, environmental variability and variability in healthcare systems. On the other hand, experts A, B, E, and G thought that direct transference was sensible. Mortality Estimates. Marked differences in estimations for the mortality questions, especially in the quantitative question, were observed between the two groups. Regarding the shape of the C−R function (Q4), experts in Group 2 determined that it was log−linear, whereas experts in Group 1 said the shape was log−linear by segments. However, none of the latter group’s members gave cutoff values for the log−linear function segments, due to the lack of evidence. While experts B, C, D, E, and F believed there was no threshold in the C−R function (Q5), experts A and G said there was a threshold. Even so, no expert included the existence of a threshold in their mortality estimates (Q6), within the range of ambient PM2.5 concentration (10−35 μg/m3) defined in this study. Figure 2 summarizes the expert panel’s estimates in the quantitative question (Q7), as well as the results of the four major cohort studies and the methodological recommendations from the WHO.61 Experts in Group 1 delivered the highest estimates in all percentiles, and showed the widest uncertainty intervals (fifth to 95th percentile). This is consistent with their experts’ responses to Q3, where they indicated that the direct transference of results was not valid. Estimates provided by experts from Group 2, who in general considered the direct transference assumption to be valid (all except expert F), were very similar to results from previous cohort studies. Experts A and B were concentrated in a region of values between those reported by Pope et al.,2,12 and lower than those of Dockery et al.11 and Laden et al.,16 while expert F provided similar values to those reported by Dockery et al.11 Experts E and G provided a distribution similar to the value recommended by the WHO.61 The main sources of uncertainty considered by each expert when making their estimates for Q7 are detailed in Table S6 in the SI. Combining Experts’ Estimates. Figure 2 shows distributions combined under equal, global and item schemes for the entire panel, and for each group, considering calibration factors estimated based on the experts’ performance in seed variables (see Table 2). Results reveal significant differences across the DM for the entire panel and groups 1 and 2not only in their mean values, but also in experts’ calibration scores. These results illustrate how experts’ estimates are combined using performancebased weights: the main virtue of the SEJ methodology. While calibration scores for Group 1 in the DM obtained using the equal weight (EW-1), global weight (GW-1) and item weight (IW-1) schemes are significantly different (0.3925, 0.0358, and 0.2627, respectively), the DM distributions are

in one country to estimate health impacts from PM2.5 exposure in another country, where no local studies have been developed. Variables Q4−Q7, mortality questions, related to the C−R function for mortality from long-term exposure to PM2.5 in Chile, were developed based on U.S. EPA.41 Experts were required to determine the shape of the C−R function (Q4) the potential existence of a threshold (Q5), as well as to decide whether to incorporate a threshold into the estimated C−R function (Q6). Finally, the quantitative question (Q7) required experts to estimate the percent change in annual nonaccidental mortality in adults caused by a permanent 1 μg/m3 reduction in PM2.5 ambient concentration throughout the entire Chilean territory. Experts’ estimates for the quantitative question were made considering annual average of ambient PM2.5 concentration in the range of 10−35 μg/m3, and were delivered as probability distributions, including maximum and minimum values, and fifth, 25th, 50th, 75th, and 95th percentiles. Complementary Material: Information and Resources. To avoid potential information gaps, the study team developed an elicitation document, a webpage with relevant information and a graphical tool to aid experts with visualizing their estimated functions. Dry-Run Session. A pilot session was implemented in order to test and validate the elicitation process and the instruments developed for the study. This session comprised three work meetings with two professionals with extensive experience in the field of interest and who were not part of the expert panel. Procedure. Elicitation Session. Three individual 1 h meetings with each expert were held. In the first meeting, the problem, goal, methodology, and seed and query variables were presented. After 2 weeks, a second meeting was held to clarify any questions and elicit seed, causality and transference variables. Finally, in the third meeting, the experts’ estimates for the mortality variables were elicited. Post-Elicitation. Experts’ estimates were combined following the Classical Model to estimate a probability distribution for the quantitative question (Q7) using the software EXCALIBUR.55 Combined distributions for Group 1, Group 2, and the entire expert panel were estimated. A summary of results for the estimates, weights, and performance was sent to each expert for them to approve or, if necessary, modify their answers. Three different combination schemes were used: equal weight, global weight, and item weight.

■

RESULTS Seed Variables. Only Expert B modified his original estimates after seeing his individual calibration and information scores. Experts’ estimates for seed variables are shown in Figure 1. In general, experts’ performance in S1, S2, and S3 was good, although only experts C and D included the real value of these three seed variables in their estimations. Note that these two experts provided the largest uncertainty intervals for S1, S2, and S3. As expected, the entire panel estimated ratios greater than one for every percentile of seed variables S4, S5, S6, and S7; nonetheless, overall experts’ performance in these variables was low. In general, all experts with the exception of experts C and E, provided narrow confidence intervals for these variables, missing the real value by large margins. Query Variables. Causality. All participants agreed that there is some degree of causality between exposure to PM2.5 and premature mortality: experts B, D, and F considered that the relationship is causal both in the short and long-term, experts C and G considered that the relationship is causal for F



Article

Figure 2. Uncertainty distributions for the relationship between PM2.5 and nonaccidental mortality in Chile. Box plots represent experts’ distributions, the three different combination schemes: equal (EW), global (GW) and item weight (IW) for each group (1 and 2) and for all the experts (A), and the results from four major international cohort studies and the WHO guidelines.

assessments and to characterize the uncertainty in these estimates. It is worth noting that the SEJ procedure is characterized by its intrinsically high level of methodological complexity, and its use in the specific context of environmental epidemiology and air pollution is very limited. While results of this study suggest that SEJ can provide relevant information to better understand how PM 2.5 exposure impacts local populations in countries where no cohort studies have been developed, its successful implementation is conditioned by a number of factors, which are discussed below. Seed Variables. The seven seed variables included in this study were intended to quantify the experts’ performance and their ability to express self-knowledge regarding two aspects of the evolution of air pollution and exposure-related mortality in the study area: changes in daily levels of PM10 in the MR (S1, S2, and S3) and changes in mortality in periods of high PM10 ambient concentration. Questions S4 and S6 address the effect of PM10 on all nonaccidental mortality. Questions S5 and S7 are similar but address the effect of PM10 on cardiovascular mortality. Regarding questions related to the second aspect, experts, in general, delivered estimates considerably different from the true values for these variables. While all experts provided ratios greater than one−thus recognizing that nonaccidental and cardiovascular mortality rates should be higher in periods of higher ambient pollutant concentration in Chile−their estimates differed significantly from the actual values. We think these results may arise from the experts possibly having developed these distributions considering only the isolated effect of air pollution on mortality (i.e., relative risk), rather than the complete set of factors that affect mortality in winter. Thus, this group of experts would underestimate the ratio of nonaccidental and cardiovascular mortality during periods of high pollution and average mortality trends. Also, we believe that the design of seed variables S4−S7 had a strong influence on experts’ performance in these variables.

markedly similar, all revealing the same value for the 50th percentile. The three DM for Group 1 are approximately two times higher than estimates from Dockery et al.,11 and Laden et al.,16 while presenting greater uncertainty intervals. The clear similarity is due to the high weighting of experts C and D in the combined distribution using the global and item schemes, estimated based on their performance in the seed variables. The DM for Group 2, where all experts are equally weighted (EW-2), received a calibration score of 0.0146. Figure 2 shows that the combined distributions are closer to results reported by Pope et al.2,12 On the other hand, the calibration score obtained in the global weight DM of Group 2 (GW-2) was significantly lower than the obtained in the item weight DM of Group 2 (IW-2). DM distributions combined under the equal, global and item schemes for the complete panel (EW-A, GW-A, and IW-A, respectively) present important differences in their central estimates (EW-A P50 = 0.9294, GW-A P50 = 2.7830, and IW-A P50 = 2.4480) as well as in uncertainty intervals. While EW-A incorporate the results from four major international cohort studies and the WHO guidelines, the GW-A and IW-A are similar to the DM’s of the Group 1, as experts C and D obtained the highest calibration scores of the panel. Robustness Analysis. The robustness analysis for variables revealed that the calibration score for Group 1 is particularly sensitive to seed variables S1, S3, and S6, while Group 2, is more sensitive to the seed variables S4, S5, and S6. For all the experts, the first three seed variables (S1, S2, and S3) retained the greatest influence on the model. In relation to the model’s sensitivity toward experts, the analysis shows that the calibration score of Group 1 is not sensitive to the experts and the calibration score of Group 2 showed greater sensibility to Experts B and E.

■

DISCUSSION In this study we explore the potential strengths and weaknesses of SEJ as a way to derive C−R functions for regulatory policy G


0.001 05 0.000 10 0.000 04 0.000 12 0.000 12 0.01464 0.02825 0.16680

0.011 02 0.004 46 0.39250 0.03583 0.26270 1.340 00 6.610 00 0.783 40 1.994 00 1.882 00 0.582 50 0.833 30 1.105 00

0.583 40 1.671 00 0.325 20 0.523 30 0.553 00

all variables

1.488 00 7.474 00 0.792 00 2.262 00 2.049 00 0.654 10 0.932 50 1.251 00

0.647 20 1.885 00 0.363 50 0.589 70 0.623 40

seed variables

relative information

0.001 57 0.000 73 0.000 03 0.000 27 0.000 25 0.009 58

0.007 13 0.008 40 0.142 70

unnormalized weightc

0.2 0.2 0.2 0.2 0.2

0.5 0.5

0.126 10 0.059 02 0.002 30 0.021 80 0.019 75 0.77110

0.045 07 0.053 08 0.90180

with DM

normalized weight without DM

0.026 34

0.001 57 0.000 73 0.000 03 0.000 27 0.000 25

0.021 13

0.007 13 0.008 40


0.550 70 0.257 80 0.010 05 0.095 21 0.086 26

0.459 20 0.540 80

without DM

0.90260

0.053 65 0.025 12 0.000 98 0.009 28 0.008 40

0.57630

0.194 50 0.229 10

with DM

normalized weight

global weight

0.208 70

0.001 57 0 0 0.000 27 0.000 25

0.16380

0.007 13 0.008 40


without DM

0.99010

0.007 43 0 0 0.001 29 0.001 16

0.91340

0.039 77 0.046 84

with DM

normalized weight

item weight

Three combination schemes are compared: equal, global and item weight. bThe calibration score is the p-value of falsely rejecting the hypothesis that an expert’s probability statements are statistically accurate. The maximal value is 1, the minimal value is 0. cRepresents the weight assigned to each expert, calculated as the product of the calibration score (second column) and the information score relative to seed variables (fourth column). dDM_EW: Equal weight combination scheme. eDM_GW: Global weight combination scheme. fDM_IW: Item weight combination scheme.

Expert A Expert B Expert E Expert F Expert G DM_EW2d DM_GW2e DM_IW2f

Group 2

a

Expert C Expert D DM_EW1d DM_GW1e DM_IW1f

Group 1

expert

calibration Scoreb

equal weight

Table 2. Results from Using Seed Variables to Obtain the Combined Distribution of the Expert Panel’S Estimates for the Mortality Questiona

Environmental Science & Technology Article

H



Article

linked to the experts’ backgrounds and professional experience. While almost the entire panel had previous experience in epidemiological studies, the experts came from two different fields. It is possible that the high estimates from Group 1 were influenced by the medical backgrounds of experts C and D, since they have had more direct contact with patients suffering from respiratory and cardiovascular illnesses worsened by air pollution exposure and are more knowledgeable about the toxicological mechanisms through which pollutants affect the respiratory and cardiovascular systems. Therefore, their estimates were more independent from previous international cohort studies. On the other hand, mortality estimates from the experts in Group 2, who have backgrounds in engineering and professional experience in exposure assessments and costbenefit analyses, are notably similar to estimates from international studies. Basically, these experts are consumers of mortality estimates reported in cohort studies, as they rely on C−R functions reported in the literature to estimate health impacts from air pollution exposure and to perform regulatory cost-benefit analyses. Consequently, it is not surprising that their estimates show an anchoring bias toward familiar C−R functions. The marked differences between combined distributions considering all experts (EW-A, GW-A, and IW-A) evidence the strong sensitivity of combined distributions with expert calibration to expert’s performance in seed variables: for example, the GW-A and IW-A are clearly dominated by estimates of experts C and D. These experts were calibrated with weights significantly higher than the rest of the panel, even though overall performance of these experts in seed variables was not good, mainly because poor performance of the rest of the experts in the seed variables. These results illustrate how experts’ estimates are combined using performance-based weights: the main virtue of the SEJ methodology. Expert Panel. The expert panel members agreed to participate in this study anonymously and voluntarily, without any kind of incentive or remuneration for their collaboration. The estimates they made required several hours of work focused on the examination, evaluation and discussion of background information. Anonymous participation without direct compensation (beyond personal satisfaction derived from collaborating) might have affected the expert panel members’ disposition to committing the time and effort necessary for this type of study. The research team suggests that future studies consider abandoning the anonymous participation method, as well as providing the appropriate incentives to motivate experts to dedicate the necessary time needed to make estimates that reflect their actual level of knowledge and expertise. Beyond the experts’ apprehensions and uncertainties regarding directly transferring results from cohort studies developed in other countries for use in impact and benefit assessments of changes in air pollution ambient concentrations due to activities, public policies and normative instruments, the resources and time necessary to develop cohort studies make it virtually impossible to conduct local studies in every country. In this context, the SEJ is a potential alternative method for estimating the increase in death risk and other health end points attributable to long-term PM2.5 exposure in countries where this information is not available, such as Chile. The results from this study provide valuable insight and experience into implementing a larger study to assess the true

Making estimates using such an unusual metric (the quotient of a specific week’s mortality rate over the annual average weekly mortality rate for different causes of deaths and for two specific years) demanded additional efforts and dedication from the experts. On the other hand, estimates for these variables required the experts to have specific knowledge of annual variations in weekly mortality rates for different causes of death. While the necessary information to make these estimates is publicly available, large amounts of time and data are required in order to do so. Moreover, there is no reason to assume that experts are familiar with how nonaccidental and cardiovascular mortality rates vary weekly over a specific year in the MR, since annual average mortality rates are usually used in epidemiology studies and health impact assessments for air pollution exposure, as these measures are less sensitive to particular events that affect daily or weekly mortality rates. Additionally, for seed variables S4−S7, a group of experts provided highly concentrated probability distributions. Similar behavior is observed for the same seed variables in Tuomisto et al.,45 who suggest that these differences could possibly owe to overconfidence in a group of experts (revealed in the small confidence intervals compared to the rest of the panel). Seed variables are helpful at determining whether experts can express when they are knowledgeable and when they are “just guessing” in the estimation of seed variables. It could be expected that experts (especially in Group 2) would have recognized the difficulty embedded in this set of seed variables, by delivering broader uncertainty intervals in their estimations. In light of our results, we believe that expert’s comprehensive understanding of seed variables (what are they used for, and how to deliver estimations) is fundamental for experts to make and communicate their estimations. This highlights the importance of conducting a workshop with the entire expert panel at the beginning of the study to ensure experts’ proper understanding of calibration variables. Finally, it is worth noting that the high complexity of the seed variables, as well as the time, research and dedication required to make good, accurate estimates, could undermine experts’ enthusiasm and commitment to the study’s main goal, affecting the quality of their estimates for query variables. Whatever the cause, low performance in estimates of these seed variables explains not only low individual calibration scores, but also the low calibration scores of combined distributions. These results evidence the relevance of defining a proper set of calibration variables in order to obtain a robust method for calibrating experts, in which a larger number of seed variables is considered and where such variables are clearly defined. Mortality Estimates for Chile. Mortality estimates delivered by experts in Group 1 were markedly similar, had the largest uncertainty intervals, were significantly different from the rest of the panel’s estimates, and, in turn, differed from previous studies. This outcome is not surprising, as experts C and D declared that directly transferring results from other countries was not valid. Therefore, it was expected that they would venture to make their own estimates, instead of relying heavily on results reported in previous cohort studies. Conversely, experts from Group 2, who in general considered the transference assumption to be valid, provided estimates that were very similar to results from previous studies. This outcome reveals that their estimates are, to varying degrees, anchored to values reported by major international cohort studies. A plausible explanation for the significant differences observed between mortality estimates by groups 1 and 2 is I



Article

pollution and mortality in six US cities. N. Engl. J. Med. 1993, 329 (24), 1753−1759. (12) Pope, C. A., III; Thun, M. J.; Namboodiri, M. M.; Dockery, D. W.; Evans, J. S.; Speizer, F. E.; Heath, C. W., Jr Particulate air pollution as a predictor of mortality in a prospective study of US adults. Am. J. Respir. Crit. Care Med. 1995, 151 (3), 669−674. (13) HEI. Reanalysis of the Harvard Six Cities Study and the American Cancer Society Study of Particulate Air Pollution and Mortality; Health Effects Institute, 2000. (14) Krewski, D.; Burnett, R.; Goldberg, M.; Hoover, B. K.; Siemiatycki, J.; Jerrett, M.; Abrahamowicz, M.; White, W. Overview of the reanalysis of the Harvard six cities study and American Cancer Society study of particulate air pollution and mortality. J. Toxicol. Environ. Health, Part A 2003, 66 (16−19), 1507−1552. (15) Pope, C. A., III; Burnett, R. T.; Thurston, G. D.; Thun, M. J.; Calle, E. E.; Krewski, D.; Godleski, J. J. Cardiovascular mortality and long-term exposure to particulate air pollution epidemiological evidence of general pathophysiological pathways of disease. Circulation 2004, 109 (1), 71−77. (16) Laden, F.; Schwartz, J.; Speizer, F. E.; Dockery, D. W. Reduction in fine particulate air pollution and mortality. Am. J. Respir. Crit. Care Med. 2006, 173 (6), 667−672. (17) Jerrett, M.; et al. Spatial analysis of air pollution and mortality in Los Angeles. Epidemiology 2005, 16 (6), 727. (18) Krewski, D.; Burnett, R.; Jerrett, M.; Pope, C. A.; Rainham, D.; Calle, E.; Thurston, G.; Thun, M. Mortality and long-term exposure to ambient air pollution: Ongoing analyses based on the American Cancer Society cohort. J. Toxicol. Environ. Health, Part A 2005, 68 (13−14), 1093−1109. (19) Beelen, R.; et al. Long-term effects of traffic-related air pollution on mortality in a Dutch cohort (NLCS-AIR study). Environ. Health Perspect. 2008, 116 (2), 196−202. (20) Enstrom, J. E. Fine particulate air pollution and total mortality among elderly Californians, 1973−2002. Inhalation Toxicol. 2005, 17 (14), 803−816. (21) Crouse, D. L.; et al. Risk of nonaccidental and cardiovascular mortality in relation to long-term exposure to low concentrations of fine particulate matter: A Canadian national-level cohort study. Environ. Health Perspect. 2012, 120 (5), 708−714. (22) Puett, R. C., Yanosky, J.D.; Hart, J.E.; Paciorek, C.J.; Schwartz, J.D.; H. H. Suh MacIntosh, Speizer, F.E.; , Laden, F.; , Chronic fine and coarse particulate exposure, mortality, and coronary heart disease in the Nurses’ Health Study. Environ. Health Perspect., 2009. (23) Puett, R. C.; et al. Chronic particulate exposure, mortality, and coronary heart disease in the nurses’ health study. Am. J. Epidemiol. 2008, 168 (10), 1161−1168. (24) EPA. Integrated Science Assessment for Particulate Matter; U.S. Environmental Protection Agency: Washington, DC, 2009. (25) Cooke, R. Experts in Uncertainty: Opinion and Subjective Probability in Science; Oxford University Press, 1991. (26) Morgan, M. G. and Small, M. Uncertainty: A Guide to Dealing with Uncertainty in Quantitative Risk and Policy Analysis; Cambridge University Press, 1992. (27) NUREG-1150 Severe Accident Risks: An Assessment for Five U.S. Nuclear Power Plants; Nuclear Regulatory Commission, 1990. (28) .Public attitudes toward nuclear power. In Nuclear Power in an Age of Uncertainty, Chapter 8; Office of Technology Assessment, 1984. p 218−219. (29) Harrison, J.; Khursheed, A.; Phipps, A.; Goossens, L.; Kraan, B.; Harper, F. Uncertainties in biokinetic parameters and dose coefficients determined by expert judgement. Radiat. Prot. Dosim. 1998, 79 (1−4), 355−358. (30) Goossens, L.; Harper, F.; Kraan, B.; Metivier, H. Expert judgement for a probabilistic accident consequence uncertainty analysis. Radiat. Prot. Dosim. 2000, 90 (3), 295−301. (31) Cooke, R.; Goossens, L. Procedures guide for structured expert judgement in accident consequence modelling. Radiat. Prot. Dosim. 2000, 90 (3), 303−309.

change in health end points from exposure to PM2.5 in Chile. It is worth noting the enormous potential of this methodological tool for aiding regulatory and normative analyses in Chile, extending beyond the field of air pollution.

■

ASSOCIATED CONTENT

S Supporting Information *

History of expert judgment, SEJ methodology, details of the Classical Model, description of the seed variables and other main results are presented in Supporting Information. This material is available free of charge via the Internet at http:// pubs.acs.org.

■

AUTHOR INFORMATION

Corresponding Author

*Phone: +56 (2) 26618644; fax: 00-562-6618623; e-mail: [email protected]. Notes

The authors declare no competing financial interest.

■

ACKNOWLEDGMENTS This research was partially funded by Chile’s National Science and Technology Commission (Conicyt) through the National Fund for Scientific and Technological Research (Fondecyt, grant 1130864), and by the National Research Center for Integrated Natural Disaster Management CONICYT/FONDAP/ 15110017.

■

REFERENCES

(1) Ostro, B.; Sanchez, J. M.; Aranda, C.; Eskeland, G. S. Air pollution and mortality: Results from a study of Santiago, Chile. J. Exposure Anal. Environ. Epidemiol. 1996, 6 (1), 97. (2) Pope, C. A., III; Burnett, R. T.; Thun, M. J.; Calle, E. E.; Krewski, D.; Ito, K.; Thurston, G. D. Lung cancer, cardiopulmonary mortality, and long-term exposure to fine particulate air pollution. J. Am. Med. Assoc. 2002, 287 (9), 1132−1141. (3) Pope, C. A., III; Dockery, D. Health effects of fine particulate air pollution: Lines that connect. J. Air Waste Manage. Assoc. 2006, 56 (6), 709−742. (4) Slaughter, J. C.; Kim, E.; Sheppard, L.; Sullivan, J. H.; Larson, T. V.; Claiborn, C. Association between particulate matter and emergency room visits, hospital admissions and mortality in Spokane, Washington. J. Exposure Anal. Environ. Epidemiol. 2004, 15 (2), 153−159. (5) Dominici, F.; Peng, R. D.; Bell, M. L.; Pham, L.; McDermott, A.; Zeger, S. L.; Samet, J. M. Fine particulate air pollution and hospital admission for cardiovascular and respiratory diseases. J. Am. Med. Assoc. 2006, 295 (10), 1127−1134. (6) Zanobetti, A.; Schwartz, J. Air pollution and emergency admissions in Boston, MA. J. Epidemiol. Community. Health 2006, 60 (10), 890−895. (7) Laden, F.; Neas, L. M.; Dockery, D. W.; Schwartz, J. Association of fine particulate matter from different sources with daily mortality in six US cities. Environ. Health Perspect. 2000, 108 (10), 941. (8) Tainio, M.; Tuomisto, J. T.; Hänninen, O.; Aarnio, P.; Koistinen, K. J.; Jantunen, M. J.; Pekkanen, J. Health effects caused by primary fine particulate matter (PM2.5) emitted from buses in the Helsinki metropolitan area, Finland. Risk Anal. 2005, 25 (1), 151−160. (9) Ostro, B.; Broadwin, R.; Green, S.; Feng, W. Y.; Lipsett, M. Fine particulate air pollution and mortality in nine California counties: Results from CALFINE. Environ. Health Perspect. 2006, 114 (1), 29. (10) Dockery, D. W.; Schwartz, J.; Spengler, J. D. Air pollution and daily mortality: Associations with particulates and acid aerosols. Environ. Res. 1992, 59 (2), 362−373. (11) Dockery, D. W.; Pope, C. A.; Xu, X.; Spengler, J. D.; Ware, J. H.; Fay, M. E.; Ferris, B. G., Jr; Speizer, F. E. An association between air J



Article

(52) Ilabaca, M.; Olaeta, I.; Campos, E.; Villaire, J.; Tellez-Rojo, M. M.; Romieu, I. Association between levels of fine particulate and emergency visits for pneumonia and other respiratory illnesses among children in Santiago, Chile. J. Air Waste Manage. Assoc. 1999, 49 (9), 154−163. (53) Ostro, B. D.; Eskeland, G. S.; Sanchez, J. M.; Feyzioglu, T. Air pollution and health effects: A study of medical visits among children in Santiago, Chile. Environ. Health Perspect. 1999, 107 (1), 69. (54) Román, A. O.; Prieto, M. J.; Mancilla F, P.; Astudillo O, P.; Acuña S, C.; Delgado B, I. Aumento del riesgo de consultas ́ cardiovasculares por contaminación atmosférica por particulas: Estudio en la ciudad de Santiago. Rev. Chilena Cardio. 2009, 28 (2), 159−164. (55) Cooke, R. and Solomatine, D. EXCALIBR Integrated System for Processing Expert Judgements version 3.0; User’s Manual, Prepared under Contract for Directorate-General XII, Delft University of Technology: Delft, 1992. (56) Cakmak, S.; Dales, R. E.; Vidal, C. B. Air pollution and mortality in Chile: Susceptibility among the elderly. Environ. Health Perspect. 2007, 115 (4), 524. (57) Clancy, L.; Goodman, P.; Sinclair, H.; Dockery, D. W. Effect of air-pollution control on death rates in Dublin, Ireland: An intervention study. Lancet 2002, 360 (9341), 1210−14. (58) Katsouyanni, K.; et al. Short term effects of air pollution on health: A European approach using epidemiologic time series data: The APHEA protocol. J. Epidemiol. Commun. H. 1996, 50 (Suppl 1), S12−S18. (59) Pope, C. A., III; Ezzati, M.; Dockery, D. W. Fine-particulate air pollution and life expectancy in the United States. N. Engl. J. Med. 2009, 360 (4), 376−386. (60) Son, J.-Y.; Bell, M. L.; Lee, J.-T. Survival analysis of long-term exposure to different sizes of airborne particulate matter and risk of infant mortality using a birth cohort in Seoul, Korea. Environ. Health Perspect. 2011, 119 (5), 725. (61) WHO. Air Quality Guidelines: Global Update 2005: Particulate Matter, Ozone, Nitrogen Dioxide and Sulfur Dioxide; World Health Organization, 2005.

(32) Kraan, B.; Cooke, R. Processing expert judgements in accident consequence modelling. Radiat. Prot. Dosim. 2000, 90 (3), 311−315. (33) McKay, M.; Meyer, M. Critique of and limitations on the use of expert judgements in accident consequence uncertainty analysis. Radiat. Prot. Dosim. 2000, 90 (3), 325−330. (34) Goossens, L.; Wakeford, R.; Little, M.; Muirhead, C.; Hasemann, I.; Jones, J. Probabilistic accident consequence uncertainty analysis of the late health effects module in the COSYMA package. Radiat. Prot. Dosim. 2000, 90 (3), 359−364. (35) Erhardt, J.; Jones, J.; Goossens, L. Probabilistic accident consequence uncertainty analysis of the whole program package COSYMA. Radiat. Prot. Dosim. 2000, 90 (3), 365−371. (36) Cooke, R.; Goossens, L. Procedures Guide for Structured Expert Judgment; European Commission, 2000. (37) Van der Fels-Klerx, H.; Cooke, R. M.; Nauta, M. N.; Goossens, L. H.; Havelaar, A. H. A structured expert judgment study for a model of campylobacter transmission during broiler-chicken processing. Risk Anal. 2005, 25 (1), 109−124. (38) Boone, I.; Van der Stede, Y.; Bollaerts, K.; Messens, W.; Vose, D.; Daube, G.; Aerts, M.; Mintiens, K. Expert judgement in a risk assessment model for Salmonella spp. in pork: The performance of different weighting schemes. d. 2009, 92 (3), 224−234. (39) Aspinall, W. Structured elicitation of expert judgment for probabilistic hazard and risk assessment in volcanic eruptions. J. Geol. Soc. London 2006, 15−30. (40) Goossens, L.; Cooke, R.; Woudenberg, F.; Van der Torn, P. Expert judgement and lethal toxicity of inhaled chemicals. J. Risk Res. 1998, 1 (2), 117−133. (41) Industrial Economics, I. Expanded Expert Judgment Assessment of the Concentration-Response Relationship Between PM2.5 Exposure and Mortality; U.S. Environmental Protection Agency, 2006. (42) Roman, H. A.; Walker, K. D.; Walsh, T. L.; Conner, L.; Richmond, H. M.; Hubbell, B. J.; Kinney, P. L. Expert judgment assessment of the mortality impact of changes in ambient fine particulate matter in the US. Environ. Sci. Technol. 2008, 42 (7), 2268− 2274. (43) Industrial Economics, I. An Expert Judgment Assessment of the Concentration-Response Relationship Between PM2.5 Exposure and Mortality; U.S. Environmental Protection Agency, 2004. (44) National Research Council. Estimating the Public Health Benefits of Proposed Air Pollution Regulations; The National Academies Press: Washington, DC, 2002. (45) Tuomisto, J. T.; Wilson, A.; Evans, J. S.; Tainio, M. Uncertainty in mortality response to airborne fine particulate matter: Combining European air pollution experts. Reliab. Eng. Syst. Safe 2008, 93 (5), 732−744. (46) Cifuentes, L. A.; Vega, J.; Köpfer, K.; Lave, L. B. Effect of the fine fraction of particulate matter versus the coarse mass and other pollutants on daily mortality in Santiago, Chile. J. Air Waste Manage. Assoc. 2000, 50 (8), 1287−1298. (47) Sanhueza, P.; Vargas, C.; Mellado, P. Impact of air pollution by fine particulate matter (PM10) on daily mortality in Temuco, Chile. Rev. Med. Chile 2006, 134 (6), 754. (48) Dales, R. E.; Cakmak, S.; Vidal, C. B. Air pollution and hospitalization for headache in Chile. Am. J. Epidemiol. 2009, 170 (8), 1057−1066. (49) Dales, R.; Cakmak, S.; Vidal, C. Air pollution and hospitalization for venous thromboembolic disease in Chile. J. Thromb. Haemostasis 2010, 8 (4), 669−674. (50) Sanhueza, P. A.; Torreblanca, M. A.; Diaz-Robles, L. A.; Schiappacasse, L. N.; Silva, M. P.; Astete, T. D. Particulate air pollution and health effects for cardiovascular and respiratory causes in Temuco, Chile: A wood-smoke-polluted urban area. J. Air Waste Manage. Assoc. 2009, 59 (12), 1481−1488. (51) Castro, P.; Vera, J.; Cifuentes, L.; Wellenius, G.; Verdejo, H.; Sepúlveda, L.; Vukasovic, J. L.; Llevaneras, S. Polución por material particulado fino (PM 2.5) incrementa las hospitalizaciones por insuficiencia cardiaca. Rev. Chilena Cardio. 2010, 29 (3), 306−314. K


Structured Expert Judgment to Characterize ... - ACS Publications

Recommend Documents