Environ. Sci. Technol. 2005, 39, 5911-5920
Use of Expert Judgment to Bound Lung Cancer Risks ELIZABETH A. CASMAN* AND M. GRANGER MORGAN Department of Engineering & Public Policy, Carnegie Mellon University, Pittsburgh, Pennsylvania 15213
A bounding analytic technique for inferring the contribution of poorly characterized risk factors to a common health endpoint is demonstrated. Lung cancer mortality was chosen for the case study because the exposures responsible for the bulk of the deaths are very well-known, and the contribution of other putative causes is a focus of ongoing research and regulatory scrutiny. We elicited expert opinions on the upper and lower bounds on the fractions of the total lung cancer mortality due to individual risk factors. Interactive second-order uncertainty analysis was used to improve the experts’ confidence in their bounds. From this information we calculated an upper bound on the residual fraction of deaths due to minor causes not selected by the experts.
1. Introduction Probabilistic risk analysis is best suited for estimating large risks with clearly defined population exposures. Especially when applied to poorly understood risks with uncertain exposures, these methods can result in very wide confidence intervals. This problem is magnified when the results of such studies are extrapolated to large populations. Since regulatory decisions must often be made before the relevant science is complete, an independent test of the plausibility of a preliminary impact assessment would be reassuring. Previously (1, 2) we have proposed a method for estimating the contribution of poorly documented risk factors to a common health endpoint where the sources of large portions of the risk are known. More precisely, given what is known about all causes of that health endpoint, we estimate the fraction of cases of the health endpoint that the poorly documented risk factors, taken as a group, could not exceed. In this paper we apply the method to the case of annual lung cancer mortality in the United States. This method is intended to supplement, not replace, existing risk analytic methods and is applicable only to situations where the contribution to mortality by some risk factors is well-known, but for others the existing data are insufficient to support the robust application of standard methods of risk assessment. As with all expert elicitation work, such estimates should be updated as more experimental and epidemiologic information becomes available and ultimately replaced by classical risk analysis. The idea of bounding analysis is to use the parts of the problem that can be characterized using conventional probabilistic risk analysis to infer an upper bound on the contribution that could be made by the causes for which there are incomplete data. The input for this method is derived from the results of conventional risk analyses. For * Corresponding author phone: (412)268-2670; fax: (412)268-3757; e-mail:
[email protected]. 10.1021/es048492t CCC: $30.25 Published on Web 07/09/2005
2005 American Chemical Society
bounding analysis to work, the impacts of some of the risk factors must be well quantified. Conservation principles (such as mass or energy mass balance calculations) are commonly invoked in science and engineering, as are order of magnitude arguments (3). Also common in engineering and risk analysis is the use of expert elicitation to provide subjective probabilistic judgments on parts of a problem that cannot be determined experimentally but for which considerable information is available (4, 5). In this method we employ similar techniques to bound the fraction of lung cancer mortality that is due to poorly understood causes. This is done by eliciting upper and lower bounds for the fraction of lung cancer mortality due to the best-understood causes and subsequently deducing the upper bound on the remainder. 1.1. Mathematics of Bounding Analysis. The mathematical theory behind this work has previously been published (2). Two principles are central to our method, consistency (eq 1) and coherence (eq 2):
∑f(s ) e 1 s ∈ Ω, 1 e j e |Ω| j
(1)
j
sj
hf (si) +
∑f(s ) ) 1 s ∈ Ω, 1 e j e |Ω| j
(2)
j
sj*si
Following the notation in our theory paper, s is a cause in Ω, the set of all causes, and hf (s) and f(s) are the upper and lower bounds on the fraction of cases due to cause s, respectively. Here, si is the subset of unspecified (residual) causes. A set of bounds is inconsistent if the sum of all the lower bounds exceeds 100%. An example of inconsistent bounds would be the following: “The fraction of lung cancer deaths due solely to cigarettes is no less than 90% and the fraction of lung cancer mortality due solely to residential radon is no less than 15%.”. If at least 90% of the deaths are from cigarettes and at least 15% are from radon, then at least 105% of the lung cancer deaths must have occurred, a logical impossibility. There is not a consistency problem when the upper bounds sum to more than 100%. Nor is it a problem if the lower or upper bounds sum to less than 100%. We illustrate this concept using triangular coordinates (Figure 1). The vertices of the triangle correspond to three causes in Ω ) {A, B, X}. Any point on or in the triangle corresponds to values of the three variables summing to 100%, measured on the perpendiculars dropped from each vertex to its opposite side. For example, at the left vertex A ) 100%, B ) X ) 0%; at the point where the perpendicular dropped from vertex A intersects the line XB, A ) 0, X ) B ) 50%; and at the point where the three axes meet A ) B ) X ) 33.33%. We have drawn upper and lower bounds for A and B on the appropriate axes in Figure 1a. The intersection of these bounds lies off the triangle (hatched), so these bounds are not consistent. In Figure 1b, we have relaxed the lower bound on A, which makes the entire set consistent (shaded triangle). In our elicitation protocol, we require that the experts’ bounds be consistent. The other basic concept is coherence. [The word “coherence” is used in the sense of holding together. That is, if one bound is implied by some members of the elicited set and the elicited bound is less stringent than the implied bound, the implied bound is used.] If a bound is implied by other bounds in the set and is different from the implied value, the bound is considered to be incoherent (Figure 2a). An example of incoherent bounds would be the following: “At least 85% VOL. 39, NO. 16, 2005 / ENVIRONMENTAL SCIENCE & TECHNOLOGY
9
5911
FIGURE 1. (a) Inconsistent bounds on A and B and (b) consistent bounds.
FIGURE 2. Incoherent (a) and coherent (b) sets of bounds. of lung cancer mortality is from cigarette smoking alone, at least 5% is from residential radon alone but no more than 50% is due to all other causes.”. The first two statements imply that no more than 10% (i.e., 100% - 85% - 5% ) 10%) could be due to all other causes, because the upper bound of the last variable is at its highest value when the other variables are at their lowest values. With this simple rule we can calculate an upper bound on the residual (Figure 2b). Once the expert gives us a consistent set of bounds, we can impose the coherency requirement to produce an upper bound on the residual. In our theory paper (2) we checked all bounds for coherence and solved a linear program to produce a globally coherent set of bounds. We have since realized that the only bound that must be coherent in order to solve the problem is the upper bound on X. However, imposing globally coherent bounds can lead to some interesting insights (see Discussion). 1.2. Expert Elicitation and Bayesian Inference. When the empirical information is insufficient and direct observation is impossible, it is common practice to supplement empirical knowledge with subjective information, rules-ofthumb derived from experience with similar problems, or information concerning the underlying mechanisms involved. The expert combines the two types of knowledge and produces a subjective probability in the Bayesian sense. By that we mean that the probability is a statement of the expert’s belief about the likely occurrence of an event conditioned on the current state of information concerning that event, P(X|i), where X is the uncertain event, and i is the expert’s state of knowledge. Because i is incomplete, the probability is considered to be a statement of degree of belief based on currently available information (4, 6). Different people or even the same person at different times may have different subjective probabilities for the occurrence of X. As long as the subjective beliefs are consistent with the axioms of probability, they can be operated on as probabilities. Expert subjective judgment in the form of probability distributions 5912
9
ENVIRONMENTAL SCIENCE & TECHNOLOGY / VOL. 39, NO. 16, 2005
has been successfully used in a variety of contexts such as nuclear reactor safety (7), air pollution regulation (8), cancer risk assessment (9, 10), and seismic hazard analysis (11). Like these examples, bounding lung cancer fatalities from poorly understood causes is a problem that involves high uncertainty. Lung cancer experts derive their knowledge of lung cancer risks for the most part from epidemiologic studies and from exposure modeling. The epidemiologic studies can suffer from small sample size, difficulty in meaningfully defining exposure, multiple confounders, and problems with selection of control populations, all complicating the estimation of “attributable fraction”, the most common metric for expressing the contribution of a risk factor to the population incidence of a health problem. It is not illogical for the sum of the expected values of attributable fractions for a single health condition with multiple contributing causes to exceed 100%. It would be a problem, however, if the sum of their lower bounds (or lower values of probabilistic confidence intervals) exceeded 100%. (See discussion on consistency, above.) In this study, we do not ask experts for bounds on the attributable fraction, which is tied to specific epidemiological studies. Instead we ask for the expert’s judgments regarding the bounds on the fraction of disease actually caused by each factor, based on their consideration of scientific evidence, analytical models, and their knowledge of mechanisms of disease. The result may be broader or narrower than those generated observationally from epidemiological studies. We must emphasize that our tool is in no way a substitute for rigorous scientific research. Expert judgment is inferior to directly relevant high quality scientific evidence and analytical methods whenever the latter are available. Furthermore, expert judgment is subject to a variety of biases produced by cognitive heuristics. (12) We only claim that expert judgment can be helpful in policy analysis when
TABLE 1. Lung Cancer Epidemiology Experts Participating in This Study name
current affiliations
Michael C. R. Alavanja, Dr. P.H.
Senior Investigator, National Cancer Institute, Division of Cancer Epidemiology and Genetics; Fellow of the American College of Epidemiology CEO International Epidemiology Institute; Professor of Medicine, Vanderbilt University Medical Center and the Vanderbilt-Ingram Comprehensive Cancer Center; adjunct professor at the Johns Hopkins University and Rutgers University Scientific Director, International Epidemiology Institute; Professor of Medicine, Vanderbilt University Medical Center and the Vanderbilt-Ingram Comprehensive Cancer Center; adjunct professor at Harvard University and the Uniformed Services University of the Health Sciences Codirector of the Risk Sciences and Public Policy Institute, Professor and Chairman of the Department of Epidemiology at the Johns Hopkins Bloomberg School of Public Health Vice President, Epidemiology and Surveillance Research, Chief Epidemiologist, American Cancer Society
William J. Blot, Ph.D.
John D. Boice, Jr., Sc.D.
Jonathan Samet, M.D. Michael J. Thun, M.D.
decisions must be made before all the necessary science is known.
2. Selection of Experts The experts were selected on the basis of their knowledge of lung cancer epidemiology as evidenced by their peer reviewed publications, monographs, and participation on important scientific panels and committees concerning lung cancer risks. Because our method involves face-to-face interviews, and the application focused on lung cancer in the United States, we restricted ourselves to experts residing in the United States. The quality of the bounding analysis result rests heavily on the knowledge of the experts. Accordingly, every effort was made to select prominent and knowledgeable scientists in the field. We also wanted the expert pool to represent a range of pertinent backgrounds, so we have tapped scientists with expertise that includes epidemiology, medicine, biostatistics, and public health. Some of our experts had deep knowledge of one or a few major causes of lung cancer but less knowledge of others. Some had wide expertise across the range of issues involved in lung cancer epidemiology. Five leading lung cancer experts were interviewed. In expert elicitation it is common to focus on a small diverse set of experts: quality of judgments being much more important than quantity of experts. The five experts we interviewed are listed alphabetically in Table 1. In reporting results in this paper we have randomized the order and do not attribute any specific judgments to specific experts.
3. Elicitation Protocol Before each elicitation we mailed each expert a briefing book containing background materials on a large number of established and suggested risk factors for lung cancer reported in the peer reviewed literature and U.S. government sponsored reports. The briefing book was available as a compact reference during the elicitations. The provision of a briefing book is standard good practice in expert elicitation, to limit the impact of bias introduced by the availability or representative heuristics (4) or a preference for round numbers. The interviews were held in the experts’ offices to facilitate consultation of other references. Nearly all the experts consulted some form of personal reference materials, and one expert had a staff member run a calculation to help him with a response during his interview. The elicitation followed a standard printed protocol [available as Supporting Information]. It began with a discussion of the theory and objectives of the elicitation and a clarification of terminology. The expert was asked to offer
an opinion on how well the total number of annual lung cancer deaths in the United States is assessed and to estimate the upper and lower bounds on the fraction of these deaths that would have occurred in the absence of exposure to any carcinogen besides cosmic radiation. We also ask for the expert’s best estimate of this fraction. The expert was then given a list of causes of lung cancer and asked to make a shorter list of the most important causes, order them in descending importance, and estimate the fraction of the population with significant exposure to each. The expert was told he could rephrase the causes or to add to the list as needed, and several chose to do so. We also asked the expert to identify quantifiable interactions between causes. Next, the expert was asked to give his best estimate of the fraction of lung cancer mortality due to each cause, both singly and in combination with all other carcinogenic exposures. He was also asked to provide the lowest and highest fractions of the total annual lung cancer mortality that he believed actually resulted from each cause and to provide his best estimate and upper and lower bounds on lung cancers due to any interactions between causes he wished to discuss. The elicited values were tabulated, and the expert was asked to discuss his level of confidence in them as a group and to revise any of the values if necessary. We also asked for the fraction of deaths due to interaction with other specific causes identified by the expert (best estimate and upper and lower bound). The lower bounds were examined for consistency (see above), and if they were inconsistent, the expert was asked to adjust them. We explored second-order uncertainty concerning the upper and lower bounds by asking the expert to rate his confidence in each lower bound on a 5-point qualitative scale running from “not at all confident” to “extremely confident”. We then asked the expert to relax the lower bounds in which he expressed less confidence until he was as confident in them as in the bound for which he expressed the most confidence. We tabulated the original lower bounds and the revised bounds and asked the expert to select the bounds with which he felt more comfortable. The same process was repeated for the upper bounds. Using eq 2, we calculated the upper bound on annual lung cancer mortality from all other causes, expressed both as a percentage and as a number of deaths, and asked the expert if the upper bound were plausible and how he would rate his confidence in this value. Because this case study is a test of a new risk analysis method, we also asked the experts if they would use this method and if there were things that made them uneasy VOL. 39, NO. 16, 2005 / ENVIRONMENTAL SCIENCE & TECHNOLOGY
9
5913
TABLE 2. Elicited Lower Bounds, Upper Bounds, and Best Estimates of the Fraction of Annual Lung Cancer Mortality Due to Major Causes and to Spontaneous Occurrencea cause air pollution, ambient air pollution, other than ambient diesel particulates air pollution: diesel particulates, ambient arsenic, occupational asbestos, all asbestos, nonoccupational asbestos, occupational beryllium chloromethyl ethers, occupational chromium, occupational cigarette smoke active coke oven emissions, occupational environmental tobacco smoke genetic factors, inherited nickel, occupational PAH, occupational pesticides, occupational poor diet previous lung disease radon, indoor radon, mining spontaneous - no carcinogen tobacco smoking, other than cigarettes X-rays, therapeutic sum of estimates a
expert 1
expert 2 0.1, 2, 4
expert 3 0.5, 2, 4
expert 4
expert 5
0.1, 0.2, 0.5 0.1, 0.5, 2 0.1, 0.4, 2 0.5, 2, 3
1, 7, 10
0.1, 0.3, 2
85, 87, 90
85, 90, 94
85, 90, 95
1, 1.9, 2.2
1, 2, 3
0.5, 1, 2 0.5, 1, 1.5
0.001, 0.01, 0.02 0.01, 0.05, 0.1 0, 0.001, 0.0015 0.001, 0.01, 0.02 0.001, 0.01, 0.02 80, 85, 95 0.001, 0.01, 0.02 0.2, 1.5, 2 1, 10, 100 0.001, 0.01, 0.02 0.001, 0.01, 0.02
0, 5, 20 0.1, 1, 2 1, 12, 14 1, 12, 15 0.5, 2, 10 0.1, 0.5, 1.5 0.01, 0.05, 0.1 0, 0.1, 0.62 0.1, 0.1, 0.2 0.45, 1, 7.13 0, 1, 3 1, 2, 4 2, 5, 8 0.001, 0.001, 0.002 88.7, 108.0, 116.8 91.2, 106.2, 116.2 88.4, 99.8, 123.5 84.0, 111.4, 235.3
88, 90, 93 0.5, 2, 4
0.1, 0.2, 2 0.2, 1, 2 0.5, 1.5, 3 1, 3, 5 0.1, 0.3, 1
90.7, 99.2, 116.0
Lower bound, best estimate, upper bound; percent.
concerning this process or the result. Their reactions and the results of the elicitation are reported below.
4. Results 4.1. Major Causes of Lung Cancer. The causes identified by the experts as responsible for the most lung cancer mortality are listed in Table 2 along with the experts’ best estimates for the fraction of annual lung cancer deaths due to each. The totals of some of the experts’ best estimates exceed 100%. This could be due to multiply counted interactions between causes, differences in opinion as to exposure history, differences in mental models of carcinogenesis, and/or to error. If we were trying to derive the best estimate for the fraction of deaths due to the unnamed causes, this would be problematic, but since we are seeking an upper bound on this residual, it is not. All five of the experts listed cigarette smoking, residential radon exposure, and environmental tobacco smoke (ETS) among the major causes of lung cancer. For these three causes there have been well-respected published estimates of the number of lung cancer deaths they cause individually nationwide (Figure 3.) Cancer is indisputably a genetic disease; however, inherited genetic factors do not fit the definition of a carcinogen. The experts chose to handle this problem in three different ways. Experts 1 and 2 adopted our suggestion that they consider inherited genetic factors to be a priori characteristics of the population at large. These experts did not estimate lung cancer mortality due to them individually. Experts 3 and 4 reasoned that inherited genetic factors determined how the individuals processed and responded to carcinogens and therefore listed genetics as a separate cause. Expert 5 reasoned that the effects of the major inherited genetic factors were experienced only in combination with carcinogenic exposures and then only at relatively low carcinogen exposures, because the genetic factor effect gets overwhelmed with high risk factor exposure, such as cigarette smoking. He therefore quantified them as interaction terms with two carcinogens. Specifically, he estimated that 20% (range 1030%) of lung cancer deaths from residential radon are the 5914
9
ENVIRONMENTAL SCIENCE & TECHNOLOGY / VOL. 39, NO. 16, 2005
result of an interaction with the GSTM1*0 mutation, as are 35% of the lung cancer deaths attributed to environmental tobacco smoke (range 20-50%) the result of interaction with this mutation. The glutathione S-transferases (GST) conjugate electrophilic procarcinogens with glutathione (GSH). GSHconjugates are more polar and more readily excreted in the urine than the procarcinogens, so active GST is thought to have a protective effect. A null mutant of GST class µ (GSTM1*0) in combination with mutations in the p53 tumor suppressor gene has been associated with lung tumors (13). Lung cancer patients are significantly more likely to show glutathione S-transferase µ deficiency than are controls (14). The GSTM1 null phenotype occurs in 40-50% of Japanese,
FIGURE 3. Elicited lower and upper bounds on the main causes contributing to lung cancer mortality in the United States. These factors were mentioned by all the experts. Environmental tobacco smoke is sometimes called passive smoking. Radon refers to residential radon, and background means lung cancers that occurred in the absence of exposure to any carcinogen except cosmic radiation. A major source of disagreement among the experts is the effect of exposure to residential radon. Some experts, following the methods used in BIER VI (15), prefer the linear, no threshold model of radon effects at low dose. Others reject this model based on epidemiological studies of populations exposed to low doses of residential radon.
TABLE 3. Estimates of the Percent of Lung Cancer Deaths Resulting from Each Risk Factor that Are Due to Interaction with Smokinga,b cause
expert 1
expert 2
expert 3
air pollution, other than ambient diesel particulates air pollution from diesel particulates, ambient arsenic, occupational asbestos, all asbestos, nonoccupational asbestos, occupational beryllium chloromethyl ethers, occupational chromium, occupational cigarette smoking, active coke oven emissions, occupational environmental tobacco smoke genetic factors, inherited nickel, occupational PAH, occupational pesticides, occupational poor diet previous lung disease radon, mining radon, residential tobacco smoking, other than cigarettes X-rays, therapeutic
expert 4
expert 5
additive, no noned (confounder not interaction interaction)
air pollution, ambient
additive, no interaction 30, 86, 100 80, 90, 100
nac
nac
nac
nac
nac
nac
nac
nac exists but cannot quantify
nac 10, 90, 100
20, 35, 50e
noned
86, 87, 87
80, 85, 90
20, 60, 80 noned
20, 50, 100 10, 50, 100 0, 90, 90 0, 50, 50
70, 90, 95
10, 20, 30e
a For example, expert 1’s best estimate for asbestos is that 86% of those deaths result from an interaction with smoking. b Lower bound, best estimate, upper bound; percent. c na - not applicable. d None - no synergy. e Percent of cases attributed to risk factor due to interaction with inherited genetic factors, not with cigarette smoking.
European, and North American populations. Expert 4, however, said that familial history of lung cancer increased the risk among smokers 2- or 3-fold, so that he listed cigarettes and inherited genetic factors as having an interaction. Interactions with cigarette smoking was the single most frequently cited quantifiable interaction effect (Table 3), often being cited as responsible for at least a majority of the lung cancer deaths from many joint exposures. For example, expert 5 said the interaction between poor diet (low in fruits and vegetables, high in fried foods and alcohol) and smoking is dominant, and between 70 and 95% of lung cancers from poor diet are from this submultiplicative interaction. A good diet is protective for smokers. Interaction effects are very difficult to measure epidemiologically, so the interactions quantified in the table are, even for the best-studied causes, educated guesses. This evidenced by the fact that there is so little agreement in this table and that the individual experts have such broad ranges around their best estimates. Though there is strong agreement that cigarette smoking is responsible for between 80 and 95% of lung cancer deaths, there is much more uncertainty concerning the next most important cause, residential radon. The expert community seems to have two major approaches to estimating this risk: (i) a linear, no-threshold model of radon risk at low dose and (ii) a threshold model derived from the inability to detect an effect in large population studies of humans exposed to low doses (Figure 3). The linear, no-threshold group’s estimates are based on dose-response relationships among underground miners exposed to high doses of radon. The latter group gives more weight to epidemiological studies of low dose exposures in residential settings, both in the United States and abroad. The overlap of the ranges of the elicited distributions indicates that both groups (except expert 4) allow for the possibility that the alternative view is correct. Another source of uncertainty is the exposure history of the lung cancer victims. We asked the experts to estimate the fraction of the U.S. population exposed at doses that could have resulted in lung cancers in the present year (Table 4.) 4.2. Definition and Upper Bound for the Residual Causes. The upper bound on the residual causes (those whose
effects were not individually quantified by the experts) is at its maximum when all the named causes are at their minima. Therefore we calculate the upper bound on the residual of unnamed causes as 1 minus the sum of the lower bounds on the named causes (Table 5). The upper bound on the residual calculated by this method is in the vicinity of 10% for all of the experts. However, in subsequent discussion none of the experts felt that the causes that they did not list could account for so many annual fatalities. 4.3. Feedback from the Experts. When we asked the experts to evaluate the upper bounds we estimated on the residual causes, they all indicated that the computed bound was much larger than their expected value for the fraction of deaths due to this group of causes. They also expressed a high degree of confidence that the upper bound could be no higher than their respective results. Though an upper bound is not supposed to be a central estimate, all the experts made the point that the real number of cases was far from the upper bound. So, though the upper bound was plausible and represented their understanding of the situation, it was so slack relative to their understanding of the true value of the residual that they were uncomfortable with the entire analysis. They could not imagine enough unaccounted carcinogenic exposures to produce as many deaths as implied by the upper bound on their residuals. (See Table 6.)
5. Discussion This analysis yielded upper bounds on the proportion of lung cancer mortality which might be caused by the group of poorly understood causative agents in the range of 1015%. Our experts, who are intimately familiar with the literature, concluded that such a bound was on the high side of what is plausible. They suggested that a bound half this size or smaller would be more consistent with their current knowledge. There is extensive evidence that people, including experts, are often seriously overconfident when they make such VOL. 39, NO. 16, 2005 / ENVIRONMENTAL SCIENCE & TECHNOLOGY
9
5915
TABLE 4. Estimated Fraction of the Population Having Been Exposed to Risk Factors at Levels that Could Be Responsible for a Lung Cancer Death This Year cause
expert 1
air pollution, ambient air pollution, other than ambient diesel particulates air pollution from diesel, particulates ambient arsenic, occupational asbestos, all asbestos, nonoccupational asbestos, occupational beryllium, occupational chloromethyl ethers, occupational chromium, occupational cigarette smoking, active
expert 3
expert 4
expert 5
100% 5% 5%
