Environ. Sci. Technol. 1998, 32, 2346-2348
Comment on “An Alternative Minimum Level Definition for Analytical Quantification” SIR: Gibbons et al. (1) contains errors and omissions in the development of a statistical procedure (referred to as an Alternate Minimum Level or AML) and errors in application of the method detection limit (MDL) procedure [defined at 40 CFR 136, Appendix B and Glaser et al. (2)], limit of quantitation (LOQ) procedure, and the AML. The errors and omissions in Gibbons et al. (1) are objectively verifiable and call into question use of the AML procedure as a mechanism for determining quantitation levels. A comment and response on Gibbons et al. (1) [Kimbrough (3) and Gibbons et al. (4)] is relevant but does not address the range of issues that concern us. Kimbrough (3) provides a useful diagnostic tool for examining the precision and bias of an analytical method and important considerations for the design of data collections. Gibbons et al. (4) contains some perplexing claims regarding data provided by EPA. These topics are discussed below. Development of the AML. The technical development of the AML is flawed in a number of areas: critical definitions and components are missing from the basic statistical model, statistical theory is used that has not been justified, and statistical theory is used that is demonstrably incorrect. (a) Specification of the basic statistical model, eq 7, for measurement is deficient: the probability distribution for measurement error is not specified explicitly, and key variables in the model, the yˆ’s, y’s, b0w, and the index variable i are not properly defined, defined in a contradictory manner, or not defined at all. The index i in the model eq 7 and associated estimator equations apparently refers to each observed measurement regardless of concentration. In defining the regression weights, i is concentration specific and refers to a particular concentration replicate variance. In addition, eqs 7, 11, and 12 use yˆ in a contradictory manner. (b) Two contradictory probability distributions are implied for measurement error. The early development of the AML model implies that the relationship between measurement and true concentration is described by the Rocke and Lorenzato (5) model in which error is a mixture of a normal and a log-normal distribution. The Rocke-Lorenzato model results in the expression for precision as a function of concentration given by eq 6 that may be approximated by the exponential form given by eq 5. Gibbons et al. (1) use eqs 5 and 6 in attempting to model precision data. Later, these authors imply that measurement is normally distributed by introducing, without justification, the t-distribution in eqs 20 and 21, which are key computational expressions for calculating the AML. Use of the t-distribution is only appropriate theoretically when data are normally distributed (see, e.g., Hogg and Craig (6)) or when the data can be transformed into a normal distribution. Data following a mixture of the normal and log-normal distributions cannot be transformed into a normal distribution. (c) The weights suggested by Gibbons et al. (1) are not appropriate. We interpret the explanatory text associated with eq 10 to mean that the estimated calibration relationship is weighted by the observed variance of measurement results obtained from spiked reagent water samples replicated at each level of spike concentration used in the experiment. Carroll and Ruppert (6) recommend against the use of such 2346 9 ENVIRONMENTAL SCIENCE & TECHNOLOGY / VOL. 32, NO. 15, 1998
a weighting scheme. When the number of replicate measurements at a particular spike concentration are small, they recommend against using observed variances as weights in estimation procedures because these observed variances “can vary wildly”. Instead, they prefer the use of weights estimated based on modeling the relationship between measurement and the associated variation. (d) No motivation (in the sense of a real world application) has been provided for the one-sided tolerance limit. Depending on the intended use for the AML, the justification for the one-sided tolerance limit is arguable. Application of the MDL Procedure. Gibbons et al. (1) are quite critical of the MDL procedure and use results they obtained in an analysis of data for cadmium to support their criticisms. However, these criticisms lack credibility because the authors did not follow the MDL procedure properly in regard to selection of data and computational method. The EPA’s MDL procedure requires “a complete, specific, and well-defined analytical method”. Gibbons et al. (1) devote a significant amount of their paper to the analysis and discussion of data for 114Cd determined using EPA Method 1638. They show no recognition that Method 1638 specifies use of m/z 111 for quantitation of cadmium. In June 1997, we specifically brought this issue to the attention of Professor Gibbons in personal correspondence. While we note that Gibbons et al. (4) presented some of the data and based analyses on 111Cd, we are troubled by the incorrect implication that this change from 114Cd to 111Cd was motivated by some technical problem with the way EPA collected these data. These cadmium data are part of a larger data set that consists of raw intensity responses, five-point calibrations, and study measurements on reagent water samples containing spiked concentrations of nine metals analyzed using draft EPA Method 1638 (8) for inductively coupled plasma with mass spectroscopy. The nine metals were silver, cadmium, copper, nickel, lead, antimony, selenium, thallium, and zinc. Five-point multi-element calibrations were conducted based on reagent water spiked at 100, 1000, 5000, 10 000, and 25 000 ng/L. For purposes of the study, all nine metals were spiked into reagent water at each of 12 concentrations. Spike concentrations were 0, 10, 20, 50, 100, 200, 500, 1000, 2000, 5000, 10 000, and 25 000 ng/L. Seven replicates at each concentration were analyzed. The 11 m/z values specified for quantitation in Method 1638 are 60Cu, 66Zn, 82Se, 107Ag, 111Cd, 60Ni, 206/207/208Pb, 123Sb, and 205Tl. Results reported at other m/z values, such as 114Cd, were never intended to be used for quantitation of the analyte. Using the format requested, the EPA transmitted the data and data elements requested by Gibbons et al. to Maddalone by e-mail. Gibbons et al. (4) states that “we had no knowledge of the background correction prior to publication of our paper,” but the transmittal message stated that “concentrations, given in parts per trillion, were calculated using a weighted linear calibration with no intercept on the blank-adjusted net intensity ratio from the calibration samples.” Gibbons et al. (4) further states that “only measured concentrations were available at the time of our analysis.” However, our understanding of the reason they only have such data is because they requested that the EPA’s data be transmitted to them in the format used by their preexisting computer software for calculating the AML. Attempting to apply the MDL procedure, Gibbons et al. (1) incorrectly obtained MDLs between 5 and 72 ppt for 114Cd measurements. Using the same data and applying the MDL S0013-936X(98)00099-6 Not subject to U.S. copyright. Publ. 1998 Am. Chem.Soc. Published on Web 06/20/1998
procedure in an appropriate fashion, we find that the procedure yields an MDL for 114Cd between 17 and 20 ppt and an MDL for 111Cd of 1.8 ppt. In calculating these MDL values, the initial estimate of the MDL was determined as the concentration equivalent of three times the standard deviation of replicate measurements of cadmium in reagent water. This is one of the four criteria provided in the MDL procedure for selecting an initial estimate of the MDL. Gibbons et al. (1) do not refer to any of these four criteria. After determining an initial estimate of the MDL, the basic MDL procedure requires seven replicate measurements on a sample spiked between one and five times the initial estimate of the MDL. The MDLs we report are based on the available spike concentrations that are between one and five times the initial estimate of the MDL. Gibbons et al. (1) computed MDLs on all available standard deviations and without reference to any initial estimate of the MDL. Since the available data were not collected for purposes of determining an MDL, it was not feasible to apply the optional iterative procedure included in 40 CFR Part 136, Appendix B, to the available data without making assumptions and building models. The iterative procedure is used to establish MDL standards that are based on a model where measurement variation is approximately constant at low concentrations. For a variety of chemical analytical methods, the EPA is currently engaged in collecting data at and below the MDL for purposes of examining the behavior of measurement variation in this range and for purposes of comparing the performance of various detection and quantitation procedures.
FIGURE 1. Diagnostic for the exponential model.
Application of the Limit of Quantitation (LOQ). Gibbons et al. (1) use the wrong data and an incorrect equation in determining an LOQ for cadmium. In fashion similar to the MDL, the LOQ (see Keith et al. (9)) requires measurements “made with properly tested and documented procedures”. Hence, it is not appropriate to determine an LOQ using cadmium measurements at m/z 114. Furthermore, eq 2 in Gibbons et al. is intended for use with gross signal data whereas the data used has been background corrected to analyte signal data (see MacDougall et al. (10)). The correct LOQ equation for analyte signal data is LOQ ) 10σ0, where σ0 is the standard deviation of replicate measurements on a sample that does not contain the target analyte or the nearest equivalent. Hence, the LOQ could never be a negative number. The correct LOQ for cadmium, at m/z 111, is 4.9 ppt. Gibbons et al. (1) reported an LOQ of -10.62 ppt for 114Cd. Application of the AML. Gibbons et al. (1) did not supply diagnostic information on their fit of precision models (exponential and Rocke-Lorenzato) to 114Cd data, and diagnostics provided here indicate that these models do not fit these data. Hence, conclusions reached based on their example are suspect. In particular, the conclusion regarding robustness of the AML procedure to departures from the assumed relationship between true concentration and measurement variation is not justified. The AML procedure is based on the strong assumption that the relationship between true concentration and measurement variation has been appropriately modeled. It is apparent, however, from inspection of Figure 2 in Gibbons et al. (1) that the models employed, the exponential and the Rocke-Lorenzato model, do not fit the 114Cd data. This lack of fit can be demonstrated in the plots for the 114Cd, shown here in Figures 1 and 2. For the exponential model, the relationship between concentration and measurement variation is described in eq 5 of Gibbons et al. (1) as σx ) a0ea1(x). If data fit this relationship, then plotting the log of the arithmetic standard
FIGURE 2. Diagnostic for the Rocke and Lorenzato model. deviations calculated at each spike concentration versus the spike concentration would show points that indicate a straight line, with some allowance for random variation. Figure 1 is such a plot of the 114Cd data. Figure 1 clearly does not indicate a straight line. Hence, the exponential model is not appropriate for describing the 114Cd data. For the Rocke and Lorenzato model, the relationship between concentration and measurement variation is deVOL. 32, NO. 15, 1998 / ENVIRONMENTAL SCIENCE & TECHNOLOGY
9
2347
TABLE 1. Comparison of AMLs to Quantitation Levels in Commonly Used Methods reagent water quantitation level (µg/L) element
AML
MCAWWa
EPA 200.9b MLd
SMEWWc LOQe
chromium nickel lead
29 33 63
5 5 5
0.3 2 2
6 3 3
a Ref 11. The low end of the specified analytical range. b Ref 12. c Ref 13. d Minimum Level ) 3.18 × MDL ) 10 × standard deviation used to calculate the MDL. e Limit of quantitation ) 10 × standard deviation of the blank.
scribed in eq 6 of Gibbons et al. (1) as σx ) xa0+a1(x)2 If data fit this relationship, then a plot of the variance (square of the standard deviation) calculated at each spike concentration versus the square of the spike concentration would show points that indicate a straight line. This is not the case as can be seen in Figure 2. Hence, the Rocke and Lorenzato model is also not appropriate for describing the data. Given that neither of the example AML calculations appear to be based on appropriate models for concentration versus measurement variation, we find that the statement by Gibbons et al. (1) regarding robustness of the AML procedure, to departures from the assumed relationship between true concentration and measurement variation, is unwarranted. Inconsistencies with Quantitation Levels in Commonly Used Analytical Methods. Table 3 in Gibbons et al. (1) presents AMLs for chromium, nickel, and lead in various matrixes. These data are from interlaboratory studies directed at characterizing variability across the analytical range of the graphite furnace method; i.e., these studies were not directed at establishing the lowest level at which reliable quantitative measurements could be made. Our Table 1 compares these AML values with quantitation levels found either in commonly used analytical methods or constructed from MDLs in those methods. The differences among the quantitation levels is more likely due to differences in data sets used for determination of each of the values than to differences in the definitions of the concepts. In fact, Gibbons et al. (1) recognize that if data sets do not contain sufficient low level measurements, their procedure can result in an overestimate of the AML. However, they apparently did not investigate whether this could have occurred with the data they present. In any case, the reported AML values are well above quantitation levels published in current methods and demonstrate that the AML procedure is quite capable of producing overestimates of quantitation levels. Data Quality Objectives. Kimbrough (3) suggests the use of data quality objectives (DQOs) for precision and bias when designing specific applications for measurement methods. We note Kimbrough’s assessment of the Minimum Reporting Level (MRL), contained in the EPA’s Information Collection Request for drinking water, as an example of such DQOs. However, his rejection of the ML is confusing. The ML is the lowest acceptable calibration point in certain EPA methods, and as such, quality control standards for both accuracy and precision have always been associated with the ML. Of course, any specific application of a measurement method may have DQOs that are more or less stringent than the quality control standards established in a preexisting measurement method. We believe that it is appropriate to select measurement methods based on the relationship between the DQOs of the specific application and the quality control standards established in preexisting methods. 2348
9
ENVIRONMENTAL SCIENCE & TECHNOLOGY / VOL. 32, NO. 15, 1998
Though 114Cd is not a quantitation m/z, we find Figure 1 in Kimbrough’s paper to be an effective illustration of why we do not recommend that extrapolations be made outside of the range where data exist. In the case of the EPA data set used to create Figure 1, the lowest spiked reagent sample used to estimate the calibration relationship was 100 ng/L. Quantitation results based on that calibration relationship, when applied to spiked reagent samples outside the original calibration range, are somewhat biased. The data in the EPA’s study were collected below 100 ng/L in order to assess variability of the method at lower concentrations; estimated variability associated with specific spike concentrations is independent of bias. Studies of this type will go below the calibrated range of the instrument by necessity, although a post hoc calibration in this range could be conducted using the raw intensity data. Conclusions. Given the errors and inconsistencies in Gibbons et al. (1), we believe that considerable additional effort would be required for development of an adequate AML concept. Furthermore, the AML is based on a collection of data from many laboratories for a given analyte in a given matrix using a given method. By implying that the AML should be substituted for the U.S. EPA’s Minimum Level (ML), the use that the authors propose for the AML is as a censoring level for regulatory compliance. This would preclude use of data from a laboratory that could demonstrate attainment of a low detection or quantitation limit. Consequently, compliance levels lower than the AML could not be enforced. To the extent that use of the AML would raise regulatory compliance levels, the AML would be inconsistent with the EPA’s mandate to protect human health and the environment and would preclude its use for environmental monitoring.
Acknowledgments These comments reflect the personal opinions of the authors and not the policy of the U. S. Environmental Protection Agency.
Literature Cited (1) Gibbons, R. D.; Coleman, D. E.; Maddalone, R. F. Environ. Sci. Technol. 1997, 31, 2071-2077. (2) Glaser, J. A.; Forest, D. L.; McKee, G. D.; Quave, S. A.; Budde, W. L. Environ. Sci. Technol. 1981, 15, 1426-1435. (3) Kimbrough, D. E. Environ. Sci. Technol. 1997, 31, 3727-3728. (4) Gibbons, R. D.; Coleman, D. E.; Maddalone, R. F. Environ. Sci. Technol. 1997, 31, 3729-3731. (5) Rocke, D. M.; Lorenzato, S. Technometrics 1995, 37, 176-184. (6) Hogg, R. V.; Craig, A. T. Introduction to Mathematical Statistics; Macmillan Publishing Co., Inc.: New York, 1978. (7) Carroll, R. J.; Ruppert, D. Transformation and Weighting in Regression; Chapman and Hall: New York, 1988. (8) U.S. EPA. Draft Method 1638; U.S. Environmental Protection Agency, Office of Water: Washington, DC, 1996. (9) Keith, L. H. Anal. Chem. 1983, 55, 2210-2218. (10) MacDougall, D. Anal. Chem. 1980, 52, 2242-2248. (11) U.S. EPA. Methods for Chemical Analysis of Water and Wastes; EPA-600/4-79-020; U.S. Government Printing Office: Washington, DC, March 1983. (12) U.S. EPA. Methods for the Determination of Metals in Environmental Samples, Supplement I (Method 200.9); EPA/600/ R-94/111; U.S. Environmental Protection Agency, Office of Research and Development: Washington, DC, May 1994. (13) Greenberg, A. E.; Clesceri, L. S.; Eaton, A. D. Standard Methods for the Examination of Water and Wastewater, 19th ed.; American Public Health Association: Washington, DC, 1995.
Henry D. Kahn,* William A. Telliard, and Charles E. White Engineering and Analysis Division (4303) Office of Science and Technology U.S. Environmental Protection Agency Washington, D.C. 20460 ES980099Y
