A comparison of laboratory performances ~
How to check the reliability of analytical environmental sewices Julie € Einerson I. Phyllis C. Pei SMdia Nmonal Loboratories Albuquerque,NM 87185
Analytical laboratory results often are used to determine whether a facility or process is in compliance with permit or regulatory conditions; therefore, credible data are an essential component of the decision-making process for an environmental manager. At Sandia National Laboratories in Albuquerqne, NM, accurate and precise data are essential for managing our envirunmental programs. Because we seek high credibility of laboratory data, a study was conducted to determine the management of measurement accuracy and precision by commercial laboratories. This article describes some procedures that will help identify those laboratories that provide precise and accurate analytical services on a consistent bask Ten laboratories (. A 4. were selected to participate in the competency tests. Laboratory qualifications were evaluated in six areas that had preassigned weights (Table 1): precision, accuracy, quality assurance documents, ability to customize reports, turnaround time, and price. Laboratories were scored and ranked by each category. Categories were then weighted and summed to produce a list of laboratories ranked in order of competence. A set of written instructions were given to the laboratories, and written price quotes and quality assurance (QA) plans were obtained from each laboratory. The laboratories were given check samples that were spiked with six inorganic contaminants commonly found in wastewater discharges. These samples were sent along with routine wastewater samples. Preparation of standards. Standard solutions containing loo0 ppm of a
metal were purchased. The standards were certified for use in atomic absorp tion spectroscopy and, except for arsenic, were secondary standards trace able to the National Bureau of Standards. The standards purchased were cadmium (Cd) as cadmium nitrate, trivalent chromium (CI(,)) as potassium dichromate, mercury (Hg) as mercuric chloride, arsenic (As) as arse-
W1393BW88NW1~1121EOl.50/0 0 1988 American Chemical Society
nic trioxide in nitric acid, copper (Cu) as copper oxide in nitric acid, and lead (Pb) as lead nitrate. These stock solutions were used to prepare the check samples. Ten batches of 2-L acidified check standards were prepared. All of the check standard solutions were poured into a 20-L container and thoroughly mixed. The final pH of the solution wa$ Envimn. Sci.Technol..Voi. 22. No. 10, 1988
1121
tested to be < 2 . The final concentrations of the metals (true values) after dilution were As, 1.0 ppm; Cd, 0.05 ppm; Cr,,), 0.05 ppm; Cu, 5.0 ppm; Pb, 5.0 ppm; and Hg, 0.5 ppm. Deionized distilled water blanks containing 5 mL of nitric acid per 1ooO mL. were also included as samples to detect any possible contaminant in the diluent. The water blanks submitbed to the laboratories all contained less than detectable amounts of contaminants.
'.
,
1122 Emimn. Sci.Technoi..Vol. 22, NO. 10, 1984
Analytical results One container was given to each of the 10 laboratories to analyze six check sample metal parameters: As, Cd, Cr,,,, Cu, Pb, and Hg. The labs also were given another duplicate set of the samples to analyze for Cr,,, and Fb' . Table 2 outlines the raw data reported by each laboratory. precision. The precision error was calculated as follows: [(X, - X,)/2]rrV x 100% (1) where X , and X, are replicate values and TV is the hue value. The smaller the difference between X , and X,, the smaller the % error and the better the precision. Laboratories G and J demonstrated 100% precision in both the chromium and lead analyses. In general, the laboratories performed bet& in the analyses of chrohium than lead. Filly percent of the labs were 100% precise in the analysis of chromium, whereas only two labs were 100% precise in the analyses of lead. The % error in precision for chromium and the % error in precision for lead were then combined by using the root mean square formula to provide an overall precision rate (Table 3). The smallest error was assigned the highest rank of 1. Except for lab A, most had good precision. Aecuraey. In order to assess the degree of agreement between the measured value and the true value, the accuracy for each lab was measured. The difference between the reported value and the true value divided by the m e value all multiplied by 100% provides the accuracy error rate [(X - T v ) / T v ]x 100% (2) where Xis the measured value and TV is the true value. Lab A showed an error of more than 1ooO-fold for arsenic. It is interesting that one lab, Lab B, consistently erred on the negative side of the true value in all six parameters, whereas labs G and H consistently erred on the positive side of the m e value (except in the case of arsenic). Table 3 illustrates the average % error from the six parameters for each of the laboratories. Labs D, G, and H all demonstrated an average ac-
curacy error rate of less than 10%. Precision and accuracy. These categories were weighted 4096,as accurate and precise numbers are essential for environmental data. AU the elements of the sampling process-from the sampling techniques to sample preservations, chains of custody, sample blanks, labratory analyses, and documentstion-contribute to the reliability of data. It must be kept in mind that if an analytical laboratory does not produce reliable results, the entire sampling effort may be rendered useless. The ultimate cost will be much greater if resampling has to be conducted. More laboratory performance8 are measured in this manner now, as is evident in the EPA Contractor Laboratory Program Statement of Work. Lab A ranked tenth in precision and ninth in accuracy; Lab G ranked first in precision and second in accuracy One may conclude from this correlation that if a laboratory has poor precision in its measurement of replicate samples, then its accuracy should also be suspect. Quality assurance. The laboratories
were requested to provide a copy of their Quality Assurance manual for r e view. The 11 criteria under evaluation were chosen for their importance in assuring the quality of analytical results. This list is by no means comprehensive but is meant to highlight critical aspects of a QA program. The total points available per criterion ranged from 1to 4. The highest w r e earned the highest rank. Table 3 summarizes the total scores earned by each laboratory. The evaluation criteria included the description and frequency of blind check samples, method detection limits, method blanks, equipment calibration, matrix spike sampl&, check samples, replicate samples, internal audits, EPAKLP (Contractor Laboratory Program) participant, chain-of-custody use, and corrective action program. A good quality assurance program is essential to the accuracy and precision of analytical results. If employees are expected to understand and follow specifications, it is important that a plan be documented and implemented. If a good QA plan is followed properly, the
degree of laboratory errors should be minimized. For example, lab G ranked second in QA and accuracy and tied for 6rst in the precision category. The expected correlation is that excellent QA plans, when executed, will result in good accuracy and precision in analyses, as seen with lab G. Lab H ranked first in the QA and accuracy but fifth in precision, and lab E ranked third in QA, but eighth in precision and sixth in accuracy. The results here suggest that not all QA plans are practiced at the lab bench. often, for the sake of standardization, the corporate division of a laboratory would produce the QA plan without input or feedback from the analysts. The QA prccedures could be inappropriate or not practicable in the field laboratcries. If no one verifies that the QA plan is being implemented, then the plan becomes a mere marketing tool; there will be no discipline or consistency in the quality of the lab’s output. There were instances in which a lab ranked low in QA and performed well in analysis. For example, lab C ranked Environ. Sci. Technol.. Val. 22,No 10. 1888 1123
ninth in QA but third in precision; Lab J ranked eighth in QA but tied for first in precision. Possibly these laboratories did not explain their full QA programs well, the evaluation criteria were not adequate in scoring these labs, or the analysts have good techniques in spite of the QA plans. In this latter case, some additional time spent on relining a QA plan should correct the problem and result in a better score in this evaluation.
a m a r o u n d time. The samples were shipped to the laboratories in ice chests on the same day via Federal Express. Although laboratories agreed to report data within 14 days, few were able to meet the deadlime. Those labs that met the turnaround time received a rank of 1. The lab providing data latest was ranked tenth (Tables 3 and 4). We found that the two laboratories (A and B) with the best turnaround time rank ninth and tenth, respectively, in the ac-
lation between the qualii of the e Ability to customize reports. The laboratories were requested in writing to provide the analytical data in the following manner. (These points were specific in the written instructions eiven to the 10 laboratories.) EPA-approved methods are to be used for sample analysis. The references to the methods are to accompany each parameter's analytical results. The analytical results will be reported with confidence limits for each parameter based on the techniques and equipment used by the laboratory. *The dates the analyses were performed are to be reported for each parameter. All chain-of-custcdy forms are to be returned to Sandia National Laboratories after the samples reach the laboratory destination. All sample damages and abnormalities are to be noted on the forms and signed by the laboratory recipient. Regulatory agencies often require different formats for the submission of environmental monitoring reports, and this facilitates the use of computers by the agencies to track permit compliance information. The instructions used in this evaluation were some of the requirements imposed by the city of Albuaueraue. Those labs that followed all th6 ins'tructions received a score of 10 and a rank of 1, and so forth (Tables 3 and 4). If Labs G and H (both of which followed all of the instructions)
.,
mation from h e laboratories. Therefore, the ability of the laboratories to tailor analytical reports to suit customers' requirements is essential. 1124 Environ. Scl.Technol..Yol. 22,No. 10, 1988
curacy category. Rushing does not pay off. 'hrnaround time is important because regulatory agencies may sampling and rewrting within a speck fied-(ofiin shortj amount of time.. ~ e lay in reporting the results is usually frowned upon by regulatory agencies. The monitoring of process wastewater, for example, requires a short turnaround time so that parameters that are out of compliance may be quickly corrected; violation penalties for out-ofcompliice parameters axe based on per-day calculations. Also, it is difficult for the environmental manager to a p proach the production manager about a high metal value that was discharged three months ago and expect effective corrective action today. Therefore, the sooner the problem is identified, the easier it is to determine the source of the problem. Price. All the laboratories were given a specific set of analysis parameters in writing and asked to quote a price. They were asked to analyze for 33 parameters from 11 samples; mutine wastewater samples and the check
que
samples were mixed in for analysis. There was a wide range of prices charged by laboratories for the same amount of work. The price vs. performance relationship is further complicated by some labs charging 150% and 2W% of the base price to compensate for the required turnaround time in the study However, there is not a definite correlation between the quality of the work and the price of the work. The laboratory that ranked lowest in the precision category, lab A, was also the secondhighest priced vendor. The highest priced vendor, lab C, ranked ninth in the QA category. It is interesting to note that lab J ranked first in price but eighth in QA, and lab F ranked second in price but loth in QA. This indicates that the cost of implementing a QA program is not reflected in the prices charged. Theoretically, a well-documented, well-executed QA plan should lower the long-term costs to the laboratory, and subsequently, the costs to the customer. Oversll ranking.The ranks for each laboratory were multiplied by the - weight assigned for each category. The rank x weight values were totaled for each lab and divided by 6,the number of categories. The lowest weighted average ranked first, and so forth (Table 4). Labs H, G, and D ranked h t , ~ e c ond, and third, respectively. Lab H ranked first in three categories, and lab G ranked first in two categories. Labs C and F, which ranked ninth and tenth in QA, respectively, tied for last in the overall ranking. No lab ranked last in more than one category. Our study confirms that there is a wide spectrum of differences in laboratory performances and that the customer should be aware of the differences before evaluating his or her data. Problems One interesting problem arose during this study. When a laboratory is aware that check samples are sent to examine its performance, more. care may be
ly by the labralory,
taken to ensure the confidence of reported data. Laboratory A was given the same samples one week apart. In the first sample, the laboratory was aware of the purpose of the samples. In the second week the same sample was given to the laboratory along with other routine wastewater samples. The results are listed in Table 5. Analysis of the first sample resulted in parameters that were very close to the true values. Without including Cu, the average accuracy error was 12%, whereas the average accuracy error for the blind check data was 270%. It appears that less care or attention was given to the blind samples. The blind As value of 1200 ppm would have been a major cause of concern if a regulatory agency submitted the sample. Based on this datum, the regulatory agency could have assessed a fine or issued a compliance order to the regulated industry. (In one of the authors’ experience, this situation has occurred.) Conclusions Ours was a small-scale study of commercial laboratories that provide sewices in the Albuquerque area. We were able to select at least three laboratories with standards acceptable to Sandia National Laboratories. Among our other conclusions and suggestions are the following. Do not take reliability for granted. One cannot blindly rely on commercial laboratories for the values of contaminants; data must be critically evaluated. Check samples must be submitted to verify the precision and accuracy of the laboratory data. Sample blanks and equipment blanks are necessary to verify the validity of the analyses. A good laboratory invites its customers to submit check samples. Establish good communications. It is important to establish specific contacts and communications with the commercial laboratory. Good communications will ensure that the requirement? of each party are understood. Sample collection and preservation techniques, sample media, labeling, and custodies must be understood by the laboratory, just as internal procedures of the laboratory should be understood by the customer. It is a good idea to physically audit the laboratory and review its adherence to the written QA program, the qualifications of its staff, and recordkeeping procedures. Continually monitor laboratory performance. A laboratory’s performance should be periodically checked and charted to ensure data reliability. Blind check samples and replicates should be submitted periodically, and the customer should inform the laboratory if it produces an unacceptable er-
ror in its accuracy or precision. Performance charts also may help to reveal trends or problems. The customer’s responsibility is to demand quality work so that he or she can intelligently manage his or her programs, but the laboratory has an obligation to investigate questionable data and to take the appropriate corrective actions in response to customers’ needs. The laboratory must provide quality work to bring the customers back. Do not expect to pay more for better performance. As the demand for laboratory analytical work increases in response to environmental legislation, laboratory capacities and turnaround times will become critical. One may tend to choose whatever laboratory may be available at the time of need. It is important, however, that laboratory products be closely scrutinized to assure that accurate data are provided and meaningful decisions are made. Furthermore, our study demonstrates that there is no relationship between the price of analyses and the reliability of the data; therefore, price is not a good indicator of performance. The quality of the results will be the final cost to the customer. Beware of comparing data from different laboratories. Many companies are instituting long-term (e.g., 30 years) groundwater-monitoring programs. If laboratory precision and accuracy are poor, one may be feeding meaningless analytical data into complex computer programs to perform groundwater modeling. The output will be disappointing because we have found so much variability in data from different laboratories. It is also unwise to try to compare data from different laboratories that were analyzed at different times. Laboratory personnel, equipment, and management often change, and these variables contribute to errors. Unless there is constant vigilance in the oversight of laboratory performance, the data may not make sense. Maintain credibility with the laboratory. The customer must maintain credibility with the laboratory. He or she should not provide misleading information or try to “trick” the laboratory. For example, it should be stated up front that the customer plans to submit 5% or 10%of quality control samples with the environmental samples. The customer must continually provide feedback to the laboratory on its performance. Check samples must be carefully prepared and verified, where possible, by another laboratory. The pursuit of true sample values should be a partnership between the laboratory and the customer.
Acknowledgment This article has been reviewed for suitahility as an ES&T feature by Richard G.
Zepp. EPA Environmental Research Lahoratory. Athens, G A 30613. and by Lawrence H. Keith, Radian Corporation, Austin, TX 78720-1088.
Additional readings Gautier. M. A. et al. Quolir? A , ~ . m r m tfor l Healfh nnd En~~ironnuvzral CIwmi.cq; Los Alamos National Laboratory. Lor Alamos. NM. 1986. Meyerhein. R . Anolwicnl and SumplinK Rrhniquer; New Mexico Scientific Laboratory Division. Xerox. Rice. C.; Brinkman. J . ; Muller. D . Rdiabilify of C h e m i d Ann1y.w of W
