Determination of the Precision of Analvtical Control Methods J
R.4YlIOND F. RIORAN, Westvaco Chlorine Products Corporation, South Charleston, W. Va.
of uncertainty) can then be calculated within which routine analyses may be guaranteed. The average of duplicate determinations made a t the same time does not result in as much improvement as may be theoretically calculated, evidently because the results are not truly random. Normal control methods were found to have 1.5 to 2.5 as much variation under routine conditions as the same method under the best conditions. The use of this method of criticism has proved a valuable tool in the author’s laboratories.
For the intelligent control of plant operations and product quality, it is essential that the precision of the analytical control methods be quantitatively known. The use of statistical reasoning based on the standard deviation has been found applicable. The analytical method is first tested under ideal conditions to find the highest precision of which the method is capable. If this precision is judged high enough, the method is then tested in routine practice for a year to discover the variability under routine laboratory conditions. An LUz (limit
I
because the precision of the analytical method is not known, There also exists a tendency on the part of many engineers. sales personnel, and even chemists to treat a single analytical result as an exact quantity and to make decisions therefrom that would be unjhstifiable if the significance of the result were known. After an extensive program of development and refinement of analytical methods, this company was faced with the determination of how well the routine control laboratories were following these new methods. Several years were spent in check sample work in which standard samples were run in duplicate by one laboratory and attempts were made to check these results in another laboratory. Although considerable information was obtained, a great many disputable differences arose that could not be traced to any assignable cause. One reason for this disagreement was found to be the use of the ‘‘average deviation” as a criterion, since any variations greater than the criterion were treated as poor analyses. I n commercial analytical practice speed is often essential for good control, even if accuracy and precision are sacrificed. It is often expedient to employ nontechnical personnel as analysts or testers for routine determinations. Any method for determining the precision of routine results must be able to evaluate these personnel factors. The procedure described in this article was developed in the summer of 1940 and has been used throughout the company with satisfaction since that time.
T IS a primary concept of nature that no one physical measurement is exact. Only those values that are accepted by definition are free from deviations in the last significant figure. The determination of the composition of any sample even by the best known technique is similarly influenced by the inability to measure weights, volumes, colors, chemical equilibria, etc., with exactitude. Practically all analytical methods contain enough small constant errors to make them somewhat empirical. It follows that some differences will be obtained between individual analytical results from the same sample even if the best possible technique and instrumentation are used. When the conditions that obtain in commercial control laboratories are considered, the above concept becomes much more important, since the variations between personal analytical techniques, solutions, apparatus, and surrounding conditions are certain to influence the precision of an analytical method to a considerable extent. This is true even if the method is followed exactly as written; a departure from standard instructions or error in judgment would give further deviations from the truth. Since these variations are known to exist, they are a constant threat to good commercial operation or product quality until they are quantitatively determined. An attempt to control a plant operation within 0.05 per cent of a standard value with a method precise to but 1 0 . 5 per cent would obviously fail, yet similar situations frequently occur in industry 361
Vol. 15, No. 6
INDUSTRIAL AND ENGINEERING CHEMISTRY
362
OF METHOD UNDER BESTCONDITIONS TABLEI. PRECISION
Method: SC-11-3 of 6-27-41for specific gravity of carbon tetrachloride by pycnometer) Specific Gravity Test KO. at 25’14’ C. 11 r i z ( X 10”’) - 0.00016 266 1 1.5849 +o. 00004 16 2 1.5s51 + O . 00024 576 1.5853 3 +0.00024 ,576 4 1.5853 - 0.00006 36 5 1.5850 -0.00016 256 6 1.5849 7 1,5847 -0 00036 1296 S 1.5851 + O 00004 16 9 1.5s52 + O 00014 196 10 1.5851 +O 00004 16 Zd* = 3240 X IO”’
LUav.
-
1.58506
=
6
1,58489to 1 58523
V G
Prior Concepts A number of investigators @,S, 5 , 6 ) have indicated that the variations of an analytical method may be treated by statistical calculations. Power (6) found that over 100 determinations had to be made before the variability of a micromethod for carbon could be accurately estimated. Informative as such an extensive investigation might be, such a study in a commercial laboratory would be difficult to justify for economic reasons. The two routines given below use 10 samples and 24 samples to obtain the precision and the results hare been satisfactory from the author’s point of view-. Although some of the calculations have the surface appearance of complexity, the most complicated operation is the extraction of square roots. The criterion by which the precision of the method is to be expressed is of considerable importance. It may be mathematically proved that the standard deviation, u, is the most accurate measure of dispersion about an arithmetical mean. Any multiple of u may be selected according to the limits of precision desired. Since the author wished to express the precision as the limits within which an analytical result could be practically guaranteed, the value *3u was chosen. If the variations are distributed about the mean according to chance and the determinations are representative, the average * 3a should include 997 out of 1000 determinations ( 1 , 7 ) . This is equivalent to one result outside the limit in a year’s routine of daily analyses and was thought to represent the precision in the best possible manner for commercial practice. The investigator must use some judgment before any unreserved guarantees are given, since these characteristics of u are strictly true only for a system of chance causes; but the viewpoint adopted was that all variations are due to chance until the cause becomes known. By intelligent usage of the data obtained from these calculations, variations due to other than chance causes may be detected and subsequently eliminated. Although the accuracy of a method is of great importance, a general procedure for its determination cannot be described. When a standard sample made from known amounts of the desired ingredients can be prepared, a definite measure of accuracy may be determined, but such samples are sometimes impossible to create with certainty because of the nature of the sample. Most commercial methods are more or less empirical in nature and are accurate by definition, their purpose being to enable two or more interested parties to obtain substantially the same results on the same material when analyzed independently. All the author’s methods were checked by whatever convenient means were possible,
such as the use of standard samples, synthetic knowns, checks with standard methods, etc., and if the limit of uncertainty of the average (described below) bracketed the known result, the method was said to be “accurate” in a qualitative sense.
Limit of Uncertainty under the Best Conditions
(Lull The highest precision that can be expected from an analytical method is the precision as shown by the best available technician working under the most closely controlled conditions. The following procedure was prescribed as standard for obtaining a measure of this precision: Prepare a synthetic mixture that exactly reproduces the commercial material and contains a known amount of the desired ingredient, or select a representative homogeneous sample of the caommercial material. Store enough material for at least fifty analyses in containers that will prevent any change in composition for at least one year. Select an analyst who is well acquainted with the method, preferably the one who developed the method. Prepare fresh reagents, standardize all solutions, and calibrate all apparatus before making the determinations. Run ten analyses as closely together as possible under the most favorable conditions. Calculate all results to one more place than is enerally reported. balculate the arithmetical average, Tl,of the ten analyses. Calculate the standard deviation of the results by the use of the following equation : 610
where
=
4%
standard deviation from the average shown by the 10 results d = individual deviation of each result from the average
u10
=
Calculate the limit of uncertainty of the average, LU,,,, by the following equation ( I ) :
Calculate the limit of uncertainty of the method under the best conditions, LU1, as follows ( I ) :
Table I indicates the method used for calculation. If the sample is of known composition and the method is accurate, Z1 * LU,,. should bracket the known value. If the result differs, judgment has to be used to decide whether the method should be improved until it is accurate or a standard correction should be applied to all results. If the concentration is not known, the accuracy cannot be determined unless the method is “accurate by definition” as previously mentioned. Since the LU1 is a measure of the ultimate precision of the method as written, a decision as to the commercial applicability of the method can be made. If the indicated precision is high enough, the method is ready for test under routine conditions, but if it is found too low the method must first be improved, since the test under routine conditions is almost certain to show an even lower precision because of the additional sources of variation introduced.
Limit of Uncertainty under Routine Conditions ( LU2) If all the above conditions have been satisfied, the method may be tested under routine conditions by the following prescribed procedure : Use the sample on which the LCl test was made. Have the sample analyzed in a routine manner by a routine technician, preferably together with routine plant samples of a
ANALYTICAL EDITION
June 15, 1943
similar nature. If possible, the techniri,in should not knou t h e prior results. Have the sample analyzed in duplicste each month for one year, using as many different routine technicians as possible. As soon as the monthly analyses have been completed, compare the results with the average and LU1 obtained during the LL'I, allou first test. If the individual results are within XI them to stand as representative. If the results are outside these limits, immediately check the technique, reagents, etc., so that any deviation from the method will be discovered before the details slip the technician's mind. If an objective difference in technique is found, have the analyses rerun by the same technician but in the correct manner. If no assignable cause is found, allow the results to remain uncorrected. When the 12-month accumulation of 24 analyses has been completed, average the 24 results to obtain z2. Also calculate the average of each month's duplicate tests. Calculate the standard deviation of the 24 individual tests, uZ4,and the standard deviations of the 12 monthly averages.
+
6:s.
Calculate the limit of uncertainty of the average as follow:
Calculate the limit of uncertainty of the method ( 1 ) :
Table I1 illustrates these calculations. If the limit of uncertainty of the average includes the known value, the method may be called accurate. If the concentration is not known but the method is "accurate by defini-
363
tion", the limits of uncertainty of the averages under both LU,and LC2 conditions must overlap for approval of the method. The LI'? result is the objective figure of the whole study and represents the precision that may be expected from the method under routine conditions. Since 997 out of 1000 determinations can be expected to represent the truth within +LT,, this result may be used to judge whether the method is precise enough for routine control and is a measure of the range within which analyses are to be trusted. If the LIT2is judged small enough, the method is approved for routine but if it is too large for good control, it must be replaced or amended. By comparison of the LC2 with the LU1, a measure of the amount of possible improvement in routine work is obtained. Since the LC1 was obtained under ideal conditions, the LU2 may be brought closer to the LC1value by further education of routine personnel, intensive standardization of technique or equipment, closer temperature control during seasonal variations, etc., but it might be found that the analytical method possesses inherent characteristics that allow personal variation that cannot be standardized without extensive modification of the method. The comparison of the standard deviations shon-n by the individual analyses and the duplicate averages represent the amount of improvement that may be obtained by the averaging of two or more analyses. For all these possibilities that may confront the investigator, the LrTland LC2are excellent toois for guidance.
Discussion of Results
The analytical method reported in Tables I and I1 represents a simple technique that is of major importance to the author's laboratories, since it is a sensitive test for the composition and purity of chlorinated hydrocarbons and their PRECISIOX OF METHOD UNDER ROUTINE COKDITIOXS TABLE 11. mixtures. This determination is made in a 25-m1. pyc(Method: SC-11-3 of 6-27-41 for specific gravity of carbon tetrachloride b y nometer containing an accurate thermometer. The pycnomepycnometer. Average of 10 analyses for LCi test = 1.58506, L ~ =I i:0.00059) ter is filled with the sample and allowed to expand to near Deviation of Deviation of equilibrium temperature in a balance room. The contents Specific Individual Duplicate are adjuded to the mark, the temperature is read, and the Gravity from Grand Average from at Average of Average, Grand pycnometer is n-eighed. The indicated specific gravity is Month Analyst 25"/4' C. Duplicates di Average, d, then corrected to 25'/4" C. by means of the appropriate - 0,00003 1.5850 January + O . 00007 + o , 00002 1.5851 1.58505 fact or. - 0.00021 1.5848 February The L I T 1test (Table I) indicated that this method nould - 0.00013 1.5850 1,58490 - 0.00003 - 0,00043 1.5846 .\larch give results within *0.0006 of the mean in 997 out of 1000 - 0,00043 - 0.00043 1.5846 1.58460 1.5851 ++0.00007 April trialb. By comparison with plant control and finished prod0,00007 + O . 00007 1.5851 1 . 5S5lO uct specifications, this indicated precision was found satis+ 0.00027 3 1.5853 May +0.00027 3 1,5863 1.58530 + O . 00027 factory. The average and limit of uncertainty of the average 1,5849 4 -0,00013 June - 0.00008 4 1.58495 -0,00003 1.5850 check closely 11-ith the 1.5850 a t 25'/4" C. characteristic cal0.00013 3 1,5849 July culated from reported data for carbon tetrachloride (4). 5 -0 00023 -0 00018 1.58485 1.5848 3 1.5854 i0.00037 August hccoidingly, the method was judged ready for the L C z test. f0.00037 3 1,58540 +0.00037 1.5854 + O .00017 3 September 1,5852 The LI-T2test (Table 11) indicated that the method was also 3 +o, 00028 1 ,58525 t0.00027 1.5833 accurate in routine work, since the average agreed very closely 1,5847 -0.00033 October 1.58490 + O , 00007 - 0.00013 1,5851 with the previously obtained figure. The LC2was found to be Sovember 1,5850 -0.00003 1.58500 - 0.00003 -0 00003 1.5850 *0.0007, which T T ~ Sonly slightly higher than the LV1 of 1.5849 -0.00013 December * 0.0006 and represents very good agreement for routine -0 00003 1.5851 1.58500 + O 00007 6 Grand average = x', = 1,58503 work. The method was therefore judged satisfactory for all routine laboratory n-ork. 1 7 Standard deviation (individual determinations) = 024 = ''dl - 0 00022 The standard deviation of the averages of the duplicate v24 determinations in this study was most enlightening. If the -1 deviations were purely random, the averaging of two indeStandard deviation (duplicate averages) = c i 2 = = 0.00021 pendent analyses should have decreased the variations by LE Limit of uncertainty of grand average = 1.58503 i: ? =? 1,5849 it o 1.5852 factor of or 1.41. The actual improvement as shovn by 4% the ratio of standard deviations was only 1.05. This ratio Limit of uncertainty of method under routine conditions = i: = indicated that duplicate analyses made at the same time are 0.968 t 0 . 0 0 0 6 8 = LVz not truly random but are rather influenced by slight variations in technique or surrounding conditions. = 1, O s Ratio of improvement by duplication = E?! = 012 0.00021 Table 111 gives the over-all results from a number of repLU 0.00068 Ratio of 2 = = resentative commercial analytical methods. For each of LUI 0.00059 "15 theqe Seven methods the precision -hewn by the LT', test
$$
*
~
di
Vol. 15, No. 6
INDUSTRIAL AND ENGINEERING CHEMISTRY
364
appeared satisfactory. The LC2 of tests 1, 3, 4, and 5 were low enough for the required closeness of control. Test 2, a determination by a modified Volhard method for chlorides, indicated an Lug that was 2.7 times as large as the LU1 and was considered too variable for good control. This method was re-examined for possible improvement. Tests 6 and 7 were obtained by a refined Engler distillation technique, but the results were found to be influenced by seasonal variations and superheating of vapors in the flask. These methods were also improred to eliminate these assignable causes.
TABLE 111.
PRECISIOK
Commercial Application Because the study is founded on statistical reasoning from the comparatively small number of samples dictated by economic reasoning, the results are open to some variations in an absolute sense. However, in practice this objection has been found academic, since the study has resulted in definite precision characteristics close enough for purposes of refinement of methods and techniques and to discover within rvhat limits routine analytical results are significmt. The
DETERXINATIOSS O F REPRESESTATIVE COMBIERCIALAKa4LYT~c.k~ METHODS Ratio I m of ~ oDuplicate vem~nt
KO.
Sample 5070 caustic soda 50% caustic soda 50% caustic soda Carbon tetrachloride Carbon bisulfide Trichloroethylene Trichloroethylene
Analysis
I n the study of some 40 basic analytical methods during the last two years, the following relationships were found from the LC2/LU1ratio: Ratios of 1 . 0 to 1 . 5 indicate that the variations result mainly from limitations of instrumentation. Ratios of 1.5 to 2.5 indicate normal relationship between analyses made under the best conditions by one man and analyses made under routine laboratory conditions. Ratios over 2.5 generally indicate considerable personal and seasonal variations that may be reduced without modification of the basic analytical method. Duplicate determinations run a t the same time are generally not truly random and the averaging of such results does not improve the precision appreciably. The only justification for running analyses in duplicate is the additional safeguard afforded against an outright mistake, such as an error in calculation or an erroneous weighing. Most commercial analyses are made on products of approximately constant composition, so that a measure of precision may be expressed in terms of the actual percentage variations on the original sample basis. Whenever several widely different levels of composition exist, it has been found desirable to make complete studies at each level to discover whether or not the extent of the variations is proportional to the amount of ingredient.
Procedure for Decomposable Samples Several cases have been encountered where the year’s study was rendered impossible because of the instability of the samples or attack on the containers. I n this case either one of two techniques has been used to approximate the true LC*: 1. If the sample decomposes a t a constant rate, the results may be plotted against time and an average curve drawn through the points. The deviations of the individual results from this curve are used for the calculation. 2. If the sample decomposes in an inconstant manner or attacks its container appreciably, or if an approximate determination of the routine precision is desired rapidly, a speedup of the technique may be used by making the LUZ tests during one day by as many routine analysts as possible. This gives only an approximation of the true L C z , because variatims encountered during a yearly study may not be observed. It does yield an approximation of personal differences.
L 1-1 *0.078 10.0102 +0.77 1 0 ,00059 10,0004 * o 11 tO.05
L C? 1.0. 147 t o . 0280
L C x / L CI 1.9
t1.00 *O. 00068 * O ,0008 *o 33
*0.30
2.7 1.3 1,l5 2.0 3.0 6.0
Determinations 1.2 1.0
1.1
1.05 1.0 1.07
1.07
procedure described is offered as a tool for such investigation and as a standard method for describing precision. The use of standard samples in the control laboratory has the additional advantages of providing periodical checks on personal techniques and for the education of new laboratory personnel. Since the basis of comparison is fair, the individual technician is more likely to cooperate than if he is expected to duplicate exactly a single analysis from some outside source. These techniques have been in use throughout this company for over two years and have formed the first common bases for describing and comparing precision. They have led to the discovery and subsequent correction of many variations previously not suspected and bhe quantitative evaluat’ion of other deviations that were known to exist. By the judicious use of LU2 figures, the men having supervision of plant operation and shipping know within what limits routine analyses are to be trusted. Some similar procedure could well be adopted as st’andard by industrial analytical chemists to provide a mutual basis of understanding when descfibing the precision of analytical methods.
Acknowledgment The writer gratefully acknowledges the assistance of Dwight, Killiams in the development of the described procedures and the preparation of this paper, and the permission of Westvaco Chlorine Products Corporation for publication.
Literature Cited (1) -4m. Soc. Testing Materials, “1933 Manual on Presentation of D a t a ” , second printing, 1937. (2) Benedetti-Pichler, A . d.,ISD. ESG. C H E Y . , AXLL. E D . , 8, 373 (1936). (3) Crumpler, T. B., and Yoe, J. H., “Chemical C o n p u t a t i o n s and E r r o r s ” , New York. J o h n Wiley & Sons, 1940. (4) International Critical Tables, 1st ed., Vol. 3, p . 28, New York, McGraw-Hill Publishing C o . , 1928. (5) Power, F. W., IND. ENG.C H E Y . ,ANAL.ED.,11, 660 (1939). (6) Power, F. W., ”Application of M o d e r n Statijtical Methods in Chemical Analydia”, presented before Division of Analytical and Micro Chemistry, A M E R I C ACHEMICAL S S O C I E T YMemphis, , Tenn. (7) Shewhart, “Economic Control of Quality of h l s n u f a c t u r e d Products”, New York, D. V a n Nostrand Co., 1931. PRESENTED before the Division of Analytical and Micro Chemistry at the 103th Meeting of the AMERICAN CHEXICAL SOCIETY. Detroit, Mich.
