In the Laboratory
Screening and Sequential Experimentation: Simulations and Flame Atomic Absorption Spectrometry Experiments Richard J. Stolzberg Department of Chemistry and Biochemistry, University of Alaska Fairbanks, Fairbanks, AK 99775-6160 An introduction to screening experiments and sequential experimentation should be a part of undergraduate chemical education. We include this topic in the senior level instrumental analytical chemistry laboratory. Hendrix (1) makes a compelling argument for the value of effective screening experiments in industry. He notes that the number of experiments needed to solve a particular problem depends primarily on the size of the random error and the size of the expected effect, but only to a minor extent on the number of variables that are to be studied (1). That is, if the effect of an independent variable is investigated by performing experiments at a low and a high level of that variable,
tal run does multiple duty. Thus, many variables can be studied for the effort that might be expended on studying a smaller number of variables using the single-factorat-a-time approach. The frequency of lost opportunities (type II errors) is minimized by being able to investigate all reasonable factors that might affect results. Furthermore, interactions between two variables may be identified. Subsequent experimentation using only the active variables can supply details or eliminate ambiguity. Instruction in these methods can avoid the unfortunate situation described by Kowalski (2), who related the exasperation of a chemistry major who discovered the existence of efficient experimentation methods only a couple of weeks before graduation. The philosophy and mechanics of sequential experimentation are developed in detail by Box, Hunter, and Hunter (3). A number of articles have addressed experiments of this general type in the chemical education literature. Palasota and Deming (4) described central composite design and construction of response surfaces in conjunction with a spot test experiment for a student lab. Van Ryswyk and Van Hecke (5) did the same with a synthe-
n ≈ 16σ2/∆2 where n = the number of experiments required at each level of the independent variable, σ = the standard deviation of a single measurement, and ∆ = the change in dependent variable caused by varying the independent variable from the low level to the high level. In a properly designed screening experiment, such as two-level fractional factorial design, each experimen-
Table 1. Design and Results of Simulated Synthesis Yield Experiment Factor
Response
A
B
C
D
Case 1
Case 2
Case 3
Case 4
Solvent pH
Light intensity (W/m2)
Ionic strength (M)
[DMC] (µM)
Yield (%)
Yielda (%)
Yieldb (%)
Yieldc (%)
1
4
1000
0.25
0.1
53
43
48
51
2
6
1000
0.25
0.1
58
48
53
56
3
4
2000
0.25
0.1
47
57
62
59
4
6
2000
0.25
0.1
55
65
70
67
5
4
1000
0.75
0.1
50
40
45
48
6
6
1000
0.75
0.1
49
39
44
47
7
4
2000
0.75
0.1
49
59
64
61
8
6
2000
0.75
0.1
59
69
74
71
Run
9
4
1000
0.25
1
46
36
31
28
10
6
1000
0.25
1
45
35
30
27
11
4
2000
0.25
1
46
56
51
54
12
6
2000
0.25
1
46
56
51
54
13
4
1000
0.75
1
52
42
37
34
14
6
1000
0.75
1
51
41
36
33
15
4
2000
0.75
1
48
58
53
56
16
6
2000
0.75
1
50
60
55
58
a
Low light intensity yields are decreased by 10%, high light intensity yields are increased by 10%, by comparison to case 1. The light effect is thus +20%. b Low [DMC] yields are increased by 5%, high [DMC] yields are decreased by 5%, by comparison to case 2. The [DMC] effect is thus {10%. c Low light intensity, low [DMC] and high light intensity, high [DMC] yields are increased by 3% by comparison to case 3. Low light intensity, high [DMC] and high light intensity, low [DMC] yields are decreased by 3% by comparison to case 3. The magnitude of the [DMC]*light intensity effect is thus +6%.
216
Journal of Chemical Education • Vol. 74 No. 2 February 1997
In the Laboratory
sis lab. Strange (6) described two-level factorial design, blocking, and empirical model construction in a general context. We are not aware of any articles in this Journal that guide students through a multiple factor fractional factorial design and a subsequent detailed investigation of active factors. We introduce students to factorial design with a spreadsheet simulation. The goal of the simulation is to familiarize students with specific designs and with the standard normal quantile plots (7) used to identify significant variables. This is followed by the first laboratory exercise: a study of the effect of six variables on the silver flame atomic absorbance signal using a blocked 26{2 fractional factorial design. (Blocking is done because students work in pairs and both students in the group acquire data. The numerical notation indicates that each variable is studied at two levels, there are six variables, but only 24 experiments.) On the basis of the identification of two significant variables in the screening experiment, students then perform further experiments to determine details of the relationship between levels of these two variables and absorbance. Results are interpreted in the context of what students know about chemical and physical factors that affect the flame atomic absorbance signal (8). Simulations and Data Evaluation A Quattro Pro template is used to generate data simulating four different experimental studies of yield of a synthetic product. The template calculates percentage yield in the 16 experiments of the complete 24 factorial design (Table 1). In each case, the effect of pH, light intensity, ionic strength, and concentration of a conjectured catalyst, DMC, on the percentage yield of product is studied. A standard deviation of approximately 4% is superimposed on an average yield of 50%. The random error component changes for each run of the simulation, so each student generates a unique set of data. In case 1, none of the factors affect yield. In case 2, high light intensity gives 20% higher yield than low light intensity (factor B effect = 20%). In case 3, the effect of the concentration of DMC is added. The yield is 10% lower in the presence of DMC than in its absence (factor D effect = {10%). In case 4, these two factors affect yield as in the previous two cases, and there is a B*D interaction of 6%. Results of the four simulations are evaluated with DesignEase 2 software (Stat-Ease, Inc., Minneapolis, MN). Students interpret results of all four simulations by visual inspection of the normal probability plots, which are similar to the normal quantile plots used in this paper. They are also encouraged to investigate the effect of one bad experimental run out of 16 by examining the residual plots available with the software. The visual evaluation of data requires a lower degree of statistical sophistication than evaluation of the numerical ANOVA data, which is displayed with the output. Students have also used a BASIC program written in-house to interpret the results of the experiments. The commercial software is superior, but the local product worked adequately. Although software is available to do all calculations of effect size, we feel it is important that students be capable of making these calculations by hand. For example, in case 2 of Table 1, the light intensity (factor B) effect is the difference between average yield for experiments with high light intensity and average yield in experiments with low light intensity:
B = {(57+65+59+69+56+56+58+60) – (43+48+40+39+36+35+42+41)}/8 = 19.5% The [DMC] (factor D) effect is the difference between average yield for experiments with high [DMC] and average yield in experiments with low [DMC] : D = {(36+35+56+56+42+41+58+60) – (43+48+57+65+40+39+59+69)}/8 = -4.5% Each experiment does multiple duty because these 16 results can be grouped in 15 different ways to calculate 15 effects. Details of calculations and the more efficient Yates algorithm are described elsewhere (6, 9, 10). The final problem is to distinguish significant effects from effects that represent random variation around a true value of zero. This is done by an examination of standard normal quantile plots of effects (7). In the absence of any factors affecting yield (case 1, Table 1), the 15 effects are random variates with a mean of near 0% and a standard deviation of 4.2%. The standard normal quantile plot shows all effects falling approximately on a straight line (Fig. 1a). The interpretation of a result showing no significant effects is that none of the investigated variables has an effect on the response, percent yield. That is, there are no active factors. When high light intensity causes a 20% higher yield than low light intensity (case 2), Figure 1b shows 14 small (i.e., nonsignificant) effects clustered near zero and a factor B effect of 19.5%. The factor B effect is translated horizontally to the right of the cluster of random variates, indicating that it is larger than can be explained by the presence of random errors. That is, factor B, light intensity, is an active factor. (The factor D effect of {4.5% is part of the cluster of random variates. In case 2, the catalyst effect is not perceived as significant.) When higher [DMC] causes 10% lower yield than lower [DMC], the factor D effect is observed to be {14.5 and the data point is translated horizontally to the left of the random variates in Figure 1c. The [DMC] effect is real, but DMC presumably promotes a side reaction, reducing the yield of product. The addition of the B*D interaction (case 4) results in displacement of the B*D effect point slightly to the right of the random variates. In each case, the calculated effect (19.5%, {14.5%, 5.5%) is an estimate of the
(a)
(b)
(c)
(d)
Figure 1. Standard normal quantile plots for simulations. Each point represents a calculated effect. (a) Case 1: no significant factors. (b) Case 2: factor B effect = 20%. (c) Case 3: factor B effect = 20%, factor D effect = {10%. (d) Case 4: factor B effect = 20%, factor D effect = {10%, factor B*D effect = 6%
Vol. 74 No. 2 February 1997 • Journal of Chemical Education
217
In the Laboratory
(a)
slit width, lamp current, and PMT gain. Then the instrument zero is adjusted. Signal strength is determined by making five pairs of alternating readings of distilled water and the appropriate silver solution. The software calculates the average value, which is used as the response. Data acquisition takes approximately one hour. Students evaluate their data using the factorial design software. They then choose two significant factors to study in detail in the second stage of experimentation. If more than two factors are significant, continuously variable and interacting factors are chosen in favor of factors that have a small number of allowable levels (such as wavelength or slit width). Students then design a two-factor matrix of experiments with four to six levels of each variable. If the concentration of acetic acid was chosen as a variable, students prepare a series of solutions containing the desired concentration of acetic acid and 2.00 ppm silver by dilution of a stock silver solution. They are instructed to use glacial acetic acid in the hood and to wear lab goggles and chemically resistant gloves. Signal measurements are then made in random order for each combination of factor levels in the design matrix. Students write a report describing the effect of all variables on the silver absorbance signal. The results are put into perspective of what might be expected, based on their knowledge of flame AA. For the two variables studied in detail, students make a graphical presentation of results and then explain the results in writing.
(b)
Figure 2. Simulation effect plots. Each point is the average of four experimental results. (a) Case 3: no interaction. (b) Case 4: B*D interaction effect = 6%.
real effect (20%, {10%, 6%). The differences are due to the presence of the 4% random error superimposed on the observed yield. An interaction between two factors exists when the magnitude of the effect of each factor depends on the level of the other factor. This can be understood pictorially by examining effects plots (Fig. 2). In the absence of a B*D interaction, the plots are nearly parallel and the [DMC] effect is a constant {14.5%. In the presence of the B*D interaction, the lines converge. The [DMC] effect is {20% at low light intensity and only {9% at high light intensity. Laboratory Experiments Students are supplied with 2.00 ppm Ag+(aq) in water and in 5% (v/v) acetic acid. The flame atomic absorption spectrometer (Perkin Elmer 103) is adjusted optically, and the A/D converter and data acquisition software are installed. Students work in pairs and follow a randomized 26{2 blocked design, which they generate before lab with the Design Ease 2 software. For each of the 16 experiments, students set the following while aspirating distilled water: burner height, fuel flow setting,
Experimental Results The observed silver signal varies by a factor of more than three in the 16 initial experiments (Table 2). The normal quantile plot (Fig. 3) indicates that three factors clearly affect the strength of the silver absorbance signal: wavelength (E), flame observation height (A), and
Table 2. Design and Results of Silver Atomic Absorption Experiment Factor
Obser- Run vation order
218
Block
Response
A
B
C
D
E
F
Flame ht above base (mm)
Flame stoichiometry
Acetic acid (%)
Lamp current (mA)
Wavelength (nm)
Slit width (nm)
Signal (mAbs)
1
6
1
6
lean
0
4
328.1
0.2
95
2
14
2
12
lean
0
4
338.3
0.2
41
3
9
2
6
rich
0
4
338.3
0.7
63
4
1
1
12
rich
0
4
328.1
0.7
83
5
7
1
6
lean
5
4
338.3
0.7
59
6
13
2
12
lean
5
4
328.1
0.7
114
7
16
2
6
rich
5
4
328.1
0.2
121
8
3
1
12
rich
5
4
338.3
0.2
59
9
11
2
6
lean
0
8
328.1
0.7
107
10
2
1
12
lean
0
8
338.3
0.7
38
11
5
1
6
rich
0
8
338.3
0.2
44
12
15
2
12
rich
0
8
328.1
0.2
73
13
10
2
6
lean
5
8
338.3
0.2
60
14
4
1
12
lean
5
8
328.1
0.2
97
15
8
1
6
rich
5
8
328.1
0.7
105
16
8
2
12
rich
5
8
338.3
0.7
53
Journal of Chemical Education • Vol. 74 No. 2 February 1997
In the Laboratory
Figure 3. Normal quantile plot for Ag atomic absorption experiment.
Figure 5. Effect plot for Ag atomic absorption experiment. Each point is the average of four experimental results.
Figure 4. Cube diagram for Ag atomic absorption experiment. Bold numbers at vertices are the average signal in mAbs units for two experiments.
Figure 6. Flame height and lamp current effect plot for Ag atomic absorption experiment.
percent acetic acid (C). Lamp current (D) and the interaction of flame observation height and percent acetic acid (A*C) are possibly active factors. The cube diagram (Fig. 4) gives a summary of the observed signals at each of the eight treatment combinations of the three major factors. Choice of wavelength has the largest effect on signal strength. The 328.1 nm wavelength produces a 90% larger signal than the 338.3 nm wavelength (compare the front and rear face.) The signal is larger when the matrix is 5% acetic acid (compare top and bottom faces) and when the light beam passes through a lower portion of the flame (compare left and right faces). The figure shows the interaction of flame observation height and acetic acid concentration, the A*C interaction. (Compare left to right changes in signal for the top face, 5% acetic acid, and the bottom face, 0% acetic acid.) This is made more obvious in the effect plot for these two variables (Fig. 5). There is little change in signal (approximately 7% relative) with change in flame observation height when the matrix is 5% acetic acid. In contrast, there is a substantial change in signal (30% relative) with change in flame observation height when acetic acid is absent. Other factors have little or no effect on signal. Lamp current has been a small but significant factor in about half of the student results. This is almost certainly an artifact of operating the lamp at extremely low light intensity so that the AA cannot be adjusted to normal op-
erating conditions. Only very small lamp currents appear to give low signal (Fig. 6). Neither flame stoichiometry nor slit width has ever been observed to be significant in six independent student studies with two instruments. Students find reassurance and surprise when they compare their observations to statements of fact in their textbook (8) and in the standard method sheet supplied by the AA manufacturer (11). The standard method sheet indicates a factor of two difference in sensitivity between the two wavelengths. This is what is observed. The organic solvent effect is present, as indicated in both sources. The silver signal changes with observation height in the flame, but in a direction opposite to that expected. Students observe that the signal strength is greater near the base of the flame than near the middle, while Figure 10-12 of the text shows a strong increase in signal with increase in flame observation height. The design of the second stage of experimentation is apparent to most students. A matrix of four or five flame observation heights and four or five values of percent acetic acid will give an experiment whose results will show details of the interaction. Results of this second set of experiments illustrate the complexity of the system (Fig. 7). In the absence of acetic acid, the maximum signal is observed at midheight in the flame. In solutions containing 2, 4, or 6% acetic acid, the signal is nearly constant slightly above the base of the flame, and
Vol. 74 No. 2 February 1997 • Journal of Chemical Education
219
In the Laboratory
Figure 7. Flame observation height and % acetic acid effect plot for Ag atomic absorption experiment
it decreases at greater heights. When the solution contains 8% acetic acid, there is a continuous and sharp decrease in signal strength above the base of the flame. Two details of the design and the results bear elaboration. First, because students work in pairs, blocking is done to eliminate the possibility that student differences will constitute an unrecognized and uncontrolled variable that might bias or complicate interpretation of the data. Second, this fractional factorial design has been chosen to minimize the number of experimental runs made to screen the six variables. The trade-off to the gain of efficiency is the confounding of some of the two-factor interactions. That is, it is not possible to unambiguously assign the calculated “A*C effect” uniquely to either a true A*C (flame observation height and percent acetic acid) interaction or a true B*E (flame stoichiometry and wavelength) interaction. The ambiguity can be minimized by appropriate choice of letter assigned to each variable during the planning stages of the experiment. This was done for this experiment, and we can be reasonably certain that the B*E interaction has no physical or chemical cause, so we can assume that the entire A*C effect is due to the reasonable interaction of flame observation height and composition of the aqueous matrix. When a highly fractionated design does lead to serious ambiguity, highly fractionated “fold-over” experiments can be done to remove ambiguity (12).
220
Summary Student response to this and an allied experiment, simplex optimization of signal-to-noise ratio (S/N) in flame AA, is strongly positive. One student extended these experiments to use fractional factorial design to study the effect of seven factors on S/N in the determination of silver by flame AA. The results showed that wavelength, data acquisition time, and flame observation height affected S/N, and the S/N ratio varied by nearly a factor of five in the trials (Dick, K., unpublished results). The methodology described in this paper is superior to the one-factor-at-a-time approach. Many factors can be screened in a relatively small number of experiments. Important factors are subsequently studied in detail. Interactions are readily identified. We owe it to our students to show them these more productive methods of experimentation (13). Acknowledgment The Spring 1994 Instrumental Analysis class supplied data used in Table 2 and in Figures 3–7. Literature Cited 1. Hendrix, C. D. Chemtech 1979, 9, 167–174. 2. Geladi, P.; Esbensen, K. J. Chemometrics 1990, 4, 337–354. 3. Box, G. E. P.; Hunter, W. G.; Hunter, J. S. Statistics for Experimenters: An Introduction to Design, Data Analysis, and Model Building, Wiley: New York, 1978; Chapter 1. 4. Palasota, J. A.; Deming, S. N. J. Chem. Educ. 1992, 69, 560–563 . 5. Van Ryswyk, H.; Van Hecke, G. R. J. Chem. Educ. 1991, 68, 878–882. 6. Strange, R. S. J. Chem. Educ. 1990, 67, 113–115. 7. Mason, R. L.; Gunst, R. F.; Hess, J. L. Statistical Design and Analysis of Experiments with Applications to Engineering and Science; Wiley: New York, 1989; p 241. 8. Skoog, D. A.; Leary, J. J. Principles of Instrumental Analysis, 4th ed.; Saunders: New York, 1992; pp 205–209, 211–222. 9. Box, G. E. P.; Hunter, W. G.; Hunter, J. S. Statistics for Experimenters: An Introduction to Design, Data Analysis, and Model Building, Wiley: New York, 1978; p 309–316. 10. Box, G. E. P.; Hunter, W. G.; Hunter, J. S. Statistics for Experimenters: An Introduction to Design, Data Analysis, and Model Building, Wiley: New York, 1978; p 323. 11. Perkin Elmer Corp. Standard Atomic Absorption Conditions for Ag; January 1982. 12. Box, G. E. P.; Hunter, W. G.; Hunter, J. S. Statistics for Experimenters: An Introduction to Design, Data Analysis, and Model Building, Wiley: New York, 1978; pp 398–404. 13. Kvalheim, O. M. Chemometrics and Intelligent Laboratory Systems, 1993, 19, iii–iv.
Journal of Chemical Education • Vol. 74 No. 2 February 1997
