Student Strategies in a Junior-Level Procedureless Laboratory Warren S. Warren and Miles Pickering' Princeton University, Princeton, NJ 08544 For some time. the chemical education communitv has been interested in free-form practical exams, in which students were expected to do lab work without written instructions (1-3). ~ ~ a r v a r dfreshman 's program, in particular, has featured many such examinations in recent years (2). The importance of studying such lab exams is that they giveus afairly direct view of how students approach lab. The work of Johnstone (4) has emvhasized the effects of limited short-term memor; bu student lab performance. Mulder and Verdonk (5) . . have emvhasized that learnine the theorv of pn~ceduresrarely takes place simultaneously with manipulative learnine. It is reasonable toexuect that a student will construct a set of instructions in such a way as to use mental resources in the most productive way. Hence studies of such student notes may hklp us develop lab manuals that are easier to use because they are consistent with the way students actually think in lab. In a previous paper (3) one of the authors put great stress on feedhack in such procedures. "Feedback" is the use of rough experimental results to modify the procedure for the rest of the exveriment. a successive refinement avvroach. This earlier study, done in an advanced freshman Eourse a t Yale. showed that the successive refinement was a verv rare (hutexceedingly effective) strategy. The work currently reported is with juniors, and it will be interesting to see if a larger proportion of the class uses feedbackin the procedure. We also need to know how verbal and mathematical aptitude scores on conventional paper-and-pencil testa correlate with the ability to solve this sort of problem. Not only does this shed light on our test, hut also upon the nature of the "mathematical" and "verbal" ability measured by the SAT. The Experiment This experiment was a "practical exam" (see Appendix) required of students taking the integrated lab course at Princeton. All of the 21 students were chemistry majors, mostly juniors, and were unusually able even by our own past standards.(meanmath SAT = 732, mean verbal SAT = 689). A day in advance, students were given a handout sheet presenting three measurement tasks, each containing two parts, one quite easy, one significantly harder. They were allowed to prepare procedures for all three before lab and were told that on test day, they would he expected to carry
I
68
Author to whom correspondence should be addressed.
Journal of Chemical Education
out only two of the three measurement tasks (assigned randomly): The third procedure (the "preparation" section of the exam) would he graded hy aTA. The test questions form an a ~ n e n d i xto this ianer. beginning of the period the students 0;'test day, at were told which two of the three pairs of measurements they would do. Exam conditions (no collaboration, silence, etc.) were enforced for the whole period. Handwritten notes, hut no hooks, were permitted in the lab. Students were then allowed 90 minutes for each pair of measurements. The grade on the measurements performed was based on how close results were to the accepted value. Grades on hoth the measurement and preparation sertion of the exam were normalized so the average would he the same, no matter which measurement tasks the student was assigned. No correction of student mathematical errors was done before grading since this did not appear to be necessary.
tee
Analysis ol Results The scores on the section eraded on results and on the TAgraded preparation section showed a normal distrihution. This is different from the results on "one-task" practical exams, in which a frequency plot of the results has an exponential shape. The normality of our distrihution allows more powerful statistical methods to he used, instead of the nonparametric statistics used in past studies. The samples used will seem very small to the casual reader. The statistical tests used are such that they will he "fail safe"; small correlations may be missed, hut correlations U
Correlation with Free-Form Exam Scores and Other Variables VSAT Practical Exam I (N = 21) Preparetwy part Experimentalpart Total Exam
0.13 0.14 0.16
Practical Exam I1 ( N = 14) Preparatory part Experimentalpart Total exam
-0.24 0.365 0.303
Correlation wilh MSAT Written Final
0.35. 0.3i5 0.37' -0.07 0.48' 0.49'
0.38. 0.28 0.36' 0.00 0.69 O.7Sa
reported are real in spite of the small sample size. The sample size also means that the levels of significance are not narticularlv hiah. .. (.'orrelntion coefficients wirh math and verl~ntSAT and with the luhcourse's uwn iinal exam %,erereuurted.'l'his test was then replicated using a different group of 14 students (mean math SAT = 734, mean verbal SAT = 654), a different grader, and a similar hut different set of problems. The correlation coefficients for the second practical exam are shown in the table also. For the experimental part, the correlation coefficients with math SAT are significant ( p < 0.1) hut the correlation coefficients with verbal SAT are much lower. In the second practical, the same correlation pattern appeared. The correlation between the total practical exam results and the course final is also significant for hoth practical exams. This is not really unexpected, as similar results have been shown for other types of practical exams (6). It is interesting that the math SAT is correlated to the ability to do a measurement. This correlation replicates over two different sets of problems. This is, to our knowledge, the first time that anyone has shown an extension of this measure of mathematical aptitude to the solving of a quantitative problem requiring a measurement rather than a paper and nencil task. ~ i correlation e of math SAT and freshman chemistry is well known and not unexoected (7-9).T h e correlation of this score withorganic has r e i e n t ~ ybeen shown (10) and together with the present result seems to he arguing that the math SAT is really measuring something beyond strictly mathematical aptitude in the narrowest sense. The preparation part showed wildly erratic correlations with ability scores on the second practical exam. The apparent correlations seen on the first exam were presumably an artifact. Since the preparatory part is subjectively graded, there is a strong grader dependence. It is also interesting that the correlation of preparatory part grades and experimental part grades was 0.28 on the first exam and -0.30 on the second exam, in neither case significant. This is further evidence of the striking divergence between TA grading and grading on results shown in previous studies (6). Grading a student's written procedure is not equiualent to grading on experimental results. In a previous study (3) we examined the student's procedures and records in denth and showed that the onlv reallv important difference was the extent to which successive refinement was built into procedure. Combining hoth practicals, we saw only 11 student procedures with any planned use of feedback or proofs of method. Manv of these were really only slight gliknerings of feedback, dot really coherent planninp. The averaae test and SAT scores were higher for those &dents using a successive refinement strategy, hut sample size precludes meaningful tests of significance. If ~oh&.tone's theories are correct, one mightexpect procedures would he organized by students in a way that would optimize the use of short-term memory. Most procedures had several striking features. First, they were very much shorter and more telerrranhic than facultv written lab handouts. This also was shown in our previous"study in which the median length was less than 150 words (3).Also, very few theoretical ldeas were present. Those that did occur-were
-
either a t the heainnina or the end (more common), not intermingled with the procedure itself. The step-by-step quality tended to occur even during the data-reduction instructions. This appears to he string anecdotal support for Johnstone's hypothesis. Students can clearly think about operations a t one time and theory a t another, hut not both a t the same time. Further, they know this unconsciously and, given the choice. will oreanize the exneriment in such a wav as to u cope with the constraints of short-term memory. I t is not surprising that long theoretical discussions in traditional lab handouts are not read or absorbed. I t looks as if, when writina lab materials, the best strategy is to embed the details of the theory in the report-writing section, because when the manipulations are finished the demand upon short-term memory is reduced. This conclusion is also in agreement with Johnstone's postulation of the need to reduce "noise" in lab text material ( 4 ) . The major conclusion from this study is the surprising fact that the math SAT score has nredictive vower for e x ~ e r i mental design p r d ~ l e m a; s well as more conventional testing iituatims. 'l'his illumina~esanother awert of its ~ r e d i r t i v e value. Literature CRed (1) Wolfonden.J.J. J. Chgm. Edue. 1959,36,490.
Appendix. The Test Questlonr Test I I. Using an assigned vacuum line, measure: a. The volume of the vacuum line, and molecular weight bulb. h. The vapor pressure of an unknown liquid at 0 'C. 11. Given a mixture of two dyes, and given absorption spectra of solutions of eaehdye at known concentrationmeasured at pH 7, deterrninv ~~. ......... ~
'The w~vrlmgrhuf nurimurn absorbance ior thr mixture h. The mole percent of rrsorufin ( o w o f the d\e., in the mixture. 111. Given the function of a thermocouple, and some basic electronic a.
equipment, measure: a. The temperature of boiling water. b. The temperature of a slush bath (known to be colder than -40 ' 0 .
Apparatus normally available in the lab could be used for the measurement. Test 11 I. Using an assigned vacuum line, measure: a. The volume of the vacuum line and molecular weight bulb. h. The amarent densitv of some molecular sieves. .. 11. Given a sample of two dyes and their absorption spectra determine: a. The molar absorptivity of the first dye at the A., b. The cram fraction of dve 1 in the mixture. 111. Given a LED and photodiode, measure: a. The LED voltage drop with 10-mA current flowing. b. The LED light output in wattslsteradian at 10-mAcurrent.
-
Volume 64 Number 1 January 1987
69
