Research: Science and Education edited by
Chemical Education Research
Diane M. Bunce The Catholic University of America Washington, D.C. 20064
Undergraduate Science and Engineering Students’ Understanding of the Reliability of Chemical Data
Bette Davidowitz* Department of Chemistry, University of Cape Town, Rondebosch 7701, South Africa;
[email protected] Fred Lubben Department of Educational Studies, University of York, York, UK Marissa Rollnick College of Science, University of the Witwatersrand, South Africa
This study investigates the status of procedural knowledge of sophomore science and chemical engineering students at the University of Cape Town. Procedural knowledge includes collection, manipulation, and interpretation of experimental data. Standard textbooks in both analytical and physical chemistry discuss several aspects of procedural understanding (1, 2). They elaborate on the concepts of variance, the different types of errors in measurements, Gaussian distributions and how to approximate other distributions, the appropriate procedures to take means and calculate standard deviations, how to use confidence testing, and least square fitting procedures. Several papers in this Journal have contributed insightful examples of teaching the statistical analysis of errors (3, 4) and how to help learners to go back to first principles when considering the propagation of statistical errors (5). Very useful suggestions have been made on the choice of experiments that allow students to apply methods of statistical analysis to experimental data they collect themselves (6, 7 ). Table 1. Progression of Ideas about the Reliability of Experimental Data Level
Student’s View of the Process of Measuring
A
Measuring once and this is the right value.
B
Unless you get a value different from what you expect, a measurement is correct.
C
Make a few trial measurements for practice, then take the measurement you want.
D
Replicate a measurement till you get a recurring value. This is the correct measurement.
E
You need to take a mean of different measurements. Slightly vary the conditions to avoid getting the same results.
F
Take a mean of several measurements to take care of variations due to imprecise measuring. Quality of the result will be judged only by an authority source.
G
Take a mean of several measurements. The spread of all measurements will indicate the quality of the results.
H
The consistency of a set of measurements can be judged and anomalous measurements need to be rejected before taking a mean.
I
The consistency of data sets can be judged by comparing the relative positions of their means in conjunction with their spreads.
Hudgins and Reilly postulate that for chemical engineers measurement forms a bridge between the real world and the mathematical world (8). Undergraduate students spend most of their scientific lives on the abstract mathematical side of this bridge, manipulating sanitized models and practicing with often-contrived experimental tasks. For these students, two options exist when experimental results do not agree with theory: there is something wrong with the apparatus, or the theory has little value in the real world. We took one step back and explored students’ untutored ideas about the sources of and ways of dealing with variation in experimental data, the treatment of spurious data, and the comparison of sets of data. The status of this understanding among sophomore students is of interest because it impinges on their choice of methods of data collection, presentation, and interpretation. These processes are an essential part of both laboratory exercises and the writing of reports. In the constructivist paradigm it is paramount to use an inventory of students’ spontaneous ideas, scientifically correct or misconstrued, as the basis of course development (9). Although a large body of research has identified students’ misconceptions in the area of their declarative knowledge (10), little has been published on students’ ideas about the methods of scientific enquiry. This paper examines one aspect of such ideas: the perceptions of the reliability of experimental measurements. We employed a model of progression of students’ ideas about experimental data that was developed by Lubben and Millar (11) using extensive data from high school students. The model was extended by Allie et al., who investigated the status of procedural understanding in a group of freshman students in physics (12). The extended progression scheme is shown in Table 1. The freshman chemistry course for science and engineering students at the University of Cape Town (UCT) provides little formal teaching about the reliability of experimental data, assuming that most of the understanding will be picked up en route. For example, no explicit instruction is provided about standard deviations. We sought to answer the following questions: 1. What procedural understanding do students entering sophomore chemistry possess? 2. What differences exist in procedural understanding in chemistry between students registered for science and students registered for chemical engineering degrees?
JChemEd.chem.wisc.edu • Vol. 78 No. 2 February 2001 • Journal of Chemical Education
247
Research: Science and Education
Methods A questionnaire probing students’ understanding of handling of experimental measurements was adapted from the physics questionnaire used by Allie et al. (12). The questionnaire consisted of six paper-and-pencil tasks based on a hypothetical experimental situation (see Fig. 1) and the adaptation used a chemistry rather than a physics context for the posited experiment. The tasks were followed by a scripted discussion by a group of fictitious students. Each respondent was asked to align himself or herself with one of the characters depicted in the tasks and to justify this choice. Each task was printed on a single sheet of paper, and after completing the task the student placed the paper in the envelope provided before proceeding to the next task. This was done to discourage students from changing their responses to earlier tasks. The questionnaire was pilot-tested on a group of matriculation pupils in a nearby school and modified in the light of the pilot study. It was then administered to sophomore students at the beginning of the university year, before any instruction had taken place. Using a sample of questionnaires, we then held a workshop to adapt the physics coding scheme (12) to the chemistry context and the more advanced level of respondents. The responses were then coded and analyzed by identifying clusters of responses based on similar procedural ideas. Sample Population The 135 sophomore students who completed the questionnaires comprised 79 chemical engineering and 56 science students. There were 85 were men and 50 women; 47 spoke English as a first language and 88 were second-language English speakers. Science and chemical engineering students are combined in one chemistry course. All science and chemical engineering students complete courses in chemistry, mathematics, and physics during their freshman year. The freshman practical course in chemistry has a strong focus on titrations, with emphasis on taking the mean
Some students are doing an experiment in the chemistry laboratory. They have a flask with a 1 M solution of lead nitrate and another flask with a 1 M solution of potassium iodide. The students pour 5 mL of lead nitrate into a test tube. When potassium iodide Clear liquid solution is added to the lead nitrate solution, they see a yellow precipitate form. They put a cork Precipitate on the test tube and leave it to stand for 4 hours. The yellow precipitate has settled to the bottom of the test tube and a clear liquid remains on top.
of several readings. Chemical engineering students register for a course in mechanical drawing as well as an entry-level course in chemical engineering. This course comprises study and communication skills, computer literacy, problem solving, and the use of modeling programs for chemical engineers. Chemistry majors can, in addition, register for a course in geology, computer science, biology, statistics, archaeology, or the humanities. The criteria for admission to UCT are based on the results of the matriculation examination and differ for the two groups. The chemical engineering course, being more in demand, is able to set its entrance qualification at a higher level. Findings The tasks tested three aspects of student understanding about handling experimental data: doing replicates, handling data, and judging the quality of data with respect to spread.
Description RA response → UR response
Science Total = 56 n
%
Total Total = 135
n
%
No replicates needed → Calculating the mean
1
1
1
2
2
1
C-D
Perfecting measurement skill → Replicating for a recurring value
1
1
3
5
4
3
n
%
C-F
Perfecting measurement skill → Calculating a mean value
15
19
9
16
24
18
C-G
Perfecting measurement skill → Determining spread
6
8
6
11
12
9
D-D
Replicates needed to identify the recurring value: consistent
3
4
1
2
4
3
D-F
Recurring value → Calculating the mean
7
9
3
5
10
7
D-G
Recurring value → Determining spread
3
4
2
4
5
4
F-D
Replicates needed to calculate the mean → Replicating for recurring value
0
0
2
4
2
1
F-F
Replicates needed to calculate the mean: consistent
24
30
16
29
40
30
F-G
Calculating the mean and determining the spread
6
8
4
7
10
7
G-G
Determining the spread: consistent
0
0
1
2
1
1
G-F
Determining the spread → Calculating the mean
5
6
3
5
8
6
Unclassifiable
8
10
5
9
13
10
aCodes
248
Engineering Total = 79
A-F
U
h
Figure 1. The hypothetical experimental situation. The students have been asked to investigate what happens to the height, h, when the volume, v, of potassium iodide is varied. They use a ruler to measure h in millimeters.
Table 2. Doing Replicates: Responses to RA and UR Tasks Codea
Test tube
A, C, D, F, and G refer to Table 1.
Journal of Chemical Education • Vol. 78 No. 2 February 2001 • JChemEd.chem.wisc.edu
Research: Science and Education
A group of students measure h for different volumes of potassium iodide solution and plot them on a graph as shown below. On this graph, draw the straight line that you think best fits these data. Explain what you have done and why.
Some students in this cluster note the variation in the measurements. A few explicitly include all readings in the calculated mean, whereas others explicitly exclude selected readings, as shown by the following two quotes:
h / mm
9 and 14 could be discarded, but they average out to between 11 and 13 anyway. I just averaged all the results, since the double appearance of 13 may just be fluke, and not an indication of the true result. (Eng. 32) The measurement number 3 is far out of the range of the average result of h. Therefore a set of only 4 measurements can only be taken into account. So add these 4 measurements and then divided by 4 to get an average. (Sci. 43)
0
V / mL Figure 2. Task: Straight-line graph (SLG).
Doing Replicates The following two tasks (RA and UR) deal with doing replicate measurements. The contents are shown below. Task: Doing Replicates Again (RA) The students work in groups on their experiment. Their first task is to find h when v = 5 mL. One member of the group adds 5 mL of potassium iodide solution to the 5 mL of lead nitrate solution and measures the height of the precipitate using a ruler. He finds h to be 11 mm. The group decide to do the experiment again with the same volume to find h when v = 5 mL. This time they find that the height of the precipitate is 13 mm. First measurement: v = 5 mL, h = 11 mm Second measurement: v = 5 mL, h = 13 mm
The following discussion takes place among the students. 1. We know enough, we don’t need to replicate the measurement again. 2. I think we should do the experiment just one more time with the same volume of potassium iodide. 3. Three results will not be enough. We should do the experiment several more times.
Task: Using Replicated Measurements (UR) The students continue to replicate the experiment using 5 mL of potassium iodide solution. Their five measurements of height of the precipitate in mm are 11, 13, 9, 14, 13 mm. The students then discuss what to write down for the their final result for h. Discussion Several categories emerged during the analysis, which was based on the levels in Table 1. These categories were clustered and the results of the clustering are shown in Table 2. The largest group of over 40% of all students (clusters F-F, F-G, G-F, and G-G) consistently replicate measurements in order to obtain a mean. About one-third of them (the last three clusters) even suggest replicating measurements to obtain a magnitude of the spread.
About 30% of the total sample (C-D, C-F, and C-G) take replicate measurements to perfect their skills. It is unclear if this is a first step to collecting several “good” measurements as a basis for taking a mean, or if the skill perfection leads to obtaining a single final measurement seen as “true” value. The UR responses (mainly F and G) may suggest the former. An average value is a fairly good estimate of the true value. (Eng. 61)
Almost one in seven students (D-D, D-F, D-G, and F-D) take replicate measurements to identify the recurring value. The fact that more than half of this cluster take a mean when provided with a set of numbers may indicate that the latter procedure is done as a routine. We noted (not shown in the table) that 12% of the engineering students but only 2% of the science students were hesitant to replicate measurements, arguing that this is a waste of time and resources.
Data Handling The responses to the UR task also provide insight into students’ ideas about handling data. An alternative way of handling data is to represent the data in graphical form. One of the tasks, Straight Line Graph (SLG), probed this issue (Fig. 2). The responses to the task on recurring readings (UR) were compared with those for drawing a straight-line graph (SLG). If the graph shows a line through two selected points (first + last, origin + last, or origin + middle), the response was coded SLG1. If the graph shows an attempt to draw the line through as many points provided in the task as possible, the response was coded SLG2. If the graph shows a line of best fit, for example with the same number of points below and above the line, the response was coded SLG3. The results of these codings are shown in Table 3. More than 60% of the students (clusters F-SLG3 and G-SLG3) consistently take a mean and then draw a best fit straight line. More than a quarter of this group (G-SLG3) offer a higher level response by considering the spread and drawing a best-fit straight line. The best line would be having about the same error on both sides of the line and so ‘cancelling’ out the error to some extent. (Eng. 15)
About a quarter of the students (clusters F-SLG1, F-SLG2, G-SLG1, and G-SLG2) determine the mean (and sometimes the spread) for numerical data, but select specific coordinates to draw a line through.
JChemEd.chem.wisc.edu • Vol. 78 No. 2 February 2001 • Journal of Chemical Education
249
Research: Science and Education Table 3. Handling Data: Responses to UR and SLG Tasks Codea
Engineering Total = 79
Description UR response → SLG response
Science Total = 56
n
%
D-SLG2 Recurring reading → Line through selected points
1
1
1
2
D-SLG3 Recurring reading → Best fit straight line
3
4
5
9
8
6
F-SLG1 Determining a mean → Line through origin and midpoint or origin and last point
7
9
3
5
10
7
F-SLG2 Determining a mean → Line through selected points
8
10
8
14
16
12
G-SLG1 Determining a spread mean → Line through origin and midpoint or origin and last point
3
4
2
4
5
4
G-SLG2 Determining a spread → Line through selected points
3
4
0
0
3
2
F-SLG3 Determining a mean → Best fit straight line
37
47
23
41
60
44
G-SLG3 Determining a spread → Best fit straight line
10
13
11
20
21
17
7
9
3
5
10
7
U
Unclassifiable aCodes
n
%
Total Total = 135 n 2
% 1
D, F, and G refer to Table 1.
Judgment of Quality of Data with Respect to Spread Three tasks (AN, SMDS, and DMSS) probed students’ ability to judge the quality of data in terms of their spread, as shown below. Task: Dealing with an Anomaly (AN) Another group of students have decided to calculate the average of all their measurements of h for v = 5 mL. The students discuss what to write down for the average value of h, having measured the following heights: 15, 11, 24, 10, 11, 13 mm. They discuss the results. 1. All we need to do is add all our measurements and divide by 6. 2. No, we should ignore h = 24 mm and then add the rest and divide by 5.
Task: Same Mean Different Spreads (SMDS) Two groups of students compare their results for h. The results are shown below. Group X: 10, 12, 14, 11, 13; average height 12 mm Group Y: 10, 16, 9, 11, 14; average height 12 mm
Discussion The data were analyzed, revealing the clusters shown in Table 4. One in ten students (V) resort to point-to-point comparison, looking at individual measurements in the two sets or checking if the calculated mean value appears as one of the measurements, even though the tasks forced them to consider all measurements together. A very small number of students (W) take no account of the anomaly and base their judgments only on the mean. Half of the students (W + X) consider the mean to be of overriding importance and almost all are willing to consider the spreads only when the means are the same. A larger cluster of about one in seven students (Y) do not mention the mean but focus on the spread only. At this level is there a difference between the science and chemical engineering students; the latter take account of the spread more frequently. Very few sophisticated responses (Z) describe sets of data by the relative position of their means and spreads. The large number of unclassifiable responses is due to the combination of responses to three tasks. This leads to an increase in the number of inconsistencies of responses.
The groups discuss the results. 1. Group X’s results are better. They are all between 10 mm and 14 mm, group Y’s are spread between 9 mm and 16 mm. 2. Group X’s results are just as good as group Y’s. Their averages are the same. They both got 12 mm for h. 3. I think that the results of group Y are better than the results of group X.
Task: Different Means Same Spread (DMSS) Two other groups of students compare their results for h. These are shown below. Group X: 14, 11, 10, 12, 13 mm; average height 12 mm Group Y: 12, 15, 14, 13, 16 mm; average height 14 mm
They discuss their results. 1. Group X’s results agree with group Y’s. 2. No, group X’s results do not agree with group Y’s.
250
Discussion
Procedural Knowledge: Second-Year Students It is disappointing that one in seven of these sophomore students perceives the purpose of replicating measurements to be the identification of a recurring value. From the formulation of the responses, this recurring value is seen as the “true” value. It is striking that for the same type of student in a physics context (12) the search for a recurring value is virtually absent. About 30% of the students replicate their measurements to perfect their measuring skills. Cross-task comparison indicates that this skill perfection is mostly seen as a routine prior to taking a mean. Both these clusters of students may have adopted these views about variation in measurements from standard titration procedures where students carry out one trial titration—and then several more, until three measurements are within a prescribed range, from which the mean is determined.
Journal of Chemical Education • Vol. 78 No. 2 February 2001 • JChemEd.chem.wisc.edu
Research: Science and Education Table 4. Judgment of Similarity of Data Sets: Responses to AN, SMDS, and DMSS Tasks Engineering Total = 79
Code Description
Science Total Total = 56 Total = 135
n
%
n
n
%
V
Compare individual measurements
9
11
5
% 9
14
10
W
Compare only the mean, take no account of spread, anomaly seen as a mistake
2
3
5
9
7
5
X
Compare means and only if they are the same consider spread 34
43
28
52
62
46
Y
Only spread is important, mean is not mentioned
16
20
4
7
20
15
Z
Consider mean, spread and overlapping spread as important
2
3
3
5
5
4
U
Unclassifiable
16
20
11
20
27
20
On the other hand, it is encouraging that over 40% of the students replicate their measurements in order to take a mean and a third of these intend also to calculate the spread. However, the intention to take a mean in itself does not indicate if this is seen as a measure of central tendency, or as an approximation of the “true” value. Clearly the students do not understand the need for randomly varying results before taking a mean has any significance. It may be that once they are presented with a series of numbers, these students are prompted by previous experience to use an algorithm to take the mean. A very small number even resort to a formula learned in high school to determine this mean. These students eliminate the lowest and highest value as a matter of course and then calculate the average. In drawing the straight-line graph, the majority of students made some attempt to draw the “best fit”. A fair proportion of students (16% of the chemical engineers and 7% of the science students) tried to fit a best straight line that went through the origin. These students may have been doing more than interpreting the results as presented. As the height of the precipitate should theoretically be zero when the volume is zero, they have linked the drawing of the graph not only to the actual results but to the chemical concepts underlying the experiments. The freshman practical course includes at least one exercise where the origin is included as one of the data points. In this experiment the instrument is zeroed before measurements are taken and thus the origin is a valid data point. Students do not see the difference between experimental results and theory. Analysis of the AN task separately revealed that over 80% of the students were able to recognize an anomalous reading. There is also some perception of an acceptable range for the readings. When dealing with spreads of results with the same mean, sophomore students appear to be able to understand the significance of a spread of readings; a sizeable number are able to see the significance of the quality of the mean. When presented with task SMDS, students are able to recognize and explain the importance of spread but are unable to apply this to compare two sets of readings, DMSS. Comparability of measurement sets and criteria to be used are dealt with extensively in the freshman physics course but there appears to be little transfer to chemistry. Most students appear to see the relationship of the spread to the mean when the means are the same, but the response is much more variable when the means are different. The mean still appears to be their main point of reference, and the spread seems to be a secondary consideration.
Presenting a task where the means are different but the spreads are the same causes almost one in ten students to resort to a comparison of the individual readings despite having being forced into holistic reasoning by the tasks.
Differences between Science and Chemical Engineering Students No major differences between the two groups were noted for the tasks on using replicated measurements or handling data. The only notable difference between the two groups was in the response to the tasks dealing with comparability of data sets. Here the chemical engineers seemed to take into account the notion that both the mean and the spread are important. This is an interesting finding in view of the fact that the entrance requirements are more stringent for chemical engineers than for science students and the former are a more highly selected group. One possible explanation may be that we are looking at a different type of ability from that tested in the school-leaving examinations. Conclusions In keeping with the constructivist paradigm, the findings of this study will have an impact on the contents of the practical manual. This will inform the design and sequencing of experiments of a modified sophomore course. There is also a need to consider the mode of presentation of titration exercises in freshman laboratories where there is an established convention of practicing both to perfect a skill and to achieve a narrow spread of readings. Resources permitting, more titrations could be achieved in a sensible laboratory time by getting rid of the buret and doing titrations semiautomatically. Care should be taken to use terms such as “accurate” and “precise” to ensure deep understanding as opposed to rote learning. The sophomore chemistry course also includes a short series of lectures on statistics and propagation of errors. The findings from this study will inform the structure and content of this section of the course. Acknowledgments Financial support was obtained from the National Research Foundation in South Africa and the University Research Committee of University of Cape Town.
JChemEd.chem.wisc.edu • Vol. 78 No. 2 February 2001 • Journal of Chemical Education
251
Research: Science and Education
Literature Cited 1. Skoog, D. A ; West, D. M.; Holler, F. J. Fundamentals of Analytical Chemistry, 7th ed.; Saunders: Fort Worth, 1996. 2. Atkins, P. W. Physical Chemistry, 6th ed.; Oxford University Press: Oxford, 1998. 3. Guedens, W. J.; Yperman, J.; Mullens, J.; Van Poucke, L. C.; Pauwels, E. J. J. Chem. Educ. 1993, 70, 776–779. 4. Guedens, W. J.; Yperman, J.; Mullens, J.; Van Poucke, L. C.; Pauwels, E. J. J. Chem. Educ. 1993, 70, 838–841. 5. Andraos, J. J. Chem. Educ. 1996, 73, 150–154.
252
6. Paselk, R. A. J. Chem. Educ. 1985, 62, 536. 7. Thomasson, K., Lofthus-Merschman, S.; Humbert, M.; Kulevsky, N. J. Chem. Educ. 1998, 75, 231–233. 8. Hudgins, R. R.; Reilly, P. M. Chem. Eng. Educ. 1989, 92–94. 9. Bodner, G. M. J. Chem. Educ. 1986, 63, 873–878. 10. Pfundt, H.; Duit, R. Students’ Alternative Frameworks and Science Education. Kiel: IPN: Kiel, 1994; bibliography. 11. Lubben, F.; Millar, R. Int. J. Sci. Educ. 1996, 18, 955–968. 12. Allie, S.; Buffler, A.; Kaunda, L.; Campbell, R.; Lubben, F. Int. J. Sci. Educ. 1998, 20, 447–459.
Journal of Chemical Education • Vol. 78 No. 2 February 2001 • JChemEd.chem.wisc.edu
