In the Classroom
Effective Use of Demonstration Assessments in the Classroom Relative to Laboratory Topics W David T. Pierce* Department of Chemistry, University of North Dakota, Grand Forks, ND 58202-9024; *
[email protected] Thomas W. Pierce Department of Psychology, Radford University, Radford, VA 24142-6946
Demonstrations help instructors provide motivation and inspiration in lecture classes, especially at introductory levels where students have had little or no contextual exposure to chemical phenomena. Students like to watch demonstrations and, in general, they pay remarkable attention to them; herein lies their pedagogical value. However, demonstrations must do more than entertain if their ultimate goal is to help students learn the principles or concepts that underlie the chemistry being presented (1). To identify characteristics that improve learning, chemical educators have turned to cognitive learning theories that have evolved out of the developmental research of Piaget (2). From this body of work, a number of specific recommendations have been put forward for effective preparation, delivery, and discussion of classroom demonstrations (3, 4). A common theme among these recommendations is that students not simply observe a demonstration; they must be challenged to create, invent, or discover for themselves a rational explanation for the chemistry they are witnessing. For students to construct such knowledge on their own, the teacher must pitch the demonstration at a concrete level that makes the intended concept accessible. The instructor must also structure the demonstration in a manner that helps students to resolve the inevitable cognitive conflicts and uncertainties that arise when they try to explain their observations. Demonstration Assessment Within this pedagogical framework, demonstrationbased learning can be engineered in a variety of ways. One of the most informative strategies for both the teacher and the student is the demonstration assessment (5–9) or what Bowen has termed demonstration-based cooperative testing (10, 11). When practiced as a classroom assessment technique (CAT) (12), demonstration assessment provides immediate feedback about learning to both the teacher and the student. The graded nature of the exercise encourages students to go beyond simple observation and to struggle with the deeper processing required for explanation. The general teaching plan for classroom-based demonstration assessment has students view a short demonstration, work together to record their findings, and then discuss what they have learned once their responses are handed in. Miller described this sequence as demonstration–exploration–discussion and first suggested that it could be integrated with formal assessment (13). Later, differential learning studies by Bowen and Phelps, as well as Deese et al., showed that demonstration-based assessment significantly improves student performance on standardized test questions emphasizing conceptual understanding (8, 11).
1150
Journal of Chemical Education
•
Although demonstration assessment helps students to learn, it does not create the optimal learning cycle that cognitive studies have suggested is the best way for students to grasp and retain concepts in science (14–16). The optimal learning cycle begins with exploration, data acquisition, or simple observation. The next step involves activities that promote concept invention. Here, students should be allowed to ask themselves and each other questions such as: What did I see? Did I see a pattern? What does it mean? Demonstration assessment aggressively stimulates this type of inductive reasoning by having students at all cognitive levels work cooperatively to record their findings—both observations and explanations—and then to review them in discussion with the instructor. For the final step of the learning cycle, students should be involved in activities that help them apply their new knowledge, predict related phenomena, or otherwise expand their general understanding of the newly invented concept. It is this deductive reasoning step that is missing from the demonstration assessment model. A Case for Connecting Demonstration Assessments in the Classroom with Experiments in the Laboratory? The laboratory is one venue where students can beneficially apply knowledge they have recently acquired in the classroom. By scheduling a lecture-based demonstration assessment to immediately precede a conceptually similar lab experiment, an optimal learning cycle could be developed and student learning might be further improved. On the other hand, if students have an adequate opportunity to understand a concept through conventional lecture and laboratory experiences, providing an additional demonstration assessment might have little or no beneficial effect on learning. To evaluate these divergent learning outcomes, we conducted a study to determine whether coupling a lecture-based demonstration assessment with a conceptually related laboratory experiment results in a significant improvement in student learning. The answer to this question, regardless of its outcome, should help teachers to evaluate whether or not to use this beneficial yet demanding form of classroom assessment. Demonstration assessment requires a heavy time commitment, both in terms of preparation and execution. More than other CATs, such as the one-minute paper or muddiest point methods (12), demonstration assessment requires detailed preparation and execution. Even with efficient classroom management and a well-practiced presentation (or possibly a prerecorded demonstration) (11), a demonstration assessment can occupy as much as 20–30 minutes of class time. For this heavy commitment of class time to be worthwhile,
Vol. 84 No. 7 July 2007
•
www.JCE.DivCHED.org
In the Classroom
Table 1. Exam Questions To Evaluate Conceptual Understanding, Pre- and Post-Treatment Question Related to a Lab Exercise?
Exam 2 Topics: Pre-Treatment
Exam 3 Topics: Post-Treatment
No
Silver
Light
Yes
Calcium
Heat
it is especially important that the exercise be focused on a topic that significantly improves learning for the widest spectrum of students. The present study should help teachers to decide whether to align demonstration assessments with accompanying laboratory experiments or to target other conceptually challenging topics.
List 1. Demonstrations and Learning Goals for the Study Treatments A reaction sequence with calcium (17, p 111 ff) Learning goal: Role of water in the formation and neutralization of a metal oxide Formation of a silver tree Learning goal: Role of electrons in an oxidation–reduction reaction Dilution and neutralization of sulfuric acid (17, p 177 ff) Learning goal: Effects of heat in chemical processes Intensity of a line-emission light source Learning goal: Distinction between light intensity and photon energy
Methods
Design The study was conducted with first-year college students enrolled in a large introductory chemistry course. The course was taught with a conventional format of separate lecture and laboratory periods. To gauge the effectiveness of lecture-based demonstration assessments, a treatment section was selected to receive demonstration assessments while a control section was taught without CATs of any kind. Both sections were taught from the same textbook at the same pace and by experienced instructors of the same gender (male) and with similar teaching styles. One of us (DTP) directly participated in the study as the instructor of the treatment section. Firstweek enrollments in the treatment and control sections were nearly the same (201 students versus 216 students, respectively). Students in both sections were given the same exams and performed the same laboratory exercises at the same times throughout the semester. The only exception to consistent instruction, assessment, and grading between the sections was that the four demonstration assessments given in the treatment section replaced four online quizzes that the control section students took. Participants Students who participated in the study were volunteers from both the treatment and control sections. The course had a co-requisite of college algebra and the majority of those enrolled were first-year students taking the class as part of a science or engineering major. The final numbers of participants (i.e., volunteers who completed the course) in the treatment and control sections were 140 and 138, respectively.
Data Collection Data were gathered from both sections through attitude surveys and the results from multiple-choice midterm exams. Additional data were gathered from the treatment section in the form of written responses to the four demonstration assessments. The surveys asked students to anonymously respond to statements on a Likert scale of 1 (strongly agree) to 5 (strongly disagree). These statements polled attitudes toward science
www.JCE.DivCHED.org
•
and chemistry, preferred modality for learning chemistry (i.e., observation versus symbolism), and opinion whether learning was transferred between lecture and laboratory. A pretreatment survey was timed so that students had completed the first midterm exam and at least three laboratory exercises. An identical post-treatment survey was given at the end of the term to assess changes in attitudes. Comparisons of the results by t-test at a 95% confidence level were used to determine whether attitudes differed significantly between the two sections or over time. Students in the treatment and control sections completed the same four midterm exams (100 points each) and a final exam (150 points) throughout the term. Exam questions had four multiple-choice responses. Except for conceptually focused questions used to assess the effects of demonstration assessments, these questions usually required algorithmic problem solving or memorization. The averaged scores from these exams were used to assess the overall knowledge of the study participants. To evaluate any differences, average scores were compared by t-test at a 95% confidence level. Differences in conceptual understanding that may have resulted from demonstration assessment were evaluated by the number of correct responses to specific multiple-choice questions placed on the second and third midterm exams (Table 1). These questions were either related to an experiment the students had performed in the laboratory or they were not. To provide a more reliable sampling, the responses for two questions with the same relatedness to the laboratory were pooled. This pooling was accomplished by counting the number of participants who responded correctly to one or both of the pooled questions. Because of timing, it was only possible to schedule two demonstration assessments per exam period. Within each exam period, one demonstration assessment was related to a laboratory exercise and the second was not. A 2 ⫻ 2 χ2 test at a 95% confidence level was used to compare the treatment and control sections based on the number of pooled responses.
Demonstration Assessments The four demonstration assessments (List 1) were not used until after Exam 1 in order to gauge whether the treat-
Vol. 84 No. 7 July 2007
•
Journal of Chemical Education
1151
In the Classroom
ment and control sections were initially similar in areas of knowledge and attitudes. Four demonstration assessments were performed during lectures with the treatment section. Before giving each demonstration assessment, the general topic (e.g., redox reactions) was reviewed in order to give each student enough vocabulary and symbolism to make informed observations. Demonstration assessments that were coordinated with the laboratory (i.e., reactions with Ca and sulfuric acid) were timed to occur before the related laboratory exercise. Each demonstration assessment was crafted as a 20–30 minute classroom exercise and used both pre- and post-assessment rubrics (available in the Supplemental MaterialW). An example is described below. Demonstration: Formation of a Silver Tree The demonstration involved the formation of metallic silver and blue copper(II) ions from the reaction of copper metal in a 1 M solution of silver(I) nitrate. This reaction was selected because of its visual clues to the chemistry taking place. The goal of the exercise was for the students to use their observations to explain the role of electrons in the reaction, even though the electrons could not be directly observed. The reaction was performed under a document camera for convenient magnification and class-wide projection. Before performing the reaction, students were shown samples of copper and silver metals, as well as 1 M solutions of both copper(II) and silver(I) nitrate in a manner that highlighted their color and luster. Students were then instructed to make observations and explain how electrons were involved in the reaction using ap-
propriate symbolism and vocabulary. They were also shown the pre-assessment grading rubric that listed grading expectations (Table 2, middle column). Because this reaction takes more than 20 minutes to develop enough blue color for students to identify copper(II) ions, it was prerecorded using time-lapse and displayed as a five-minute video clip. The clip was shown repeatedly as students talked among themselves to complete the grading forms, the forms were collected, and finally a ten-minute discussion culminating in the post-assessment rubric (Table 2, rightmost column) was held. As with any classroom assessment technique, the submitted responses were graded and returned to the students at the beginning of the next class period, at which time the general results were briefly discussed. By structuring the assessment in this way, common misconceptions not addressed in the original discussion were identified and clarified. Results and Discussion
Initial Parity of the Student Sections Although both student sections were similar with respect to the instruction they received, student numbers, and grading, it was important that we determine whether the initial knowledge and learning attitudes between the treatment and control participants were similar. Students’ initial chemical knowledge was evaluated by comparing participant scores for the first midterm exam (Table 3) while attitudes were evaluated with a Likert-scale survey (Table 4). No significant differences were apparent between the first exam averages or even between quartile and median scores of the two sections. The students in both sections also showed similar attitudes with
Table 2. Rubric for the Silver Tree Demonstration Assessment Score
Pre-Assessment Rubric Grading Expectations
Post-Assessment Rubric Wrap-up Discussions
0
No observation or inaccurate observations
1
Accurate observations, but no explanations
The silver(I) nitrate solution was colorless, while copper(II) nitrate solution is blue. Copper metal was shiny and yellowish-brown in color. Silver metal was also shiny but with a white or colorless appearance. When placed in the silver(I) nitrate solution, the copper metal started to accumulate a lighter-colored solid on its surface. Over time, more of this solid formed in shiny white spikes that covered the entire surface of the copper and grew outward. When scraped off, the brown copper could still be seen below, but its surface was rapidly covered again with the same shiny solid. After a long time, the solution surrounding the copper turned blue.
2–3
Accurate observations; some accurate explanations
The reaction of copper metal occurred with silver ions in the solution. The silver(I) ions seemed to be converting to silver metal on the surface of the copper. The shiny spikes were actually silver metal growing out from the surface of the copper. At the same time, the copper metal seemed to be dissolving as copper(II) ions, giving the solution its blue color. [Note: This description does not mention the role of electrons.]
4–5
Accurate observations and accurate explanations; appropriate use of symbolism and vocabulary
The products of the reaction were silver metal and copper(II) ions: Cu(s) ⫹ 2 Ag⫹(aq) → 2 Ag(s) ⫹ Cu 2⫹(aq) Electrons were involved in this reaction because they were transferred from the copper metal to silver(I) ions at the surface of the copper. Half-reactions that explain the loss and gain of electrons are: Cu(s) → Cu 2⫹(aq) ⫹ 2e᎑ and Ag⫹(aq) ⫹ e᎑ → Ag(s)
1152
Journal of Chemical Education
•
Vol. 84 No. 7 July 2007
•
www.JCE.DivCHED.org
In the Classroom
respect to the subject matter and learning. The only significant difference in initial attitude was a slightly lower interest in science by the treatment section (df ⫽ 220, p ⫽ 0.040). Taken together, these results indicated that the study populations were initially quite similar and well suited for differential evaluation of learning outcomes.
Demonstration Assessment Results Figure 1 illustrates the distribution of scores earned by the treatment section participants for the four demonstration assessments performed in the study. The most striking characteristic was a marked improvement in the quality of explanations after the first exercise. This change reflected the timely feedback provided by the classroom assessment format, as well as a better understanding of the exercise or additional motivation to meet the grading standards. Because the preassessment rubric was explicit about grading expectations, nearly all of the respondents made accurate observations and provided at least some explanations (i.e., scored greater than 1 point). This result supported our assertion that demonstration assessment and the cooperative nature of the exercise aggressively stimulates inductive reasoning by students at all ability levels. However, nearly a quarter of the respondents in each exercise failed to use appropriate symbolism or vocabulary to form their explanations (i.e., scored 2–3 points). In
Figure 1. Distribution of demonstration assessment scores of the treatment-section participants by demonstration topic.
about half of these cases the failure seemed to reflect an unfamiliarity with new terminology or symbolism. In the other half, the failure seemed to reflect a consistent inability to relate concrete observations to more abstract (e.g., symbolic) constructs, such as chemical equations.
Table 3. Comparison of Average Exam Scores, by Study Participation Exam
Total Points
Treatment Section Mean Values (SD), N ⫽ 201
Control Section Mean Values (SD), N ⫽ 216
t-Test Values
p Values a
1
100
081.4 (10.5)
083.0 (9.9)
᎑1.28
0.200
2
100
071.0 (17.7)
076.2 (17.1)
᎑2.51
0.013
3
100
067.3 (16.3)
068.9 (15.6)
᎑0.83
0.410
4
100
066.7 (16.8)
071.0 (16.3)
᎑2.11
0.036
Final
150
108.3 (20.7)
113.3 (22.0)
᎑1.89
0.060
aA
value in which p < 0.05 indicates a significant difference between the treatment and control sections.
Table 4. Distribution of Students’ Averaged Ratings on Attitude Sur vey Statements, Pre- and Post-Treatment Pre-Treatment Averaged Ratings a
Statements for Response
Post-Treatment Averaged Ratings a
Treatment (N ⫽ 201)
Control (N ⫽ 216)
Treatment (N ⫽ 140)
Control (N ⫽ 138)
NI am interested in the sciences.
2.3
2.0
2.4
2.2
NI am interested in chemistry.
2.4
2.6
2.7
2.6
NI learn chemistry better when I can observe NNNthe reactions and reagents involved.
2.4
2.4
2.4
2.3
NI learn chemistry better when I can see NNNthe equations and formulas involved.
2.3
2.2
2.5
2.2
NWhat I learn in class helps me on exams.
2.2
2.3
2.2
2.6
NWhat I learn in lab helps me on exams.
2.6
2.5
2.7
2.7
a
Ratings were based on a Likert-type scale of 1–5 in which 1 indicates “strongly agree” and 5 represents “strongly disagree”.
www.JCE.DivCHED.org
•
Vol. 84 No. 7 July 2007
•
Journal of Chemical Education
1153
In the Classroom
Table 5. Comparison of Pooled Responses to the Percentage of Correct Answers on Common Exam Questions Pooled Questions
Percentage of Students with Correct Responses
Results of χ2 Test (df ⫽ 1)
p Values a
Treatment Group (N ⫽ 140)
Control Group (N ⫽ 138)
NQuestion not related to a lab exercise
81.4
69.6
5.283
0.021
NQuestion related to a lab exercise
91.4
95.7
2.047
0.152
a
A value in which p < 0.05 indicates a significant difference between the treatment and control sections.
Effects of Demonstration Assessment on Learning and Attitudes Table 5 compares the pooled responses from exam questions that were related to demonstration assessments. The results showed that both sections had nearly the same number of correct responses to the lab-related questions. In fact, both sections showed higher numbers of correct responses to the lab-related questions and therefore showed a better grasp of these concepts. Similar performance between the treatment and control sections was apparent for participants who placed in the upper ( p ⫽ 0.314) and lower quartiles ( p ⫽ 0.256) of their respective sections. Clearly, coupling a demonstration assessment with a conceptually similar laboratory exercise had no discernible effect on learning. However, when a demonstration assessment topic was not reinforced in the laboratory the students in the treatment section performed significantly better on questions tapping the same concepts ( p ⫽ 0.021). The effect was small, which may be due to the fact that the intervention was rather limited (only four exercises), yet it was clearly significant. This better performance was also apparent for participants scoring in the lower quartiles (p ⫽ 0.039) as well as in the upper quartiles (p ⫽ 0.024) of their respective sections and was not the result of uniformly better performance of the treatment section on the second and third exams (Table 4). Table 5 shows that attitudes toward learning did not change significantly for either section during the study. Again, this could have been due to the limited number of interventions that could be applied. However, when post-intervention survey responses were compared between sections, the treatment section did show a more positive attitude that classroom learning helped on exams (t ⫽ ᎑2.13, df ⫽ 176, p ⫽ 0.034). This difference seems consistent with the use of demonstration assessments for teaching concepts in the classroom and the placement of related, concept-oriented questions on course exams.
Perspectives Because of the significant investment in class time required to implement demonstration assessment, the choice of topic for this technique is crucial to maximizing its effectiveness. Our results suggest that demonstration assessment is most effective when it targets concepts or principles that are not part of an accompanying laboratory curriculum or cannot be explored in the laboratory because of issues of class size, safety, or expense. Our results also suggest that conceptual understanding can be improved when students have an
1154
Journal of Chemical Education
•
opportunity to observe and rationalize conceptually oriented chemical phenomena. This improvement was certainly observed in treatment students participating in demonstration assessment. However, it also evident in control students when they were able to expand and apply concepts in the laboratory. These findings suggest that demonstration assessments and laboratory exercises may provide similar pedagogical benefits to students; it is important for instructors to select the most effective venue for students to learn a particular concept or principle. Because students can only participate in a limited number of laboratory exercises during a given semester, a great deal of leeway is available for instructors to design effective demonstration assessments. One direction that needs further exploration is the appropriate use of prerecorded videos and computer animations (especially of hazardous or molecular– nanoscale chemical processes) in place of live demonstrations. This choice would remove some of the presentation burden from the instructor and it would give students an opportunity to see the phenomena additional times. It would also help in large classrooms where some students may find it difficult to see a chemical process without magnification and largescreen projection. However, important elements such as spontaneity, enthusiasm, and showmanship can be easily lost with prerecorded media demonstrations. These losses could potentially offset the gains stated above and lessen the effectiveness of the activity. Our study appears to show that concept reinforcement in the laboratory helped all students, regardless of their participation in demonstration assessment. This outcome masked any benefit that resulted from coupling laboratory exercises with lecture-based demonstration assessments and made it difficult to judge the importance of completing an optimal learning cycle. However, a different experimental design might offer a greater level of sensitivity and discrimination. The main strength of the demonstration assessment method is that students are given an opportunity to explain chemical phenomena within a conceptual framework. If completing an optimal learning cycle with demonstration assessment does help with conceptual understanding, a larger effect might be apparent if students provided written explanations for concept-based questions, rather than multiple-choice responses. Although this design would be difficult to apply with the large classes available for our study, smaller classes might provide better venues in which to test the full effects of the demonstration assessment method. Two limitations of the study should be mentioned. First, our results are based on data collected during a single semester.
Vol. 84 No. 7 July 2007
•
www.JCE.DivCHED.org
In the Classroom
Second, neither the instructor for the treatment condition (and one of the authors, DTP) nor the volunteer instructor for the control condition were blind to the expected results of the study. Unfortunately, it is very difficult for instructors to evaluate instructional techniques using the type of doubleblind design commonly employed in behavioral research. However, in the present study, steps to reduce the likelihood of experimenter bias included: • Use of identical exam questions in the treatment and control sections to equate the sections on the degree of difficulty of test items • Computerized scoring of exam questions
To the authors’ knowledge, students in the treatment section did not receive extra help on demonstration assessment topics that might have helped them to perform better on test questions covering these topics. Conclusions The learning improvements derived from demonstration assessment, as performed in this study, were most apparent when the assessment was not related to an accompanying laboratory exercise. Because these interventions generally require a higher level of preparation and a greater commitment of classroom time than ordinary classroom assessments, this guideline should help instructors to make better choices of topics. The study also showed that when demonstration assessments do improve learning, the enhancements are evident for both strong and weak students. Helping students to learn at both of these extremes is a tremendous challenge in the large lecture courses often encountered at the introductory level. However, demonstration assessment appears to be an effective learning tool for both types of students. Acknowledgments We would like to thank Harmon Abrahamson and Libby Rankin for their advice and helpful assistance in this study and the Archibald Busch Foundation for providing funding through the Office of Instructional Development at the University of North Dakota.
www.JCE.DivCHED.org
•
W
Supplemental Material
Discussion rubrics and example exam questions are available in this issue of JCE Online. Literature Cited 1. Meyer, L. S.; Schmidt, S.; Nozawa, F.; Panee, D.; Kisler, M. J. Chem. Educ. 2003, 80, 431–435. 2. Nurrenbern, S. C. J. Chem. Educ. 2001, 78, 1107–1110. 3. O’Brien, T. J. Chem. Educ. 1991, 68, 933–936. 4. Roadruck, M. D. J. Chem. Educ. 1993, 70, 1025–1028. 5. Radford, D.; Ramsey, L.; Deese, W. Sci. Teach. 1995, 62, 52– 55. 6. Markham, L. M.; Knoespel, S. L. Do Students Understand Our Demonstrations? Using Demonstration Assessment in General Chemistry. In Abstracts of Papers, 213th ACS National Meeting, San Francisco, CA, April 1997; abstract CHED-814. 7. Deese, W. C. Demonstration Assessments in General Chemistry. In Abstracts of Papers, 214th ACS National Meeting, Las Vegas, NV, September 1997; abstract CHED-007. 8. Deese, W. C.; Ramsey, L. L.; Walczyk, J.; Eddy, D. J. Chem. Educ. 2000, 77, 1511–1516. 9. McKee, E.; Williamson, V.; Peck, M. L. Use of Demonstrations in Teaching Chemistry. In Abstracts of Papers, 225th ACS National Meeting, New Orleans, LA, March 2003; abstract CHED-128. 10. Simmons, J.; Bowen, C. W.; Elakovich, S. D. DemonstrationBased Assessment in General and Organic Chemistry. In Abstracts of Papers, 211th ACS National Meeting, New Orleans, LA, March 1996; abstract CHED-716. 11. Bowen, C. W.; Phelps, A. J. J. Chem. Educ. 1997, 74, 715– 719. 12. Angelo, T.; Cross, K. Classroom Assessment Techniques: A Handbook for College Teachers; Jossey-Bass: San Francisco, CA, 1993. 13. Miller, T. J. Chem. Educ. 1993, 70, 187–189. 14. Karplus, R. J. Res. Sci Teach. 1977, 14, 169–175. 15. Abraham, M. R.; Renner J. W. J. Res. Sci. Teach. 1986, 23, 121–130. 16. Spencer, J. N. J. Chem. Educ. 1999, 76, 566–569. 17. Postma, J. M.; Robert, J. L., Jr.; Hollenberg, J. L. Chemistry in the Laboratory; W. H. Freeman: New York, 2000.
Vol. 84 No. 7 July 2007
•
Journal of Chemical Education
1155
