In the Laboratory
Application of Calibrated Peer Review (CPR) Writing Assignments To Enhance Experiments with an Environmental Chemistry Focus
W
Lawrence D. Margerum,* Maren Gulsrud, Ronald Manlapez, Rachelle Rebong, and Austin Love Department of Chemistry, University of San Francisco, San Francisco, CA 94117; *
[email protected] Active learning strategies and alternate assessments are slowly replacing the traditional chemistry lab experience and the traditional classroom lecture format at the college and university level (1). A well-developed body of literature now exists for educators interested in implementing strategies such as in-class concept tests (2), cooperative learning groups or guided inquiry (3–10), peer-led team learning (11–13), writing in chemistry (14–21), and modular chemistry (10, 22, 23). Many faculty members are skeptical of the outcome of these changes or lack the time, tools, or training necessary to implement them. At this university, a national liberal arts university of 4000 undergraduates, we are concerned that many of our beginning students are not well-prepared or do not have the depth of understanding to clearly express technical concepts in oral and written form. We are completing an NSF-sponsored grant that adapts experiments in environmental chemistry using atomic absorption spectroscopy (AAS) and implements a Web-based writing-to-learn program to help address these concerns. First-semester general chemistry students complete prelab and postlab writing assignments around a lead-in-paint analysis using AAS (24). Our objectives are to stimulate student interest in chemistry and to give students more confidence with quantitative chemistry while developing their skills in reading for content and technical writing. We deliver our project via Calibrated Peer Review (CPR) software, developed by the Molecular Science Project.1 This browser-based tool enables instructors to create structured writing assignments in which students learn by writing and reading for content. This technology may lower the barrier that faculty see in producing and grading writing assignments in large classes, since the CPR program takes over most of these functions. The initial results of the project show that our students are successfully completing the quantitative experiments and are mastering the reading and writing assignments.2 In addition, students are actively engaged in prelab and postlab technical writing and reading about an important environmental issue, without an increase in faculty grading time. Project Description We developed three writing assignments using the CPR authoring tools and the writing guidelines of Kovac and Sherwood (25). We report the results from 2003 and 2004 for all general chemistry students who completed the project called: Is there lead in my paint? Each CPR writing assignment lasts one week, with a four-hour lab held between the end of the prelab CPR and the beginning of the postlab CPR. We give students an additional week to work up their own results for a total project time of four weeks.
292
Journal of Chemical Education
•
Understanding Absorption and Emission In the sixth week of the lab course, the instructor presents the project goals along with information on the CPR program to lab sections of 30–40 students (this will be their peer group). The first writing assignment is to submit a 175word essay (with the help of guiding questions) describing the outcome of a simulation on absorption and emission in the hydrogen atom.3 The assignment introduces students to peer review, gives them practice with the software grading system, and replaces related homework from the lecture course. They have four days to draft and electronically submit their essay to the CPR program. The calibration or review stage follows, where the student demonstrates reviewing competency by reading, answering questions, and trying to match scores on three instructor-generated essays of high, medium, and low quality (10 point scale). Once trained, students score three anonymous peer essays and their own essay resulting in seven content reviews per essay. This stage lasts for three days, after which students check all of their scores and receive feedback. The student grade consists of the average peer score of their submitted essay (20%), their calibration performance (30%), their review performance (30%), and on how well they self-assess (20%). For example, a student may write a weak essay, recognize the weakness after the calibration and review stage, and give a low self-assessment score. If this score matches the peer average, the student gets full credit for self-assessment. Prelab Writing Assignment The second week of the project is a prelab in which students use guiding questions to study the source materials.4 Some examples of guiding questions include (i) What are the health concerns for children exposed to lead paint? (ii) How will you use a Mohr pipet to make standards? and (iii) What are the five main components of an AAS? The writing prompt is to answer these questions in the form of a letter (250 words) to their high school chemistry teacher that explains the experiment. The calibration and review stages focus on specific content. Students review an essay to find answers to such questions as “Does the essay say that lead atoms in the flame absorb some of the light from the hollow cathode lamp, causing a drop in the PMT signal?” This prelab assignment requires students to use higher-level skills than the first assignment in that they must synthesize several concepts into a professional letter. Experimental We adapted Markow’s lead-in-paint experiment to firstsemester general chemistry (24). Our alterations included: adding a pipet calibration using mechanical pipet pumps
Vol. 84 No. 2 February 2007
•
www.JCE.DivCHED.org
In the Laboratory
bration answers, and calibration exemplars. We sought electronic source material that provided clear, concise scientific explanations and appropriate use of visual learning aids or animations (27). The real time savings is apparent during the assignments, as there is no faculty time spent grading the 400+ student essays generated each fall.
(VWR International); using heating blocks instead of a water bath for secondary digestion; and adding filtering steps. In addition to the materials listed in the original experiment, one needs a paint standard (National Institutes of Standards and Technology, NIST, SRM 2589, Powdered Paint Sample), 0.45µm syringe filters (VWR International), disposable plastic syringes, pipet pumps, and fixed-volume and Mohr pipets. All students complete a fixed-volume pipet calibration while groups complete microscale acid digestions of the NIST sample and a campus paint sample on a hot plate and prepare five calibration standards by dilution of a 100-ppm lead solution. It was most efficient to start half of the groups on standards and half on digestions. Most groups load the AAS autosampler (Shimadzu 6650), light the flame, and obtain results with instruction from a trained research assistant. Some groups observe an analysis, but have their samples analyzed later.
Student Participation and CPR Evaluation The pilot project showed that students had several concerns. First, they wanted more time to research and write the short essays, so the first stage expanded from two to four days. A second concern was the fairness of peer grading on a tenpoint scale. Based on student interviews and performance, we inserted more guidance as to the value of each content or style question. As a result, each assignment was limited to ten questions, with assigned point values adding to ten. Third, we found that adding the 30-minute training session on the first day helped students understand the project and the value of peer review. A final challenge for students was remembering the multiple deadlines for each online assignment. It helped to hand out deadline sheets and to send email reminders. Our initial concern was that student participation could be low, since this project had multiple deadlines over four weeks. On average, students submitted essays about 85% of the time (the range was 63–99% over 20 lab sections) and most finished the calibrations and reviews (81% in 2004 and 76% in 2003). Participation in each CPR remained constant despite the increasing point value. A breakdown of student submissions via lab section revealed that the lowest participation rates correlated with the experience level and commitment (instructor-perceived) of the graduate student TA. The two most frequent student excuses for not completing CPR assignments was confusion on the deadlines and forgetfulness. We are still seeking ways to increase participation via better TA training and better student reminders. The CPR scoring outcomes for students completing each assignment are listed in Table 1. The assignment scores are a weighted average of each stage and use the software “easy” scale for CPR 1 and the “moderate” grading scale for the preand postlab assignments. The scales differ in the latitude students get on matching scores in the different stages. The relative standard deviation (% RSD) for self-assessment is a measure of how well students recognize the deficiencies in their essay compared to their peers. Two themes emerge from the data. First, the higher scores for the completed assignment compared to the initial essay (see mean essay score from peer-review in Table 1) indicate that most students are mastering the material via calibrations and reviews (80% of the grade). This is not
Postlab Writing Assignment The final writing assignment starts after the experiment and guides students to produce a short formal laboratory report (450 words). The source material links to Web sites on calibration curves and has instructor generated AAS results. The students must calibrate and review a common set of data. After completing this postlab CPR, each group uses their own data to construct a calibration curve, calculate the weight percent lead in their samples, and compare the on-campus sample to the act cited above. They write a conclusion comparing their findings to levels cited in Title X of the 1992 Residential Lead-based Paint Hazard Reduction Act (26). This last CPR writing assignment replaces a traditional written report and is worth 10% of the course lab grade. Hazards The concentrated or dilute (0.1 M) nitric acid solutions should be handled with care, as they are corrosive oxidizers that are highly hazardous to skin and eyes. All acid digestions on hot plates must be carried out in a fume hood, as a potential byproduct is brown, toxic nitrogen dioxide gas. Lead is toxic so proper safety precautions and disposal methods should be adopted. We instruct students to clean up by adding small quantities of water to all glassware and pouring the rinse directly into heavy metal waste containers. Protective gloves and safety glasses are mandatory throughout the experiment. Results and Discussion The CPR authoring tools provide templates for learning goals, links to source materials, calibration questions, cali-
Table 1. Student Performance on CPR Assignments, Peer-Reviewed Essays, and Self-Assessment Writing Assignment
Mean Score (%) for Completed Assignment
N (Essays Submitted)
Mean Essay Score (%) from Peer-Review
% RSD for SelfAssessment
2003
2004
2003
2004
2003
2004
2003
2004
CPR 1
155
139
87
87
64
59
12
13
CPR 2
132
132
78
79
60
56
12
13
CPR 3
137
125
85
87
63
62
11
11
NOTE: The writing assignments are CPR 1, Understanding absorption and emission, CPR 2, Prelab on AAS, and CPR 3, Postlab report on lead in paint.
www.JCE.DivCHED.org
•
Vol. 84 No. 2 February 2007
•
Journal of Chemical Education
293
In the Laboratory
Table 2. Student Results for Lead in Paint Lead Found, % wt (std dev)
Paint
2003 (n = 32)
2004 (n = 27)
NIST standard (9.99%)
14.90 (6.76)
9.37 (2.27)
Campus paint chip
00.24 (0.22)
0.17 (0.15)
surprising, since students must synthesize source material into an essay without the benefit of seeing the content and style questions posed in the calibrations and reviews. The relatively small RSD for self-assessments implies that students agree with their peers’ review score and recognize the shortcomings of their initial work. The second point is that overall scores (range 79–87%) and essay scores (range 56–64%) remain about the same, despite the arguably more difficult assignments (harder grading scale, longer essays, and more quantitative analysis). One may argue this is due to an increase in experience and comfort level with the process, clearer instructions or is due to improved skills in technical writing and reading for content. We note in the data (not shown) that calibration scores on content questions remain the same for all assignments, while style scores improve. This shows up as a 20% improvement in the software-generated Reviewer Competency Index (RCI) from CPR 1 to CPR 3. This may not be surprising either, since each assignment has new content, while students get better at identifying style issues (topic sentences, spelling and simple grammar). Regardless, we see these results as partially meeting our goals for improved student skills and confidence in technical communication.
Analytical Results The student results from the AAS analysis of lead in the NIST standard and the on-campus paint chip are shown in Table 2. Each group is digesting 10–20 mg samples, weighed on a digital pan balance (±0.001 g), followed by a secondary digestion, dilution, and filtering. Many students are using volumetric pipets and flasks for the first time and their lab results reflect this. There are large standard deviations and poor accuracy (49% high) in the 2003 results. Students now complete a short pipet calibration and use fixed volume pipets for paint sample transfers instead of Mohr pipets. The 2004 results show significant improvements in accuracy (relative standard deviation of ᎑8%) and precision over the previous year, giving us more confidence that a first-semester student can do meaningful quantitative analysis on this scale. The campus paint chips give large standard deviations in both years, partially due to absorbance readings at the low end of the calibration curve (< 2 ppm). However, they have higher lead values than allowed by the U.S. Consumer Product Safety Commission in new paint.5 We are testing a new multi-vessel microwave digestion oven (CEM MARS Express) for this experiment that uses larger sample sizes, is safer, and is faster than the existing method.
mending the project to future chemistry students. Secondly, students obtain good analytical results on the standard paint sample (in 2004) and discover that paint from a building on campus has elevated levels of lead. Third, students are becoming more adept at technical reading and writing based on the aforementioned assignment scores and increased RCI. More than 50% of students agree or strongly agree (33% neutral) with this statement: “With each CPR, I am becoming a better technical reviewer.” There are slightly more students who disagree than agree with the statement “the CPR assignments help more than traditional pre- or postlab assignments”, perhaps because they have not written a traditional report or because of the perception that the CPR assignments are more work. Finally, there is a modest gain (10% improvement) comparing student performance on a pre-project versus post-semester quiz on the wavelengths and energies of absorption and emission events in the H-atom (the subject of CPR 1). We recommend using or modifying one of the CPR library assignments before authoring new assignments. In addition, a limit of three to four CPR assignments per semester is reasonable if they replace existing assignments. Students report there is a higher “hassle” factor using CPR, as many are not adept at writing in chemistry or in using online tools for science courses. Unlike the movement away from cookbook-like laboratory experiments, the student reviewers use a prescribed scoring rubric. This is a direct result of the philosophy of the CPR approach and our own assessment-based revision of the materials. Student complaints about peer grading are now minimal. While the project covers only one experiment in general chemistry, it may provide lab instructors with a method to deliver technology-based lab materials and may train students in technical writing for upcoming chemistry courses. We see encouraging results for a CPR adaptation of the multielement analysis of wastewater using AAS in the second-year analytical laboratory (28). Many students understand the benefit of writing-to-learn despite the extra time needed to complete these assignments. They are also excited to see a modern analytical instrument in action after reading and writing about its operation, especially when we tell them how much it cost. Students report that these CPR pre- and postlab assignments help them learn, because “they better your understanding of what you are really doing because you have to explain it in a way to make others understand, which in return shows what you truly know” (Anonymous student, 2003) Acknowledgments We acknowledge funding from the NSF-CCLI-A&I program (DUE-0127174), the University of San Francisco Faculty Development Fund, and The Lily Drake Cancer Research Fund at USF, all of which provided summer undergraduate research stipends for MG, RM, RR and AL.
Conclusion W
The quantitative data above and subsequent survey results allow us to conclude that we are meeting the project objectives. First, end-of-semester surveys reveal a high interest level in the project, with the majority of students recom294
Journal of Chemical Education
•
Supplemental Material
Instructions for the students, notes for the instructor, and the pre- and postlab survey questions are available in this issue of JCE Online.
Vol. 84 No. 2 February 2007
•
www.JCE.DivCHED.org
In the Laboratory
Notes 1. Information and free registration for CPR is at http:// cpr.molsci.ucla.edu (accessed Aug 2006). 2. A preliminary report appeared online: Spring 2003 CONFCHEM, Paper 4 at http://www.ched-ccce.org/confchem/2003/ a/index.html (accessed September 2005). 3. This assignment is in the public CPR Server Library as: Understanding Absorption and Emission-General Chemistry (by L. D. Margerum and M. Gulsrud). The simulation is available on General Chemistry Interactive CD-ROM, Version 3.0 by William Vining, John C. Kotz, Patrick Harman; Brooks-Cole: Belmont, CA, 2002. 4. The URLs for the prelab are http://www.chem.yorku.ca/ courses/chem1000/equipment/pipette.html and http://www.shsu.edu/ %7Echemistry/primers/AAS.html (both accessed Aug 2006). 5. In 1978, the U.S. Consumer Product Safety Commission lowered the maximum lead content in formulated paint to 0.06% by weight (CPSC Document #5054). Title X of the 1992 Residential Lead-Based Paint Hazard Reduction Act requires remediation if paint samples contain more than 0.5% Pb by weight (http:// www.cpsc.gov/cpscpub/pubs/lead/6007.html, accessed Aug 2006).
Literature Cited 1. Chemists’ Guide to Effective Teaching; Pienta, N. P., Cooper, M. M., Greenbowe, T. J., Eds.; Prentice Hall: Upper Saddle River, NJ, 2005. 2. Mazur, E. Peer Instruction—A User’s Manual; Prentice-Hall: Upper Saddle River, NJ, 1997. 3. Pavelich, M. J.; Abraham, M. R. J. Coll. Sci. Teach. 1977, 7, 23–26. 4. Smith, M. E.; Hinckley, C. C.; Volk, G. L. J. Chem. Educ. 1991, 68, 413–415. 5. Dougherty, R. C.; Bowen, C. W.; Berger, T.; Rees, W.; Mellon, E. K.; Pulliam, E. J. Chem. Educ. 1995, 72, 793–797. 6. Wenzel, T. J. Anal. Chem. 1998, 70, 790A–795A.
www.JCE.DivCHED.org
•
7. Farrell, J. J.; Moog, R. S.; Spencer, J. N. J. Chem. Educ. 1999, 76, 570–574. 8. Paulson, D. R. J. Chem. Educ. 1999, 76, 1136–1140. 9. Bowen, C. W. J. Chem. Educ. 2000, 77, 116–119. 10. Burke, K. A.; Greenbowe, T. J.; Lewis, E. L.; Peace, G. E. J. Chem. Educ. 2002, 79, 699. 11. Gosser, D. K.; Strozak, V. S.; Cracolice, M. S. Peer-Led Team Learning: General Chemistry; Prentice-Hall: Upper Saddle River, NJ, 2001. 12. Lyle, K. S.; Robinson, W. R. J. Chem. Educ. 2003, 80, 132–134. 13. Lewis, S. E.; Lewis, J. E. J. Chem. Educ. 2005, 82, 135–139. 14. Rosenthal, L. C. J. Chem. Educ. 1987, 64, 996–998. 15. Cooper, M. M. J. Chem. Educ. 1993, 70, 476–477. 16. Russell, A. A.; Chapman, O. L.; Wegner, P. A. J. Chem. Educ. 1998, 75, 578–579. 17. Kovac, J.; Sherwood, D. W. Writing Across the Chemistry Curriculum: An Instructor’s Handbook; Prentice Hall: Upper Saddle River, NJ, 2001. 18. Rudd, J. A., II; Greenbowe, T. J.; Hand, B. M.; Legg, M. J. J. Chem. Educ. 2001, 78, 1680–1686. 19. Magner, J. T.; Chadwick, J. E.; Chickering, J.; Collins, C.; Su, T.; Villarba, M. J. Chem. Educ. 2002, 79, 544–547. 20. Oliver-Hoyo, M. T. J. Chem. Educ. 2003, 80, 899–903. 21. Van Ryswyk, H. J. Chem. Educ. 2005, 82, 70–72. 22. Stewart, J. L.; Wilkerson, V. L. ChemConnections: A Guide to Teaching with Modules; John Wiley & Sons: New York, 2000. 23. Casiday, R. E.; Holten, D.; Krathen, R.; Frey, R. F. J. Chem. Educ. 2001, 78, 1210–1215. 24. Markow, P. G. J. Chem. Educ. 1996, 73, 178–179. 25. Kovac, J.; Sherwood, D. W. J. Chem. Educ. 1999, 76, 1399– 1403. 26. Public Law 102-550 [H.R. 5334], Residential Lead-Based Paint Hazard Reduction Act of 1992—Title X. http:// www.epa.gov/lead/pubs/titleten.html (accessed Aug 2006). 27. Chasteen, T. G. J. Chem. Educ. 2001, 78, 1144–1146. 28. Buffin, B. P. J. Chem. Educ. 1999, 76, 1678–1679.
Vol. 84 No. 2 February 2007
•
Journal of Chemical Education
295
