Development, Implementation, and Evaluation of a Web-Based Tool

On the Web edited by

Teaching with Technology

Gabriela C. Weaver Purdue University West Lafayette, IN 47907

Development, Implementation, and Evaluation of a Web-Based Tool, Today’s Science for Tomorrow’s Scientists Caroline Pharr† Department of Chemistry, University of Wisconsin–Madison, Madison, WI 53706; [email protected]

Formal investigation of students’ attitudes toward science has been pursued by researchers for the past 30–40 years (1, 2). In 2001, opinion research and consulting firm Wirthlin Worldwide studied the perceptions of “millennials” (the generation of Americans born in 1982 or later) with respect to scientists and science careers (3, 4). The study found that many millennials perceive scientists as reclusive and socially awkward. Positive findings included millennials’ belief that scientists have the ability to find cures and make medicines, make a decent salary, and have the opportunity to win awards and gain recognition. However, these positive beliefs seem to be overshadowed by the perception that scientists are constantly frustrated, alone, and working too much to enjoy their families or lives. The Wirthlin study suggested a number of recommendations to help make a chemistry career more appealing to young people, including: engaging students with fun, hands-on experiments; challenging existing stereotypes; and celebrating the successes of chemists. Well before the Wirthlin study emerged, much work had been done on science outreach to foster interest and awareness †Current address: Department of Chemistry and Biochemistry, Mercyhurst College, Erie, PA 16546

among young people. These efforts include chemistry camps, science competitions, mentoring programs, and demonstrations (5–10). This type of work continues (11, 12), and has lately been supplemented by the incorporation of technology, particularly in the form of Web tutorials, as a tool to introduce science and mathematics to undergraduate students (13–17). Combining outreach ideas and technology, we have designed and developed a Web-based tutorial that introduces current scientific research into middle school and high school classrooms while simultaneously correlating with National Science Education Standards (NSES; see ref 18). This effort aims to reach students early in their education, and has the potential to be incorporated into middle and high school curricula in place of a more traditional format, such as a lecture. The development, testing, and evaluation of this Web site, Today’s Science for Tomorrow’s Scientists (TSTS) (19) will be discussed here. Development of TSTS Today’s Science for Tomorrow’s Scientists was designed with several goals in mind: to improve students’ attitudes toward science; to improve students’ understanding and awareness of research and the research environment; and to teach students in-

Figure 1. Screenshot of an example Web page for grades 5–8 on biochemistry and peptide design (Gellman group) of the Today’s Science for Tomorrow’s Scientists Web site.

© Division of Chemical Education • www.JCE.DivCHED.org • Vol. 86 No. 6 June 2009 • Journal of Chemical Education

757

On the Web Table 1. Research Concepts and National Science Education Standards Presented in Research Tutorials for Grades 5–8 Researcher and Tutorial

Research Concepts Taught

Corresponding Standards

McMahon: Organic chemistry and astrochemistry

Interstellar space; Radio astronomy; Energy transfer in reactions; Matrix isolation; Photochemical reactions; Synthesis

Energy and its different forms; Transfer of energy; Light absorption; Transformation of one chemical substance to another

Gellman: Biochemistry and peptide design

Peptides; Proteins; Design and synthesis of molecules; Secondary structure; Tertiary structure; Self-assembly; Lock and key model

Structure and function in living systems; Protection from disease; Systems in the human body

Stahl: Inorganic chemistry and catalysis

Catalysts; Environmentally friendly reactions; Transition metal chemistry; Molecular oxygen

Elements combine to form different compounds; Chemical reactions yield substances with different properties from the individual parts; Characteristic properties of elements; Grouping of the elements into categories; Characteristic chemical reactions

Table 2. Research Concepts and National Science Education Standards Presented in Research Tutorials for Grades 9–12 Researcher and Tutorial

Research Concepts Taught

Corresponding Standards

McMahon: Organic chemistry and astrochemistry

Interstellar space; Radio astronomy; Spectral data; Reactive intermediates; Matrix isolation; Photochemical reactions; Synthesis

Transfer/sharing of electrons; Chemical bonds; Physical properties of molecules: solids, liquids, gasses; Carbon chemistry; Chemical energy; Chemical reactivity

Peptides;

Molecules in cells: protein function; Structure of molecule determines interaction with other molecules; Reaction rates dependent on shape; Designing systems to study

Gellman: Biochemistry and peptide design

Stahl: Inorganic chemistry and catalysis

758

β-Peptides; Peptide

bonds; Design and synthesis of molecules; Secondary structure; Tertiary structure; Self-assembly; Lock and key model Catalysts; Environmentally friendly reactions; Catalytic cycles; Transition metal chemistry; Molecular oxygen; Oxidizing and reducing agent

Outer electrons involved in chemical reactions; Periodic table trends: transition metals; Redox reactions; Catalysis

formation corresponding to the NSES. Currently, the research of three chemistry groups at the University of Wisconsin–Madison is presented on TSTS. Each research area is presented at two levels, one for students in middle school (grades 5–8) and one for students in high school (grades 9–12). Broadly defined, the research areas presented encompass organic chemistry, inorganic chemistry, and biochemistry. These are introduced in conjunction with basic chemistry concepts that meet the NSES. These concepts are then built upon to illustrate what each individual research group does. The Web site design program Adobe GoLive was used to make TSTS. Throughout the Web site movies, Jmol molecular images, and puzzles help to make the learning process interactive. Terminology specific to the research area and words the students may not know are colored green and italicized. Holding the mouse over these words causes a box with the definition of the word to pop up. This allows the student to read the information without having to move to a separate page, and then to continue with little interruption. After completing the first version, TSTS was reviewed and edited by middle and high school teachers and the researchers whose work was presented. Following incorporation of these revisions, TSTS was beta tested at a local high school where feedback was received from students and teachers. This information was used to revise the site further, resulting in the final product. Figure 1 is a screenshot of a typical page from a TSTS tutorial. Tables 1 and 2 indicate the topics presented in each tutorial. Surveys Used in Evaluating TSTS Three surveys were used to evaluate the effectiveness of TSTS in reaching its intended goals: an attitudinal survey, a research environment survey, and a content survey (see the online supplement). The attitudinal survey consisted of Likert-scale questions from the revised Scientific Attitude Inventory (SAI II) (subscales 5-AB and 6-AB were used) (20). The original SAI was developed and field tested 36 years ago. It has been used extensively throughout the scientific community and was revised in 1996 to improve readability and eliminate genderbiased language (20). A panel of judges evaluated the validity of the questions when the survey was developed, and subsequent statistical tests of the SAI II demonstrate the survey’s ability to distinguish between low and high scores. Students’ scores were adjusted for negative questions to give an attitudinal survey total. This technique was used for each subscale and for the survey in its entirety by the researchers who designed the SAI II (20). The research environment survey consisted of four openended questions concerning who conducts research, including when, how, and where it is carried out. These questions were formulated by the researcher and were scored by level of understanding demonstrated in the answer. Points earned for each question on the research environment survey were summed to give a research environment survey total score. The content surveys consisted of five open-ended or multiple-choice questions relevant to the topic area covered. Questions were chosen from middle (21, 22) and high (23, 24) school science textbooks. Open-ended questions were scored for level of understanding demonstrated in the answer. Points earned for each question on the content survey were summed to give a content survey total score. Inter-rater comparisons were not used in the scoring of questions based on level of understanding in either the research

Journal of Chemical Education • Vol. 86 No. 6 June 2009 • www.JCE.DivCHED.org • © Division of Chemical Education

On the Web

environment or the content surveys. The potential effect this could have on results is discussed in the data analysis section. Surveys were administered pre- and post-treatment. Pre and post surveys consist of identical questions in a different order. Surveys were originally in paper-and-pencil format, but upon the request of teachers, were moved online for ease of use. Students were assigned a login ID and password to access the surveys. Implementing TSTS in High School Classrooms After the Web site was completed, the surveys chosen and constructed, and an implementation plan laid out, TSTS was submitted to the University of Wisconsin–Madison Educational Institutional Review Board (IRB). IRB approval was gained to implement TSTS in middle and high school classrooms. Because the participants were under 18 years of age they were required to have their parents or guardians sign a consent form in order for data collected from the study to be evaluated and published. Students whose parents or guardians did not sign the consent form were allowed to view TSTS, but did not participate in the survey portion of the study. Beta Tests at Middleton High School and Adams–Friendship High School Beta tests of the three high school tutorials (McMahon, Stahl, and Gellman) were carried out at Middleton High School in Middleton, WI and Adams–Friendship High School in Adams, WI in May and December 2006, respectively. These beta tests results were used mainly to determine challenges associated with implementing TSTS. Clarity and usefulness of some questions as well as redundancy issues with respect to the attitudinal and research environment surveys were found to be problematic. Specifically, because the attitudinal and research environment surveys ask the same questions regardless of which tutorial is viewed, students grew tired of filling out the same survey in rapid succession. In the December implementation students viewed the Stahl and Gellman tutorials back to back, and subsequently grew tired of the attitudinal and research environment surveys. Difficulties associated with online surveys (looking up answers online, chatting with classmates about answers) were also encountered. It was at this point that the content questions were taken from grade-level appropriate text books and the wording of the questions on the research environment survey reworked. With respect to redundancy, it was determined that if two or more tutorials were viewed close to each other in time, there should be a pre attitudinal and research environment survey before the unit starts and then post attitudinal and research environment surveys upon completion of the entire unit. Lastly, in future implementations teachers would instruct students not to communicate with each other or look up information online while completing the pre and post surveys. Middleton High School Full-Scale Implementation Informed by the results of the beta tests, TSTS was implemented at Middleton High School in Middleton, WI again in June of 2007 on a larger scale (~200 students). In this implementation students viewed the Stahl 9–12 tutorial (inorganic chemistry). Students were assigned IDs ending in numbers generated randomly by a computer (25) (provided by the researcher). Students were then divided into control and treatment groups

by teachers according to whether their ID number was odd or even. Thirteen classes with a total of five teachers followed this procedure. In this large-scale implementation students in the control group received a lecture on content material similar to that presented on the Web site while students in the treatment group viewed TSTS. Because of time constraints and computer lab scheduling difficulties, all students took three pre surveys (attitudinal, content, research environment) as a homework assignment two to four days before the viewing or lecture instead of taking them during class time as a group. Upon completion of the lecture or viewing of TSTS, students took the three post surveys two to four days later, again as a homework assignment. Students were instructed not to communicate with their peers or look things up on the Internet while taking the online surveys. Edgewood Middle School Full-Scale Implementation TSTS was implemented at Edgewood Middle School in Madison, WI throughout November and December of 2007. Students at Edgewood Middle School viewed all three tutorials currently presented on TSTS (Stahl, McMahon, and Gellman) over a span of three to four weeks. Students were assigned IDs and divided into control and treatment groups by the teacher in the same fashion described for the Middleton High School procedure. Two classes with the same teacher followed this procedure, giving a sample size of 35–40 students. Implementation of TSTS followed the same format as described for Middleton High School students with three exceptions. First, students in the control group received a lecture from an outreach educator of the chemistry department of the University of Wisconsin– Madison instead of a teacher at Edgewood Middle School. Second, students took the surveys in class instead of as a homework assignment in this implementation. Third, because of the redundancy of multiple tutorial viewings revealed by the beta testing, pre attitudinal and research environment surveys were taken two to four days before the start of TSTS. After viewing all three of the tutorials, students took post attitudinal and research environment surveys two to four days later. Again, students were instructed not to communicate with their peers or look things up on the Internet while taking the online surveys. Results TSTS Implementation, Middleton High School, June 2007 Pre and post survey data were used for statistical evaluation of the implementation of TSTS. Comparisons between classes were made using an independent-groups, one-way analysis of variance (ANOVA). Comparisons between control and treatment groups with respect to pre and post survey scores and gender were made using the general linear model repeated measures ANOVA function of SPSS 15 (26). Both quantitative tests and qualitative arguments were made in support of the decision to group together control students and treatment students from each class (based on mean scores on pre surveys) to make one large control and one large treatment group. Quantitatively, a one-way ANOVA was carried out for each type of survey to compare the classes to one another. The null hypothesis that a difference exists between students in each class is rejected for the attitudinal survey (F194 = 1.745, p = 0.064), the research environment survey (F203 = 1.132, p = 0.336), and the content survey (F202 = 1.668, p = 0.077).


759

On the Web

The large sample size resulting from grouping control and treatment students from individual classes together to give one large control and one large treatment group is more representative of students taking general chemistry at Middleton High School. All students in this study attend Middleton High School, and were taking the same level general chemistry course. While not all students had the same teacher, they encountered the same content material for this study. One could also make the argument that not all students learn the same way, and therefore the fact that different teachers delivered the information becomes a moot point. Most importantly, each class was divided into control and treatment groups randomly. Any differences among students in different classes would be accounted for in both the control and the treatment group. While inter-rater comparisons for questions scored based on level of understanding would have been ideal, it is unlikely that the results of the study were affected by this issue. Two versions of each of the surveys (labeled type H and type T) were provided to the teachers using TSTS, and it was they who were in control of which type was to be used for pre survey and which type was to be used for post survey. Upon getting the surveys back for analysis, they were scored blindly, without the researcher knowing which surveys were pre or post or which were from the control group and which were from the treatment group. Any anomalies in scoring would have therefore been present in both the pre and post survey and the control and the treatment group data, correcting for a potential skewing of the results. After grouping the classes, a repeated measures ANOVA was carried out to look for changes from pre survey to post survey with respect to treatment (whether students received a lecture or viewed TSTS) and gender. Attitudes toward science were similar for both pre survey and post survey (F194 = 0.342, p = 0.560) regardless of whether students received a lecture or viewed the Web site (F194 = 0.916 p = 0.340). Attitudes toward science were also similar for pre survey and post survey with respect to gender (F194 = 0.827, p = 0.364). Understanding of the research environment was similar for pre survey and post survey (F203 = 2.22, p = 0.138) regardless of whether students received a lecture or viewed the Web site (F203 = 0.566, p = 0.453). Interestingly, there was a significant difference in understanding of the research environment with respect to gender (F203 = 4.36, p = 0.038). In this sampling of high school students the mean score of females illustrates that they knew more about the research environment than the males. There was no statistically significant difference in the change in understanding with respect to gender from research environment pre survey to post survey (F203 = 0.20, p = 0.899). Students performed significantly better relative to content on the post survey than the pre survey (F202 = 57.4, p < 0.001) regardless of whether they received a lecture or viewed the Web site (F202 = 2.474, p = 0.117). Improvement from content pre survey to post survey was similar with respect to gender (F202 = 1.050, p = 0.307). TSTS Implementation, Edgewood Middle School, December 2007 Identical statistical analysis of pre and post survey data was used in the evaluation of implementation of TSTS at Edgewood Middle School as described for Middleton High School. Again, quantitative tests and qualitative arguments were used to group the control students and treatment students from the two 760

classes (based on mean scores on pre surveys) together to make one large control and one large treatment group. A one-way ANOVA was carried out for each type of survey to compare the classes to one another. The null hypothesis that a difference exists between students in the two classes can be rejected for the attitudinal survey (F32= 0.319, p = 0.576), the research environment survey (F38 = 0.972, p = 0.331), the McMahon content survey (F42 = 2.236, p = 0.143), and the Gellman content survey (F38 = 0.550, p = 0.463). The null hypothesis cannot be rejected for the Stahl content survey (F39 = 4.325, p = 0.045). However, the implementation at Edgewood Middle School had a small number of students. Only one survey out of five total showed a significant difference between the two classes with respect to the mean scores on the pre survey. Because of the small sample sizes and the majority of surveys allowing for the rejection of the null hypothesis, the assumption that the two classes are similar with respect to their abilities was maintained. The same qualitative arguments apply to the students at Edgewood Middle School as those given above for the students at Middleton High School. The larger sample size resulting from grouping control and treatment students from the two classes together is more representative of students taking this science course at Edgewood Middle School. Lastly, the same argument that was made for Middleton High School data with respect to inter-rater comparison issues applies to the data collected from Edgewood Middle School. After grouping the classes to make one control and one treatment group, a repeated measures ANOVA was carried out to look for changes from pre survey to post survey with respect to treatment (whether students received a lecture or viewed TSTS) and gender. Attitudes toward science were similar for pre survey and post survey (F32 = 0.248, p = 0.623) regardless of whether students received a lecture or viewed the TSTS Web site (F32 = 0.620 p = 0.438). Attitudes toward science were also similar for pre survey and post survey with respect to gender (F32 = 3.834, p = 0.060). The data show that understanding of the research environment decreased by a significant amount from pre survey to post survey (F38 = 5.951, p = 0.020) regardless of whether students received a lecture or viewed the TSTS Web site (F38 = 2.635, p = 0.114). However, after reviewing a graph showing a distribution of the results, it was clear that the control group post survey scores decreased by a much larger amount than those of the treatment group. For this reason, the data were separated and further analyzed for each group (control and treatment). This analysis showed that the control students did indeed score lower on their post survey compared to their pre survey by a significant amount (F17 =12.179, p = 0.003). Analysis of the treatment students showed no significant decrease in performance from research environment pre survey to post survey (F21 = 0.282, p = 0.601). Suspected reasons for this trend will be discussed subsequently. Understanding of the research environment was similar with respect to gender (F38 = 0.012, p = 0.915). Students performed significantly better on the McMahon content post survey than at pre survey (F42 = 19.973, p < 0.001) regardless of whether they received a lecture or viewed the TSTS Web site (F42 = 0.041, p = 0.840). Improvement relative to content from the pre survey to the post survey was similar with respect to gender (F42 = 0.248, p = 0.621). Students performed significantly better on the Gellman content post survey than at the pre survey (F38 =26.642, p < 0.001),


On the Web

regardless of whether students received a lecture or viewed the TSTS Web site (F38 = 0.248, p = 0.622). Improvement from the content pre survey to post survey was similar with respect to gender (F38 = 3.133, p = 0.086). However, a significant difference in mean scores was found between females and males about the content material, with female students demonstrating a better understanding of the material (F38 = 5.002, p = 0.032). Lastly, students were just shy of performing significantly better at the Stahl content on the post survey than on the pre survey (F39 = 3.738, p = 0.061). No significant improvement was seen from pre to post survey regardless of whether students received a lecture or viewed the Web site (F39 =0.116, p = 0.736). Differences in scores from the content pre survey to post survey were similar with respect to gender (F39 =0.351, p = 0.588). Discussion of Results Middleton High School Results Given that viewing a TSTS tutorial takes roughly 45 minutes, it may be overambitious to expect a large change in high school students’ attitudes toward science. Additionally, high school students are further along in their academic career, and may have more firmly set attitudes toward science than younger students, and therefore, their perceptions may be harder to change. Trends from the data show students who viewed the TSTS Web site had a more positive attitude toward science from pre survey to post survey, while students who received a lecture instead showed a slight decrease in attitudes toward science. However, this difference is not statistically significant. It is encouraging that attitudes toward science are similar for both males and females. Much work has gone into studying and identifying factors that keep women out of scientific careers as well as increasing awareness and promoting women in science because of the lower number of females who chose a scientific career (27–31). Fostering a positive attitude toward science at a younger age is one way to promote interest and possibly increase the number of women entering undergraduate and graduate school to study science. Trends from pre survey to post survey show females demonstrating a more positive attitude toward science, although, again, the difference is not statistically significant. Viewing one TSTS tutorial did not increase students’ understanding of the research environment by a significant amount. It may be ambitious to expect a large change as a result of one, 45-minute encounter with the material; this gap is being addressed to make more clear how, with whom, and where researchers work. Most of the Web pages that introduce information about the research environment consist mainly of text; now pictures to complement the written explanations are provided. For example, in the segment that introduces the concept of scientific journals and conferences as a means for sharing information with others, pictures show people presenting their work as well as a picture of popular science journals. The hope is that pictures will increase the likelihood that the description is read as well as leave a more lasting impression on the students. It is interesting that to begin with, females knew significantly more about the research environment than males. The reason for this discrepancy is unknown. This is particularly odd, because as referenced earlier, women are an underrepresented group in scientific careers. For example, tenure-track positions in research institutions are held disproportionately by males

(32–34). The change in understanding from pre survey to post survey was not significant and was similar for both sexes. This is expected because of the low exposure time discussed earlier. Statistically significant improvements from content pre survey to post survey regardless of whether students received a traditional lecture or viewed TSTS are very encouraging results. The aspiration in designing this Web site was that teachers could substitute the Web tutorials on TSTS in place of standard lectures in their classrooms, and the data suggest that this is a possibility. Using TSTS to introduce content in an innovative and different way that also incorporates current science research into the lesson is an appealing alternative. Because TSTS is able to deliver content as defined by the NSES, teachers can incorporate traditional material, technology, and current science research into their classrooms simultaneously. It is also encouraging that improvement from content pre survey to post survey was similar for males and females. Studies have shown that learning styles can sometimes differ between men and women (35–37). Based on this study, the TSTS tutorial content seems to be reaching students equally and effectively regardless of gender. Edgewood Middle School Results Again, the lack of improvement in students’ attitudes toward science was disappointing. It was hoped that with the longer time frame, increased exposure, and lack of repetition in taking the attitudinal survey a more significant effect on students’ attitudes toward science would emerge. However, three, 45-minute sessions is still not an overwhelming amount of time to effect change. Another problem was that the students participating in the study had previously completed a unit in which they regularly took pre and post surveys. The post attitudinal survey came at the end of a four–six week period in which the students took surveys frequently. The teacher participating in the study even noted that he had to encourage the students to take the surveys seriously and put their best effort into them, as the students expressed frustration at taking more surveys. The decrease in understanding of the research environment was also particularly disappointing. While dividing the data into control and treatment groups showed that only the control group decreased significantly from pre survey to post survey, replication and larger sample sizes are needed to fully verify this result. It is encouraging that trends in the data show treatment students’ scores decreasing much less than for the control students, however, an increase in understanding is the ultimate goal. While these results were also likely influenced by an overabundance of pre and post surveys taken by the students, it re-illustrates the point that the portrayal of the research environment could have been clearer. The incorporation of pictures described earlier has also been made to the middle school tutorials. In both the Middleton High School and Edgewood Middle School implementations it was clear that students who viewed the TSTS tutorials knew little more about the research environment than students who did not. The significant increase from pre survey to post survey for the McMahon and Gellman content surveys is encouraging. While the increase from pre survey to post survey for the Stahl tutorial fell just short of being significant, a trend of improvement was seen in the data. Furthermore, lack of significant change is present for both the control and treatment groups, which may be an indicator that the topic presented was difficult for the students. The previously discussed number of surveys


761

On the Web

taken by the students may also have played a role in the lack of significant change. As in the Middleton High School implementation of TSTS, students at Edgewood Middle School demonstrated similar attitudes toward science, understanding of the research environment, and improvement from pre to post survey regardless of gender. In one case (the Gellman tutorial) female students had a better understanding of the material on the content survey. For reasons discussed earlier, it is encouraging that the TSTS material reaches students of both genders equally and effectively. Conclusion No significant change emerged in attitudes toward science for the middle or high school students who participated in this study. Understanding of the research environment was not increased by using TSTS tutorials for middle or high school students. High school students’ content knowledge improved by a significant amount on the content survey from pre survey to post survey regardless of whether they received a lecture or viewed TSTS tutorials. Middle school students also improved by a significant amount from pre to post survey on two of the three content surveys. Improvement on the third content survey was just short of being significant, but this was true regardless of the method used to introduce the material (TSTS or lecture). These results suggest that TSTS can be used in classroom environments in place of traditional formats, such as lecture. TSTS is published by the National Science Digital Library and is free for teachers or interested citizens to use (19). Scientists in academia and industry are invited to submit tutorials. It is the hope of those involved that TSTS will one day serve as a bank of tutorials about current science research in different disciplines contributed by a diverse scientific community. Acknowledgments For guidance and direction in developing and evaluating TSTS I thank John Moore. I also wish to thank Erik Hadley, Will Pomerantz, Amanda King, Robert McMahon, Shannon Stahl, and Sam Gellman for providing materials, ideas, and answering questions about content. I thank Ken Robertson and Jenny Powell for teaching middle and high school classes during the implementation of TSTS. Lastly, I am forever thankful to the teachers who helped implement TSTS: Julie Jensen, Joe Spolar, Terri Kowieski, Jack Markin, Debbie Weitzel, Stacy Bohacek, Kevin Moore, and Dan Toomey. The development of this tutorial and statistical evaluation of the data were supported by a grant from the University of Wisconsin–Madison Nanoscale Science and Engineering Center, NSF MDR-0425880. Literature Cited Simmons, J.; Esler, W. Sch. Sci. Math. 1973, 72, 633–636. Leavers, D. R. J. Chem. Educ. 1975, 52, 804. Oliver, B. Chem. Eng. News 2003, 81 (29), 35. Dagani, R. Chem. Eng. News 2004, 82, (36), 46. Moore, J. W. J. Chem. Educ. 1999, 76, 1469. Exstrom, C. L.; Mosherj, M. D. J. Chem. Educ. 2000, 77, 1295– 1297. 7. Martin, A.; O’Conner, K.; Mix, K.; Nicolette, D.; Chrysanthe, D. J. Chem. Educ. 1997, 74, 452.

1. 2. 3. 4. 5. 6.

762

8. Keck, P. J.; Hendrickson, H.; Johnson, K.; DeClue, M. E. J. Chem. Educ. 2000, 77, 608–609. 9. Lee, N. E.; Schreiber, K. G. J. Chem. Educ. 1999, 76, 917–918. 10. Scaife, C. W. J. J. Chem. Educ. 1986, 63, 790–791. 11. Voegel, P. D.; Quashnock, K. A.; Heil, K., M. J. Chem. Educ. 2004, 81, 681–684. 12. Voegel, P. D.; Quashnock, K. A.; Heil, K. M. J. Chem. Educ. 2005, 82, 634–636. 13. Dickstein, R.; Bracke, P. J. Ref. Serv. Rev. 2002, 30, 330–337. 14. Frey, R. F.; Holten, D.; Spees, W. M.; Gilbertson, M.; Deppe, R. K.; Casiday, R. E.; Herman, C. J. Chem. Educ. 2005, 82, 1871–1879. 15. Oldknow, A. Mathematics Teaching 2006, 196, 16–21. 16. Luck, L. A.; Urmi, R. Biochem. Mol. Biol. Educ. 2007, 35, 238–243. 17. Harwood, W. S.; Korkmaz, A. J. Sci. Educ. Technol. 2004, 13, 243–253. 18. National Science Education Standards. http://www.nap.edu/ openbook.php?record_id=4962 (accessed Mar 2009). 19. Today’s Science for Tomorrow’s Scientists. http://www.chemeddl. org/collections/TSTS/ (accessed Mar 2009). 20. Moore, R. W.; Foy, R. L. H. J. Res. In Sci. Teach. 1997, 34, 327–336. 21. Jones, G.; Jones, M.; Marchington, P.; Acaster, D. Coordinated Science 2; Cambridge University Press: New York, 1995. 22. Essenfeld, B. E.; Gontang, C. R.; Moore, R. Biology; AddisonWesley Publishing Company: Menlo Park, CA. 23. Myers, R. T.; Oldham, K. B.; Tocci, S. Chemistry; Holt, Rinehart, and Winston: Austin, Texas, 2004. 24. Carle, M. A.; Sarquis, M.; Mary, N. L. Physical Science: The Challenge of Discovery; D.C. Heath and Company: Lexington, 1991. 25. Random Integer Generator. http://www.random.org/integers/ (accessed Mar 2009). 26. SPSS Inc. http://www.spss.com (accessed Mar 2009). 27. Nature 1999, 400, 195. 28. Dewandre, N. Science 2002, 295, 278–279. 29. Li, L. J. Chem. Educ. 2002, 79, 418–419. 30. Milakofsky, J.; Milakofsky, L.; Shibley, I. A.; Bender, D. S.; Patterson, H. O. J. Chem. Educ. 2003, 80, 569–573. 31. Lindsay, H. A.; Turner, R. C. J. Chem. Educ. 2003, 80, 563–568. 32. Long, J. R. Chem. Eng. News 2000, 78 (39), 56. 33. Kuck, V.; Marzabadi, C. H.; Nolan, S. A.; Buckner, J. P. J. Chem. Educ. 2004, 81, 356–363. 34. Kuck, V.; Marzabadi, C. H.; Buckner, J. P.; Nolan, S. A. J. Chem. Educ. 2007, 84, 277–284. 35. Cavallo, A. M. L. P.; W. H., R. M. Sch. Sci. Math. 2004, 104 288–300. 36. Gokhale, A. A.; Stier, K. J. Women Minor. Scien. Eng. 2004, 10, 149–160. 37. Greene, S. V.; Wheeler, H. R.; Riley, W. D. J. Women Minor. Scien. Eng. 2004, 10, 217–228.

Supporting JCE Online Material

http://www.jce.divched.org/Journal/Issues/2009/Jun/abs757.html Abstract and keywords Full text (PDF) Links to cited URLs and JCE articles Supplement Attitudinal and research environment surveys

Content surveys by grade level and topics taught


Development, Implementation, and Evaluation of a Web-Based Tool

Recommend Documents