Research: Science and Education edited by
Chemical Education Research
Diane M. Bunce The Catholic University of America Washington, D.C. 20064
Leveraging Technology for Chemical Sciences Education: An Early Assessment of WebCT Usage in First-Year Chemistry Courses
W
Paul Charlesworth* Department of Chemistry, Michigan Technological University, Houghton, MI 49931; *
[email protected] Chelley Vician School of Business and Economics, Michigan Technological University, Houghton, MI 49931
Chemical science educators face numerous challenges in today’s higher education setting. Large class sizes, differing analytical abilities on the part of students, and variable student motivational levels are but a few of the common challenges awaiting the chemical science educator. The unintended effects of ignoring these challenges are manifested in poor student performance in the course, increased drop rates and repeated classes, and a reduction in student selection of upper-level chemical science courses and chemical science-related extracurricular activities (1–5). How can chemical science educators address these performance issues yet still balance the competing demands set by a diverse student population? In this article, we present the early results of combining information technologies (e.g., WebCT and Microsoft PowerPoint) with the intent of improving the chemical science learning environment in terms of student motivation and learning. Our assessment focuses on student reactions to these instructional innovations. By analyzing the perceptual consequences of new information technology introduction, we gain knowledge necessary for subsequent technology introduction efforts (6–9). Background At the time this study was undertaken (1999–2000 school year), the chemistry sequence at Michigan Technological University (MTU) involved a three-course sequence over an entire academic year. The courses were taught as service to the University with class sizes of several hundred students. Through our teaching efforts, we were able to observe two negative factors at work in these student-learning experiences: (i) widely varying student analytical abilities and (ii) widely varying student interest and motivation in the subject. In fact, we found that even with the minimum ACT math score of 26 set for entry into the main first-year program at our University, we encountered groups of students incapable of working with simple equations. These students shared a classroom with students who quickly grasped complex equilibrium and kinetics problems. Further, it became clear that for many of our current students learning chemistry is a tiresome and complex process that requires something extra beyond the material presented in a textbook or lecture. As Mesher notes, this magic “something” is hard to identify, harder to quantify, and
almost impossible to control (10). Thus, it was of paramount importance to develop ways to engage students and personalize instruction under these circumstances. Technology-Supported Learning (TSL) Environment Our approach was to incorporate information technology into the first-year chemistry program. Our vision for the chemistry-learning experience is twofold: to automate and to informate selective portions of the learning environment (6, 7). Automating means to use information technology to improve quality by performing structured, routine, operational tasks reliably and efficiently. Informating means to use information technology to provide students and instructors with access to more information. This was achieved by providing a Web-based learning environment facilitated through WebCT and by the use of a media-based classroom environment facilitated through Microsoft PowerPoint. We believed that the combination of these two products would assist the instructor in providing a learning environment that would foster an increased level of motivation and enhance learning. (6, 7, 10). The benefit of supplementing classroom instruction with Web-based content and exercises is that the classroom provides a visual focus for the online material, while the online material provides the student interactions, accessibility, convenience, and repeatability not found in many classroom environments (10). WebCT, a course management tool, served as a means of organizing our TSL environment. WebCT contains a number of features that were used to engage students and personalize the learning environment. In our situation, we utilized: (i) the Path Tool to organize presentation of course topical material; (ii) the Quiz and Survey Tool to coordinate online quizzes, examinations, and student satisfaction surveys; (iii) the Email and Threaded Discussion Tools to facilitate communication between students and the instructor; and (iv) the Gradebook Tool to provide student retrieval of grading evidence scores and course grade progress. A summary of these features and how we used them to automate and informate the learning environment is contained in Table 1 (6, 7). Course content was presented to students via the Path Tool of WebCT. The Path Tool allows the instructor to quickly construct a content tree from the existing classroom content,
JChemEd.chem.wisc.edu • Vol. 80 No. 11 November 2003 • Journal of Chemical Education
1333
Research: Science and Education
including HTML, PDF, and DOC files. By using the Path tool, an instructor is able to view the total number of “hits” by each student, the last time they visited the site, and a breakdown of where the student visited while at the site. When studying a student’s use of material in the Path, an instructor is provided with information about which pages a student viewed, how long pages were viewed, and how often the student viewed each page. For the student, the Path Tool provides a well-defined and searchable structure; it allows each student to compile individual parts into a printable set of notes tailored to his or her individual needs. Furthermore, students can quickly return to the topic they looked at last by selecting the “resume” icon within WebCT. The Path for the chemistry learning experience included the following components: •
An online “textbook” developed by the instructor (WebNotes).
•
Problems sets for each chapter.
•
Answers to the problem sets for each chapter.
•
Partial lecture notes prepared as PowerPoint handouts (RediNotes).
WebNotes are HTML-based class notes reflecting the material being presented to students in lectures. Essentially, the WebNotes behave as a media enhanced textbook interpretation that students may view or collate and print for use as study notes. In addition to the WebNotes, PowerPoint slides were output to Adobe Acrobat PDF format at a rate of three slides per page with the lines removed. These RediNotes were available for download and printing as a classroom supplement. Bound copies of the RediNotes were also available for purchase from the university bookstore. McGraw-Hill’s Custom Division published these for the instructor. Approximately 85% of the students used copies of the RediNotes in
class and identified that this course element made a very positive contribution to the chemistry learning experience (11, 12). The final primary component of WebCT, which differentiates the technology from “online notes”, was the use of online weekly quizzes and examinations that were computer scored. Students were able to view their own scores from these quizzes and other inclass assignments through WebCT’s gradebook function. The quiz and survey function permits instructor development of several question types and storage of these in a database for later incorporation into quizzes and surveys. Some of these types are computer-scorable; others require the instructor to intervene in the scoring process. The first-year chemistry database at our institution currently contains about 5000 different computer-scored questions that are subdivided into textbook chapters and surveys. These three main question types are appropriate for computer-scoring: •
Multiple Choice, which allows the instructor to specify not only the number of possible answers, but to give partial credit by assigning a percentage of the total points to each answer.
•
Calculated, which allows the instructor to write a numerical question and enter an algorithm that generates a series of possible questions and answers from a set of variables. Students answer these questions by entering a numerical value and units. The instructor sets the tolerance for each answer as a percentage of the correct answer.
•
Matching, which allows the instructor to create a question and present the students with a list of statements. The student selects the correct answer from a pulldown menu. Credit may be assigned for each answer independently, or students may be required to correctly match all statements and answers for full credit.
Table 1. Automating and Informating the Chemistry-Learning Experience with WebCT Learning Vision
WebCT Feature
Using These Features To Teach
Automate
Path Tool
Placing course content (WebNotes, Problem Sets, Answers, and RediNotes) here enables the learner to control the sequence of information presentation (similar to interactive learning systems), thus permitting personalization of the learning process.
Gradebook
Online, secure retrieval of routine, structured information permits the instructor to focus on more semi-structured aspects of the the learning environment and the learner to personalize the environment.
Path Tool
Providing a nonlinear means of browsing and searching through course information permits the learner to organize knowledge in a manner tailored to his or her own logic. The tracking features provide the instructor with feedback about student progress.
Quizzes, Exams, and Surveys
Online access provides the learners with a means of personalizing the evaluation aspect of their learning experience. Online grading of such elements provides the instructor with feedback on learner mastery of material.
Email and Threaded Discussion
Usage permits further exploration of learner understanding, thus supporting more thorough cognitive information processing.
Informate
1334
Journal of Chemical Education • Vol. 80 No. 11 November 2003 • JChemEd.chem.wisc.edu
Research: Science and Education
During this initial implementation, it was found that the multiple-choice questions were most effective for fact recollection and conceptual questions. The calculated questions were most effective for extended calculations such as those found in titrations or analytical problem solving. The matching questions were only used for questions of the type “Which element is a metal?” or “Which element is a nonmetal?”. The quizzes and exams were administered as follows: each quiz consisted of six randomly-selected questions, and each exam of twenty randomly-selected questions from a database of several hundred per chapter. Students were allowed to make up to three attempts at a quiz and up to two attempts at an examination; each attempt contained a different randomized set of questions. The randomized sets were grouped such that question 01 on the student quiz would be randomly selected from, for example, a set of limiting reagent problems. This meant that for each attempt, the student would get a different limiting reagent problem at question 01. Since there are few ways to monitor the activities of students when they are not present in the traditional classroom, no restrictions were placed on approaches to quizzes and exams. Deadlines were set at 11:55 p.m., one week from the date that an assignment was released, which allowed students to obtain help from the Chemistry Learning Center prior to a deadline. WebCT presents students with a new freedom to work at their own pace, regulate their learning, and collaborate with others in the class. During our initial implementation study, we wanted to allow our students maximum freedom to innovate in their use of the new method, and report this in surveys. It was decided therefore that requiring a student to work alone, or under timed conditions, not only defeated many of our goals, but would be very difficult to enforce when students are working in computer labs and offcampus. Concerns that students may take adverse advantage of this freedom were addressed by assigning 300 points, of the 900-point total grade, to a 40-question final examination and the ACS standardized test. The three online examinations and nine online quizzes were assigned 400 points. The remaining 200 points were assigned to class assignments and weekly recitation quizzes. No normalization of online grades based on classroom grades was performed. Analysis Analyses reported in this article are based on observations and interviews with the instructor, an analysis of WebCT tracking data, and student responses to survey questionnaires administered via WebCT. Copies of the surveys, in a WebCT-ready format, and all raw data are available in the Supplemental Material.W
Overall Analyses WebCT tracking data indicated that students spent short, but successive quantities of time viewing the problems sets and their respective answer keys, suggesting that students were viewing and then printing pages for use away from the computer. Some students would access WebCT several times per day, but use very few path-related components. This suggested that the students are simply checking for messages or working on quizzes and then leaving the TSL environment
again. Access to the online “textbook” was generally concentrated at times when online assignments were due, suggesting that students were using this material to help with the quizzes and exams. An analysis of exam and quiz data demonstrated that online exam and quiz scores were generally higher than those for written exams and quizzes, but the pattern of online assignment grades mirrored those of written assignments. Students who performed poorly on the online assignments also performed poorly on the written assignments. There were no cases of students with very high performance on the online assignments and very low performance on the written assignments. This is consistent with the “no significant difference” phenomenon that is well cited in the literature (13); basically, it argues that the venue alone (online versus traditional) shows no difference in performance outcomes. In this initial study, we chose to examine individual learner performance in the technology and nontechnology realms by checking their performance on WebCT quizzes and exams in contrast with their normal classroom exams. We found no significant difference between the two approaches; in fact, we found that students who performed poorly on WebCT also performed poorly in the normal classroom exams. This suggests that students who take advantage of WebCT and perform well online improve their offline scores in a similar manner. This result also suggests that students who perform poorly in offline assignments are not bolstering their grade with online assignments. Clearly, making technological changes is not sufficient for “significant difference” and the goal for future development will be a learning environment where changes are not purely technological. Although WebCT provides a complement of communication tools, only the email and threaded discussion tools received any significant use. The real-time chat and the whiteboard were not used. The primary use of the email and threaded discussion was for communication with the class instructor about technical issues relating to WebCT and the course as a whole (e.g., my computer crashed five minutes before a deadline, or what will be on the exam) and rarely for communication between students. Students were encouraged to discuss suggested topics online or to post interesting Webbased articles, but were not required to do so. Because the students’ primary communication was with the instructor, the asynchronous email and threaded discussion were found to be most efficient for them. Students who needed immediate instructor assistance would generally meet with the instructor in person or visit the Chemistry Learning Center. Student Perceptions Several survey questionnaires were developed to capture self-report variables of interest to this study. Basic demographic information (age, gender, major, year in school, and GPA) was captured along with the quantity of prior computer experience, usage data for the WebCT quiz module, and satisfaction with the learning experience. The satisfaction variables were measured with questionnaires adapted from existing Web-based surveys (14–17), whereas the other variables were measured with surveys developed specifically for this study. The demographic characteristics of our student sample are presented in Table 2.
JChemEd.chem.wisc.edu • Vol. 80 No. 11 November 2003 • Journal of Chemical Education
1335
Research: Science and Education
Use of Quizzing Module Students were not restricted in the way they approached quizzes. As a result, we were interested in what approaches students actually used. Of 180 students responding, 65% reported working alone, 25% reported working in pairs, 5% reported working in groups of three, and only 1% reported working in larger groups. The remaining students reported seeking help in the Chemistry Learning Center. This seems like a fairly typical response from science and engineering students who have previously studied in environments where group work is discouraged. Students were then asked how they rated the convenience of taking online quizzes relative to traditional quizzes. Of the 180 students who responded, 88% reported the online quizzes to be more convenient and only 1% reported them as being less convenient. When asked to justify their answers, students responded with such comments as: “I like the option of being able to take a break if I want to finish the quiz later” and “I can work from the comfort of my room.” When asked about their thoughts on problems with the convenience, the primary concern was the availability of quiet computer labs on campus. Computer Usage Since prior computer experience might influence their use of WebCT, students were surveyed on their overall experience with computers and the way they use computers both on- and off-campus. Only 36% of students had their own computer with them during the school year; 28% of those surveyed had over one year of Internet experience, 30% had over two years of experience, and 27% had over three years of computer experience. It was interesting that at the time of this study (1999–2000), 15% of the students had never used the Internet prior to attending MTU. Satisfaction Students were asked to rate their level of satisfaction by answering a series of survey questions. Students were first asked about their perspective on the relative effectiveness of WebCT’s implementation in the course and how it compared to a more traditional course. Based on a class of 180 students, 86% of the class felt that WebCT was used effectively in the class, 75% felt that it was a more effective method of helping them learn than a chalkboard and slide-based lecture alone, 72% felt that WebCT was important to their learning, and 75% reported that the use of WebCT helped them enjoy the class more than the traditional chemistry classes they had taken previously. The major reasons cited by students were the convenience and the ability to learn from multiple attempts at online quizzes, reduced stress levels, and the improved access to materials that support the classroom discussion. Of the 180 students, 6% felt that WebCT was not used effectively, 15% felt that WebCT was less effective at helping them learn than a more traditional classroom situation, and 17% felt that WebCT was not important to their learning. The major reasons cited by students were a perceived increase in workload because of the “extra online quizzes,” and various computer-related technical issues. To gain a better understanding of motivating factors, students were asked to report on their perception of relative impact of several factors on overall learning and on confidence. The results obtained are summarized in Table 3. These re1336
Table 2. Student Sample Demographic Characteristics Characteristic
Demographics (%)
Age (years) 18
12.8
19
35.9
20
23.1
21
12.8
over 22
15.4
Gender Female
50.0
Male
47.0
No Response
3.0
Current GPA x ≤ 2.5
5.2
2.5 < x ≤ 3.0
38.4
3.0 < x ≤ 3.5
38.4
3.5 < x ≤ 4.0
18.0
Year in School Freshman
12.8
Sophomore
52.6
Junior
20.5
Senior
14.1
Major Biological sciences Biomedical engineering Chemical engineering Chemistry or physics Civil or environmental engineering Mathematical sciences Othera
26.9 2.6 17.9 2.6 21.8 7.7 20.5
a Includes geological, metallurgical, computer, and social sciences, and technical communication.
sults suggest that while WebCT has a positive relative impact on the students’ learning and confidence, its effect is not perceived to be significantly different from the effect of the professor or other instructional components. Where WebCT excels over the traditional environment is in the way students are tested. Students reported significantly greater positive impact on both their perceived learning and their confidence for online exams than they did for handwritten exams. We posit the following possible explanations for this finding: •
Students have more time to work on online exams and quizzes. Students reported spending an average of 60 minutes on a six-question quiz and 3.5 hours on a 20question exam.
•
Students can ask for help with online exams and quizzes.
•
Students get multiple attempts at quizzes allowing them to master a topic.
•
Students have ample opportunity to work with other students and exchange ideas.
•
Students receive immediate feedback on incorrect answers and can then make a second attempt at a similar question.
•
Elimination of test anxiety.
Journal of Chemical Education • Vol. 80 No. 11 November 2003 • JChemEd.chem.wisc.edu
Research: Science and Education Table 3. Relative Impact Data for Different Classroom Interventions Relative Impact on Learning (%)
Factor
Relative Impact on Confidence (%)
Positive
Neutral
Positive
Professor
88
16
Neutral
6
75
16
Lecture
77
12
9
11
65
18
17
PowerPoint
83
WebCT
78
12
5
68
27
5
10
12
75
11
14
Handwritten exams
47
18
35
31
9
60
Online exams through WebCT
79
7
14
67
8
25
Additional research is necessary to better understand the exact nature of how learning antecedents of perceived learning, perceived confidence, and motivation manifest themselves in TSL outcomes. These early results are promising, though, as the TSL environment for the first-year chemistry program does appear to improve students’ perceived learning and their confidence that can ultimately be of much importance to student long-term success and motivation. Additional Analysis As a follow-up analysis in this study, we investigated whether our sample evidenced a gender bias in the use of WebCT and computing in general. Analysis of questions pertaining to WebCT effectiveness and computer experience indicated very little difference in the responses. The major difference was that males showed an increased number of hours online relative to females and slightly more computer experience, reporting that on average they considered themselves intermediate to advanced users with about three years Internet experience and an average of between 4 and 8 specific uses of the Web per day. Females, however, reported being beginning to intermediate users with about two years Internet experience and an average of 1 or 2 specific uses of the Web per day. Comparing computer experience to level of satisfaction with WebCT (WebCT effectiveness) we found that students with more experience do not automatically have a greater level of satisfaction. We do find, however, that female students, regardless of their experience level, have a slightly higher level of satisfaction with WebCT than male students. Conclusion In this study, we assessed the effects of using a TSL environment to support first-year chemistry instruction. We utilized information technology to both automate and informate selective portions of the learning experience. Students appeared to appreciate the scheduling flexibility found in the TSL environment and took advantage of this flexibility to maximize their scores on online quizzes and exams. Our findings indicate that students perceived more positive impact on their perceived learning and confidence when using the TSL environment. We also found that students appeared to use the TSL environment as a complement to traditional instructional tools, rather than as a replacement. Although more research is needed to understand the antecedents and consequences of employing TSL environments, these early results shed considerable light on the effects of
Negative
Negative
such environments on student perceptions of learning and confidence. For the first-year chemistry program at our university, these results show promise for unraveling issues with retention and student motivation. W
Supplemental Material
Files containing the raw data and surveys described in this article are available in this issue of JCE Online. Literature Cited 1. Carrell, L. J.; Menzel, K. E. Commun. Educ. 1997, 46, 262– 272. 2. Church, M. A.; Elliot, A. J.; Gable, S. L. J. Educ. Psych. 2001, 93, 43–54. 3. House, J. D. Int. J. Instr. Media 1994, 21, 1–11. 4. Rieck, D. F. J. Chem. Educ. 1998, 75, 850. 5. White, L. F. Coll. Student J. 1998, 32, 190–196. 6. Leidner, D. E.; Jarvenpaa, S. L. Inform. Syst. Res. 1993, 4, 24– 54. 7. Leidner, D. E.; Jarvenpaa, S. L. MIS Quarterly 1995, 19, 265– 291. 8. Rogers, E. M. Diffusion of Innovations, 4th ed.; Free Press: New York, 1995. 9. Webster, J.; Hackley, P. Acad. Management. J. 1997, 40, 1282– 1309. 10. Mesher, D. Syllabus 1999, 12, 16–20. 11. Charlesworth, P.; Vician, C.; West, A. Using Information Technology to Rejuvenate General Chemistry Instruction: An Investigation of WebCT. Presented at the 220th ACS National Meeting, Washington, DC, 2000; Abstract number 339. 12. Charlesworth, P.; Vician, C.; West, A. WebCT@MTU: Revitalizing General Chemistry. Presented at the 16th Biennial Conference on Chemical Education. University of Michigan, Ann Arbor, MI, 2000; Abstract number 524. 13. The “No Significant Difference Phenomenon” Home Page. http://teleeducation.nb.ca/nosignificantdifference/ (accessed Aug 2003). 14. Project ADEPT (Assessment of Distance Education Pedagogy and Technology) Home Page. http://EMPLOYEES.csbsju.edu/ TCREED/adept/ (accessed Aug 2003). 15. GVU Center WWW User Surveys Home Page. http:// www.cc.gatech.edu/gvu/user_surveys/ (accessed Aug 2003). 16. Field-tested Learning Assessment Guide Home Page. http:// www.wcer.wisc.edu/cl1/flag/ (accessed Aug 2003). 17. Distance Education on the Web: Technology Resources Questionnaire Home Page. http://library.kcc.hawaii.edu/~illdoc/de/ survey.htm (accessed Aug 2003).
JChemEd.chem.wisc.edu • Vol. 80 No. 11 November 2003 • Journal of Chemical Education
1337
