Using- Computer-Assisted Personalized Assignments for Freshman Chemistry D. J. Morrissey Department of Chemistry, Michigan State University, East Lansing, Mi 48823 and National Superconducting Cyclotron Laboratory E. Kashy Department of Physics and Astronomy, Michigan State University, East Lansing, MI 48823 and National Superconducting Cyclotron Laboratoly
I. Tsai Department of Computer Science, Michigan State University, East Lansing, MI 48823 The number of computer software modules to assist undergraduate chemical education has grown explosively. An appreciation for the very large number of specialized tutorial. simulation. and visualization computer modules in che&stly can be obtained by ~cannin~specialized publications such a s the Journal of Chemical Education: Software, edited by J. W. Moore and J. L. Holmes, or the Computer Series edited by J. P. Birk and the Computer Bulletin Board, edited by R. H. Batt, i n this-the Journal of Chemical Education. The traditional instruction mode of lectures with assigned problem sets remains the central means of instructing the bulk of our students and often is augr Because chemistw has mented bv the c o m ~ u t e modules. a quantitative aspect, though, the students must be challenged to achieve and demonstrate certain levels of understanding. Such individual challenges can be provided by individual computer-based instruction modules but more often is done with assigned problem sets and examinations. Thompson has described the development and utility of a computer-based problem-set system for juniorlevel chemistry ( I ) . The recently developed Computer-Assisted Personalized Assignment (CAPA) system offers a powerful way to apply computers to assist instructors and students i n the traditional framework of lectures and assigned problem sets without necessarily forcing students to use the comuuter svstem ( 2 ) . CAPAis a system to create personalized, that is, individual numerical and multiple-choice problem sets to be distributed to and solved by'the studeits. The instructor uses the system to create problem sets for the class material, much as would be done in a traditional lecture class, with solutions and hints. An individualized problem set i s printed for each student i n the class with'the student's name a t the top and a personal identification number (PIN). The students have the option to login onto a host computer and verify their answers to any or all of the problems without oenaltv before the assienment is due. (We believe the importance of the printed problem set and the "optional" nature of the computer is large, see below.) The computer generation of individual numerical problems for students is not new, in fact. i t has a long tradition a t Michigan State (3) and, more recently, elseGhere (1,4, 5). The system differs significantly i n its features, implementation and philosophy as discussed in reference (2). We decided to use the CAPA system of directed problem solving in the freshman chemistry for several reasons. We wanted the following.
.
To provide students with the opportunity of timely (and accurate) feedback on problem solving. Students often work an
homework problems outside of the usual nine-to-five workday. If students solve problems incorrectly in the traditional framework of written assimments. the" mieht not be told student.
To provide a realistic and individually challenging opportunity for collaborative learning. Collaboration among peers is very important far learning new material ( 6 ) .The peers should find and reinforce the correct solution. Also, the individuals should be able to demonstrate their own competence. h shift the role of judge to the computer, if only in perception, and away from the instructor. The instructor does not generally know the correct answer to the individualized problems. After helping sdve the student's problem, the instructor can suggest "verifying the result with the computer." From a practical standpoint,to reduce the tedious grading of individual assignments and tc see ifthe CAPAsystem can be sealed to large-sized classes. Orhrr ntrrnrrlvr frnturrs nf t h e system are r h u t rt reward. d l . l u m ~work and c;m help reduce t h e imper-.malnature of instruction in a large college class. We report the first application of the CAPA system, developed with a (small) nonscience physics course, to a large freshman chemistry course. he-el-ass, Chemistry 152Principles of Chemistry-, is the second semester of freshman chemistry for science and engineering students. The material covered included a review of stiochiometry, an introduction to chemical kinetics. thermodvnamics and eauilibrium. The class enrollment ktabilizedat approximately 250 students. Abrief description of the system is provided i n the following section. The statistical results and discussion of the student usage also is provided. In this report, we will not attempt to analyze why the system appears to have succeeded but will focus on the system's operation and some representative student reaction to the system. Description of CAP4 The features of this svstem have been described previously (2) and so we willgive only an overview here. CAPA is a n integrated system developed to create individual assignments for students in quantitative or semi-quantitative subjects. I t is a tool that relies on modern networked computer technology, and its success in a particular setting clearly depends on the skill and dedication of the instructor who must "create" the problems sets. The present model system used in the chemistry course used a single NeXT station computer to handle all the problem sets and was connected to the campus ethernet system for all the Volume 72 Number 2 February 1995
141
remote student interactions1. The students onlv needed W 1 0 0 terminal emulation to connect to the system, see below. The nhilosonhv is that the student is eiven a nrinted assignment to work from and computer usage is optional; therefore, the computer display contains only the minimum information (the printed text) to remind the student of a given problem anddoes not contain any graphical images. etc. This keens the cost of the eauinment on the side with the large muitiplier (the numhe; o i students) a s low a s possible. The proffered answers are judged to be either "correct" or not, there is no partial credit. A distribution of scores is built up over time by having the students answer a large number of problems. As the first step i n using the system, the instructor writes a prototype problem set in a language that includes substitution of random numbers for variables and codes the solutions based on these variables. The nroblems are written in plain text and tha solutions to each problem are coded into the file. The CAPAS\stem orowdcs a tad to edit and preview the problem set on the NeXTstation but any word processor can be used and the files ported to the lorrin-host. After the problem set and solutions have been debugged, hlgh quallt) prmtcd vcr.i~on.sartacrtvited fi~reach student and dlstnbuted in class The level of differences among these personalized assignments is entirely controlled by the instructor when the prototype problem set is written. I t is relatively easy to have several variables with large ranges so that literally no two students have the same answer. All of the problem sets for the course were written during the semester. This was a iarge task but once accomplished the problem sets are trulv reusable. The instkctor can include hints for each problem into the prototype set that become available to the student after a n incorrect answer has been input via remote loein. Because the assignments are unique; students are enciuraged to study together, to discuss and understand the concepts, and yet each must still do hisher own work to obtain their individuallv "correct" answers. The order of mnltinle choice problems-can be randomized so that the studeits must convey the correct answers to their friends not simply the correct letter choices. We routinely have multiple ehoice questions that include a n unspecified numher of correct choices in a personally randomized order. All the correct statements must be identified, so combinatorial factors work aeainst svstematic euessine. Random euessing, particularly of numerical answers, is not generally useful. The student mav elect to turn in written answers to each problem on the d"e date to he graded by a teaching assistant in the traditional manner. To ease the grading task, the TA uses a computer tool that can display the personalized answers and modifv the comouter file with the erades. On the other hand, the student'may choose to enter answers directly via the campus network and is immediately informed that the proffered solution is correct (or incorrect). I n this case, the student may try a problem a s often a s helshe desires before the due date with NO penalty for incorrect answers. This feature gives students of different skill levels the opportunity to achieve the goals of the course without being judged during the learning process. Because the computer data entry and on-line grading has been optional, no student is "forced" to use the computer,
-
-
'The recenl news tnal NeXT compJler haroware wll no onger oe prodJceo snoL d not pose a proomem lor CAPA Lsers beca~setne system does not require any specific features of NeXT computers. The NextStep operating system is now available for IBM-PC's and the same code can run on these machines. Further, the code can be adaoted for other machines with a moderate effort. 142
.
Journal of Chemical Education
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
. ,
Problem Sets --Chem152/ Spring 93
-
.-
-
.
Tues.
Wed.
Thun.
Fn.
Sat.
Sun.
Mon.
Figure 1. Number of "cpu-minutes"used by students per day of the week for each assignment. The assignments were due on Tuesdays and corrections to the two in-class exams were due on Friday. hut of course, nearly everyone "chooses" to use the computer. The students were given one week to work on each problem set. The written solutions were due in lecture (at 11:30 a.m.) but for various logistical reasons we decided to close the comnuter entrv a t 08:OO a.m. on the morning the nrohlems were due. After the course was underway, on the order of one of the students turned in only written solutions to the entire problem set without using the computer. Each week it turned out that different students had not used the computer for various reasons. We also accepted written solutions to individual problems to be added to the computer scores, and a s many a s five of the students would turn in a written solution to a single problem each week. When asked, these students said that thev had realized how to solve a challenging problem only aftkr the computer deadline and wanted to get credit for findine the correct solu-
-
..-... Students were given detailed instructions for connecting to the host machine us in^ the network. The Michirran State campus is equipped with numerous, easily accessible personal computers (PC's) with terminal emulation software located invarious places: the large dormitories, many buildings with dedicated labs including the chemistry building, and other places such a s the computer center and the student union. A number of students have their own PC's, equipped with modems, and can obtain access to the svstem over normal nhone lines. I n the state of Michiean. the ~ ~ ~ I ~ / ~ i c h ~ e t allows ' s y s tconnection em to any ma: chine on the MSU camous ethernet from almost anvwhere " ~~-~ in the state by a local telephone call. The login records iudicate that approximately 40 different locations were used by the 250 students in class. A few students encountered some difficulties in their initial attempts to connect to the systt:m that were caused by nut p n ~ p t d ysetting the tcrminnl emulation specifically to V1'100. Students that uwd the 1aborator.y in the chrmistrv buildma wcrtr nrovided with a menu system to automa~icallylogin to the proper host. Some students suggested that it would help if the computer laboratories would be open all night. The distribution of comnuter time used bv the students during t h e semester, shown in Figure 1, peaks very strongly the day before the assignment is due. Note that even with this large numher of students, the absolute number of cpu-minutes per day used by the students remained below 200. Closer scrutiny of the login records showed that the students were using the computer on Monday evening and early Tuesday morning; thus, the "load" occurred outside the normal working hours. The
k
1
Apr/W, Three Components
C
0
"
"
~
W
~
'
" ~ 10
"
" ~ 00
"
"
80
~
'
" ' 100
~ 0
LOGIN T i m e (Minutes) Figure 2. Distribution of the number minutes that the students remained logged onto the host computer. The distribution was best-fit by a sum of three exponential curves with half-times of 1.9. 7.6, and 29 min, shown by the solid curve. computer usage for the exams, discussed below, was less sharply peaked. The students were specifically instructed to use the computer to check their answers and not to "hang" on the com~ u t e while r thev solved the ~ r o b l e m s We . felt these iustructions were necessary due to t h e relatively large number of students that required access to the single computer. The distribution of the number of minutes that the students stayed logged into the computer i s shown in Figure 2. The distribution falls sharply with time but extends off-scale to many hours. This distribution was fitted to the sum of three exponential functions with a nonlinear leastsquares technique; the resulting half-times were 1.9, 7.6, and 29 min. We attribute the three groups to students that check only one or two answers and get off, those that work on a few ~roblems.and those that hane on the comwter while the; work on the entire problem let. More tho;ough analvsis of the loein records is ~ o s s i b l ebut has not been done. The distribution of the number of the students that are logged into the computer was sampled early in the semester and toward the end. These distributions, shown in Figure 3, are well described by single exponential functions with a surprisingly large mean number of simultaneous users (-5 later compared to =8 earlier). We believe that these numbers are so large because the bulk of the students use the computer system a t the same time the night before the problem set is due. The computer system keeps separate records of all the problems that each student has correctly solved and the pattern of correct answers obtained each time the student logged in. There i s no record of incorrect responses. Students can also display their cumulative'scores during a session. Some analysis of this data for the chemistry class is given below.
-
Assignments
The problem sets are output through LATEX, a document preparation language that is well known for its ability to express mathematical symbols and equations. CAPA with LATEX readily allows the inclusion of complex graphics prepared as PostScript files that can he generated, for example, with the well-known Mathematica system or with various drawing programs. The powerful features of the LATEX document preparation option thus provides a n excellent wav. of oroviding materials . .. high .. qualitv . . printed . for the students. The problem sets also can he printed out usinz ASCII characters u,ithout using I.ATF:X. This w ~ u l d allow instructors to use the system without knowing the
'
'
'
'
l
10
'
s
l
l
l
.
PO
.
l
-
l
I 30
Terminal n u m b e r Fioure 3. The distribution of numbers of students loooed onto the c&p.ter a1 one tme taken near tne beglnn ng an,:^ 1993, and near the end. Aprd 1993. of the c o m e The 0 slr bdllons are approx male y exponent a wdh mean n.moen of st~oenlsof 7 8 and 5 0 students, respectively. LATEXlanguage. As part ofthe development of the system we expect that the capability to print with other document preparation packages will be added. I t should he noted that a significant effort is required from the instructor to prepare correct prototype source files before the assimments are distributed to the class. The instructor muscbe certain that these coded solutions are complete and correct before the assignment is printed for the 'ntire class. Prior to printing, t h e assignments can be previewed on the NeXT computer i n the exact form that they will appear on paper. In addition, single assignments can he printed and solved by the teaching assistants as a check on the coding. I t is essential that the answers be correctly coded, a s a s h e s of errors might shake the students' faith that the computer has the correct answer. The inclusion of high quality hints can be important a s this permits the assienment of more challeneine ~ r o b l e m sbecause the instructor can "instruct" outs%euo~class time or oftice hours. Comdicated problems can be broken down into pieces so that the &dent can be led through a series of steps to the ultimate answer. In addition, logical functions can be used to chain the correct answers to multiple-choice questions to the numerical results of previous problems chat contain random variables. The timeliness of the grading of students' work should not be underestimated. The students are informed immediately a t the time that they are thinking about the problem that they have obtained the proper answer. In the traditional method of turning i n written problem sets for hand grading by a teaching assistant, the feedback might not be given to the student until a s much a s a week has passed and the class has moved on to new subject material. No partial credit i s given; an accuracy of numerical answers in percent can be specified for each problem allowing a little flexibility, while questions requiring the selection of all the correct answers from a list must be completely correct. Working out the assignments, where any error includine a missed decimal ~ o i n lead t to an "incorrect" response from t h e computer generally resulted i n some frustration but eventually improved calculating skills. On the other hand, the computer's simple response that a n answer is "incorrect" does not indicate to the student if they are on the right track or completely wrong. At present this information has to come from the hints or from consultation with the fellow students, teaching assistants, or the instructor. Many students requested that the system iudicate when the answer is close or way-off. We will add a n additional response, close, to t h e system i n which the closeness criterion can be determined by the instructor.
-
Volume 72 Number 2 February 1995
143
Chemistry 152 Poll 6 May 93
Chernislry 152 Poll 16 Mar93 30 -
.8 3 Cc
-
Tj
20-
E
L
10
0
-
I
I MSh
Shorter
Sam
Longer
ML
MSh
I Shorter
Same
Longer
ML
Figure 4. The distributions of answers to the question: "The time I have spent working on assignments using the CAPAsystem is probably (a) much shorter than, (b) shorter than, (c)about the same as, (d)longer than, (e)much longer than, forTA-graded assignments"from polls taken a few weeks into the semester (left)and at the end (right). CAPA Use in Examinations We believe that mid-term examinations are a n integral and important part of the learning experience. The system was used to generate individual mid-term examinations analogous to the problems assimments. Students were givena separate 'nswer sheet to record their answers to individualized problems during the examination period. Most orohlems had numerical answers: two auestions re-
nature of the exams, seating students in close proximity did not generate anv concern with cheating. a n imoortant practicai detail for farge classes. As a n additional precaution, the students were assigned to seats randomlv.1 The students kept the question sheet and, after the examination, were given the opportunity to use the computer to check their performance and thus find out which problems they had answered correctly. The CAPAsystem printed a n individual answer sheet for each student and the teaching assistants scored the examinations by comparing the written answers to each answer kev. In addition, the students were &ven the ouoortunitv .. to enter the cwrect answers to tlien'exani quc.;tlt~~~s on the wrnptrter wirhin 3 days, w ~ r hthe inducement of 1 :)credit fir cmwcting mii.;(d pn~hl(,m.z..Again, ts;t.ntinIIy all ol'the studcllls w:tilcd th(mselvei of this ~~ppi~rtunit): wirh the great majority getting a perfect comp;ter score.. This partial credit scheme generated many student-hours of concentrated effort to learn the material on the exam which would otherwise might not have taken place! I t was particularly satisfying to the observe the growing record of student effort to correct mistakes on the exam, shown for example in Figure 1. It should he noted that a number of students felt that the CAPA examinations were too difiicuul Several complained that they were not being tested on their chemistry knowledge but on their calculating skills. There is no doubt that personalized examinations are more difficult than multiole-choice examinations. nartlv because there is no feedback to the students that (hey cave completed the problem .nrouerlv. . "Therefore. we would like to trv the CAPA svstem in on-line examinations. The students would have the opportunity to know and correct the classical 'dumb mistakes and the instructor would have the grades for the exam a t the end of the examination period. Another small benefit i s that the computer would force the students to complete their work a t the appropriate time. We have not been able to try on-line examinations in this class for practical reasons, for example, one NeXTstation can not simul144
Journal of Chemical Education
taneously handle a 250-student class and the computer laboratories tend to he widely distributed and small (approximately 20 seats each). Discussion The CAPA system of personalized problem sets was introduced to the students in the very first lecture. The class list was obtained from the registrar a few days earlier and a personalized assignment was prepared for each student on the class list. The majority of the students in the class were second-semester freshman with a relatively small amount of experience in a large university environment, but the general reaction to the idea that the professor had prepared individual assignments with their names on them was utter disbelief. On entering the lecture hall, the students were asked to find their paper a t the front ofthe room and a significant number just grabbed any paper. The students were asked to check the name a t the top and return wrongly-chosen papers. A moderate number of assignments had to be reprinted (-15%) because the original could not be found. In one of the worst cases, three papers belonging to other students were retrieved from one student after he complained that he could not find his. The assignments have a PIN number keved to the course. the student number, and the assignmentnumber; thus, taking another students' assignment does not allow computer access. The students quickly realized this feature of the system and subsequently did not take the wrong assignment. Throughout the couise, the assignments &at were not picked up provided the instructor with specific information on those students who were not coming to lecture and not working on the material. When the students understood that they could use the comuuter svstem to enter their answers and that thev wouid not be penalized for the number of incorrect attempts, the typical reaction was again disbelief: "You mean I can get ali'the problems right?!" The distribution of scores was very skewed with a median above 90%, the average dropped very slightly with time partly due to the influence of the small but systematically increasing number of students that gave up doing the work. These science and engineering students immediately realized the potential of the system and used i t at a high rate more readily than the nonscience students in the original trial course (21. The average number of times each student logged into the system was surprising constant, between four and five, indicating that the students put significant effort into obtaining the solutions. As might be expected, the detailed records indicated that some students managed to get all correct an-
I
Chemistry 152 Poll 40
.s cc8
-
30
-
Cheminrv 152 ~oii
16 Mar 93
-
a
mb
10
-
0
(ON)
Negative
I
lnd'lnennt
Helpful
OH
Figure 5. The distributions of answers to the question: "As a learninglunderstanding tool. I find using CAPA (a) Quite negative, (b) Somewhat negative, (c) Indifferent,(d) Somewhat helpful,(e)Quite helpful"from polls taken a few weeks into the semester (left)and at the end (right).Note that in the second 0011. five choices were aiven for svmmetw about the "Indifferent"choice. In the first poll 15%of the students chose Neaative or Quite negative. and in the second 6011;12% chose either S~mewhatne~atiie swers quickly, while others entered the correct answers over several days. A few students said that they entered incorrect answers in order to get the hints before trying the problem sets. There was no penalty nor record of this, but these students clearly put themselves a t a disadvantage for exams. The first assignment which was purposely short and relatively easy including several calculator-proficiency exercises in order to acquaint the students with the system. Polls were taken toward the middle of the semester (16 Mar 93) and a t the end (6 May 93) in the same format as that used for polls of the students in the first trial (2).The polls were unannounced and included three multiple choice questions and two requests for written comments. The students responded to the first question, "I have used the CAPA system (a) a t almost every opportunity, (b) Sometimes, (c) Not a tall" in the ratios of 10:l:O. The distribution of student responses to the second question, "The time I have spent working on assignments using the CAPA system is probably (a) Much shorter than, (b) Shorter than, (c) About the same as, (dl Longer than, (el Much longer than, for TA-graded assignments" from the two polls is shown in Figure 4. The students clearly feel that they are working longer (harder?) to complete the assignments. The third question was "As a learninglunderstanding tool, I find using CAPA (a) Quite negative, (b) Somewhat negative, (c) Indifferent, (d) Somewhat helpful, (el Quite helpful" and the distribution of choices is shown in Figure 5. The results were overwhelmingly positive, approaching 7 to 1in being helpful. The written comments reinforced the favorable response and gave some clues to the small negative response. The data entry format was very strict for answers that required a specific number of significant figures. The level of emphasis that should be given to significant figures in expressing numerical values has been the subject of conside r a b l e discussion a n d differs among instructors. Thompson has discussed some of the pitfalls and empahsizes that there is "no single adaquate significant figure criterion" in a recent article in this Journal (7).The student login code will be modified to test for the significant figures separately from the numerical value in the same sense as the difference between accuracy and precision. I t is interesting to note that the students did not change their opinion of the system with increased use, this was also seen in similar polls of the students in the physics trial (7).
Instructors' Observations The following comments are based on the experience usine the CAPA svstem and are. to a certain extent. anecdot a r The newly ieveloped systdm had been used once before with a class of ao~roximatelv 90 nonscience students. This .. was the first large-scale test for science and engineering students. All of the problem sets had to be created for this class. The time Dressure under which the creation of new. correct and teste'd problems sets for a large number of students was large. The CAPA system is open-ended in terms of application and particularly easy for students to use. Significant effort is reauired of the instructor to create new roblem sets. By tge same token, the problems sets can be refined and reused the next time the course is offered. An annoying problem was that the problem set must always use the exact values of constants, thermodynamic data, etc. from the same sources that the students are expected to use (i.e., the textbook). The students can be brought to focus on the important types of problems and can be honestly encouraged to work together. Occasionally students indicated that they were having difficulty getting the computer to accept their answer. They would start the discussion by saying that they "did the problem the same way a s their friend" but the answer was deemed INCORRECT by the computer. The problem was usually that the student did not do the problem in exactly the same way. The teaching. assistants can be asked (or reauired) to do the assignme& using the login system befdre they are printed and distributed to the class. The login record from the host computer can be used to verify compliance. This provides a needed check on the coding, a check on the detailed understandine of the teachine " assistants. and the opportunity for the teaching assistant to encounter some of the students' problems. The three teaching assistants developed a rivalry similar to that of the students to get a perfect computer score. The fact that the "Computer" is the final arbiter of what is "correct" or "incorrect" is very important. This can help create the impression that the instructor is not the judge, rather a friendly coach helping the student. The teaching assistants honestly can tell the students that they are all in exactly the same position because they had to enter their own solutions to the problem sets. Last, but not least, the on-line knowledge that students were working hard, particularly a t the correction of missed exam problems, was a source of encouragement and was
-
Volume 72 Number 2
February 1995
145
tangible evidence of the efforts of the dedicated and interested students. All too often, the voices of the disaffected and complaining students ring loudest. Conclusion The CAF'Asystem has been shown to work well in a large (traditional) freshman chemistry class. I t has received remarkable use and approval from students, and motivated them to work more diligently. I t requires the instructor to write and test problem sets for each particular course and a t the same time allows the instructor to emphasize and focus student effort on the important problems. The optional nature of the computer interaction has many advantages, some of which are perceptual but, nonetheless, important. Several additions and changes are planned i n the near future to enhance the ease of use and the capabilities. Implementations of this authoring software for other computer systems are being considered. We have been excited by the student reaction to the system and its apparent po-
146
Journal of Chemical Education
tential as new way to apply modern computer technology efficiently i n the framework of traditional instruction. Acknowledgment We wish to thank Dean F. Hoppenstaedt of the College of Natural Science for his continued support of this project. The efforts of the graduate teaching assistants, W. Hartmann, C. Powell, and S. Tjahahadiputra, to check scrupulously the problem sets before they were distributed to the students were extremely important and especially helpful. Literature Cited 1. Thornwon, H.B. J. Comp Moth. Sci. Teach. IS32, 11(11 , 63-73. and references
7. Thompson, H: B. J Chsm.Educ I W l ,68.4MC102.
