Miles Pickering Princeton University Princeton, NJ 08544
II
Teaching Assistant Predictions of Student Performance in General Chemistry
This paper is the second in a series1about teaching assistant oredictions of grades in general chemistry. The ability to bredict grades i;s important for many reasohs. First, we need to he able to answer truthfully the student question, "How well will I probably do?" second, many inititutions have tutoring arrangements to which a student is referred by TAs. If the student in difficulty is not reliably identified, this resource will not be used efficiently. Third, such studies give us information about how lone it takes for TAs to eet to know their stud& whirh is impo&nt in any gradings&m which asks for TA ratines of student wrformance. Last. ifa TA d w s he or she will be a passive not know who w i i do well or responder to student questions rather than aggressively seeking out and helping students in-difficulty. This latter is especially important in the lab, since students often do not realize that they are ma kin^ important mistakes in technique and theretort will not ask ior help until too late. In our previous study, TA ratings of student laboratory techniques were correlated with results on a quantitative analvsis . exneriment in Columbia's freshman chemistrv laboratory course. The nveragr predictive ability was very poor, and there was a startline variahilitv from TA toTA. SomeTAs were overwhelmingly s;ccessful(90%) in predicting the third of the class into which each student would fall. Others did worse than chance. The ability to judge the student's capahilities seemed to correlate with our suhiective opinion of the TA's skill. There are many possible explanations for these results. It may he that wsts of laboratory technique are hard to predict: thus, teaching assistants wuuld do better predicting lccture exams. It mav~alsobe that if the conditions for the TA to know the studentwere more favorable (small classes, more contact hours), the predictability would he better. ~~~
~
~.
~~
Method
The study reported here was done a t Princeton rather than a t Columbia as in the previous work. There are a number of
354 1 Journal of Chemical Education
differences between the programs a t the two institutions. Classes in freshman chemistry a t Princeton are smaller. Each TA has only 20 students instead of 30, and besides the lab, he or she has a recitation with the same students. The TAs also made student homework sets and lab renorts. Hence if the ~. prior predictahility arises from simply not knowing the mldrnt well enoueh, there ghould he verv few 1'As who dt, not have time to g z to know the student'scapabilities very well by the end of the term. In this studv TA nredictions for both lecture hour tests and lah prartirals wili he made so that the prediccabilitv of both sorts of tests can hc camonred. Test grades and TA predictions were coliected for the 200 students in freshman chemistry in the fall of 1976. TAs were asked to predict the third of the class into which each of their students would fall on each of the hour tests, the final exam, and the laboratory practical examination. A small monetary award was given for the most accurate set of predictions on each test. The study was repeated in the fall of 1977, except that TA predictions for the final exam were not collected. No re dictions were collected for the snrine . .. term because the sections werr mixtures of some wntin~lingstudents and some new students. In 19'76 there were three 1 -hr tests: in 1977 there were only two. The fraction of correct predictions by the most and least accurate TA and the overall accuracy are displayed in Tables 1and 2. A c o m ~ u t emediction r was also made for each of the lecture tests. his pre&tion was done hy averaging each studrnt's percentile rank on all previous exams. N o trends were eiamined, and nu d a b a h t labratory or homework performance was included. In hoth the 1976 and 1977 academir yews, there was a eradual erowth in nredictive accuracv the TA h&ame " as -~~ acquainted wch hisiherstudeuts, but the limiting value, even for the best predictor, seems to be about 75%correct on lecture exams. I t is also noteworthy that the computer prediction
.
~
~~
~~~~
~~
~~~~
~
~
' Pickering, M. and Kolks, G.,J. CHEM. EDUC., 53,313 (1976).
Table 1. TA Predbtlons on Lecture Exams 1976
Hour Test 3
1
Fraction correct predictionsfor TA Most accurate TA Least accurate TA Computer ' X computer versus TA Week of exam
0.372 0.55 0.33
4
0.485 0.65 0.30 0.497 0.063 8
Final Exam 3 0.566 0.74 0.37 0.612 1.23 12
0.634 0.71 0.62 0.690 1.36 15
Hour Test 1
2
predictions for ail TAs Most accurate TA Least accurate TA Computer x2, computer versus TA
0.394 0.55 0.18
Week of exam
6
0.418 0.68 0.19 0.487 3.418 12
1477
Fraction correct
-
exams? In 1976 there was a marked difference, in which the laboratorv exam moved to be much harder to oredict. The TAs said'that thi's was a particularly difficult prediction to make because the task was oresented as a relative one ("Which third of the class will the s'tudent be in?"). In 1977 the study was repeated, and this time the task was presented as an absolute one ("Will the student's accuracy be better than I%?", etc.). This time there was no important difference between the accuracy of TA predictions on this test and on the l-hr test held in the lecture course the week before. I t should be noted. thoueh. .. . that the second mouo. of TAs seemed to be less skillrd generally at making predictions than the.vre\C~usvear's erouo. (This mav he rehted to the fact that there was one less f-hr test, and soiess chance to learn how to make accurate ~redictions.) However, in both cases an important minority of the TAs was unable to predict the laboratorv exam results a t a level exceeding chance. This replicates the result of the previous study. I t is therefore likely that the inability to predict these results is widesuread and not an artifact of the Columhin ~ ~ ~-~ - ~ study. It may wlll reflect the inability of TAs to spot faults in laboratorv techniaue. a worrvine - - hvnothesis. .. These iesults h'ave obvious implications for student advising. I t is clear that the student's own "track record" is a signiccantly better predictor than the student's TA. Thus, to the student question, "How well will I do?" the most truthful answer is something like, "Usually, hut not always, youf standing in class on previous exams is the best clue to how well you will do in the future.'' Similarly, tutoring should be offered to students on the hasis of test scores, not TA referral. There is also sienificant data in the literature areuine that students themGlves are good predictors of their grades.2.3 Thus the comhination of low test standine and self referral should he a good way of getting tutoring tb those who need it. The subjective prediction of student results seems to he a task of considerable difficulty. !+'here it is necessary to make such predictions, it is best to look a t the student's-Past performance. This look to the past is the best clue as to what he will do in the future.
..
~~
Table 2. TA Predlctlonr on Lab Practloal Exams 1976
1977
0.364 0.65 0.25 0.566
0.404 0.55 0.29 0.418 2
Fraction correct prediction All TAs Most accurate TA Least accurate TA Ail TAs on corresponding lecture exam Number of TAs with accuracy below 0.33 Total number ol TAs
4 10
10
becomes more accurate as the term progresses, and that the computer regularly does better than the overall TA prediction. (The individual x2 values for 1976 tests do not indicate a high level of statistical significance, hut if all the l-hr test predictions are combined, the resulting x2 indicates significance at the 90% confidence level.) This findine is surorisine because the computer has far 1es9da1ato workkith than t h g ~The ~ . T A k n o w the student, the level of his lab orrformance, his homework grades, and his motivational staterwhile some TAs outnredicted the machine occasionallv, thev were rarely consistently better than the romputer on a section hy section hasis. Clearly, the presenceof the extra dnta i n theTA's minds is an impediment to hisher predirtion ahiliries. Presumnhly this impediment is some man~festationof the well-known "halo effect." Are laboratory practical exams hard to predict from lecture ~
~
~~
~
~
- -
~
~~~
Acknowledgment
D. Monts made some useful suggestions about an early draft of this manuscript. Gadzells, B. M., Cochran, S. W., Parkam, L., andFournet,G. P., J. Educ. Res., 70,75 (1976). SHalen, M. C. and Newhouse, R. C., J , Educ. Res., 69, 219 (1976).
Volume 57, Number 5, May 1980 1 355
-
