Nelson W. Hovey and Albertine Krohn , n v c r c I , c f T 3 eo3 To PIIO. 0+3
I
I
An Evaluation of the Toledo Chemistry Placement Examination
Eight years ago the authors, being disturbed by the high percentage of failures in the general chemistry course a t the University of Toledo, began a study of this prohlem. This study' showed that t,he best single criterion for prediction of failure in general chemistry was a combination of the scores on the Iowa Chemistry Aptitude Test and an achievement test based on knowledge of high school chemistry. At that time we suggested that these two tests might be replaced by a new examination containing items of the type found to he statistically most valid on the other two tests. The advantage in this was to administer a single examination in one hour, rather than to give both tests which required two hours. The demand for reprints of the published paper and the number of letters of inquiry concerning the availability of such an examination lead to the decision to develop, publish, and arrange for distribution of the Toledo Chemistry Placement Examination (TCPE). The test was made available in the fall of 1959. Sales have totaled 9000 tests and 24,000 answer sheets in addition to 230 specimen sets. The extensive use of the present examination imposes a clear obligation on its authors to make an alternate form available for reasons of security. Presented before the Division of Chemiesl Education at the 144th Meeting of the American Chemical Soeiety, Los Angeles, April 1963.
' HOVEY,N. R., AN,) KROHN,A,, J. CHEM.EDUC.,35, 507 (1955).
370 / Journal o f Chemical Education
Before publishing a new form of the examination, it seemed appropriate to undertake a thorough evaluation of the original examination and the consequences of its use in sectioning beginning chemistry students at the University of Toledo. The examination is of the multiple-choice single response type. The time allowed is 55 minutes. The organization into parts and the method of scoring are indicated in Table 1. The IBM answer sheets were designed so that they could be scored by hand or by machine. Table 1.
I. 11. 111. IV. V. VI.
Organization and Scoring of the Toledo Chemistry Placement Examination
Arithmetic and algebra. General knowledge Formulas and nomenrlature Equations Algebraic formulations Chemical problems T
~
~
~
I
15 25 10 6 6 5 67
RightsX1 Rights X 1 Rights X 1 Rights X 2 Rights X 3 Kights X 4
15 25 10
12 18 20
inn
Readers desiring additional informa.tion on the TCPE are advised to write the Research Foundation, University of Toledo, Toledo 6, Ohio. Specimen sets are available at 506 each. Testa are priced at $5.00 per package of 26 and answer sheets are 46 each. The new form will be wailable for late summer and ertrly fdl testing.
The Statistical Analysis
All answer sheets for Toledo students for the academic years 19594, 1960-1, and 1961-2 were arranged in order of decreasing score. These tests had been administered to prospective freshmen with at least one year of high school chemistry who planned to take the regular general chemistry course. By removing a small number above and below the median, the number of answer sheets was reduced to 1000. These were then used to establish norms (percentiles), the standard dwiation, and the reliability coefficient (Table 2).
decreased from over 30% in 1954 to approximately 10% per year for the past three years. The deficiency course for those scoring below 40 on the placement test has been drastically revised since our 1958 report. The present course (Chem 109) cousists of three hours of lecture and quiz and one three-hour laboratory period per week for one semester. Students entering without credit in high school chemistry are also required to take this course. Students in hoth groups must earn a minimum grade of C in Chem 109 hefore being admitted to Chem 117. Predicting Achievement
Table 2. Norms for the Toledo Chemistry Placement Examination
Pewentile rank
50 St,nndnrd deviation
Score
Percentile rank
Score
5.5 =
16.8
Reliability coefficient = 0.921
The reliability coefficient was estimated by means of the Iiuder-Richardson formula number 21.l (This is t h reliability ~ of the test itself. Correlation of these scorps with actual performance in the course is given later.) The highest 27% and the lowest 27% were used to ohtain the percentage in each group which chose each of the four possible responses to each of the 67 items. U&g these data, the values for the item-difficulty index and the item-discrimination index were obtained by use of the Item Analysis Table published by the Educational Testing Service.8 A summary of the item analysis is available to users of the test on request. The reliability coefficient of 0.921 is entirely satisfactory. The spread of responses in the low group indicated that the incorrect answers (distractors) were effective. The item-discrimination indexes are very high. Of the 67 items, 20 are outstanding (discrimination index r = 5(t69), 25 are very good (r = 4049), 18 are acceptable (r = 30-39) and only 4 could he rated unacceptable (r less than 30). I t is somewhat shocking to observe that the least discriminating question involved recognition of HNOa as a n oxidiz~ngagent. In the high group 31% and in the low group 34y0 chose "reducing agent" as the correct response! The TCPE has proved to be very satisfactory a t the University of Toledo for the purpose of determining which students should be denied direct admission to general chemistry. The cut-off score of 40 was low enough to insure that very few students capable of earniug a C grade in the first semester of general chemistry (Chem 117) are denied direct admission to this course. The number of F grades in Chem 117 has KIIDER.G. F..AND RICHARDSON, M. W., P~ychometrika,2, 251 (1937): 8 FAN, C., "Item Analysis T;tble," Educational Test.ing Servire, Princeton, N. J., 1952.
We also have data on the reliability of the TCPE as a predictor of achievement. We realize that motivation plays such an important role in determining course grades of those above the cut-off point that a high correlation between the TCPE scores and grades in Chem 117 is unlikely. Educational psychologists state that the maximum corlel,ation between placement test scores and college grades is probably 0.7, even with a perfect test battery."rediction of a single course grade by means of a single test is even more difficult. To obtain some measure of the motivation factor as related to the students' professional goal, we divided the 900 students whose records were used for this study into five groups based on the program in n hich they were enrolled. We also sought to determine Thether there is any significant difference in the chemistry grades of students from different geographical areas. Since our present placement program in general chemistry went into effect in 1958, we have kept a complete record of every student enrolling in Chem 109 (the deficiency course) and in Chem 117 and Chem 118 (the college-level general chemistry courses!. From these records for the academic years 1959-60, 1960-1, and 1961-2, the records of 900 students were selected for this study. (After eliminating the records of foreign students and all who had taken Chem 109 before enrolling in Chem 117, about 930 records remained. By removing records of students who dropped out for reasons other than to avoid failure, and by withdrawing a few records from the middle range, the number was adjusted to exactly 900.) The grade distribution for these students was next obtained. The 900 placement scores were then assigned letter grades t o give approximately the same distribution as for the grades actually earned in Chem 117for the three-year period. The letter grades earned by the 900 students in Chem 117 are compared with their grades on the TCPE in Table 3. We see that 45.lyO of these students earned the grade predicted by the test. The correlation coefficient based on this table was estimated to be 0.54. We can predict that 83% of those who score above 84 on the placement test will rate A or B in Chem 117 and 68y0 of those who score below 46 will rate D or F. Our conclusion is that, in combination with other criteria, the TCPE should be of value in selecting students for an Honors course in General Chemistry. I t also shows that prediction is less accurate in the middle score ranges where motivation and other outside factors have a greater impact on the student's performance. 4 TRAVERS, R. M. W., "Educational Measurement," Maemillan Co., New York, 1955, p. 65.
Volume 40, Number 7, July 1963
/
371
It should be kept in mind that the TLPE mas not designed to predict achievement, but rather to separate those who were capable of passing Chem 117 from those who were not. This it has done, since nearly 95% of those who scosed above 45 on the placement test passed the course (90% of those above 40). Our previous study indicated that only about 10% of those below the cut-offscore of 40 would have passed the course had they been admitted. I t is interesting to compare our experience with the test with that of 11 Junior Colleges in California who have been using the test for the past two years? Their data were in different form, so the Toledo grades were re-tabulated for comparison as shown in Table 4. About 65% of the California students who scored below 40 in the TCPE failed to pass the course. No direct comparison call be made with the Toledo students in this range since they were excluded from the regular course. I t is not surprising that the correlation in the case of the California schools is not quite as good as for Toledo. Grading practices would be expected to vary more in 11 different schools than in a single department. Our somewhat limited study of the motivation factor indicates that neither professional goal nor high school location are very significant in relation to achievement as predicted by the placement score. We found that chemistry majors and chemical engineering students,
although they had higher test scores than the other groups, did not actually earn as high grades as might be expected. The pre-pharmacy group, on the other hand, had the lowest average on the TCPE hot earned grades higher than those predicted. The pre-medical group also tended to perform higher than their placement scores predicted. We suspect that motivation is so complex and so personal that any statistical approach one might devise will yield very little useful information. Summary and Conclusions
Although a single one-hour placement examination such as the Toledo Chemistry Placement Examination predicts with satisfactory accuracy which students should be deuied direct admission to a college level general chemistry course, it is of much less value as a predictor of achievement in the higher score ranges. The examination could be used as one criterion in selecting students for houors programs in general chemistry. I t is unlikely that any student scoriug less thau 70 on the TCPE should he considered for such a program, since only about 15y0 of those below this score made A or B grades in the regular general chemistry course. On the other hand, it would be unwise to automatically accept all students with high placement scores in light of the fact that only 49y0 of those in the A placement range (84100) actually earned an A grade in the general chemistry course. Based on the statistical analysis, it is our conclusion that the TCPE is statistically sound and that no drastic revision is needed in the preparation of an alternate form.
6 H ~M., ~ AND ~ , REIFSNYDER, H., "Grades in Chemistry 1A Related to Toledo Chemistry Placement Examination Scores," private communication, Mt. San Antonio College, 1962. Table 3.
TCPE score
4M5
Letter Grodes Earned in Chem 1 17 versus Grodes Predicted b y TCPE Score.
Letter grade TCPE
Numher in range
F
93
0.
0
30
9W
74 8.2
161 17.9
405 45.1
Totals Per Cent
Table 4.
TCPE score
---CasesCal.
9(t100 80-80 70-79 60-69 50-59 40-49 3039 2S29 10-19 0-9 Totals
18 87 260 429 6'38 481 253 02 16 1 2275
Chem 117 Grades
--A
/
D
F
30
33 91 10.0
169 18.8
Per cent predicted correctly
33
35.5%
406
45.1%
A ~omporironof California and Toledo Grades versus TCPE Scores. Tal. 36 80 146 203 243 192
900
-%
Cd.
Journal o f Chemical Education
-*
A, B, C-Tol.
94.4 85.0 76.1 65.1 49.0 36.8 20.0 0.8 0.0 0.0 49.2
+ 372
C
B
Number predicted correctly
Cal.
Tol.
94.4 96.2 86.3 80.8 66.7 40.0
21.8
2.8 3.8 10.3 15.3 22.6 33.4
71.2
1'3.2 17.4 37.5 0.0 16.5
18.8
+
0.0 3.5 7.7 12.1 20.1
h D-
+
----%
Cal.
5.6 11.5 16.2 22.8 30.9 41.4 61.7 72.8 62.5 100.0 34.3
F--
Tol. 2.8 0.0 3.4 3.9 10.7 26.6
10.0
