0. G. Gowenlock Heriot-Watt University Edinburgh, Scotland D. M. Mclntosh and A. W. Mackaill Moray House College of Education Edinburgh, Scotland
I ACS Standardized Exams versus Conventional Papers at a British University
I
Heriot-Watt University was chosen as the object of this study, and it is not expected that this University varies significantly from other U.K. universities. Degree examinations take place a t the end of each academic year (usuallv in June) and consist of conventional theorv . .Dauers . in eaEh of the 'major branches of chemistry. In the majority of universities. candidates are afforded a wide choice of questions. It has already been shown that these traditional examinations in chemistry at both school and university levels are functioning in a hitherto unsuspected way.* Two conclusions were drawn from this work. First, present university degree examinations are distributing their marks over the ohiectives in a different wav from school examinations. second, new developments inschool curricula are resulting in an examination policy which includes more questionswhich test higher ir%ellectual skills than recall. Our understanding of the nature of chemical ahility appears to be undergoing revision among examiners at present, with positive policy changes and the introduction of other techniques. One such technique, whose use in Britain has been primarily restricted to pre-university examinations, is ohjective testing and in order to obtain some further indication of the success, particularly of final university examinations as measuring instruments, it was decided to administer three ohjeqtive tests which had been .constructed with specific objectives in mind. The tests chosen were the American Chemical Society Co-operative Examinations in Inorganic Chemistry.(Form 19611, Organic Chemistry (Form 1965-0) and Physical Chemistry (Form 1964P). (For ease of reference these have been coded M.C.Q.1, M.C.Q.II and M.C.Q.111, respectively.) These papers were studied hv the staff of Heriot-Watt Universitv, judged to he relevant to the work of final year students-and administered in the academic years 1968-69 and 1969-70. It was felt however that the M.C.Q.111 was restricted in character in that its coverage of physical chemistry was related more towards classical thermodynamics with less attention paid to kinetics than was the case with the HeriotWatt course. Each item was classified as a "recall" item (A) or a "higher-processes" item (B) according to the type of intellectual ability it tested.2 The facility value of each item was obtained from pre-test data supplied by Dr. T. A. Ashford of the University of South Florida. The mean facility value was worked out for category A items separate from category B items, when it was found that for the organic paper only, the "higher processes" items were on a slightly different difficulty level from the recall items. Correlation coefficients were worked out between the individual tests and the scores of a verbal reasoning test (Moray House Verbal Reasoning Test Adult I) and Moray House Space Test Advanced I, and the scores obtained in the conventional final papers in organic, physical, and inorganic chemistry. These are shown in Table 1. (The
Table 1. Correlation Coefficientsa 1968-69
%
01.8
Phys 76 74 Inar V.R.I.Q. -01 Space -25 27 M.C.Q. I 32 M.C.Q. I1 M . c . Q . ~ -16 Org Phys
Phy
In 15 -08 30
67
-07 -36
20
19 -20
43
-08
V.R.
25
47
53 37
47
50
I
75 38
I1 36
m
Org
Decimal point omitted. Table 2.
Standard Deviations for Conventional Papers and Obiective Tests for 1969
Subiect
Conventional muers
Inorganic Organic
10.32 15.02
Phvsiral
Objective test
1r1.56
Moray House Space Test measures the ability to handle spatial concepts which the'constructors assumed to occur in such subjects as engineering drawing, geometry, and geography. It is apparent that chemistry also makes use of spatial concepts.) The 1968-69 figures show that a better relationship exists between the objective tests and the two "intelligence" tests than between the conventional papers and the "intelligence" tests. This was surprising since a higher correlation between a conventional examinat~onand a test of verbal reasoning might have been expected. This relationship exists in 1969-70 only,so far as the verbal reasoning test is concerned. The coefficients for the space test with M.C.Q.1 and M.C.Q.11 are not significantly different from zero. On this evidence it cannot therefore he said that these objective tests are necessarily better instruments for the measurement of chemical ability; indeed it might be argued that such tests, like some conventional papers (accordine to Smith-?.., are biased towards the student of hieh verhalability. 'Gowenlock. B. G.. MeIntosh. D. M.. and Mackaill. A. W.. Chem. i n ~ r i t .6,341 ; (1970). ZBloom, B. S., "Taxonomy of Educational Objectives," Longmans, New York, 1956. %mith, I. M., "Spatial Ability," University of London Press, 1964.
Volume 50, Number 2,February 1973 / 139
The standard deviations for the three objective tests were worked out for 1969 and compared with those for the conventional papers. The results show (Table 2) that a greater spread of marks is obtained with the objective papers than with the conventional papers. The increase in standard deviation is probably not enough to justify their use for this reason alone. The scores obtained by the students in M.C.Q.III are much lower than those for M.C.Q.1 and M.C.Q.II which suggests that the test covers material less familiar for Heriot-Watt University students as earlier suggested. It is interesting to note that the mean scores for M.C.Q.1 and M.C.Q.11 were 10% u p on the American mean facility values whereas those for M.C.Q.111 were lower. Analysis of Objective Test Results in Terms of Social, Environmental, and Intellectual Factors An analysis of the objective test scores was carried out by considering the three tests M.C.Q.1, M.C.Q.11, and M.C.Q.III as one test consisting of 200 items consisting of ninety-one purely "recall" items (A) and one hundred and nine "higher processes" items (B). The total student scores in category A items were noted over the three tests with the corresponding scores in category B items. The class was then divided into several groups based on the following criteria. 1. First class honours with upper seconds and lower seconds with
third class honours
2. Verbal reas?"ing intelligence quotient 2127 Verbal reasoning intelligence quotient < 127 3. Space test store 288 Space test score < 88 4. Socio-economicgroups I to IV and socio-economicgroups V to
VII
Note: The categories of parental occupation were based on the Hall-Jones scale of occupational prestige for males4 and are as follows. I Professional and high administrative I1 Managerial and Executive I11 and IV Inspectional and supervisory higher and lower grades V (a) Routine (non-manual) (b) Skilled manual VI Semi-skilledmanual W Manual routine
The interesting point arising from these comparisons is that nerformance in obiective tests of "recall" material compared with "higher processes" material is related neither to the class of degree obtained, nor to the socioeconomic background of the student. A relationship does appear to exist between objective test performance and the test of verbal reasoning. A simple explanation of this might well be the objective nature of M.H.T. (Adult I). It is interesting to note that in both the academic years, the more intelligent students (i.e. IQ 2 127) did significantly better on those it6ms which tested intellectual ability more complex than recall, whereas those students with an IQ of less than 127 performed equally on either kind of item. It might be argued that any difference in $he scores on one group of items compared with another cohld well be due to difficulty differences in the items themselves. This is not the case with these tests. Data suppli6d by Dr. T. A. Ashford of the University of South Florid&
140 /Journal of Chemical Education
shows the mean facility values to be virtually the same for hoth "recall" and "hieher orocesses" tvnes. The fieures .. . also suggest that the more intelligent stldents score i;igher marks in both "recall"and "hinher proces~es"items. These observations are reinforced to some extent by the test of spatial ability. Those students who score more than 88 on the space test do significantly better on "higher processes" items than on recall items (1968-69 only) and those same students do better on "higher processes" items than those who score less than 88 in the space test (again 1968-69 only). These significant differences were not shown in 1969-70. Summary and Conclusions The relatively high correlation coefficients obtained between the Moray House Tests and the Co-operative Examinations are interesting, and it was this which prompted the investigation of performance on recall and higher processes items. The interpretation of correlation coefficients must, however, be approached with caution. Decisions on the nature of a good correlation must be made with other considerations, such as the nature of the sample, kept in mind. Correlations found in educational research tend to be rather smaller than those calculated in other areas of research. The reason for this is that, as the variables examined are usually complex ,in nature, i t is unreasonable to expect high correlations among them. The most i m ~ o r t a n tdecision which has to be made is whether a perceptible correlation exists. In this context a correlation of 0.3 can he considered interestine and one of 0.5 to be of some importance. Correlation between two tests does not necessarily indicate a measure of causation. The comparison of means for both 1968-69 and 1969-70 indicates that students of higher than average verbal ability scored better on higher processes than on recall items whereas students of less than average verbal ability scored equally on both types. Students of higher than average verbal ability also scored significantly higher on both types of items than students of less than average verbal ability. It has long been thought that objective tests offer an opportunity to students of lower verbal ability to score better marks. The evidence of this present research indicates that, while this might very well he the case, i t is clear that obiective tests do not have a deleterious effect on the performance of the verhally more able students. The lack of sienificant difference in the ~erformancesof students from drfferent home backgroundsAwassurprising. Early investigations showed that the children of routine workers tend, on the whole, t o be less able verbally, and that objective testing might well suit them in examinations. The lack of significance in this investigation might well be explained by a "masking effect" of individuals in the middle range-say socio-economic groups 111 to VI. Examination of g r o u p I and II compared with group VII miebt well show significant differences but was not nossiu hle in this case as the number of students was too small. It is worth considerine that if. as sus~ected.conventional examinations place undue emphasis bn the' ability to recall plain facts, there is not much point in replacing them with objective tests if all that would then be measured was verbal ability or intelligence quotient. 'Oppenheim, A. N., "Questionnaire Design and Attitude Measurement," Heinemann, London, 1966.
