Administration of an ACS-NSTA Exam to Italian Pupils Rinaldo Cewellati Dipartimento di Chirnica, "G. Ciamician". Universita di Bologna, via Selmi 2, 40126 Bologna, ltaly Bruno Piacenza I.T.I.S. "M. Delpozzo", Cuneo, ltaly Paola Ambrogi I.P.S.I.A. "Sidoli-Bellerio", Reggio Emilia, ltaly Gianmario Bonati Liceo Scientific0 "Lussano", Bergarno, ltaly We have reported a small-scale experiment in which an examination from the USA (Part I of the 1984 Regents High School Examination of the State of New York) was used with Italian students. The result of that experiment indicated that multiple-choice tests can be nsed a s a useful tool ibr obtaining u &Teatdeal of information on the learningof cherniitn 11,. The present article describes a larger scale experiment done wiih a larger test. The test is a literal translation of the 1989 ACS-NSTA High School Chemistry Exam. We chose this test because it has been tested on a large number of American students, approximately 6,000, distributed throughout that country. Abooklet, available from the ACS Examinations Institute, gives the composite norms and statistics obtained with this test. Our Aims To pretest the American Exam on representative sam-
ples of the student population in suitable Italian upper secondary schools; To compare the results with those obtained in traditional Italian examinations.
862
Journal of Chemical Education
To investigate whether this kind of objective assess-
ment technique can be used besides (or as a replaeement for) the subjective evaluation methods currently used in Italy. Experimental Procedure The Sample
We administered the exam to 17-year-old students who were taking the basic chemistry course in the Scientific Lyceum. This type of school may be regarded a s approximately equivalent to the "academic track" of American High Schools. The course taught in the Scientific Lyceum is equivalent to the American basic chemistry course (2). Some translations of American textbooks (3-5) are widely nsed in the Scientific Lyceum. The cities involved in the research work were Bari, Bergamo, Bologna, Cagliari, Cuneo, Genova, Napoli, Roma, and Savona. These cities and their environs cover most of Italy and can be considered representative of the national situation. However, the sample is not completely random, because the schools and the classes engaged in the experi-
ment were chosen on the basis of the willingness of the teachers to take part in the experiment. Anyway, as in Italy all classes are composed of students with mixed ability, the bias introduced with this accessibility sampling method can be considered small, and our sample can be reearded as an average - mouD - . of the Scientific Lvceum population. A total of 507 ~ u ~ iwere l s examined. This s a m ~ l eis about 5% of the students attending the Scientific~ y i e u m s in the cities mentioned above. On the basis of the Italian school situation and organization (6) we think that the considerations we can make on this sample can be reasonably extended to the whole Scientific Lyceum population.
-
The Exam The 1989 ACSNSTA High School Chemistry Exam was furnished by the Examinations Institute of the American Chemical Society. It consists of 80 questions, with four options and one correct answer. It is split into two equivalent parts. The exam booklet, instructions, and answer sheets were translated literally into Italian. The exam was administered at the end of the school year, in May-June 1990. The time allotted was two hours, longer than the 80 min. prescribed in the test booklet, because Italian students are not familiar with this kind of exam. The students were invited to answer as many questions as possible. They were told that the test was for research purposes only, and that the results would not be used in calculating the grade for the course. Data reduction was carried out using the Excel package with the suitable macro functions. Results and Discussion The general results obtained are tabulated below The reliability coefficient, calculated with the KR formula 20 (7)was 0.90. This conf~rmsthe high reliability of the ACS-NSTAExams. It is of interest t o compare these results with those obtained by the same students in the traditional Italian examinations. In Italy, the assessment is generally based on oral exams. The teacher tells the students to state what thev know about a s~ecifictooic.. and. in the course of the ex& the teacher usually intervenes asking for furtherexolanations of what the student is saving. At the end of the kxam the student's performance is Lvaiuated subjectively on a "scale" from 0 to 10. with six beiue- the minimum Dassing grade.
.
In our sample the percentage of students who passed the traditional exam was 73%. We have converted the scores obtained in the ACS-NSTA Exam into the usual Italian marks by the following formula (8): Mark (i) = 6 + (Score (i) - Mean Score)/o where a is the standard deviation of the distribution of scores. This formula assumes that the marks' distribution of the whole population has a mean of six and a standard deviation of one (9). On this basis, 355 pupils out of 507 (70%)would pass the ACS-NSTAtest. In other words, the results of these two assessment techniques are in good agreement, in spite of their differences. This work suggests that the ACS-NSTA High School Chemistry Exams could be utilized well as an assessment technique in Italian Scientific Lyceums. This is of great importance in view of the introduction of objective assessment techniques in the Italian High School System now under renewal (6). Acknowledgment The authors wish to thank the teachers and the pupils of the schools who took part in this investigation. We also thank D. I. Eubanks, Director of the ACS Examinations Institute that gave us the booklet of the 1989 High School Chemistry Exam and the related composite norms and statistics, and Pasquale Fetto, Dipartimento di Chimica "G. Ciamician", Universita di Bologna for his technical assistance. This work was financially supported by a grant from the Ministry of University and Scientific and Technological Research (MURST). Literature Cited 1.Ceweliati, R: Piac-a, 8.; Olmi. F Rieieie Edvmtiuo 1889,6,7&81. 2. Gailey, K D. Chemistry Edumtlon in Uppr Smmiory Sehaalofrhz USA Joint ItalyUSA Seminar on Mathematics and Stience Education in the Upper Secondary Schml. Athens, GA, June 3-7.1985, unpublished paper 3. Di&on, T. R. Intmduction to Chomislry; John Wiley: New York, 1979: Italian translation: Corso intmduaiuo di Chimim; Zaniehelli: Bologna, 1982. 4. Metealfe. H. C.: Williams, J. E.: Caatka, J. F M&m Chemktry; Holt, Funehart and Winston, 1974: Italian translation and adaptation by Cotta Ramusino, F. and P e p , A. Chimica M&mo: Cremonese: &ma, 1975. 6. Parry,R. W ; Die*, P. M.; IkUefsen, R. L.; Seiner, L. E. Chemistry: Expetimntal Foundotlona: PrentiecHall: Englevood Cliffs, NJ, 1975: Italian translation: Chlmim-Fandamnti Spetimntdi: Zaniehelli: Bologna, 1977. 6. Cewelisti, R.;Guardo, M. Edw. Chem. 1992.29.43-45. 7. Bodner.0.M. J Chern. Edue. lsB0,57,18B-190. 8. Gattvllo M. Didoetior ondDaeimalogy; Armando: Roma, 1976, p 3541in Italianl. 9. See ref (8)p 376. This assvmption le baaed on'e number of research works on the disVibutiona of the marks of several thousands of Italian pupils m &ffffre"t subject matters. In fad, the di8tzibutionof the fvld marks of oursample shows these statistics: mean 6.18, median 6, atd dev 1.20.This makes reasonable the application of the equation used in the conversion srnresirnarks.
Volume 71 Number 10 October 1994
863
