A Rationally Designed General Chemistry Diagnostic Test Arlene A. Russell University of California,Los Angeles Los Angeles, CA 90024 What knowledge, skills, and abilities must students have to succeed in a traditional general chemistry course? Chemical educators have asked this question for decades ( I d ) .The California Chemistry Diagnostic Test Project also addressed this question in answering the challenge and charge from the State: "Develop a new Freshman Chemistry Placement Test". A supportive funding agency provided for a careful, methodical, multi-year project in which many items and topics from high school chemistry and math were developed, field-tested, revised or discarded, field-tested again and re-evaluated. The final 44question tests emanating from the Project were constructed from those items found relevant to student success in first-term general chemistry. A computer summary provides an individual diagnostic summary of strengths and deficiencies in preparation for each student. Presented at the 204th National American Chemical Society Meeting, Washington, DC, August 1992.
Project History The original State charge resulted from a 1985 survey of the 129 public, higher-education institutions in California. A one-page questionnaire asked each Chemistry Department if they pre-tested general chemistry students and if so, what test was used, what enmllment decisions were made, whether they would he interested in a new test, and whether they would work on a new test. A rapid response resulted. Within four weeks of the survey mailing, 69% of the schools returned questionnaires, and 72 people volunteered to work on a new test. The survey elicited a tremendous desire to diagnose students' readiness for general chemistry, and an overwhelming dissatisfaction with existing tests. Thus, the UC-CSU Joint Projects Steering Committee funded our effort to resolve the pretesting dilemma. The 16-member Chemistry Diagnostic Test Project (CDTP), which represented a cross-section of chemistry educators in the State, quickly learned the basis for the general dissatisfaction. Analysis of 10 of the tests in use in
Table 1. Topicsa on Nine Tests in Use In California in 1985 and on New CCDT Test #
1
2
3
4
10 1 42 3 Compounds and Elements (Compounds,Chemical Formulas) 1 2 12 States of Matter (Gases,Solids, and Liquids) 6 7 12 4 Reactions of Matter (Calculationswith Chemical Formulas and Equations) Structure of Matter 4 3 7 3 (AtomicNature of Matter, Atomic Structure, Molecular and Ionic Geometry, Chemical Bonding) Periodic Properties 1 5 6 (PeriodicTablelntroduction and Trends) Solutions (Concentrationsand Stoichiometry) 4 10 9 Dynamics (QualitativeKinetics and Thermodynamics) 1 5 Lab Skills Additional Topics (Introductory Concepts) 4 3 (Measurement) 4 (Advanced Topics--Organic, Thermodynamics, etc.) 1 7 2 (Other-not classified in "Statement') 6 4 Total Number of Chemistry Questions 40 46 106 10 Topics Classified by "Statement on Preparation Expected of Students Entering the Natural Sciences".
314
Journal of Chemical Education
5
6
7
8
9
CCDT
1
4
6
4
1
5
1
4
15
2
1
3
12
6
16
4
4
5
5
14
3
6
9
6
3
5
11
15
2
2
1
4
1
1 17
1
1
4 2 51
82
3
1
3 20
11
37
Table 2. Distribution of Questions Between Mathematics and Chemistry
Test # #ofchemistry Questions #of Mathematics Questions %Chemistry
1 40
46
3 106
10
20
51
20
21
25
20
4
67
69
33
83
2
4
5
82
8 20
11
10
18
26
40
84
82
7
6
California a t that time revealed a wide variability both in topics (Tables 1 and 2) and difficulty. A new test was needed. In considering the basis for a new test, the CDTP Committee turned to the recently issued report, "Statement on P r e ~ a r a t i o nin Natural Science E x ~ e c t e dof Enterine ~'re~hmen'" prepared by members oithe Academic senates of the University of California. (IJCJ the California State Universities and Colleges (CSU),and the California Community Colleges (CCC). Shortly after its release, the State Department of Education adopted it as a curriculum guideline for teachers. We knew, however, that the Statement topics were determined by consensus, not experiment and that what students are "expected" to know is not always what they "actually" know. Nonetheless, the Committee identified eight broad Statement categories normally taught in a first-term general chemistry course: (1) compounds and elements, (2) states of matter, (3) reactions of matter, (4) structure of matter, ( 5 ) periodic properties, (6)solutions, (7) qualitative concepts in thermdynamics and kinetics, and
(8) lab skills. In contrast to a general accord on the chemistry topics, marked disagreement existed among the Committee on the relative importance of a background in mathematics versus chemistry. The variation in the percent of mathematics on the various diagnostic tests we examined (Table 2) indicated that our Committee was not alone in the debate. We knew also that, notwithstanding catalog prerequisites of a year of high school chemistry, many chemical Chemistry Concept Question How many atoms of oxygen are represented in the molecular formulaC7H6(N02)3?
Chemistry "word problem" question What is the approximate percent-by-massof oxygen in N H a N 0 3 (molar mass = 80 g)? (A) 20
I
(8)30 (C) 40 (D) 60
Math 'Word problem" question A gift box contains xapples, y oranges, and zpears. The formula for calculating the percent of oranges in the box is (A) EWE xl00 Y
(D)
I
IOOV
( ~ ) r ; + ' ; T - ~ x 1(C)-xl00) 00 (X+ z)
x 100)
Figure 1. Example of linked questions from first field test.
I
9
10 CCDT 37
30
7
educators hold a strong belief t h a t if their students know math, then previousexposure to chemistry is not necessary. Having the luxury of time to fieldtest an exam, the CDTP set out to the importance of
mathematics versus chemistry preparation while a t the same time d o t testincr the efficacv. if any, i f knowled& in the che&stry topics selected from the "Statement". We set up triads of questions to examine the mathematics-chemistry conundrum. If the Committee deemed the ability to solve a chemistry "word problem" necessary for success in general chemistry, a parallel math problem iuvolving exactly the same mathematical skills was designed along with a parallel chemistry concept question devoid of mathematical calculations. A typical example, is given in Figure 1. 43
20
O
44
First Field Test Two 45-minute, 44item tests, (Fom Aand B)containing seven triads and a total overlap of 55%were ready for fieldtesting in Fall 1987. Participating schools, who were sent equal numbers of the two forms, guaranteed that they would return the answer sheets ungraded, that they would submit grade sheets a t the end of the first term, and that the results would not be used for student placement or assessment. We obtained results from 53 schools with information on 1304 students from the University of California system, 1790 students from the California State University system and 3431 students from public community wlleges in the State. Because the California Master Plan for Higher Education requires that general chemistry courses in all public higher education institutions articulate with each other, we fvst assessed whether we had a reliable test across the system, wmmunity colleges through universities. The results supported this goal: the means for Forms A and B a t the University of California, the State Universities and Colleges,and the community colleges were 24.48 and 24.21; 22.28 and 21.18; and 22.20 and 21.31, respectively. Since admission to the various sectors is based on high school performance, we had expected that the test differences might be larger. Although the differences are significant by virtue of the large sample size, they are much less than expected. We attribute this small difference to the fact that most students in the wmmunity wlleges begin with preparatory chemistry courses, which we conclude are effectivelybringing students up to the level of knowledge, skills, and abilities necessary to handle general chemistry. Equally important as the reliability of the test was its validity. Did the test identify those students who had the necessary background to pass their first term of general chemistry? Here again the answer was yes. The correlation coefficients between the test scores and the final course grade for the two forms in the various sectors were 0.40 and 0.41; 0.35 and 0.45; and 0.32 and 0.35. Finally, we needed to wnsider the mathematics-chemistry question. Regression analysis determined the predictive value of the three kinds of triad questions toward the course grade. The means of the categories were allowed to enter the regression equation in order of importance. As can be seen from the means, the quantitative chemistry questions were more difficult than the questions in the
'Available from CaliforniaState Department of Education. Sacramento, CA. Volume 71
Number 4 April 1994
315
item analysis, which included both reliability index and grade wrrelation, the hest questions from forms A and B, reSignifi- ~ u m F ? gardless of their origin and relationship cance with each other, were chosen for Form C.
Table 3. Regression of Triad Questions to Course Grade
Variable
Mean
Correlation
Beta
Part
Partial
Form A (F? = 0.12, N = 2046, #Triads = 6) Chem Word Problem
2.51
0.28
0.171
0.15
0.15
Math Word Problem
4.11
0.26
0.182
0.17
0.18
Chem Concept
3.87
0.23
0.112
0.10
0.11
Form B (F? = 0.14, N = 1980, #Triads = 5) Chem Word Problem
2.05
0.33
0.248
0.22
0.24
Math Word Problem
3.70
0.27
0.183
0.17
0.19
Chem Concept
3.12
0.19
0.075
0.07
0.08
other categories and in the test as a whole (Table 3). Taken as a set, these categories were not contributing their share to the predictive value of the two forms. Nonetheless, when the analysis of the questions in the triads was considered separately from the rest of the test questions, we found that prior skills in two of the variables, quantitative chemistry pmhlem-solving and mathematics problem-solving, independently contribute to success in the course with little additional predictive value coming from the descriptive chemistry questions. On Form A, the quantitative chemistry and algebra questions demonstrated equal predictive power with the edge going to algebra. On Form B, the quantitative chemistry category was superior to algebra in oredictive vower. Clearlv. ".mathematics ~. r e.~ a r a t i oisnnot sufficient;prior practice with chemistry "word problems" is imoortant. This is consistent with other data indicating &dents show little evidence of transfer of higher-order thinking skills between disciplines (6-7).The chemistry prerequisite in wurse catalogs has validity. Mathematical skills are, however, important. In fact, when the triad questions were examined individually, performance on the mathematics questions predicted course grade best; the quantitative chemistry questions predicted next best; the qualitative chemistry questions entered the regression equation last. Although not all questions were equally powerful in predicting the grade in general chemistry, the nine categories (eight chemistry and mathematics) all proved valuable and these categories were retained in the revised version (Form C) of the test. The triads were not kept. Based on an Table 4. Field-Test Sample Demographics
yes
0.08
Second Field Test Form C was field tested in Fall 1988 yes 0.11 following the same procedure used the yes 0.12 previous year. Again three levels of schools participated and a cross-section of California students were repreyes 0.11 sented. Table 4 shows the gender and ethnicity data. We had not anticipated yes 0.14 the much higher difference between the yes 0.14 percentage of females attempting general chemistry in the University of California system compared with the other sectors. Anv ex~lanationis s~eculative. but the lack of women in the physical sciences in the other two sectors bears careful scrutiny. Our data also support the national concern that minority students are not pursuing academic goals in the sciences in proportion to their numbers in society. The test statistics for the total sample and by sector are presented inTable 5.As with the prior field test, the differences are mainly between the University of California students and the students in the other two sectors. All the differences are wnsistent with those we found the year before. Figure 2 shows a graph of the first-term grade correlation with the Diagnostic Test score for the total sample of 4023 students. Only data points containing 10 or more points are plotted. The correlation coefficient is 0.42 for this total sample with a variation between 0.32and 0.50 for the individual sectors. In other terms, the students' background in chemistry that is being measured by the test acwunts for between 10% and 25% of the variance in the final grades in the first term of general chemistry. Thus, the test appears to be a valid diagnostictool for evaluating all students' preparation for general chemistry. Because of the variation, however, if schools wish to use the test to place students in given wurses, we rewmmend that they establish their own local norms and cutoffs.
" .
Table 5. Field Test and National Statistics
Total
UC
CSU
National
CC
#Correct
23.72
26.23
22.02
22.37
22.85
CC
%Correct
53.91
59.61
50.04
50.84
51.93
1789
Median
23
26
22
22
22.2
Gender
Mode
24
22
17
21
3
Male
Minimum Score
2
2
4
4
7
Female
Maximum Score 44
44
43
44
42
Ethnicity
Standard Deviation
Sample size
Total
UC
4023
1473
CSU 761
7.83
7.60
7.25
StandardErrorof 0.12 the Mean
0.20
0.26
Asian Black
Reliability
0.87
0.84
Hispanic
Grade Correlation 0.42 0.5 0.41 0.32 'Standard Error of the Measurement 'California data based on KR-20;National data based on KR-21
White
Other 316
Journal of Chemical Education
0.87
7.73 0.18 (2.82)a 0.86
7.11 3.12' .Bob
Test Score - Course Grade Correlation (N = 2836; at least 10 studentslpoint)
Cabfornia Diagnostic Exam Result? IMPORTANT: SAVE THESE RESULTS Name: Deborah Brown Test Date: 09/03/93 ID: 1234-56-7890
-
I . .
I
ScorelPossible: 21/44 Result: PASS You may enroll in Chemistry 1A The followingareas were tested. Review is recommended in the "ed areas. (315)Compounds and Elements '(013) States of Matter (515)Reactions of Matter '(116) Structure of Matter (515)Periodic Properties (417)Solutions (112)Dynamics '(014) Lab Skills '(217)Mathematics
L ,
l O i 5 2 0 i s 3 b i 5 4 0 4 5 DiagnosticTest Score Figure 2. Graph of thefirst-term grade correleation with the Diagnostic Test score for the total sample of 4023 students.
National Results Having fulfilled the original charter of the Project, we now needed a vehicle for disseminating the test. Discussions with the ACS Division of Chemical Education Examinations InstituteZled to a formal designation of the test as the "California Chemistry Diagnostic Test" and its inclusion in the Institute distribution list. In Fall 1989, the Examinations Institute obtained normative statistics on internal consistency of the test on a national sample. Although slightly different statistics were used, a comparison of the California data with the national data, shows no significant difference. Between 1990 and 1992, the CDTP committee prepared and field-tested a new version of the Diagnostic Test. The Examinations Institute released this new version in 1993. The 1989 and 1993 tests can be used together as A-B forms. Student Feedback Althoueh .. the orieinal ..eoal was to develo~a la cement test, the resulting mstrument IS in fact a diagnostic tool, wh~chcan assess a student's backeround and hkclihood for success in general chemistry. Individual schools may want to use the results for dacement. but that must be a local decision. Since the test is most beneficial to the students for its diaenostic vower, we felt that each student, whether they did well or poorli or placed in general chemistry or not, would benefit from an analysis of their own strengths and weaknesses before entering the course. As a further component of the test package, we have prepared a DOSbased program3to analyze the computer-scanneddata and prepare a half-page summary (Fig. 3) for each student. Student response to this individual "counselling" has been positive. One final design feature was to try to develop a "calculator neutral" test. During the testing of Forms A and B, we asked students if they had a calculator with them during the test. Students who answered Yes' did slightly but significantly better than those who answered 'No'. Given the nature of the arithmetic and algebra required in the test (Fig. 1)we hesitate to interpret this as an effect of the test
. .
-
'ACS DivCHED Examinations Institute, 223 Bra&ett Hall, Ciemson University, Box 341913, Clemson, SC 29634-1913. 3Contactthe author for availability of this program.
These results are valid for placement for one academic year after the test date. 'Italicized lines locally determined L
Figure 3. Sample student report questions. It may involve peripheral issues such a s a student's background in chemistry classes or in test taking. Regardless of the explanation, if the test results are to be used for placement in various courses, we recommend that students bring calculators to the test in order to demonstrate their best ability. Conclusion Although we initially set out to develop a placement test for general chemistry, we realized that what was actually needed was a test with the diagnostic ability to detect areas of student deficiency and strength. Placement in courses is a local issue not an academic issue. Regardless of the placement decisions a school makes, both students and instructors can benefit from knowing students' weak areas. Given the current content of the first term of general chemistry courses, students have a much better chance of succeeding if they know where to concentrate their study efforts. The California Diagnostic Test can provide this advantage to students. Acknowledgment I thank my CDTP test-writing colleagues: J. Casanova, B. Gonzalez, M. R. Henley, J. C. Hill, F. W. Hiller, L. S. Meyers, G. E. Miller, W. T.Mooney, Jr, J. A. Neptune, M. P. Noel, D. Pribyl, S. P. Ruiz, G. Schlesinger, L. Soloff, and F. E. Wood for their diligence and persistence and the UCCSU Joint Projects Steering Committee and the President's Opportunity Fund for their continuing and patient support of this project. Literature Cited 1.Have%N. W :. Kmhn,A.J.J. ChPm Edur. 1958,35,607609. Z.Ozsogamonyan,A; M u a , D. J. Chem. Edac 197% 56,173-175. 3. Niedzieisld, R. J.;Wahaley, F. J. ChemEdrrc. 19s!a, 59, 14s151. 4. Greenbow , T. J.; Golen, J. k: Sdlwe, M. I. Tansidenng the lbledo Chunidry Placement Exam (Fmm 1974) as e Rediclor for Sueeesa in General Chemjem" Symposium on Placement Roeeduree for Genersl Chemistry, 192nd National Meeting af the ACS, Anaheim, CA, 1986. 5. Mitchell, T JChem. Edue. 1889.66, 562664. 6. Resnic*, L. B.Edvcofion and Learning lo Thin*, Nat. Ad. Reas: WashhgtOn, DC, 1987. 7. Rath, W-M.; Royehoudhu?i kJ. Rep S d M.1893,30,127-152.
Volume 71 Number4 April 1994
317
