Research: Science and Education
Assessing Student Preparation through Placement Tests Craig McFate and John Olmsted III* Department of Chemistry and Biochemistry, California State University, Fullerton, Fullerton, CA 92834
Background
Test Design
A low rate of student success is a widespread and persistent characteristic of college general chemistry. Typically, up to 20% of students fail the first-semester course, and when attrition rates are included, the proportion of those enrolling who succeed (i.e., complete the course with a grade of C or better) may be only 65–70%. A contributing factor is that not all students who desire to take college chemistry possess the proficiency in “thinking like a chemist” that is required for success in the first chemistry course. One way to improve the success rate is to identify those who are lacking and provide them with “remedial” or “preparatory” instruction. To this end, placement tests have been developed and administered at many American colleges and universities. The desirability of determining factors that can predict student success in chemistry was recognized as early as 1921 (1), and a study was published in 1942 (2). However, the earliest reported use of a chemistry placement test dates to 1955. In that year, the University of Toledo began using the Iowa Chemistry Aptitude Test to determine admission into first-semester chemistry (3). Over the next few years, the Toledo Achievement Test was also developed, and these two tests subsequently spawned the Toledo Chemistry Placement Exam (TCPE), which found extensive use (4). This exam was revised in the 1980s and is currently available to ACSaccredited institutions (5). By the mid-1980s, higher-education institutions in California had indicated an interest in developing a new placement test. The result was the Chemistry Diagnostic Test Project, a consortium of faculty members from California schools which developed and validated the California Chemistry Diagnostic Test (CCDT). The validity of this test has been extensively evaluated (6 ). A parallel thread in chemistry placement is the correlation between mathematics aptitude and success in general chemistry. An early study showed that scores on a mathematics entrance exam were a good predictor of chemistry success (2). Later, studies on the East Coast (7) and West Coast (8) revealed a significant correlation between mathematics SAT scores and the grade achieved in general chemistry. At California State University Fullerton, a placement test has been applied as a major determinant of eligibility to enroll in first-semester general chemistry for the past 30 years. Initially, a mathematics placement test administered by the mathematics department doubled as the chemistry placement test. In the mid-1970s, this test was augmented by a supplement containing chemistry-based questions. In the late 1980s, the chemistry department opted for a separate placement test, for reasons of ease in administration and advising. The test developed for this purpose, which contains both math-related and chemistry-related questions, is the subject of this report.
The function of a placement test is to separate students into two groups: those who lack the abilities required to succeed in the course and those who possess these abilities. The proper design of such a test requires identification of the abilities required to succeed. This, however, has been a vexing question. For example, there is little agreement among different levels of instructors concerning the importance of chemical knowledge and basic chemical skills (9). Indeed, studies suggest that more than one variable is likely to be involved (2, 8, 10). There are indications that formal operational reasoning ability, as defined by Piaget, plays an important role (11–13). Finally, as noted some 35 years ago, “motivation plays…an important role in determining course grades” (4), and placement test questions cannot determine level of motivation. The CSUF placement test was designed assuming that the following abilities contribute significantly to success in general chemistry: (i) ability to perform various mathematical manipulations; (ii) ability to visualize atoms and molecules; (iii) ability to reason proportionally; (iv) ability to understand chemical formulas and equations; (v) ability to interpret graphs. Additionally, it was assumed that while specific knowledge of chemical facts, theories, and nomenclature may be useful to incoming students, such knowledge is significantly less important as a predictor of success. Because we wished to emphasize abilities more than knowledge, we excluded several sorts of questions that often appear on placement tests, such as (i) nomenclature questions (e.g., What is the name of the compound NaNO3?); (ii) calculations requiring the recall of an equation (e.g., What is the volume occupied by 2.0 mol of an ideal gas at 400 K and 2.0 atm?); (iii) identification of chemical classifications (e.g., Which of the following is a nonmetal?). The final result is a 25-item multiple-choice test, of which 8 questions deal with mathematical abilities, 7 with chemical formulas and equations, 4 with proportional reasoning, 3 with atomic/molecular visualization, and 2 with graph interpretation (one question asks about the law of conservation of energy). This form of the Fullerton Chemistry Placement Test has been in use for 8 years. This test is administered during the first 45 minutes of the first scheduled laboratory session of the first-semester general chemistry course, to all students enrolled in or petitioning to enroll in the course. Scantron answer sheets are machine-graded, allowing a rapid turnaround time. Before the next scheduled laboratory session, the laboratory coordinator (a senior faculty member with substantial experience teaching general chemistry) confers with all students scoring below 50%. Remedial procedures are recommended depending on the student’s background. These range from taking preparatory chemistry and/or introductory mathematics before enrolling in general chemistry to setting aside additional study time and/or designing special study strategies to improve the likelihood of success. Although students are seldom barred from
*Corresponding author. Email:
[email protected].
562
Journal of Chemical Education • Vol. 76 No. 4 April 1999 • JChemEd.chem.wisc.edu
Research: Science and Education
the course on the basis of placement test scores, students with low scores are strongly advised not to take the course.
Instrument
School
Year
R2
Ref
Toledo Chemistry Achievement Test
Toledo
1955
.3 4
3
Results and Discussion
Math SAT
Berkeley
1976
.26
8
Campus GPA
CSUF
1988
.4 4
13
CSUF
1988
.10
13
all California
1987
.10–.20
6
all California
1988
.10–.25
6
Table 1. Statistical Evaluations of Placement Instruments
0.6
0.4
0.2
CSUF, R^2=0.82 Toledo, R^2=0.98 Calif, R^2=0.87
0.0 0
20
40
60
Test Score, %
80
100
Figure 1. Plot of fraction of students succeeding in first-semester general chemistry as a function of placement test score. Solid circles are data from this study, open circles from ref 4, squares from ref 17.
1.0
Fraction of Students
Test Effectiveness The validity of placement tests has typically been assayed by correlating test scores and course grades, using the correlation coefficient (R 2) as the measure of validity. Russell plotted average course grade as a function of diagnostic test score (6 ), Pickering plotted math SAT score against chemistry grade (7), Ozsogomonyan and Loftus tabulated chemistry grades for various math SAT scores (8), and Hovey and Krohn tabulated chemistry grades for various test score ranges (3, 4). A previous Fullerton study correlated general chemistry grades against a variety of indicators including placement test and SAT scores (15). A sampling of these correlations appears in Table 1. Accepted statistical standards for correlational studies define R 2 values between .09 and .36 as indicating only a “moderate” correlation (16 ). On this basis, none of the tests is a good predictor of course grade, and more recent placement tests may be inferior to earlier versions, despite concerted efforts to improve them. One study even showed that campus GPA is a better predictor of course grade than any test result. However, whether or not placement test results correlate strongly with course grades is irrelevant to the purpose of placement tests, which is to determine whether or not a student can pass the course, not to predetermine what grade the student will receive. Consequently, we correlated placement test performance and success in the first-semester general chemistry course, “success” being defined as completing the course with a grade of C or better. For comparison purposes, we carried out similar analyses on data from the two other studies for which published information allowed this: the Toledo study of the TCPE (4 ) and a validation study (17 ) of the CCDT for California community college students. Figure 1 shows the fraction of successful students as a function of test score for the three studies. “Toledo” refers to data for the TCPE from the University of Toledo for 1959– 61 (4 ). “Calif ” refers to data for the CCDT from Glendale Community College, California for 1991–93 (17 ). “CSUF” refers to data from this study of the CSUF placement test for 1993–95. Each shows a satisfactory linear fit with R 2 > .80, indicating high degrees of correlation. Given that these are three different test instruments applied to three distinct student populations, the similarity in the fits is striking.
Fraction Succeeding
The sample selected for this study comCSUF Chemistry/Math Placement Test prised all students who registered for the first general chemistry course and took the place- California Chemistry Diagnostic Test ment test at CSUF during the four semesters California Chemistry Diagnostic Test, C from Fall 1993 through Spring 1995, a total of 845. Comparison from one semester to another showed no significant differences in test performance or correlation with course success, in contrast with an earlier study (14 ), so data from all four semesters were analyzed together. The 1.0 questions studied were (i) How effective is the placement test in predicting success in the course? (ii) Is there a subset of questions of any particular type(s) that is especially effective in pre0.8 dicting success in the course? (iii) What characteristics, if any, are shared by the questions that have higher predictive power?
0.8
0.6
0.4
CSUF Toledo
0.2 0
20
40
60
80
100
Test Score, %
Figure 2. Plot, as a function of placement test score, of the fraction of students falling at or below a test score who do not succeed (triangles) and fraction of students falling above a test score who do succeed (circles) in first semester general chemistry. Solid symbols are data from this study; open symbols, from ref 4.
JChemEd.chem.wisc.edu • Vol. 76 No. 4 April 1999 • Journal of Chemical Education
563
Research: Science and Education
of determination that were more than one standard deviation above the mean.
Fraction Correctly Placed
0.8
1. If sodium carbonate is Na 2CO3, ammonium chloride is NH4Cl, and sodium chloride is NaCl, then ammonium carbonate is: (a) (NH4)2CO3 (b) NH4CO 3 (c) NH4(CO3)2 (d) NaNH 4CO3 (e) none of these
0.7
0.6
2. If x =
0.5
0.4
CSUF Toledo
3.
0.3 0
20
40
60
80
100
Test Score, %
Figure 3. Plot of the fraction of students correctly placed (students scoring at or below the cutoff do not succeed, students scoring above the cutoff succeed) as a function of cutoff score. Solid symbols are data from this study; open symbols, from ref 4.
For placement purposes, a more important feature is a “cutoff ” placement test score that distinguishes successful from unsuccessful students. A perfect placement instrument would correctly place 100% of students: all students with scores below the cutoff would be destined not to succeed and all with scores at or above the cutoff would be destined to succeed. Figure 2 shows the fraction of students scoring below a given test score who do not succeed and the fraction of students scoring above a given test score who do succeed. Data are included for this study and the Toledo study, the data available from the Glendale study being insufficient to carry out this analysis. It is clear from the figure that there is no cutoff score that approaches “perfect” placement. Cutoff scores around 25% would ensure that very few students who were capable of passing were barred from the course, but would allow into the course many students who were destined to fail. Higher cutoff scores exclude more of those who are destined to fail, but at the expense of excluding significant numbers who are capable of passing. Another way to analyze the effectiveness of a placement test is by determining the fraction of students who are correctly placed for each possible cutoff score. This analysis is presented in Figure 3, which shows that cutoff scores around 50% do the best job of placing students correctly. On the CSUF test, the result is ≥67% correct placement (≥69% including those low-scoring students who voluntarily withdrew). The Toledo study shows a sharper variation with a maximum correct placement at 80%. Karpp’s analysis for the CCDT test applied to Glendale Community College students (17) gave a maximum correct placement of 77+%. By this measure, the CSUF test is not as effective as either the CCDT or the TCPE.
Question Types To compare the effectiveness of individual questions as predictors of course success, the coefficient of determination (R 2) value was calculated for each test question. The 25 test items had a mean R 2 value of .0107, with a standard deviation of 0.0089. The following six questions had coefficients 564
{2
2
4.
5.
6.
1.00 × 10
8.00 × 10 {3
4.00 × 10
, the value of x,
4.00
properly rounded, is: (a) 5.00 × 10 {3 (b) 0.500 (c) 5.00 × 102 (d) 5.00 × 103 (e) none of these. The volume of a soft drink bottle is 0.750 L. Given that 1 L = 103 cm3 and 1 cm = 10 mm, this volume in mm3 is: (a) 7.50 × 102 (b) 7.50 × 103 (c) 7.50 × 105 (d) 7.50 × 106 (e) none of these If PV = nRT and n = m/W, then a correct expression for W is: (a) PV/mRT (b) mRT/PV (c) m/PV (d) PV/RT (e) none of these After an atom of potassium, K, loses an electron, e¯, chemists represent it with which symbol? (a) K (b) K¯ (c) K+ (d) K {e (e) none of these The reaction of carbon and hydrogen to form ethane can be represented by the chemical equation xC + y H2 → → zC2H6 This equation is properly balanced when the values of x, y, and z are (a) 1, 1, 1 (b) 2, 3, 1 (c) 2, 6, 1 (d) 1, 3, 1 (e) none of these
These questions have little in common. Questions 2 and 3 test mathematical ability, questions 5 and 6 test basic chemical knowledge, and questions 1 and 4 have a logical/mathematical basis while using chemical “vocabulary”. Moreover, when the full set of questions is divided among these three categories (math ability, chemistry knowledge, mixed character), each category contains weak discriminators as well as strong ones. Furthermore, discriminatory power is not correlated with question difficulty. Question difficulty ranged from 0.41 (41% of students answered correctly) to 0.96, with a mean of 0.69. The six best discriminators had an almost identical difficulty range, from 0.43 to 0.90, with a mean of 0.65. Two features present in some of the best discriminators and absent in all other questions are requirements for multistep mathematical operations (questions 3 and 4) and formal reasoning (questions 1, 5, and 6). Question 3 requires a sequence of unit conversions, and question 4 requires a sequence of algebraic manipulations (another question, requiring a single algebraic manipulation, is not a good discriminator). Question 1 requires either good knowledge of the rules of chemical nomenclature or strong logical reasoning ability. Questions 5 and 6 require knowledge of conservation principles. Placement test questions probing students’ visualization ability were among the weakest discriminators. Two graph interpretation questions fell about 0.5 standard deviation below the average, and two questions relating to visualization of atomic-sized objects were more than one standard deviation below the average. One visualization question, a “molecular
Journal of Chemical Education • Vol. 76 No. 4 April 1999 • JChemEd.chem.wisc.edu
Research: Science and Education
This work leads us to the following conclusions concerning placement tests for college general chemistry. 1. Placement tests should assess a variety of student abilities, because no study yet reported has identified specific abilities that are superior predictors of student success in chemistry. 2. An appropriate vehicle for assessing placement test validity and selecting the most appropriate cutoff score is a graph of percentage of students correctly placed as a function of cutoff score. 3. Even the best placement tests yield incorrect placements of at least 20% and bar some students from the course who are capable of succeeding.
Figure 4. Reproduction of placement test item designed to assess students’ ability to visualize molecular processes.
picture” of a chemical reaction (shown in Fig. 4), ranked about average among all the questions. We had expected that the ability to visualize, particularly at the molecular level, is an important tool in a chemist’s mental arsenal. However, the first-semester general chemistry course at Fullerton is heavily oriented toward quantitative manipulations, which may be why limited visualization ability appears not to be an impediment to success in this course. Conclusions In common with other placement tests, the Fullerton test shows a modest correlation between test score and course grade in first-semester general chemistry and a considerably stronger correlation between test score and course success. The comparable correlations suggest that a test designed specifically to test skill levels (Fullerton test) is no more effective as a placement tool than tests that include questions requiring factual recall (Toledo, California tests). This, in turn, suggests that success in first-semester general chemistry is promoted not only by possession of the requisite cognitive skills but also by a strong factual grounding in chemistry. Our analysis of the correlation between placement test performance and success in general chemistry and similar analyses of TCPE and CCDT test results indicate a 20–30% “misplacement” of students when placement test scores are used to bar students from enrolling in the general chemistry course. Given that a measurable fraction of low-scoring students will pass if allowed to take the course, institutions considering the use of placement tests must weigh the penalty imposed on those students by barring them against the costs to students and the institution incurred when underprepared students enroll in, and fail, the course. Our institution has attempted to find a middle ground by using placement test results only as an advising tool. In the advising process, nontestable factors such as motivation and maturity are taken into account.
Often, a significant fraction of students who succeed in first-semester general chemistry do not succeed when they reach the second-semester course or the beginning organic chemistry course (at CSUF, nonsuccess rates in each of these courses typically hovers around 20%). This indicates that the skills needed for success in subsequent courses are different from those needed for success in the first course. In light of this, it would be instructive to carry out a longitudinal study correlating placement test performance with ultimate success in core chemistry courses (emerging successfully from the two-year general–organic sequence required for many life sciences majors as well as chemistry majors). To our knowledge, such a study has not been carried out. Acknowledgments This work is derived from the thesis study carried out by Craig McFate in partial fulfillment of the requirements for the M.S. degree in chemistry from California State University Fullerton. We acknowledge the expert assistance of Patricia Keig and Victoria Costa, Department of Elementary and Bilingual Education, CSUF. A portion of this work was reported at the Undergraduate Chemistry Learning Assessment Symposium at the 213th National Meeting of the American Chemical Society. Literature Cited 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17.
Ogden, W. R. Sch. Sci. Math. 1977, 77, 122–126. Martin, F. D. J. Chem. Educ. 1942, 19, 274–277. Hovey, N. W.; Krohn, A. J. Chem. Educ. 1958, 35, 507–509. Hovey, N. W.; Krohn, A. J. Chem. Educ. 1963, 40, 370–372. Examinations Institute of the American Chemical Society Division of Chemical Education; URL: http://tigerched.clemson.edu/ (accessed Dec 1998). Russell, A. A. J. Chem. Educ. 1994, 71, 314–317. Pickering, M. J. Chem. Educ. 1975, 52, 512–514. Ozsogomonoyan, A.; Loftus, D. J. Chem. Educ. 1979, 56, 173–175. Mitchell, T. J. Chem. Educ. 1989, 66, 562–564. Coley, N. R. J. Chem. Educ. 1973, 50, 613–615. Herron, J. D. J. Chem. Educ. 1975, 52, 146–150. Beistel, D. W. J. Chem. Educ. 1975, 52, 151–152. Goodstein, M. P.; Howe, A. C. J. Chem. Educ. 1978, 55, 171–173. Schelar, V. M.; Cluff, R. B.; Roth, B. J. Chem. Educ. 1963, 40, 369–370. Gillis, J.; Butler, K. Chemistry Placement Study; Unpublished report, California State University, Fullerton: Fullerton, CA, 1989. Mason, E. J.; Bramble, W. J. Research in Education and the Behavioral Sciences: Concepts and Methods; Brown and Benchmark: Madison, WI, 1997; p 167. Karpp, E. R. Paths to Success, Volume III. Validating the California Chemistry Diagnostic Test for Local Use; Glendale Community College Planning and Research Office: Glendale, CA, 1995.
JChemEd.chem.wisc.edu • Vol. 76 No. 4 April 1999 • Journal of Chemical Education
565
