Content Validation Studies of the AP Chemistry Examination m y I.. Tan Educational Testing Service, Princeton, NJ 08541
The Advanced Placement (AP) Program of the College Board is a national cooperative educational endeavor designed t o offer secondary school students the opportunity to pursue college-level studies and, where appropriate, receive colleee-level credit while still in hieh school. The Droeram . .. ~rovidesdesrriptions of college-level courses to interested hieh schools and administers end-of-course or final examinations, containing both multiple-choice and free-response sections. to hieh school students who have taken the AP course. ~xaminationgrades are sent to the college of the student's choice which, in turn, grants credit andlor appropriate placement to students who have done well on the examinations. The nublication of a course descrintion and the develoDment of the concomitant AP final inaminations for eaih subject are the responsihility of a committee that consists of college faculty and secondary school teachers in a particular suhject area and that is appointed by the College Board to overlapping terms. From time to time it becomes necessary to assess the content validity of an AP course and its final examination to make sure -that these examinations adequately measure achievement corresponding to the course ohjectives at a broad spectrum of postsecondary institutions. In 1985-1986. the develooment committee for AP Chemistry undertook an ambitibus two-part study aimed at a critical evaluation of both the appropriateness of content and the level of difficulty of the A P chemistry examination. The two components of the study were the following: (1) a curriculum survey of general chemistry courses a t colleges and universities that typically receive large numbers of AP students .~~~~~~ ~-and (2) , . an evaluation survev of the multi~le-choice section of an actual A P Chemistry kxamination by college faculty. ~
~
.~ ~
~
~~~
~
~~
The Curriculum Survey Procedure
The purpose of the curriculum study was to assess the demee to which AP Chemistry topics and objectives are covered in introductory college chkmistry courses and to determine what, if any, common college topics are neglected in the AP course. For this study, a questionnaire outlining a typical college general chemistry course was developed by the College Board AP Chemistry Test Development Committee.' The ouestionnaire was developed with three princiual comuonents in mind. First, it sought to determine the relative emphasis on the maior topics included in the college curriculum for general chemistry. In addition, this surve; sought to obtain more detailed information on inclusion or exclusion of specific subtopics within each major category. The list generated for this purpose consisted of 115 aspects of chemistry that were described in the 1986 AP Chemistry course description. A companion list, containing other topics that members of the committee thought were sometimes covered in introductory chemistry courses designed for science and
engineering majors, was included in a separate part of the survey in order to obtain information on the extent of coverage of certain more advanced topics. College faculty were asked to rate the relative coverage of each of the subtopics in their courses as extensive, moderate, brief, or no coverage a t all. Finally, this survey elicited information on the time devoted to the laboratory in college general chemistry courses and tbe nature of the expe&nents conducted h i students in these courses. The entire questionnaire was sent to the chemistrv-denartment at the 200 colleees and univer. sities receiving the largest numbers of AP-students each vear. These institutions all offer four-vear undereraduate bducation and represent both small cofieges and l&e universities, public and private institutions from all regions of e s obtained from 114 i f these the country. ~ e s ~ o n i were institutions. All faculty who responded were associated with the general chemistry course attheir institution. Results and Dlscusslon The overall results of the survey are summarized by the broad content categories contained in Figure 1, which gives the averaee Dercentaee that each maior topic is allotted in thefull-year generalchemistry courses of participatinginstitutions. As is apparent from the results, distribution of coverage among the topics generally f o l l o k a well-established pattern with no topic receiving a preponderance of detailed attention. The more detailed analysis of subcategories within each of the major topics also given in Figure 1 yielded considerable variation with reeard to the extent of coverage " a t different institutions. The AP Chemistrv examination will be modified to take intoaccount the ratings given.Topics that are not commonly covered in colleee courses will bedropped from the 1989 and subsequent editions of the AP ~l;ekistry examinations. These include the Bohr atom, mass spectrometry, crystalline structures, osmotic pressures, concentration cells, chel a t e ~nonideal , gas equation (mathema~ical),and functional groups of organic compounds. Coverage of descriptive chemistry will he retained in the A P svllabus desnite the aooarent lack of broad emohasis on descpiptive chemistry in Glegegeneral chemistry coursesat the present time. Only the study of the alkali metals, the halogens, the transition metals and their compounds, and the properties of hvdrogen . . and oxygen received as much as 10-20";'extensir.e coverage in the college courses surveyed. Yet the reported overall coverage of descriptive chemistry as a major topic in college general chemistry courses wasnot
-
-
-
-
-
-
' The members who participated in the development of this ques-
tionnaire were: Graham P. Glass. Rice University. Chairman; Frank G. Cardulla, North Niles High School. Skokie, IL; Gregory R. Choppin, University of Florida; John W. Macklln, University of Washington; Norman Nachtrieb, University of Chicago; Audrey A. Sterenfeld,Hewlen High School, Hewlen. NY; D. Montgomery Wells. Newton High School. Newton, MA; Ronald Archer. University of Massaschusens, Amherst, MA, (ChiefReader for AP Chemistry). Volume 67
Number 3 March 1990
241
STOICHIOMETRY and CHEMICAL EQUATIONS 7.4%*
;:
~ e l a t i v ecoveraqe/Topic
e he mole concept and A v o q a d r ~ 'number ~
%E
%M
%B
6o
24
l3
%N
I
30. 31.
Empirical formulas and molecular formulas
I
I
Dipole moments
20
42
35
3
Electranegativity
33
53
13
1 I
39
40
32.
18
elation of properties
6
4
to bond type
1
8
9
chemical equations PROPERTIES OF GASES and KINETIC-M0LECULJ.R THEORY 5.6%*
6.
Limiting reagent and yield of product
27
48
5
20
34.
Kinetic molecular theory
30
52
16
2
35.
partial pressures
21
52
26
1
136.
NO"-ideal equation
1
137.
~ffusion
1 1 / 2 7 6 4 1 8 1
&TOMIC THEORY and STRUCTURE OF ATOMS 11.1%* 38.
2
1
23
1
64
1
11
I
axw well-~oltzrnann 4
distribution
33
48
15
THE LIQUID and SOLID STATES 4.6%'
1 and sublimation 41. 12.
19.
~guationsfor nuclear reactions
28
Critical temperature and pressure
SOLUTIONS 5.6%' 5
26
37
32
46.
Colliqative properties of non-electrolvtes
47.
Colligative properties / 1 1 1 5 2 3 1 1 of electrolytes
CHEMICAL BONDING 10.2%' 20.
phase diagrams for one-comp.onent systems
Types of bonding (i.e., ionic, covalent, hydro en bordina van der WaalsI 6 8
I I I 26
2
4
5
28
56
11
21.
Metallic bonds
22.
L ~ W structure ~ S
65
29
4
2
23.
Resonance
20
54
24
2
24.
valence shell e-pair repulsion model
61
32
4
3
I
/ 48.
Osmotic pressure
1 2 3 9 / 4 3 1 1 6 1
ACIDS, BASES and SALTS 7.4% 149.
Arrhenius concept of acids and bases
I
Br@nsted-Lowry concept of acids and bases 1 6 0 3 0 /
50.
151. Lewis concept of
1 3 0 / 3 1 1 3 7 / 2
2l
9
38
36
37
51
and bases ..
28.
Sigma and pi bonds
9.
Polarity of bonds
30
50
17
3
1 6 1
52.
Amphoterism
53.
Acid-base indicators
10
37
16
54.
Titration curves
26
68
21
1l
1
~igure1. Psrcentof faculty indicating coverageof topics in collegegeneral chemistrycourses(total N= 114);E = extensivecoverage:M = moderate coverage: B = brief coverage; N = no coverage (' = average percentage of one-year course). Figure continued on nea page.
242
Journal of Chemical Education
~ e l a t i v eCoverage/Topi
. tained from the estimates of student performance on questions receiving some inappropriate ratings are greater than those obtained for performance estimates on questions for which there was unanimity among the faculty on appropriateness. (For the latter questions the estimated performance ranged froma mean percent correct,P+, of 60%to 72%). The average standard deviation for the percent of students likely to answer correctly (P+) the questions that received some inappropriate rating is 24.41. Yet the average standard deviation for the percent of students likely to answer correctly the auestions for which there was 100% aereement bv the faculty on appropriateness is 15.34. he large standarddeviation and the relativelv . high - mean percent correct for estimated performance (mean Pt = 4iw0 to 5690) for the questions that received some inappropriate ratings indicate that the 85% of faculty who rated these questions as appropriate for collegegeneralchemistry also thought they wereof medium or averaee difficultv. This soread of ~erceiveddifficultv is predictab'ie given t h i recognized differences in expected outcomes for general chemistry courses a t different institutions and the variability of background for students taking different courses. The evaluation survey also sought to obtain some indication of the relative class sizes of thegeneral chemistry classes
at the participating institutions. Faculty were asked to describe their class size as being 1100, 2101 but 5500, 2501 but 11000, or >1000. The results indicated, not surprisingly, that there was a significant drop in enrollment a t most participating institutions for the second semester or second and third quarter of a full-year general chemistry sequence. I t is relevant t o keep in mind that students taking the AP Chemistry examination are those who have completed the fullyear AP Chemistry course. I t is of interest to examine some of the questions on the AP Chemistry examination (multiple-choice section) that were rated as "inappropriate" by a t least 15%of the respondents and also to look at a sample question for which there was unanimous agreement on "appropriateness" for a college general chemistry course. Sample questions from the 1984 AP Chemistry examination are shown in Table 4. Question 31 was judged as too esoteric for a general chemistry course by 37% of respondents. This response is not particularly unexpected for a question on descriptive chemistry. Yet the mean percent of students a t the participating institutions likely to answer this question correctly was estimated to be 52%. Forty-six percent of the AP Chemistry students answered this question correctly. Question 45 is one that cannot be answered by simple recall. It requires a quantitative application of stoichiometry and an understanding of the meaning of empirical formulas. I t does not require any knowledge of organic chemistry. The question was judged to be inappropriately difficult by 15%of the respondents and too esoteric by 10%.Estimates of likely student performance for obtaining the correct answer (P+) were given at 46%. Fifty-nine percent of the AP students answered this question correctly. Question 56 calls on qualitative application of thermodynamics principles and covers topics extensively covered in college chemistry courses, according to the curriculum survey described in this paper. I t was judged to be inappropriately difficult by 10%of the respondents and too esoteric by another 10%.Yet the mean percent of students estimated to
Figure 4. Questions from the 1984 Advanced Placement Chemisby Examination (reprimed by permission of Educational Testing Service): answers: 31-E; 52-A: 5 6 4 .
248
Journal of Chemical Education
45-A:
answer the question correctly was 53%. This is an example of considerable disameement amone facultv in the colleees and universities participating in the study regarding course difficultv andlor anticipated student outcomes. Thirtv-six oercent of the AP chemistry students answered the-question correctlv. ~ u e s i i o n52 is one on which there were both unanimity by facultv and excellent correspondence on predicted performance. i t is a s t o i ~ h i o m e t r ~ - ~ u e s tinvolving ion limiting reagents. One hundred percent of the respondents judged this question to be appropriate for college general chemistry courses. T h e mean percent of students estimated by the facultv to answer this ouestion correctlv was 64%. Sixtvthreepercent of the AP students answered this correctlv. over&, college teachers judged the AP Chemistry multiple-choice test to be appropriate in both content and difficulty as the data in ~ i g u r 3 e show. Of those who responded, 47% thought the AP Chemistry examination (multiplechoice section) is comparable to college examinations, 23% thought i t was easier, and 30% thought i t was more difficult. This bell-shaped distribution suggests that no changes in the overall difficulty of the multiple-choice section of the examination are warranted. In fact, is reasonable to infer that college faculty generally consider the AP Chemistry examination to be challenging for its intended purposes. This is further corroborated by additional data obtained from college facultv regarding timing. College teachers would have-allotted theirstudents an average of131 min for this part of the examination, whereas AP students are allotted 90 min. However, data collected on actual performance of AP candidates do not show test speededness to be as severe a problem a s the faculty survey suggests. In 1987, 99.5% of AP candidates completed 75% of the multiplechoice questions on the examination. Another measure of test speededness is to check the number of questions reached by 80% of students taking the test. I n 1987,80% of AP Chemistry students reached question 78 (out of 80 questions). Data for the past five gears are comparable. Nevertheless, the- 90-min &ultiple-choice section of the AP Chemistry examination has been reduced to 75 questions. as of Mav 1988. This adiustment was a ~ o r o v e dbv the ~ e v e l o p m e ncommittee t for ;\P Chemistry & !f two reasons:
-
(1) it reflects more accurately the expectations demanded of college students, and (2) it allows AP students more time to work on problems without reducing the complexity of such problems to levels measuring additional simple recall of facts.
Conclusions The major findings resulting from studies of a nationwide curriculum survey of general chemistry courses at institutions receivinglargenumbersof APstudentsand of a faculty evaluation survey of the objective section of an AP Chemistry examination were the following: (1) The college general chemistry course is crowded with respect to
the number of topics it covers. This is true despite the oft-cited criticism that the curriculum of introductory college chemistry attempts to cover more topics than students can reasonably be expected to learn. As long as this is the case, the AP Chemistry course will continue to require similar broad coverage since it is designed to reflect the college course. Minor adjustments to the AP Chemistry course have been made as a result of the survey. (2) Topics in descriptive chemistry dominate the list for which 20% or more of resoondents indicated no coveraee in the colleee ~ ~ general chemiatrv rourse. However, hased on the analyzmp: of topics that did receive moderate and extensive coterage, the Cvllr~eBoard Tent Developn~entCumm~tteefor Al' Chemistry voted to retain the present coverage uf dewiptive chemistry described in the AP Chemistry Courre Descrrption. (31 The time devoted to the chemistry laboratory is rianificantlv greater in colleges than it is in secondary schools. The revised Course Description for AP Chemistry (1989) will encourage secondary schools to increase their current laboratory offerings for AP Chemistry and will publish an expanded list of experiments appropriate for AP Chemistry. (4) Faculty ratings of AP Chemistry multiple-choice questions and predictions of college student performance indicate that the AP Chemistry examination is an appropriate measure of the content in typical college.general chemistry courses and that the examination meets the needs of a national examination intended to he representative of a broad spectrum of corresponding college courses. The fact that 82% of the multiple-choice questions were rated as appropriate by at least 85% of the respondents lends content validity to the AP Chemistry examination.
~..~ ~~~~
~
Acknowledgment We are gratefully indebted to the colleges and universities that participated in this nationwide survey.
ASTS Introduces Advanced Placement Chemistry Course The A m and Sciences Teleconferencing Service at Oklahoma State University is broadcasting Advanced Placement Chemistry as one of four new courses this yew. Advanced I'lacrment Chemistry is the result of a $374,0U0 grant from the National Science Foundation's science and engineering education directorate and was developed by I. Dwaine Eubanks and John Gelder. It includes video graphics, demonstrations, and computer-generated animation. AP Chemistry is aimed at students who have had one year of high school chemistry. It meets College Board requirements and allows students who do well in the course to take the Advanced Placement Exam and possibly receive college credit. AP Chemistry is broadcast three days a week and the remaining days consist of laboratory experiments. Prerequisites for AP Chemistry are Algehra I and 11, Geometry, and one year of high-school chemistry. Currently 21 schools in 12 states are participating in AP Chemistry by satellite. For further information contact: Leigh Walters, Program Director at (405) 744-7895.
Volume 67
Number 3 March 1990
247
- "
