Research: Science and Education
Item Design Considerations for Computer-Based Testing of Student Learning in Chemistry Craig W . Bowen Deportment of Chemistry and Biochemistry, University of Southern Mississippi, Box 5043, Hattiesburg, MS 39406-5043 Across the nation there are many efforts going on to reform the general chemisrry curriculum. A great deal of discussion has occurred concerning general chemisrry contem and pedagogy (see selections in New Directions for General Chemistry, ref J). While litde about assessing learning has accompanied these discussions, a workshop sponsored by the National Science Foundation included a component on assessment of learning. One recommendation in the execurive summary addressed chis issue by scaring (2): The methods we use for assessing our students and our teaching must change so that they no longer focus on the lowest levels of learning and so that they provide us with the insight into our methods and our tools that we need to drive change. The purpose of this paper is ro consider how computers might help to broaden assessment pracrices by examining the types of test items chat might be used in com purer environments-both for originally developed items and by "repurposing" exisring media. Ir examines ideas from cognitive psychology and research on problem solving in chemistry that can be used co guide irem design, and chen briefly reviews how compurers have been used for assessing student learning in the past. T he paper should be ofimeresr to people developing computer-based insrrucrional materials in chemistry because ir can provide guidance on item writing for computer environmentS. lc should also be of interesr ro chemisrry instructors because it can help rhem ro chink abour how to evaluare com purer-based materials for facilitating and assessing studem learning. Because its focus is on designing com purerbased test irems (rather than test administrarion), the paper will leave discussion of security issues associated wirh computer-based resring to ocher papers.
Ideas from Cognitive Psychology and Problem Solving Research in Chemistry It is recognized that chemisrry is a multifaceted science requiring complex thinking. Cognirive psychology offers a way for considering how thinking takes place (3-4). Representarions are models about the world that people create from their existing knowledge while solving problems. D ifferent kinds of representations (e.g., external/internal or verb~picrorial) have different advantages and disadvantages for problem solving. Problem solving in chemistry involves represeming phenomena in ar least three ways: macroscopically, symbolically, and at the particulate level (5-8). These representational approaches are summarized as follows. Macroscopic Representations. Models of the world that are based on knowledge and operations involving an understanding of observable chemical phenomena. Symbolic Representations. Models of the world that are based on knowledge and operations involving descriptions or explanations of chemical phenomena that have been translated inro a different symbolic form (e.g., mathematical or verbal). 1172
Particulate Representations. Models of the world that are based on knowledge and operations grounded in imagining what atoms and molecules do during various chemical and physical changes. As an example, the macroscopic level involves what is observed in a laboratory situation or through a demonstration. The mental model one builds from noticing that a white milky solid forms when aqueous solutions of sodium chloride and silver nitrate are mixed is a macroscopic representation. The symbolic level is most common ro chemists because it involves using symbols (of either a chemical or mathematical narure} ro represent chemical phenomena. For example, the equation below symbolically represents the macroscopic reaction of hydrogen and oxygen reacting to form water.
0 2 + 2H2 ~ 2H2 0
+ heat
The ideal gas law, PV = nRT is another symbolic represemation mathematically relating various properties of ideal gases. ,Chemists also think about what atoms and molecules do when undergoing chemical or physical changes. When water boils, for example, chemises imagine molecules moving further apart instead of breaking bonds within a molecule as in Figure I. These ways of represeming chemical phenomena are important because they allow chemists to solve many rypes of chemical problems. While chemists realize and understand the imP.ortance of these ways of representing phenomena, many studems do not. Herron, in his address to the Royal Society of Chemistry, expressed the importance of representations in problem solving (Herron, J. D. Students' Understanding ofChemistry: An Issue in Chemical Education; paper presented at the Nyholm Symposium, Royal Society of Chemisrry, London, 1983}: What the experienced chemist writes on paper and what the novice writes on paper may appear to be similar, bur examination of problem-solving protocols suggests that the thought processes may be very different. Whereas the expert is using symbols ro represent physical events that he imagines ro be raking place in accordance with general laws of nature, the student is using symbols to represent symbols which he then manipulates according to memorized rules which have no connection with physical reality. Given char these ways of represeming chemical phenomena-macroscopically, symbolically, or ar rhe parriculare level-are used by chemists while solving chemical problems,
Figure 1. Representation of what happens duri ng phase chonge from liquid water to steam.
Journal of Chemical Education • Vol. 7 5 No. 9 September 1998 • JChemEd.chem. wisc.edu
Research : Science & Education ic makes sense co use them as a framework for measuring chemistry learning. Bowen and Bunce recendy reported che development of a new examination published by the ACS Examinations Institute that involves measuring student understanding of macroscopic and particulate knowledge of chemical concepts (9). An important point of this work is that the three ways of representing chemical phenomena can be used to design assessment activities. Paper-and-pencil tests can be developed to measure some of these aspecrs of learning, but other computer-based assessment might be better suited for determining student understanding of chemistry across these representational approaches.
Past Uses of Computers in Assessing Student Learning Computers have been utilized in testing student learning in chemistry for at lease 30 years. They have been used primarily for administrative reasons (e.g:, producing customized tests for makeup exams) rather than for the unique type of items they might include. One of the first reported uses involved computer scoring of student answer sheers or lab results in large classes (I 0, 11). Computers have since found many more roles in the assessment process. For example, Kumar and Helgeson ( 12) explained that computers are now used for the conscruccion, application, and scoring of multiple-choice exams, the development and administration of constructed response exams, and even computer-adaptive testing systems, which are very useful in the assessment area where solution pathway analysis is important. The recent increase in Internet usage has even opened the door for online testing practices (13, 14). These articles and others primarily indicate che computer as a useful management tool (for construCting or administering tests or keeping track of student grades). The types ofquiz or test items that are often presented via computer could just as easily appear on paper. However, measurement of student chemistry learning can be much broader today because of che video and animation power available on computers.
Types of Items in Computer-Based Testing Like paper-and-pencil resrs, computers might be used to measure student understanding of chemistry in at least two very different ways. On one hand, computer-based testing can present items co students that are very open-ended. For example, work being done at the ACS Examinations Institute is focusing on developing icems that require high school scudenrs to integrate several areas of chemistry in order to answer scenario-based problems (15) . One scenario item being developed is the Consumer Watchdog. AI , .,..._......,. . . . . ..,.b a W'bi:k, ~ .....'-olf•
........ --w.o .... ..... _ .. .....,.,.......,...,
00
2.
.,. . . .
The srudenr assumes the role of a laborarory technician in an organization that concerns irself with the chemical composition of consumer products. In this instance, the organization is concerned about the fact that, while soft drink labels specify the total amounr of suga r in the beverage, they do not specify which sugars are present. In a series of inreractions with the laboratory director (which keep the studenr on track), the student uses a spectrophotometric technique (the Folin-Wu method, which is based on Beer's law) to find the relative amounts offructose and glucose in the beverages. The student prepares a report of the invescigacion for Watchdog Reports.
Multiple-choice type items represent a second approach to computer-based testing. Our research group is working co develop a test-item database of multiple-choice items that measure student understanding across the multiple representational levels described earlier (16; Bowen, C. W. Consequmw of Cognitive Science for Assessing Student Learning in Chemistry, paper presented at the Gordon Research Conference on Innovations in Teaching College Chemistry; Plymouth, NH, July 1996.). Several sample items across the macroscopic, particulate, symbolic, and integrated levels are given below. Macroscopic Items. Items focus on measuring student understanding of observable chemical phenomena as shown in Figure 2. In these cases students are presenred digitized video of various chemical situations. Questions are asked to measure their ability to interpret macroscopic phenomena. These items would be more difficult to pose in a paper-andpencil environment-panicularly a dynamic item such as the polar and nonpolar liquids in sample item 2. Particulate Items. Items focus on measuring student understanding of what atoms and molecules are doing during various chemical and physical changes. With sample items in Figure 3 we are trying to measure srudencs' understanding of what is occurring at the nano-level. Because of the dynamic nature of chemistry, animations in a computer environment are more useful for measuring student learning in this area than are paper-and-pencil approaches. These images may be static in nature (e.g., the crystal structure question), or need animation (e.g., raising the temperature of the gas). Symbolic Items. Items focus on measuring student understanding of descriptions or explanations of chemical phenomena that have been translated into a different symbolic form (e.g., mathematical or verbal) shown in Figure 4. These types of items are typically found on paper-and-pencil general chemistry tesrs (if multiple-choice items are used). They do not call for heavy video or animation capabilities of computers. However, they should be included because we do expect students to think about chemistry and solve problems across all three levels.
... ,..,.
Afll-.W"dwp4-'d:IM~...., ......... .,...,.
~. ~ilpi··-....,
o,,.,..
O K-wo.y.,. o w..... 0 0sytMWAit
0
0
0
0
Figure 2 . Three sample macroscopic items. (Available from author.)
JChemEd.chem.wisc.edu • Vol. 75 N o. 9 September 199 8 • Journal of Chemical Education
1173
Research: Science and Education
t.
WNch ~4 ~ iht Mtt tttt-nla* .. a «nbtlft COCIWNCIJ up uocntuMid~Mf~M.....,..?
__
,
l. O.tow I• • ,..,., wbicb ~ a picture of .om. •'*IMOe II a •i• iM\anl ia lim.. WN.th ehoiot *taiN-t lht pt»M ..
3·. Tbtatimuiontoiht~~moltooutM.. apmo~lllplnieuiV~IUN . ltf» ~-.MNM4, wblcllolWdloioMt.low
~
•••
b.t~- Chtf't
