Using Ordered Multiple-Choice Items To Assess Students

Article pubs.acs.org/jchemeduc

Using Ordered Multiple-Choice Items To Assess Students’ Understanding of the Structure and Composition of Matter Jan C. Hadenfeldt,*,† Sascha Bernholt,† Xiufeng Liu,‡ Knut Neumann,† and Ilka Parchmann† †

Leibniz Institute for Science and Mathematics Education (IPN), Kiel, Germany Department of Learning and Instruction, State University of New York at Buffalo, Buffalo, New York 14260-1000, United States

‡

S Supporting Information *

ABSTRACT: Helping students develop a sound understanding of scientific concepts can be a major challenge. Lately, learning progressions have received increasing attention as a means to support students in developing understanding of core scientific concepts. At the center of a learning progression is a sequence of developmental levels reflecting an idealized progression toward understanding a particular core concept. This sequence is supposed to serve as a basis for designing instruction that can foster learning as well as assessments that can monitor students’ progression. So-called ordered multiple-choice (OMC) items have recently been suggested as a simple and effective way of assessing students’ level of understanding of a core concept. This article details our efforts in developing an instrument for assessing students’ understanding of the structure and composition of matter based on OMC items. Ten OMC items were developed and administered to a sample of N = 294 students in grades 6−12. Rasch analysis was used to investigate instrument functioning and to determine linear measures of person abilities and item difficulties. In addition to the OMC items, students were administered corresponding open-ended items in order to investigate the validity of the results obtained through the OMC items. Our findings suggest assessing students’ understanding of scientific concepts through OMC items is indeed quite worthwhile and should be subject to further research. KEYWORDS: Elementary/Middle School Science, High School/Introductory Chemistry, Chemical Education Research, Testing/Assessment FEATURE: Chemical Education Research

■

INTRODUCTION One important goal in science education is fostering students’ understanding of core scientific concepts.1,2 However, science education research has shown that students are struggling with developing a deeper understanding of these concepts.3 Recently, learning progressions have been suggested as means to foster students’ progression in understanding core concepts.4 Learning progressions build on the idea that instruction and assessment need to be aligned. At the core of a learning progression is a sequence of levels reflecting an increasingly sophisticated understanding of the concept. As students’ pathways to scientific understanding are oftentimes not linear and not the same for every student, this sequence is thought to describe an idealized pathway.5 And while students are not expected to all have the same level of understanding or progress in the same way, each level is considered to mark a major step toward a deeper scientific understanding. Instructional components specifically designed for each level of understanding are expected to help students progress to the next level. However, in order to choose the proper instructional component for their students (or individual groups of students), teachers need to know about students’ current level of understanding.6 Hence, assessment instruments are © 2013 American Chemical Society and Division of Chemical Education, Inc.

needed that can elicit individual students’ level of understanding. While interviews or open-ended items are considered to provide a rich view of one student’s understanding, such methods do not efficiently provide an overview of students’ understanding in class. Although multiple-choice items are much more efficient for larger groups, they do not elicit students’ understanding as well as interviews can. Recently, Briggs, Alonzo, Schwab, and Wilson7 suggested that ordered multiple-choice (OMC) items might offer deeper insights into students’ thinking while still being efficient to analyze. In an OMC item, each response option reflects a particular level of understanding. In contrast to concept inventories, OMC items incorporate a developmental perspective. An instrument based on OMC items will not only provide information about which alternative conceptions of a scientific concept students hold but also allow for determining where students are on their way toward a deeper understanding of the respective concept. Students who consistently choose response options related to one level across a set of OMC items can be expected to have obtained that level of understanding. Thus, OMC items provide Published: November 5, 2013 1602

dx.doi.org/10.1021/ed3006192 | J. Chem. Educ. 2013, 90, 1602−1608

Journal of Chemical Education

Article

age levels provides a good picture of how students progress in their understanding of the structure and composition of matter.17−21 We synthesized five hierarchically ordered levels of students’ understanding of the structure and composition of matter from a review of this research. These levels describe a progression from an understanding of the macroscopic properties of matter toward an understanding of the particulate structure of matter and its relation to macroscopic properties. The lowest level, level 1, relates to an everyday conception of matter. At this level students do not use the idea of a particulate nature of matter when trying to explain phenomena relating to science. At the next level, level 2, students have learned about the particulate nature of matter. However, instead of using a particle model, students’ will explain phenomena using a hybrid model, in which particles are thought to be embedded in a substance.17,22 At level 3, students are able to make use of a simple particle model to explain phenomena. Not having learned about the substructure of these particles, they believe that matter is built by particles as the smallest unit.18,23 At level 4, students can make use of a differentiated particle model, in which each particle (i.e., atom) is made up of smaller particles (i.e., protons, electrons, and neutrons).14,24 And at level 5, students finally can explain macroscopic properties of matter as a result of the properties of the particles the matter is made from and the complex interactions between these particles.25,26 For a more detailed description of the individual levels, see the Supporting Information. It is important to note that while individual students may have a different level of understanding and how they progress through this sequence of levels may vary, the sequence represents a reasonable basis for assessing students’ level of understanding of the structure and composition of matter.

a more detailed picture of students’ understanding compared to multiple-choice items. Previous research suggests that OMC items have the potential to assess students’ level of understanding almost as well as interviews.8 The purpose of this paper is to detail our efforts in developing an instrument for assessing students’ understanding of matter based on OMC items. The specific questions to be answered by this paper are as follows: 1. How can a successful process of designing an instrument using OMC items to assess students’ understanding of matter be characterized? 2. To what extent does an instrument based on OMC items provide reliable and valid information about the level of students’ understanding of matter?

■

METHOD One approach to design assessment instruments has been suggested by Wilson.9 This approach describes four building blocks in the process of designing assessment instrument: the construct map, the items’ design, the outcome space, and the measurement model (see Figure 1). In the past, Wilson’s

Items’ Design

The second building block in the process of developing is the items’ design. This building block relates to specifying the type of item used to elicit evidence about students’ level of understanding of the construct with respect to the construct map.9 Recently, Briggs et al.7 suggested a novel type of items, so-called ordered multiple-choice items that aim to facilitate diagnostic assessment linking the response options in multiplechoice items to different levels of understanding of a construct. In an OMC item, each response option is assigned one level of understanding (Figure 2). Similar to regular multiple-choice

Figure 1. The four building blocks.9

approach has been successfully used to develop assessment instruments for a variety of different constructs.7,8,10 We will describe each of the building blocks in greater detail and discuss how to use them to design an instrument assessing students’ understanding of matter based on OMC items. Construct Map

The first building block entails the construction of a construct map. A construct map provides a substantive definition of the construct and an ordering of qualitatively different levels of increasing sophistication of the construct.10 That is, construct map defines what students should understand and how students progress in developing this understanding.11 In the case of our instrument, what students should understand is the concept of matter. Students’ understanding of matter is well researched.12 Although different conceptions of what students should understand about matter can be found in the literature,13−15 researchers agree that an understanding of the structure and composition of matter is central to understanding the concept of matter as a whole.16 And while there are only few longitudinal studies on students’ progression,17,18 the extensive research based on students’ conceptions of matter at different

Figure 2. Sample ordered multiple-choice item. In order to recognize option A as the correct answer, an understanding at level 4 is required. 1603



Article

item response theory (IRT) has proven to be fruitful. The IRT approach we used to analyze our data set is commonly known as Rasch analysis.32 For a detailed description of how Rasch analysis can guide the process of developing measurement instruments, see for example Liu and Boone.33 The first step in Rasch analysis is to define a scoring scheme. As all response options of an OMC item are linked to a specific level of understanding, each response option was assigned the value corresponding to the respective level of understanding, based on the framework described (from 1 to 5). Finally, a polytomous Rasch model, more specifically the partial credit Rasch model (PCM), was applied to the data. Analyzing the data set using the PCM can provide valuable information about item functioning with respect to the construct map. To detail this point, the design and findings of a study in which an instrument based on OMC items has been used will be reported in the next two sections.

items, one response option is correct. This option reflects the highest level of understanding that can be tested with this item. All other response options are incorrect. These options reflect (scientifically) incorrect ideas of students at lower levels of understanding. To develop OMC items assessing students’ understanding of the structure and composition of matter, in our study we used information from both the curriculum and research on students’ alternative conceptions. Based on the curriculum and respective textbooks, the item context and question were formulated.1,27 Once the question and correct response option were formulated, the response option was assigned a level of understanding. Subsequently, the response options for lower levels of understanding were formulated based on typical alternative conceptions students on the respective levels of understanding would typically have.28−30 On the one hand, the aim was to develop response options meaningful to students. On the other hand, it should not be obvious to students which answer is correct. A sample item is shown in Figure 3. As can be

■

DESIGN OF THE STUDY In order to investigate the functioning of the instrument assessing students’ understanding of the structure and composition of matter based on OMC items, we administered it to a sample of 294 students in grades 6−12 in a German grammar school. The instrument included a set of 10 OMC items. Because the school’s science curriculum would not allow for students to develop an understanding on level 5 yet, only items assessing an understanding up to level 4 were included in the instrument. In order to investigate to what extent the OMC items can be used to assess students’ understanding, openended items corresponding to the OMC items were included in the instrument. These items were basically open-ended versions of the OMC items (same question, but no multiple-choice response format). The testing was done on a single day. It was up to the teachers to decide whether a given class would participate in the study. For the classes that did participate, the instrument was administered to all students in those classes. (See Table 1 for

Figure 3. An OMC item for the category of structure and composition.31 Additional OMC items, as well as a more detailed description of the levels of understanding, can be found in the Supporting Information.

Table 1. Distribution of Students across Grades seen from this item, it is not that all levels of understanding have to be covered by response options.7 As a matter of fact, the item shown in Figure 3 can only be used to differentiate between the first three levels of understanding. The OMC items were authored and compiled into an instrument by a team of science educators and science teachers.

Grade Levels Students

6

7

8

9

10

11

12

Number

41

73

40

47

34

29

30

information about the distribution of participating students.) Due to limited testing time, each student was presented with a different version of the instrument. Each version included the 10 OMC items and three open-ended items. The different versions were designed so that each of the 10 open-ended items was answered by the same number of students. To avoid an influence of OMC items on the open-ended items, the openended items were given first. Students were allowed to proceed to the OMC items only after completing the open-ended items and were not allowed to go back to the open-ended items afterward. To obtain additional information for item refinement, each student received a questionnaire designed specifically to collect such information for a subset of three OMC items. This questionnaire included questions regarding, for example, words students would not understand, or their reasoning for choosing a particular option.34 Again, we made sure we would receive feedback for each item from about the same number of students.

Outcome Space

The third building block, outcome space, is about the relationship between the construct map and the items.9 In the case of our OMC items, this means how the response options relate to the levels of understanding of the structure and composition of matter. In order to investigate the validity of the assignment of levels to response options, the assignment was carried out independently by three researchers. A Fleiss κ measure was used to obtain information about the agreement of the three researchers. A value of κ = 0.88 suggests the researchers agreed quite well in their assignment of levels of understanding to the response options. Measurement Model

In the fourth and last building block, the measurement model (i.e., the relation between the scored outcomes and students’ level of understanding as described in the construct map) is defined.9 For the process of developing assessment instruments, 1604



■

Article

FINDINGS In the first step of our analysis, we compared students’ answers to the open-ended version of the OMC items and the response option chosen in the corresponding OMC item. Students’ answers to the open-ended items were assigned one level of understanding. This assignment was independently carried out by two different researchers. Cohen’s κ was used to determine inter-rater reliability. A value of κ = 0.91 indicates that the two researchers agreed in nearly every case. Examples of students’ responses and the respective level of understanding assigned to these responses are provided in Box 1.

In the second step of our analysis, we investigated whether the OMC items test a unidimensional trait (i.e., students’ understanding of the structure and composition of matter). Rasch analysis, specifically the partial credit Rasch model, was used to investigate to what extent the individual items fit on a one-dimensional latent scale on which students in grades 6−12 might progress. For analyzing item quality, the mean square residual (MNSQ) and the standardized mean square residual (ZSTD) are two important indicators. They resemble the difference between what is observed and what is expected by the Rasch model. Items with a good model fit should have MNSQ values within the range of 0.8 to 1.2 and ZSTD within the range of −2.0 to 2.0.35 Analysis of item quality reveals that the items meet the standards for item quality criteria except for one, item 8. (See the Supporting Information for detailed item characteristics.) This item addressed students’ understanding of particle dynamics, whereas the other items focused mainly on structure and structure−property ideas. Another indicator for instrument functioning is the WLE reliability, which can be interpreted similarly to Cronbach’s κ, in which values above 0.7 are supportive of reliability inferences.36 The WLE reliability was found to be 0.71. This can be considered as adequate given the complex nature of the trait and the small number of items.36 In summary, the fit statistics support the assumption that the partial credit Rasch model fits to the data and that the obtained data from the students can be analyzed with the Rasch model, indicating the local stochastic independence of the items and their alignment to a single unidimensional latent trait. One of the benefits of using Rasch analysis is that a student’s ability can be analyzed in relation to item difficulty, as the values for both item difficulty and student ability are transformed to the same scale.35 Regarding the current item design, a student gets a specific score in accordance to his or her selected answer: The higher the conceptual understanding represented in the chosen answer in the framework, the higher the score the student gets. The PCM analysis provides threshold values (so-called Thurstonian thresholds) that reflect students’ progression from one level of understanding to the next. If a student’s ability is above the threshold for a score category, the probability of achieving that score or higher is above 50% for that particular student.37 With regard to the Thurstonian thresholds for the items 1 to 10 (Figure 4), the items cover a wide range of ability levels. In addition, the thresholds for all items are in the correct order, that is, it is more difficult to reach a higher level of understanding than a lower level. The results displayed in Figure 4 thus indicate that

Box 1. Students’ Responses to the Open-Ended Items by Level of Understanding

In 79% of the cases, the level assigned to a student’s response to an open-ended item matched the level of understanding students demonstrated in the corresponding OMC item. (This is the average of all values reported in the second column of Table 2). In order to analyze to which extent the level of understanding identified through OMC and open ended items is the same, we calculated the following measure of agreement: The level assigned to a student’s open answer was subtracted from the level of understanding identified through the corresponding OMC item (leading to the number of paired answers listed in column 3 of Table 2). For each item we then calculated the mean and standard deviation of this discrepancy. The overall mean across all items was M = 0.1 with a standard deviation of SD = 0.67. A Wilcoxon signed ranks test revealed that in four cases the discrepancy is statistically significantly different from zero (see Table 2). These results indicate that the majority of the OMC items conform to their open-ended counterparts regarding the assessment of students’ level of understanding of the structure and composition of matter. Table 2. Comparison of OMC Items and Open-Ended Items Item

Conformity of Students’ Level of Understanding Assessed through OMC and Open-Ended Items, %

Mean Discrepancy of Levels

Number of Paired Answers

Standard Deviation of the Discrepancy of Levels

Wilcoxon p Valuesa

1 2 3 4 5 6 7 8 9 10

89 79 77 90 84 81 92 60 67 68

−0.12 0.21 0.28 0.03 0.12 −0.10 0.03 0.06 0.36 0.56

59 50 41 59 33 33 41 51 36 59

0.50 0.55 0.56 0.32 0.49 0.54 0.28 1.10 0.86 0.96

0.09 0.02 0.01 0.48 0.20 0.37 0.77 0.57 0.02 0.00

a

N = 294. 1605



Article

level, the average person ability increases constantly. The boxplot reveals a substantial overlap of person ability between two adjacent grades; nevertheless, 54% of the ability measure can be explained by the person’s grade: τ = 0.55, p < 0.001; F(6,287) = 28.92; p < 0.001; R2 = 0.536. Although the data were obtained from a cross-sectional design and though no inferences about individual learning paths can be made, this result again supports the structural validity of the data.38

■

CONCLUSIONS AND OUTLOOK In this paper, we described a procedure for successfully developing an instrument based on ordered multiple-choice items to assess students’ understanding of the structure and composition of matter (Question 1). Moreover, we investigated the extent to which the instrument based on OMC items is suitable to provide science educators with reliable and valid information about how students progress in their understanding of matter (Question 2). Our findings suggest that an instrument based on OMC items can assess students’ understanding of the structure and composition of matter as well as open-ended items. We found OMC items to be able to detect different levels of understanding and to detect students’ progress in understanding the structure and composition of matter. Overall, these findings provide initial evidence that OMC items can be used to design instruments for a valid assessment of students’ understanding of core concepts in science. Although our findings create a sound base for further research on how to assess students’ understanding of the structure and composition of matter using OMC items, some issues remain in terms of instrument development. The procedure described in the Method section resulted in an initial version of an instrument that already could produce meaningful results. However, instrument development is an iterative process.9 Based on the findings, we identified the following points that need to be addressed in refinement of the instrument. First, as this instrument was administered to students in grades 6−12, students’ response behavior needs to be more closely analyzed in terms of the items’ wording and clarity. Findings from a modified AAAS questionnaire indicate that although some students in grades 6 and 7 were not familiar with all scientific terms provided in the OMC items, about 66% were sure that only their chosen answer was correct. This percentage increases with grade. This result indicates that OMC items need to be carefully developed with respect to the target population. Without the usage of scientific terms, it is difficult to assess students’ understanding at higher levels; nonetheless, including scientific terms may confuse younger students. To address this problem, research needs to be carried out, especially with younger students (e.g., using think-aloud interviews), to identify commonly understood as well as problematic terms in the items. Second, as Table 2 shows, some open-ended counterparts of the administered OMC items reflect students’ response behavior better than others. This result indicates that not all of students’ possible answers were covered with the response options provided in the current version of the OMC items. Some open-ended answers could not be assigned a level of understanding. However, these answers can guide the development of additional response options for a revised version of the items. In return, analyzing the distribution of chosen response options per item might reveal those options that appear unattractive or implausible and thus rarely chosen by students.

Figure 4. Thresholds for items 1 to 10.

the assumption of ordered multiple-choice options in fact holds true for the developed set of items. In line with the framework, a higher level of understanding as represented in the OMC options was difficult to reach for the students. An analysis of variance indicates that about 68% of the variance in the thresholds is explained by the assigned level of understanding: τ = 0.69, p < 0.001; F(2,22) = 23.23; p < 0.001; R2 = 0.679. Figure 5 (left) illustrates the distribution of the

Figure 5. Threshold over level of understanding required to solve the item (left), person ability over grade (right).

threshold parameters for the three levels of understanding. Accordingly, higher levels of understanding are on average more difficult than lower levels of understanding. In addition, the overlap between two adjacent levels is quite small, supporting the idea of hierarchically ordered levels of understanding. (However, no perfectly disjunctive hierarchy is expected.) This suggests that the OMC items can indeed assess different levels of understanding of the structure and composition of matter. In the third and last step of our analysis, we investigated the extent to which the OMC items were suitable to measure students’ progression in understanding the structure and composition of matter. For this purpose, we investigated to which extent students from higher grades (i.e., students who had received more instruction on the concept of matter) were showing higher person ability measures. As shown in Figure 5 (right), person ability measures were indeed found to increase with increasing grade. Despite the small sample size per grade 1606



■

Third, although the levels linked to the response options of the OMC have a significant influence on item difficulty (see Figure 5), item difficulty still overlaps for some items from different levels. That is, levels of understanding cannot necessarily be distinguished from each other. Therefore, response options should be refined in order to better represent the different levels of understanding. For example, in item 8 (Figure 4), the difference in difficulty between level 2 and level 3 needs to be adjusted. Response options that were not attractive to students could be revised. Conversely, response options that reflected quite common misconceptions could be revised in order to make the right answer more attractive. Although the items cover the latent trait well, more items need to be developed. Particularly, more items are needed with thresholds between 0.7 to 1.7 logits. As it is well-known that students do not respond consistently to similar problems set in different contexts,6 more items need to be developed to get more information about the role of context with respect to the structure and composition of matter. The clear progression (see Figure 5) might be due to the fact that most of the OMC items focused on structure−property ideas. It would be of interest to determine to what extent the progression in students’ understanding was due to items that focused on single ideas (i.e., dynamics or structure itself). As the only misfitting item was the one that addressed dynamics of particles, it might be assumed that students might progress at different rates along each idea. This would be another issue to be tackled in further studies. The possibility of establishing a link between a proposed framework (i.e., a construct map) and students’ performance on the instrument (i.e., students’ scores) is one of the main advantages of using Rasch analysis. Rasch analysis allowed us to link students’ performance to the difficulty of (different levels of understanding in) OMC items. In the case of our instrument, it seems that students performing well on the instrument have indeed reached a high level of understanding with respect to the underlying framework. Students with a lower achievement were found to prefer response options assigned to lower levels of understanding. Thus, the instrument enables differentiation between high- and low-performing students, as well as allowing interpretation of this performance in terms of qualitatively different levels of understanding of the composition and structure of matter. The possibility of this kind of interpretation makes the instrument interesting for further research and for classroom practice. Teachers can use the instrument in two important ways: (i) as a formative test to see which level of understanding students enter class with and to monitor student learning throughout instruction; and (ii) as a tool for evaluation to assess student learning and give feedback to students about their achievement. Analysis can focus on the distribution of students’ responses to single items as well as students’ performance on the entire test. While Rasch analysis needs specialized software (and an analysis with data from a single classroom is not necessarily meaningful), once instrument functioning has been fully established, teachers can simply evaluate their students’ thinking by assigning them the level of understanding reflected by the chosen response option. Based on the analysis of both students’ performance on single items and the whole instrument, teachers may then design instructional components to specifically address their students’ needs. In summary, instruments based on OMC items seem to offer teachers a simple and efficient way to obtain a detailed picture about students’ understanding of core concepts.

Article

ASSOCIATED CONTENT

S Supporting Information *

Detailed description of the individual levels and of the OMC items. This material is available via the Internet at http://pubs. acs.org.

■

AUTHOR INFORMATION

Corresponding Author

*E-mail: [email protected]. Notes

The authors declare no competing financial interest.

■

ACKNOWLEDGMENTS We would like to thank the journal reviewers and editors for their detailed and helpful recommendations. The research reported here was supported by the German Federal Ministry of Education and Research.

■

REFERENCES

(1) National Research Council. National Science Education Standards; National Academy Press: Washington, DC, 1996. (2) Waddington, D.; Nentwig, P.; Schanze, S. Making It Comparable: Standards in Science Education; Waxmann: Münster, Germany, 2007. (3) Vosniadou, S. International Handbook of Research on Conceptual Change; Lawrence Erlbaum: Mahwah, NJ, 2008. (4) Duschl, R. A.; Schweingruber, H. A.; Shouse, A. W. Taking science to school: Learning and teaching science in grades K-8; National Academy Press: Washington, DC, 2007. (5) Neumann, K.; Viering, T.; Boone, W. J.; Fischer, H. E. J. Res. Sci. Teach. 2013, 50, 162−188. (6) Pellegrino, J. W.; Chudowsky, N.; Glaser, R., Eds. Knowing What Students Know: The Science and Design of Educational Assessment; National Academies Press: Washington, DC, 2001. (7) Briggs, D. C.; Alonzo, A. C.; Schwab, C.; Wilson, M. Educ. Assessment 2006, 11 (1), 33−63. (8) Alonzo, A. C.; Steedle, J. T. Sci. Educ. 2009, 93 (3), 389−421. (9) Wilson, M. Constructing Measures: An Item Response Modeling Approach; Lawrence Erlbaum: Mahwah, NJ, 2005. (10) Wilson, M. Res. Sci. Teach. 2009, 46 (6), 716−730. (11) Wilson, M. Responding to a Challenge That Learning Progressions Pose To Measurement Practice. In Learning Progressions in Science, Alonzo, A., Gotwals, A., Eds.; Sense Publishers: Rotterdam, 2012; pp 317−344. (12) Duit, R. Bibliography for Students’ and Teachers’ Conceptions and Science Education. http://www.ipn.uni-kiel.de/aktuell/stcse/stcse.html (accessed Oct 2013). (13) Smith, C.; Wiser, M.; Anderson, C. W.; Krajcik, J.; Coppola, B. Measurement 2006, 14 (1, 2), 1−98. (14) Stevens, S.; Delgado, C.; Krajcik, J. S. J. Res. Sci. Teach. 2010, 47 (6), 687−715. (15) Liu, X.; Lesniak, K. J. Res. Sci. Teach. 2005, 43 (3), 320−347. (16) Othman, J.; Treagust, D. F.; Chandrasegaran, A. L. Int. J. Sci. Educ. 2008, 30 (11), 1531−1550. (17) Johnson, P. Int. J. Sci. Educ. 2002, 24 (10), 1037−1054. (18) Löfgren, L.; Helldén, G. Int. J. Sci. Educ. 2009, 31 (12), 1631− 1655. (19) Liu, X. Int. J. Sci. Educ. 2001, 23 (1), 55−81. (20) Krnel, D.; Watson, R.; Glazar, S. A. Int. J. Sci. Educ. 1998, 20 (3), 257−289. (21) Anderson, B. Stud. Sci. Educ. 1990, 18, 53−85. (22) Renström, L.; Andersson, B.; Marton, F. J. Educ. Psych. 1990, 82 (3), 555−569. (23) Pfundt, H. Chim. Didakt. 1981, 7, 75−94. (24) Harrison, A. G.; Treagust, D. F. Sci. Educ. 1996, 80 (5), 509− 534. 1607



Article

(25) Taber, K. S. Sci. Educ. 2005, 89 (1), 94−116. (26) Ayas, A.; Ö zmen, H.; Ç alik, M. Int. J. Sci. Math. Educ. 2010, 8 (1), 165−184. (27) Baumert, J.; Bos, W.; Watermann, R. TIMSS/III. Schülerleistungen in Mathematik und den Naturwissenschaften am Ende der Sekundarstufe II im internationalen Vergleich. Zusammen fassung deskriptiver Ergebnisse; Max-Planck-Institut für Bildungsforschung: Berlin, 1998. (28) Kind, V. Beyond Appearances: Students’ Misconceptions about Basic Chemical Ideas, 2nd ed.; Durham University: Durham, U.K., 2004. http://www.rsc.org/images/Misconceptions_update_tcm18188603.pdf (accessed Oct 2013). (29) Taber, K. Chemical MisconceptionsPrevention, Diagnoses and Cure; Royal Society of Chemistry: London, 2002. (30) Talanquer, V. Int. J. Sci. Educ. 2009, 31 (15), 2123−2136. (31) Mullis, I. V. S.; Martin, M. O.; Foy, P. IEA’s TIMSS 2003 International Report on Achievement in the Mathematics Cognitive Domains: Findings from a Developmental Project; TIMSS & PIRLS International Study Center, Boston College: Chestnut Hill, MA, 2005. (32) Rasch, G. Probabilistic Models for Some Intelligence and Attainment Tests; University of Chicago Press: Chicago, 1980. (Originally published as Rasch, G. Probabilistic Models for Some Intelligence and Attainment Tests, Studies in Mathematical Psychology I; Danmarks Pædagogiske Institut: Copenhagen, 1960.) (33) Liu, X., Boone, W., Eds. Application of Rasch Measurement in Science Education; JAM Press: Maple Grove, MN, 2006. (34) American Association for the Advancement of Science. Getting Assessment Right: Rigorous Criteria, Student Data Inform Project 2061’s Assessment Design. 2061 Today 2007, 17 (1), 1−5. (35) Bond, T. G.; Fox, C. M. Applying the Rasch Model: Fundamental Measurement in the Human Sciences; Lawrence Erlbaum: Mahwah, NJ, 2001. (36) Field, A. Discovering Statistics Using SPSS; SAGE: London, 2009. (37) Wu, M.; Adams, R. Applying the Rasch Model to Psycho-Social Measurement: A Practical Approach; Educational Measurement Solutions: Melbourne, 2007. (38) Messick, S. Am. Psychol. 1995, 50 (9), 741−749.

1608


Using Ordered Multiple-Choice Items To Assess Students

Recommend Documents