Evaluating the Content and Response Process ... - ACS Publications

Article pubs.acs.org/jchemeduc

Evaluating the Content and Response Process Validity of Data from the Chemical Concepts Inventory Paul Schwartz and Jack Barbera* Department of Chemistry and Biochemistry, University of Northern Colorado, Greeley, Colorado 80639 United States ABSTRACT: Data produced by psychometric instruments are often used to inform understanding about a certain population’s knowledge of ideas or perspectives about specific topics. Concept inventories are an example of psychometric instruments used to probe students’ content knowledge within a defined framework. Concept inventories have been used as a measure of student understanding in science education for over two decades. The Chemical Concepts Inventory (CCI), published in 2002, is one of the most often referenced inventories in chemical education. However, not until the past year has detailed psychometric evidence been published about data from the instrument. Although this prior study focused on quantitative evidence to support the CCI as a whole, there are still no qualitative studies published to investigate the reasoning behind students’ responses. Therefore, the purpose of this study was to examine the response process validity of the CCI in order to understand the nature of student responses. In addition to evaluating individual items, this study examined the content validity of the CCI as a whole instrument. To do this, a survey was employed to understand expert judgments about the content being covered on the CCI. This information was used to group items by content prior to investigating student responses. It was found that a majority of the content on the CCI was limited to only two concepts (phase change and conservation of mass−matter). Student reasoning for their responses to items from these groupings was examined through think-aloud interviews. Both support for and threats to the response process validity for several items was observed. The results from this qualitative investigation can be used as a guide to improving the current CCI items, writing new items, and for the development of other concept inventories. KEYWORDS: High School/Introductory Chemistry, First-Year Undergraduate/General, Chemical Education Research, Misconceptions/Discrepant Events, Testing/Assessment FEATURE: Chemical Education Research

■

INTRODUCTION Diagnostic tools for assessing students’ conceptual understanding in various areas and levels of science have been developed and discussed for over 25 years.1−3 Recently several instruments have been developed specific to measuring students’ conceptual understanding in chemistry.4−17 Of these concept inventories, the Chemical Concepts Inventory (CCI),8 developed at Purdue University in 2002, is one of the most often referenced. Mulford and Robinson developed the instrument with the intent of measuring students’ alternate conceptions when entering a traditional first-semester general chemistry course, as well as their change in these alternate conceptions upon completion of the course. The test developers defined alternate conceptions as “concepts that are not consistent with the consensus of the scientific community”.8 The inventory contains 22 multiple-choice items, half of which are two-tiered. Two-tiered items are linked multiplechoice items wherein the first tier of an item asks about specific content and provides a number of option choices and the second tier of the item asks about the reason for choosing the option in the previous tier. Option choices in the second tier typically reflect various reasons related to the first tier option choices. The initial sample used to evaluate the instrument was 928 participants from a midwest land grant university. The researchers also used eight volunteer interviewees to verify that the items on the inventory were interpreted as intended by the © 2014 American Chemical Society and Division of Chemical Education, Inc.

authors. Overall, the researchers’ development of the CCI was modeled upon Treagust’s ten-step development method.3 A search using the Web of Science database found that the CCI has 76 citations (accessed August 2013) since its publication in 2002. The majority of these listings cite the CCI as a general measure of alternate conceptions in chemistry14,18−27 and also as a measure of specific alternate conceptions (e.g., phase change, conservation of mass−matter, concentration, stoichiometry).28−35 Some of the individual items from the CCI have been used in studies to measure students’ conceptual understanding.36−41 Other studies have used the entire instrument to measure students’ alternate conceptions38,39,42 as well as those of instructors.43 Because so many of these studies use the CCI to make claims about students’ alternate conceptions, it is imperative to understand the psychometric properties of the instrument. In order for items on a concept inventory to function properly, there should be consistency both within an individual item and among items covering a specific concept. Within an individual item, a student’s option selection should match their reasoning for choosing that option. For example, if a student chooses the correct answer, they should have an explanation for that choice based on a correct conception. Among a series of items meant to probe the same concept, a true measure of the Published: April 15, 2014 630

dx.doi.org/10.1021/ed400716p | J. Chem. Educ. 2014, 91, 630−640

Journal of Chemical Education

Article

Interview Protocols

items’ ability to detect the presence of an alternative conception is consistency among a student’s responses. This implies that there are multiple items on an inventory measuring the same construct in order to provide triangulation of data for a specific concept. A recent study has highlighted the lack of psychometric data published for the CCI.44 This study used quantitative tools to investigate the validity and reliability of data generated with the CCI and found that the CCI, “functions reasonably well as an instrument”.44 However, the author points out the need for further quantitative and qualitative investigation. Another recent research study has listed the CCI specifically as one of several concept inventories in the chemical education literature still in need of evidence based on response processes.45 The Standards for Educational and Psychological Testing46 describes the response process as, “questioning test takers about their performance strategies or responses to particular items.” As there is a lack of qualitative evidence to support the intended purpose of CCI data (measuring students’ alternate conceptions), the purpose of this study was to examine the content validity of the CCI as a whole; in addition, students’ responses were investigated for evidence of response process validity. Therefore, the following research questions will be addressed: 1. What concepts are covered on the CCI? 2. Are there any threats to the response process validity of the data obtained from the CCI?

Questioning students about their thought process can help to make meaning of trends in quantitative data and provide rich information regarding an individuals’ understanding. These prompts could be open-ended explanatory questions; however, richer data can be captured through the use of interview protocols, as this allows for the researcher to follow up with the interviewee to provide clarity on their thought process. In relation to concept inventory functioning, interviews can often give insight into student understanding that quantitative data cannot. For example, it may be found that the response frequency for certain options on an inventory item is either excessively high or low. However, without using interviews to probe students’ reasoning for choosing or not choosing these options, one does not know why they are so attractive or unattractive. All interviews were audio recorded and transcribed verbatim. During each interview, students read through and answered each multiple-choice item individually. Students circled their intended answer on the test paper provided and were asked by the interviewer why they chose their particular answer. Depending upon their initial response for each item, students were then asked follow-up questions to provide clarity. After all follow-up questions were complete for an individual item, the student went on to the next item. These types of think-aloud interviews have been used in other studies16,47−50 to elicit the thought process of an interviewee as they respond to questions. As the goal of these interviews is to evaluate students’ reasoning for selecting a response option and how this matches their conceptual understanding, the follow-up questions probe their understanding of keywords used in an explanation or present in the item or option choices.

■

METHODOLOGY Approval from the Institutional Review Board at the University of Northern Colorado was obtained prior to the collection of all data within this study. Consent form content, interview protocols, and data security procedures followed the approved methodologies.

■

RESULTS AND DISCUSSION

Content Analysis of CCI Items

Test Content Survey

The first author reviewed the responses for each participant filling out the test content survey. For each item, responses were categorized into the various topics reported and the percentages of responses for each topic were calculated. A pool of 38 participants (30 chemistry instructors and 8 chemistry education graduate students) completed the content analysis of the CCI items; for each item, the concepts described by the raters are presented in Table 1. Three items (2, 3, and 6) were rated as covering the concept of phase change. Six items (1, 4, 7/8, 10/11, 12/13, and 18/19) were rated as covering the concept of conservation of mass−matter. No other concepts were covered by more than two items. In order to accurately evaluate understanding of a concept, more than one item is required for triangulation. On the basis of the content analysis, the researchers determined that the concepts of phase change and conservation of mass−matter were covered by enough items on the CCI that further analysis of student understanding could be conducted. Therefore, the 11 items that represent these concepts were further investigated for evidence of response process validity. The two-tiered item pair 10/11 was excluded from either group as there was a lower than 50% agreement for either category.

The CCI purports to measure the alternate conceptions of students on topics found in many first-semester general chemistry courses. The developers of the CCI report developing a content list from material found in textbooks, journal articles, and exams produced by the American Chemical Society, Division of Chemical Education, Exams Institute.8 In order to evaluate the content validity of the CCI, a survey was created asking participants to review all 22 items and determine what topic they believed each covered. Each participant was given a copy of the CCI with a blank space placed after each item in which they were asked to write their response. Participants were recruited through a variety of means. Attendants at the 2012 Biennial Conference on Chemical Education were surveyed at a poster session during the conference. Other responses were gathered through e-mail or face-to-face requests by the researchers. Response Process Interview Participants

Because a goal of this study is to investigate how well items from the CCI uncover student conceptions, a series of student interviews were conducted. These interviews are meant to provide evidence for the response process validity of CCI data. Student participants were enrolled in either the first- or secondsemester of general chemistry at a midsize public research university. Volunteers were recruited during lecture and all interviews took place outside of class in a private interview room.

Interview Analysis

A total of 25 participants were interviewed. Six of the participants (24%) were enrolled in the first-semester course and 19 (76%) were enrolled in the second-semester course. 631



Article

response choices for each interview phase were ultimately combined and analyzed together. These response choices were organized by response patterns that arose from each interview. The first author analyzed each interview transcript in detail, coding the response process of each participant. The second author reviewed sample transcript sections for each item; coded sections were compared between the two authors. When discrepancies arose, the transcript and codes were discussed until agreement was reached. Each interview response was coded on the basis of whether a participant provided evidence of conceptual understanding, used the intended alternate conception, expressed confusion or misunderstanding about the item or response options, or used test-taking strategies. Coded responses across all interviews were grouped by common ideas communicated for an individual item. Where two or more participants used similar discrepant reasoning to answer an item either correctly or incorrectly, this discrepancy was labeled as a potential threat to the response process validity. Coded excerpts for an individual were reviewed for consistency within an item group; these comparisons provided further support for individual item functioning. Results from the interviews for each concept area will be presented separately. Within each section, a breakdown of the responses will be given followed by details about the function of each item or item pair. At the end of each section, details about the consistency among items will be presented.

Table 1. Range of Concepts Rated for an Individual Item or Item Pair, Shown with Percentage Agreement among Raters Item 1 2 3 4 5 6 7/8 9 10/11 12/13 14 15 16/17 18/19 20/21 22

Concepta

Agreement, %b (N = 38)

Conservation of Mass−Matter Stoichiometry/Limiting Reagent Phase Change Physical vs Chemical Change Phase Change Conservation of Mass−Matter Stoichiometry/Limiting Reagent Phase Change Physical vs Chemical Change Conservation of Mass−Matter Bond Energy Conservation of Mass−Matter Phase Change Conservation of Mass−Matter Size of an Atom/Mole Scale/Estimation/Proportion Solutions Specific Heat Capacity Thermodynamics Conservation of Mass−Matter Stoichiometry/Limiting Reagent Solutions Equilibrium Macro vs Microscopic Properties

81 22 84 13 83 89 87 72 13 84 92 47 38 92 74 26 87 49 37 70 33 90 13 82

Phase Change Items

The three items on the CCI shown to cover the concept of phase change are meant to uncover the extent of the alternate conception that water dissociates into hydrogen and oxygen when it evaporates.8 Table 2 summarizes the results for the

a

Only concepts with 10% or more agreement are listed. bSome raters categorized more than one concept per item; therefore, totals for an individual item may sum to more than 100%.

This split is reflective of the relatively smaller size of the firstsemester course during the data collection semesters. No data trends were found based on course enrollment. Interviews were conducted in two phases. The first interview phase (spring 2012) involved fifteen participants. During these interviews, each participant responded to all 11 items identified in the content analysis as covering either phase change or conservation of mass−matter. Upon analysis of these interviews, it was determined that more information was needed to evaluate possible trends in the data for each individual concept area. In order to investigate possible trends within a concept area, a second phase (spring 2013) of interviews was conducted. The second phase involved ten participants. During these interviews, participants only responded to items from one of the two groupings of items. This allowed additional time for follow-up questions within a grouping without increasing the overall length of the interview. During this second phase, three of the ten interviews focused on the phase change items and seven of the ten interviews focused on the conservation of mass−matter items. Across both phases there were a total of 18 interviews on the phase change items and 22 interviews on the conservation of mass−matter items. As there were more items in the conservation of mass−matter grouping, it took more total interviews to reconcile trends within the data. No differences were found between the student responses from phase one and those from phase two. The phase two interviews simply provided the researchers additional data points to categorize response trends. In analyzing the interview data, tables for both phase change items and conservation of mass−matter items were developed to record the response choices from each interview. The

Table 2. Number of Interview Responses by Category for Items Covering Phase Changea Item

Totally Correct, %

Totally Incorrect, %

Potential Threat, %

Other,b%

2 3 6

10 (55) 12 (66) 13 (72)

5 (28) 2 (11) 4 (22)

3 (17) 3 (17) 0 (0)

0 (0) 1(6) 1 (6)

N = 18 interview responses for each item. bThe category “Other” indicates interview explanations that do not fit within the prior categories.

a

items covering phase change. The Totally Correct category is the number of interviewees who selected the correct multiplechoice option for an item and provided a correct reasoning when explaining their answer selection. The Totally Incorrect category is the number of interviewees who selected an incorrect multiple-choice option and also explained the reasoning for their selection using the appropriate alternate conception being tested. The Threat category reflects the number of interviewees whose multiple-choice selection is influenced by a common theme beyond that of the correct or alternate conception. An explanation is only seen as a Threat if multiple individuals report a similar influence. The Other category captures the explanations that do not fit within the prior categories. These responses reflect explanations that do not support the multiple-choice option and do not have a common theme to the Threat category or additional responses in the Other category. Item 3. Students who choose the correct option for item 3 (Figure 1), option C, should have a correct conception about 632



Article

responding correctly to the item. This would severely deflate the difficulty of the item (determined by the fraction of students correctly responding to the item) and overestimate its utility in understanding students’ knowledge content. Item 2. Item 2 (Figure 2) also intends to measure the alternate conception of water dissociating into hydrogen and

Figure 1. Item 3 from the Chemical Concepts Inventory.8

what occurs on the molecular level during the process of evaporation and condensation. Students who choose incorrect option D should have incorrect understanding and hold to the alternate conception. However, it does not appear that the first two option choices, A and B, correspond with an alternate conception regarding phase change. In fact, the response frequency for each individual option choice appears in Table 3 and reveals that


oxygen when it boils. Option choice B corresponds to this alternate conception. However, three interviews revealed that students might be confused about the physical process of boiling, leading to an incorrect option choice. These students responded incorrectly to this item, even though they showed an overall correct conception about the phase change of water during the response process for the other items in this category. In describing their reasoning, one participant talked about water vapor as being something that escapes from the surface of the water but is not in the bubbles of the boiling water. This was the most common theme reported from the other participants who responded to item 2 incorrectly but showed a correct conception on the other phase change items. The following transcript excerpt reflects students’ confusion between the bubbles and water vapor. Interviewee 8014: So it was either oxygen gas or oxygen and hydrogen so I just, between the two, I just chose C. Because, I do not know, I had a choice between an answer so I went with it. Interviewer: Now you had said something about the water vapor being above the... Interviewee 8014: Yea... Water vapor to me is like what you feel when you put your hand over the steam and then water kind of condenses on your hand. So to me that does not seem like, that is not what’s in the bubbles... This confusion may indicate that the item is not always measuring a conceptual understanding of phase change but rather how familiar the students are with how water boils at the physical level. Because the item asks about the bubbles in the water, many of them see what is going on in the boiling water as separated from what they understand about the phase change process and, therefore, answer the item incorrectly even though they may have a correct conception about phase change. Item 6. Item 6 (Figure 3) is a pictorial item that asks students about the molecular level view after all of the water evaporates in a closed container. This item showed no apparent threats to the response process validity. Thirteen participants answered this item correctly and each of these participants used correct reasoning for choosing option E. Four participants who answered incorrectly also held an alternate conception about the process of water evaporating and included this as their reasoning for choosing each incorrect option. All but one participant interpreted the graphics and wording correctly. The one participant who answered option C, showing no water molecules left in the container at all, did appear to have a

Table 3. Response Frequency by Percentage for Item 3 Option

This study (N = 18)

Barbera44 (N = 3025)

Mulford and Robinson8 (N = 928)

A B C D

0 0 83 17

11 4 53 32

6 2 67 25

no interviewee thought that either option A or B were viable answers. These options also received extremely low (less than 15%) response rates in two quantitative studies.8,44 The low response rate for options A and B may seem trivial, as it would be expected that students who hold the alternate conception should choose D. However, three interviews revealed that students used process of elimination to exclude options A and B. Therefore, these students were able to arrive at the correct answer by reducing the item to no more than 50/ 50 guessing. An example of a student’s response process using guessing can be seen in the following transcript excerpt. Interviewee 8004: This I honestly did not really know so I just looked at each answer and I chose water vapor condenses from the air. And I just, out of all them, it just made the most sense to me water evaporates from the milk and condense on the outside of the glass. There probably is water in milk but I did not really feel like water would separate from milk and condense on the outside of the glass, that did not really make sense to me, and then the glass acts as a semipermeable membrane and allows water to pass but not the milk, that just does not sound possible to me and then, the coldness cause oxygen and hydrogen from the air to combine on the glass forming water it was kind of between this one and the one I chose but this one did not make as much sense to me because just that the coldness of the glass would make oxygen and hydrogen fuse together did not really seem like very reasonable to me. Interviewer: So you read through all the options first? Interviewee 8004: And then I just choose which one I felt like made the most sense even though I do not really know the answer to the question so I was just guessing In a large quantitative study, these types of responses would lead to a larger than expected percentage of students 633



Article

conception about the process of phase change. The other participant used process of elimination to answer some of the items correctly. This evidence for proper functioning of the items is further supported by the responses of two additional participants who answered all three items incorrectly while clearly displaying the alternate conception that water dissociates into oxygen and hydrogen when it evaporates. Therefore, all but 1 of these 11 interviews supports the response process validity of these phase change items. The remaining seven interviews covering the phase change items revealed inconsistencies between the participants’ option choices and their reason for choosing these options, with regard to their conceptual understanding. These inconsistencies point to possible threats to the response process validity for two of the three items covering the phase change concept. Looking across all three phase change items provides additional support for their functioning. The students who responded incorrectly to item 2 due to their confusion between the bubbles in the water and water vapor showed an overall correct conception about the phase change of water during the response process for the other two items in this category. This inconsistency further legitimizes the potential for misdiagnosis by an item due to a response threat. Support for the potential threat of guessing due to process of elimination on item 3 is also found among the other items. Students’ who answered item 3 correct due to guessing showed they actually do hold the alternate conception about water dissociating into hydrogen and oxygen when it evaporates when responding to item 2 and item 6 incorrectly. To summarize, the three phase change items on the CCI are meant to measure the extent of student alternate conceptions about water dissociating into hydrogen and oxygen when it evaporates. The interview data showed that two of the three phase change items have potential response process validity threats. The threat for item 2 involved students with a correct conception answering this item incorrectly because of unfamiliarity or confusion with the boiling process. Item 3 on the other hand involved some students answering correctly through an educated guess, even though their responses to the other items showed that they clearly did not have the right conception. No response process validity threats were identified for item 6, including potential threats due to interpretation of particulate representations. Conservation of Mass−Matter Items. The breakdown of responses for each of the items or item pairs is displayed in Table 4. The eight items covering conservation of mass−matter include three two-tiered items (7/8, 12/13, and 18/19), whereas the main concept for all items is the conservation of mass−matter they can be broken down into two subcategories.


correct conception but believed the water vapor was leaking out of the container. However, this appears to be anomalous compared to the rest of the population; therefore, overall, item 6 appears to be the best measure on the CCI for the concept of phase change. This item consistently diagnoses a student’s alternate or correct conception when compared with the other two items. Item 6 may be more effective in measuring conceptual understanding than the other two items because it represents the molecular level of evaporation pictorially. Item 6 also explicitly uses the terms liquid and evaporated water in the item stem. Examples of participant’s response process in answering item 6 appear below. The first interview quote is from a student who shows an alternate conception when explaining their reason for choosing option D on item 6. Interviewee 8004: So evaporating, I know the phase shift it goes to gas so I looked at each box and I looked for the molecules that were both individually hydrogen and oxygen because I know when water evaporates it breaks apart into oxygen and hydrogen so I looked in each box and all the boxes were either bonded to hydrogen and oxygen. They were bonded together so there was only one box where all of them were separately, not bonded together so I chose that box. Interviewer: These terms here, liquid water and evaporated water, can you just talk about those, how you understood those? Interviewee 8004: Liquid water I just saw as a liquid so it would be H2O bonded together. Evaporated makes me think of gas so in a gas it made me think of hydrogen and oxygen so hydrogen and oxygen would not be bonded together anymore, they would be in individual molecules, does that make sense? The next interview quote is from a participant who shows correct understanding about the concept when giving his or her reasoning for choosing option E on item 6. Interviewee 8014: The liquid water evaporates into gas. And because it is evaporated water, it is still water, the molecules are still there so it is not like they broke off. So that marked off A, B, and C or A, B, and D. Because they have to be broken up molecules in there. It is just evaporated water, the molecule has not changed, just the spacing in between. So it had to be (E). Consistency among Phase Change Items. Fifty percent of the participants (9 total) responded correctly to all three items. As the reason for choosing the correct option choices, all but one of these nine participants expressed the correct

Table 4. Number of Interview Responses by Category for Items Covering Conservation of Mass−Mattera Item

Totally Correct

Totally Incorrect

1 4 7/8 12/13 18/19

9 (41) 11 (50) 9 (41) 14 (63) 11 (50)

10 (45) 5 (23) 4 (18) 1 (5) 7 (31)

Potential Threat 0 6 9 6 3

(0) (27) (41) (27) (14)

Otherb 3 0 0 1 1

(14) (0) (0) (5) (5)

N = 22 interview responses for each item. bThe category “Other” indicates interview explanations that do not fit within the prior categories.

a

634



Article

incorrectly. Despite the superior functioning of this item, it cannot be taken alone in making conclusions about students’ understanding of the concept of conservation of mass−matter. Items 7/8 and 18/19. Item 7/8 (Figure 5) and 18/19 (Figure 6) probe the application of the concept of the

Item 7/8 and 18/19 measure students’ understanding of the conservation of mass−matter in relation to a chemical reaction. Item 4 and item 12/13 measure conservation of mass−matter in terms of physical processes. Item 1 appears to stand-alone as a measure of students’ understanding of the definition of the conservation of mass−matter. Item 1. Item 1 has the highest percentage of accurately supported responses (86%, totally correct + totally incorrect) within this category and appears to have no response process validity threats. Although several responses (14%) did not have accurate support for the option chosen, there were no consistent themes among the explanations that would lead to a potential threat. The question stem in item 1 (Figure 4) does

Figure 5. Item 7/8 from the Chemical Concepts Inventory.8


not discuss a specific chemical reaction. Item 1 instead probes students’ understanding of the definition of the conservation of mass−matter as it relates to a chemical reaction. Of the nine participants who answered this item correctly (option D), seven presented a correct conception in explaining their answer. The other two participants showed some confusion and had initially selected option E before talking through the problem and changing their answer to the correct option. Option E was chosen by eight participants. All of these participants conveyed a misunderstanding about how the number of molecules may change after a chemical reaction. One student who answered option E stated, Interviewee 8007: And I chose all of the above [Option E] because the sum of the masses, the number of molecules and the number of atoms because I just thought that in a reaction nothing is created or destroyed, it is just converted around so the numbers should all be the same, I guess. Interviewer: What do you mean by converted around? Interviewee 8007: Well, like the mass, let us see, I guess the number of molecules and atoms cannot just disappear they can be converted to other things but should not the number still be the same after? The transcript excerpt shows that the participant does not appear to make a distinction between atoms and molecules. They use the term “converted” and yet claim everything is the same. In their analysis of item 1, Mulford and Robinson mentioned that almost all the incorrect answers to this question other than option B, the number of molecules of all substances involved, shows “partial understanding” of the concept.8 Only 8% of pre- and 5% of postsemester participants in their study selected option B; no participants in this study selected option B. Although some students chose other response options for item 1, there were no apparent themes that would lead to response process validity threats. This item measures if a student can apply the definition of the law of conservation of mass−matter to chemical reactions; therefore, students who simply have a lack of knowledge about the nature of molecules and their rearrangement in chemical reactions answer this item

Figure 6. Item 18/19 from the Chemical Concepts Inventory.8

conservation of mass−matter during a chemical reaction. The context of item 7/8 involves a combustion reaction and requires students to conceptualize the reaction without specifically being given the chemical reaction itself. Item 18/ 19 involves the oxidation of an iron nail in forming rust. This item explicitly explains the reaction but also does not give a chemical equation. The test developers reported a percentage correct rate of about 90% pre- and postsemester for the two-tiered item 7/8.8 Similar response rates were found in the most recent quantitative study also.44 The test developers explained that item 7/8 might be prompting recall rather than conceptual understanding. A large percentage of students (82%) selected the correct multiple-choice answer choice for both parts of the item. However, during the interviews, only 41% of the responses were totally correct, that is, they were supported with accurate descriptions. The remaining 41% of interviewees gave explanations indicating that they did not understand the conservation of matter−mass and were responding based simply on recall of the definition. Overall, only 59% of the responses (41% totally correct and 18% totally incorrect) to item 7/8 were supported by the interview responses. The 635



Article

following interview excerpt is an example of the recall-based responses. Interviewee 8014: I do not know the exact wording of the law, I just remember nothings created or destroyed. And it could be matter. I think it was last semester but I also learned it in high school too...technically it is a physics law but I know we talked about it in chemistry. I just remember that nothing is created or destroyed. And I’m pretty sure it was matter but I wanted to say energy and I’m thinking it was matter. But I do not know the wording of the law, I just know that nothing is created or destroyed, that is what I remember, that was the concept. So again, that was the whole thing with the balanced reaction...Everything equals out in the end. In discussing the second tier, item 8, Mulford and Robinson remind their audience that “many students can use chemical terms without understanding the concepts they are intended to convey.”8 This was supported by several interviews in this study. The use of recall by one of these participants (who correctly answered the multiple-choice version item 7/8) is shown below. Item 7 Response. Interviewee 8022: False, it turns into ash. Interviewer: What do you mean ash? Interviewee 8022: So it changes its chemical form. That is because mass cannot be destroyed. Matter cannot be destroyed. Interviewer: What do you mean by matter cannot be destroyed? What do you mean by destroyed? Interviewee 8022: It just cannot go to nowhere. It will always exist. Because it cannot be created or destroyed Interviewer: You said that it cannot be created or destroyed can you just talk about that a little bit? Interviewee 8022: So it is someone’s law. Newton... Not Newton. Newton? No? I do not know someone’s law it is his first law. Interviewer: First law? Interviewee 8022: Of something. I do not know so the match just... well the particles come apart and they’re all just separate instead of being formed together. The student goes on to explain their understanding of the reason in the second tier of the item. Item 8 Response. Interviewee 8022: I would say the atoms are not destroyed they are only rearranged because the flame cannot actually destroy the atoms. Interviewer: And how do you interpret that word rearranged? Interviewee 8022: So when they’re in the match they are all together. And then when they’re burned they would just be smaller particles then just one big chunk of the match Interviewer: What do you mean smaller particles? Interviewee 8022: Instead of having them all stuck together on the match they have all broken apart so it is a bunch of little pieces. Interviewer: Little pieces of what? Interviewee 8022: If the match was wood I guess it would be the little pieces of wood that are burned I guess. Interviewer: What do you mean by “that are burned?” Interviewee 8022: That were, that the match caused to fall apart or the flame. Interviewer: What do you mean by “fall apart?” Interviewee 8022: They would have broken the bonds between the part of the match. Interviewer: What do you mean “broken the bonds?”

Interviewee 8022: So it would have made them weak enough that they would have fallen apart. Interviewer: And what would you be left with? Interviewee 8022: Smaller pieces of the match. The response process for item 7/8 from this interview is telling. The use of the word “destroyed” in the item stem fits the language the participant knows from the law of conservation of mass−matter. However, the participant does not understand what is meant by the term rearranged in option D of item 8. Their choice of the correct option (D), is due to that option simply stating, “the atoms are not destroyed...” The participant’s overall lack of understanding of what occurs at the molecular level is further supported by their naı̈ve use of the term “burned” and that burning would create “smaller pieces of match”. In contrast to the performance of item 7/8, item 18/19 had a large percentage of accurately supported responses (81%). Therefore, item 18/19 appears to be a better evaluation of students understanding of the concept within the context of a chemical reaction. For example, the participant from the excerpt above (interviewee 8022), who correctly responded to item 7/8 yet could not accurately support their reasoning, later answered incorrectly to item 18/19 and explained their answer using nothing more than their basic understanding of the definition of the conservation of mass−matter as shown in the following excerpt. Item 18 Response. Interviewer: What do you mean by oxidized? Interviewee 8022: The oxygen broke it down. Interviewer: And what do you mean by broke it down? Interviewee 8022: Broke it from a solid one piece to a bunch of little pieces. Interviewer: Ok, so can you talk about oxidizing? Interviewee 8022: When the oxygen gets to something it takes away its chemical properties. Interviewer: What do you mean takes away its chemical properties? Interviewee 8022: It affects its chemical properties and makes it into something else. Breaks it down. Interviewer: What do you mean by this “something else?” Interviewee 8022: Into a different form than what it is. Interviewer: Could you define form? Interviewee 8022: So the nail goes from a silver heavy single piece to a brown powdery substance. Interviewer: And what is that brown powdery substance? Interviewee 8022: The rust. Interviewer: Okay, and just to restate it, how are you answering this question? Interviewee 8022: The rust would still weigh the same because the iron nail was oxidized. But the oxidation would not take away any part of its weight. Interviewer: What do you mean “take away part of its weight?” Interviewee 8022: No part of the nail will disappear when it is oxidized it will still be there Item 19 Response. Interviewee 8022: I would say the nail flakes away. Interviewer: What do you mean by that? Interviewee 8022: The oxygen causes it to fall apart. Judging from their reasoning on item 18/19, the participant believes simply that mass would not be lost in the reaction, as they only focus on the iron that makes up the nail, not the rest of the reaction components. This fits with recall of the law of 636



Article

conservation of mass−matter, “matter cannot be created or destroyed”. The participant’s limited conceptual understanding leads them to incorrectly choose option B, “the same as the nail it came from”, for the second tier. A few students (14%) struggled with the context of item 18/ 19, leading them to incorrect responses. One participant believed the nail was only rusting on the outside, and not “completely” as the item stem states. The other participants believed the rust was flaking away and “leaving the system” and therefore could not be measured. While these misinterpretations were rare, they could certainly lead to response process validity threats. Item 4 and 12/13. Items 4 (Figure 7) and 12/13 (Figure 8) appear to measure whether or not students have an alternate

therefore, it will be 21 pounds. And I did not even really look at any of the other options, I just picked 21 pounds because I felt that that one was right. Interviewer: How about when you are looking at them now? How do you feel, what do you think? Interviewee 8004: I feel like it could be between 20 and 21 pounds because maybe the mass of salt could, not the mass but I guess the weight of salt could decrease as it disassociates in the water so I feel like it could be between 20 and 21 pounds. I’m not really familiar with how mass changes after the compound is put into different states like liquid, solid or gas. I’m not that familiar with that so I just went with it stays the same even though it is very possible it does not. Interviewer: And how come you went with it stays the same? Interviewee 8004: Because I was not sure if, well I do not know those characteristics so I just went with what I knew already. Interviewer: What do you mean? Interviewee 8004: I was not positive it makes sense to me if salt dissolves into water it could decrease in weight...but I’m not positive so I could either guess that it is either increases or decreases but with that guess it goes either way or I could just stick with it stays the same, which only gives me one answer instead of maybe being two other answers, if that makes sense. Once this participant began to reason through this question, their vague understanding of dissolution is evident. However, this participant correctly deduces the mass after salt is dissolved without using anything more than simple logic. Several other participants use similar logic arguments in explaining their answer choice of 21 pounds. Therefore, item 4 may not be measuring the extent of conceptual understanding. This same use of logic, without a connection to the physical process, also appears in responses to item 12/13; see excerpts below for examples. Interviewee 8025: I would say 27 g. Interviewer: How come you say that? Interviewee 8025: I feel like it should be 27 g but I feel like gas weighs less. This student gets the correct answer but is not sure of the reason why; in addition, they report thinking that a gas weighs less yet still choose the correct response. The following students choose the correct response and give a proper explanation. Interviewee 8010: Yeah, so if the solid is a one gram sample and you heat it up and it is completely gone or evaporated then I just kind of left it at 27 because it is there just in the gas form. Interviewee8018: “C” 27 g, because the mass is not being changed, it is just the state of the matter that changes from a solid to a gas. It should also be noted that the correct option (B) in item 13 contains the phrase “mass is conserved.” This verbiage coincides with common terminology in the statement of the conservation of mass−matter and, therefore, may be more attractive to students who only recall the definition and have no further understanding. Consistency Among Conservation of Mass−Matter Items. With the larger number of conservation of mass−matter items spanning distinct subcategories, consistency among items is harder to establish than for the phase change item category. Overall, there were 22 interviews covering the eight items. Of these 22, only four participants answered all the items correctly and had totally correct conceptions when responding to the items. In addition, two students had totally correct responses to

Figure 7. Item 4 from Chemical Concepts Inventory.8

Figure 8. Item 12/13 from Chemical Concepts Inventory.8

conception about the loss of mass during a physical process. However, this basic conceptual understanding may not be being probed effectively. For both items, only around 70% of the responses are accurately supported. The main threat for these items (27% of responses for each) is that they are simply a measure of logic or deduction. That is, students can disconnect the question from any chemical concept or idea and still answer the items correctly. An example of this can be seen in one participant’s reasoning for item 4, shown in the excerpt below. Interviewee 8004: I chose 21 pounds of solution because it asked for the mass of solution so you already have 20 pounds of water and you are adding a pound of salt to make solution; 637



Article

participants who answered incorrectly on item 2 due to confusion showed correct conceptual understanding on item 6, whereas some participants who answered correctly due to guessing on item 3 answered incorrectly for item 6 and revealed a lack of understanding of the concept in their reasoning. Threats to response process validity were observed for a few of the conservation of mass−matter items as well. Most notably data for item 7/8 confirmed the test developers’ suspicions that the item may be measuring recall of the conservation of mass− matter definition rather than conceptual understanding. Items 4 and 12/13 also showed threats to the validity evidence from the response process as students could answer the items using simple logic rather than a conceptual understanding of the chemistry topic.

all conservation of mass−matter items except for item 18; in both cases the students misinterpreted the item stem. No participant answered all the conservation of mass items incorrectly. Within the grouping of items, some trends do exist. For the items covering the subcategory of conservation of mass−matter in relation to a chemical reaction, item 18/19 provides a more accurate reflection of student conceptions than item 7/8. In eight of the nine interviews on item 7/8 where the interviewee got the correct answer through only definition recall methods, item 18/19 consistently showed that the participants did not have a robust conception of the conservation of mass−matter. One such example was presented by the interview excerpts (interviewee 8022) presented within the discussion of this group of items. Little difference was observed when comparing the items covering the subcategory of mass−matter in relation to a physical process (items 4 and 12/13). The use of logic reasoning in responding to the items was consistent for participants across both items.

■

Support for the Response Process Validity of CCI Items

The response process data derived from this study provided evidence for the validity of student responses. Item 6 (Figure 3) consistently diagnosed students’ understanding of the phase change concept. This may be due in part to the particulate-level images present within the item, which were not part of the other two-phase change items. Within the set of conservation of mass−matter items, items 1 and 18/19 have significant response process validity support. Nearly 90% of the item 1 option choices and over 80% of item 18/19 option choices were accurately supported by the students. All other items in both groups had less than 75% of responses that were accurately supported. When any of these items are used to evaluate students’ alternate conceptions, more confidence can be had in the results that stem from the items with response process validity evidence. Results from weaker items should be interpreted with caution.

CONCLUSION

Content Validity of the CCI

Content validity measures if the items on an inventory cover the spectrum of ideas or topics in the subject domain. Content validity is a subjective or logical measure that is found through expert judgments about the appropriateness, breadth, and relevance of items. It also includes the need for multiple items probing the same concept in order for triangulation to be implemented. The range of topics for the CCI was determined by use of a content survey. This portion of the study found that the concepts of the conservation of mass−matter and phase change are the only concepts covered by more than two items on the CCI. In fact, three items were shown to cover the concept of phase change, whereas ten items, three of which were two-tiered, were found to cover the concept of conservation of mass−matter (Table 1). This means that more than half of the 22 items on the CCI encompass only two distinct topics covered in the first semester of a traditional general chemistry course. When a total score from the CCI is determined, this total is meant to show the existence of students’ alternate conceptions. However, if a majority of the CCI covers only two distinct topics, of the multitude of those encountered in first-semester general chemistry, then it falls short in examining all of the important elements of the alternate conceptions students may have entering a first-semester general chemistry course. This lack of content breadth is referred to as construct underrepresentation46 and constitutes a threat to the content validity.

Depth of Coverage to Measure Student Conceptions

Although some of the items on the CCI do have some evidence to support the validity of students’ responses, there is not enough consistency among items covering an individual concept to make significant inferences from the data as a whole. The instrument developers sought to “measure the extent of entering student’s alternate conceptions about topics found in the first semester of many traditional general chemistry courses.”8 Therefore, there is a need for multiple items, all with validity evidence, to measure students’ alternate conceptions about an individual concept or topic area. There are several items on the CCI working toward that end. There is response process validity evidence to support single items within both groups as well as the phase change group itself. In considering the number of items needed to accurately measure the extent of one’s knowledge of a concept, it is also important to acknowledge that there are also multiple applications for some concepts. This is revealed by the conservation of mass− matter items on the CCI. Probing the concept effectively would involve ensuring students had knowledge across the various applications (e.g., working definition, application to chemical reactions, application to physical process).

Threats to the Response Process Validity of CCI Items

The response process data revealed threats to the item functioning for some of the items within the phase change and conservation on mass−matter groupings. Specifically, the phase change items 2 and 3 involved students answering correctly and incorrectly for the wrong reason. For item 3 (Figure 1), students were able to use educated guessing or good test-taking skills to answer the item correct, even when they showed an alternate conception in answering other phase change items. Item 2 (Figure 2) involved participants who otherwise showed a correct conception about the phase change process for water answering this item incorrectly due to confusion with the boiling of water. Student responses to item 6 (Figure 3) supported these results for items 2 and 3. The

Future Directions

The development of concept inventories should include continuous refinements.3 Therefore, future research should explore the utility of changing the CCI items that showed threats to the response process validity as well as adding different items and examining how these items function within the item sets. Items from the CCI could be used as the first step 638



Article

in developing “single concept” inventories to target alternate conceptions more directly and effectively for specific chemistry topics. As the goal of concept inventory items is to elicit student misunderstandings, all future studies should be grounded in providing proper evidence to support the validity and reliability of data as outlined in the Standards.46 Results from prior studies that have made claims based on items from the CCI, or the instrument in its entirety, should be carefully considered based on the evidence provided to support the claims.

■

(16) Wren, D.; Barbera, J. Gathering Evidence for the Validity During the Design, Development and qualitative Evaluation of Thermochemistry Concept Inventory Items. J. Chem. Educ. 2013, 90 (12), 1590− 1601. (17) Yeo, S.; Zadnik, M. Introductory thermal concept evaluation: assessing students’ understanding. Phys. Teach. 2001, 39, 496−504. (18) Bernholt, S.; Parchmann, I. Assessing the complexity of students’ knowledge in chemistry. Chem. Educ. Res. Pract. 2011, 12 (2), 167−173. (19) Canpolat, N. Turkish undergraduates’ misconceptions of evaporation, evaporation rate, and vapour pressure. Int. J. Sci. Educ. 2006, 28 (15), 1757−1770. (20) Canpolat, N.; Pinarbasi, T.; Sozbilir, M. Prospective teachers’ misconceptions of vaporization and vapor pressure. J. Chem. Educ. 2006, 83 (8), 1237−1242. (21) Cliff, W. H. Chemistry misconceptions associated with understanding calcium and phosphate homeostasis. Adv. Physiol. Educ. 2009, 33 (4), 323−328. (22) Devetak, I.; Urbancic, M.; Grm, K. S. W.; et al. Submicroscopic representations as a tool for evaluating students’ chemical conceptions. Acta Chim. Slov. 2004, 51 (4), 799−814. (23) Holme, T.; Bretz, S. L.; Cooper, M.; et al. Enhancing the role of assessment in curriculum reform in chemistry. Chem. Educ. Res. Pract. 2010, 11 (2), 92−97. (24) Marbach-Ad, G.; McAdams, K. C.; Benson, S.; et al. A Model for Using a Concept Inventory as a Tool for Students’ Assessment and Faculty Professional Development. CBE Life Sci. Educ. 2010, 9 (4), 408−416. (25) Nyachwaya, J. M.; Mohamed, A. R.; Roehrig, G. H.; et al. The development of an open-ended drawing tool: an alternative diagnostic tool for assessing students’ understanding of the particulate nature of matter. Chem. Educ. Res. Pract. 2011, 12 (2), 121−132. (26) Potgieter, M.; Ackermann, M.; Fletcher, L. Inaccuracy of selfevaluation as additional variable for prediction of students at risk of failing first-year chemistry. Chem. Educ. Res. Pract. 2010, 11 (1), 17− 24. (27) Yalcin, F. A. Pre-service primary science teachers’ understandings of the effect of temperature and pressure on solid-liquid phase transition of water. Chem. Educ. Res. Pract. 2012, 13 (3), 369− 377. (28) Adadan, E.; Savasci, F. An analysis of 16−17-year-old students’ understanding of solution chemistry concepts using a two-tier diagnostic instrument. Int. J. Sci. Educ. 2012, 34 (4), 513−544. (29) Chandrasegaran, A. L.; Treagust, D. F.; Waldrip, B. G.; et al. Students’ dilemmas in reaction stoichiometry problem solving: deducing the limiting reagent in chemical reactions. Chem. Educ. Res. Pract. 2009, 10 (1), 14−23. (30) Costu, B.; Ayas, A.; Niaz, M. Promoting conceptual change in first year students’ understanding of evaporation. Chem. Educ. Res. Pract. 2010, 11 (1), 5−16. (31) Nakiboglu, C.; Yildirir, H. E. Analysis of Turkish High School Chemistry Textbooks and Teacher-Generated Questions about Gas Laws. Int. J. Sci. Math. Educ. 2011, 9 (5), 1047−1071. (32) Schultz, E. Simple dynamic models for hydrogen bonding using Velcro-polarized molecular models. J. Chem. Educ. 2005, 82 (3), 401− 405. (33) Tan, K. C. D.; Treagust, D. F.; Chandrasegaran, A. L.; et al. Kinetics of acid reactions: making sense of associated concepts. Chem. Educ. Res. Pract. 2010, 11 (4), 267−280. (34) Uzuntiryaki, E.; Geban, O. Effect of conceptual change approach accompanied with concept mapping on understanding of solution concepts. Instr. Sci. 2005, 33 (4), 311−339. (35) Wood, C.; Breyfogle, B. Interactive demonstrations for mole ratios and limiting reagents. J. Chem. Educ. 2006, 83 (5), 741−748. (36) Halakova, Z.; Proksa, M. Two kinds of conceptual problems in chemistry teaching. J. Chem. Educ. 2007, 84 (1), 172−174. (37) Jurisevic, M.; Glazar, S. A.; Pucko, C. R.; et al. Intrinsic motivation of pre-service primary school teachers for learning

AUTHOR INFORMATION

Corresponding Author

*J. Barbera. E-mail: [email protected]. Notes

The authors declare no competing financial interest.

■

REFERENCES

(1) Halloun, I. A.; Hestenes, D. The initial knowledge state of college physics students. Am. J. Phys. 1985, 53 (11), 1043−1048. (2) Treagust, D. F. Evaluating Students’ Misconceptions by Means of Diagnostic Multiple Choice Items. Res. Sci. Ed. 1986, 16, 199−207. (3) Treagust, D. F. Development and use of diagnostic tests to evaluate students’ misconceptions in science. Int. J. Sci. Educ. 1988, 10 (2), 159−169. (4) Krause, S.; Birk, J.; Bauer, R., et al. Development, Testing, and Applications of a Chemistry Concept Inventory. In Frontiers in Education Conference, 34th Annual; IEEE: New York, 2004. (5) Krishnan, S. R.; Howe, A. C. J. Chem. Educ. 1994, 71, 653−655. (6) Lee, S. T.; Treagust, D. F.; Lin, H. J. Chemistry Teachers’ Estimations of their Students’ Learning Achievement. J. Chem. Educ. 2005, 82 (10), 1565−1569. (7) McClary, L.; Bretz, S. L. Development and Assessment of A Diagnostic Tool to Identify Organic Chemistry Students’ Alternative Conceptions Related to Acid Strength. Int. J. Sci. Educ. 2012, 34 (15), 2317−2341. (8) Mulford, D. R.; Robinson, W. R. An Inventory for Alternate Conceptions among first-Semester General Chemistry Students. J. Chem. Educ. 2002, 79 (6), 739−744. (9) Oloruntegbe, K. O.; Ikpe, A. J. Ecocultural Factors in Students’ Ability to Relate Science Concepts Learned at School and Experienced at Home: Implications for Chemistry Education. J. Chem. Educ. 2011, 88 (3), 266−271. (10) Othman, J.; Treagust, D. F.; Chandrasegaran, A. L. An investigation into the relationship between students’ conceptions of the particulate nature of matter and their understanding of chemical bonding. Int. J. Sci. Educ. 2008, 30 (11), 1531−1550. (11) Ozkaya, A. R.; Musa, U.; Saricayir, H.; Sahin, M. Effectiveness of Conceptual Change-Oriented Teaching Strategy To Improve Students' Understanding of Galvanic Cells. J. Chem. Educ. 2006, 83 (11), 1719−1723. (12) Stains, M.; Esciu-Sune, M.; Molina Alveraz de Santiz, M. L.; et al. Assessing Secondary and College Students’ Implicit Assumptions About the Particulate Nature of Matter: Development and Validation of the Structure and Motion of Matter Survey. J. Chem. Educ. 2011, 88 (10), 1359−1365. (13) Vachliotis, T.; Salta, K.; Vasiliou, P.; et al. Exploring Novel Tools for Assessing High School Students’ Meaningful Understanding of Organic Reactions. J. Chem. Educ. 2011, 88 (3), 337−345. (14) Villafane, S. M.; Bailey, C. P.; Loertscher, J.; et al. Development and Analysis of an Instrument to Assess Student Understanding of Foundational Concepts Before Biochemistry Coursework. Biochem. Mol. Biol. Educ. 2011, 39 (2), 102−109. (15) Voska, K. W.; Heikkinen, H. W. Identification and Analysis of Student Conceptions Used to Solve Chemical Equilibrium Problems. J. Res. Sci. Teach. 2000, 37, 160−176. 639



Article

chemistry in relation to their academic achievement. Int. J. Sci. Educ. 2008, 30 (1), 87−107. (38) Mayer, K. Addressing Students’ Misconceptions about Gases, Mass, and Composition. J. Chem. Educ. 2011, 88 (1), 111−115. (39) Regan, A.; Childs, P.; Hayes, S. The use of an intervention programme to improve undergraduate students’ chemical knowledge and address their misconceptions. Chem. Educ. Res. Pract. 2011, 12 (2), 219−227. (40) Stains, M.; Talanquer, V. A(2): Element or compound? J. Chem. Educ. 2007, 84 (5), 880−883. (41) Streveler, R. A.; Miller, R. L.; Santiago-Roman, A. I.; et al. Rigorous Methodology for Concept Inventory Development: Using the ‘Assessment Triangle’ to Develop and Test the Thermal and Transport Science Inventory. Int. J. Eng. Educ. 2011, 27 (5), 968−984. (42) Cacciatore, K. L.; Sevian, H. Incrementally Approaching an Inquiry Lab Curriculum: Can Changing a Single Laboratory Experiment Improve Student Performance in General Chemistry? J. Chem. Educ. 2009, 86 (4), 498−505. (43) Kruse, R. A.; Roehrig, G. H. A Comparison Study; Assessing Teacher’s Conceptions with the Chemistry Concepts Inventory. J. Chem. Educ. 2005, 82 (8), 1246−1250. (44) Barbera, J. A Psychometric Analysis of the Chemical Concepts Inventory. J. Chem. Educ. 2013, 90 (5), 546−553. (45) Arjoon, J. A.; Xu, X. Y.; Lewis, J. E. Understanding the State of the Art for Measurement in Chemistry Education Research: Examining the Psychometric Evidence. J. Chem. Educ. 2013, 90 (5), 536−545. (46) Standards for Educational and Psychological Testing; American Educational Research Association: Washington, DC, 1999; http:// www.apa.org/science/programs/testing/standards.aspx (accessed Apr 2014). (47) Bowen, C. W. Think-Aloud Methods in Chemistry Education: Understanding Student Thinking. J. Chem. Educ. 1994, 71 (3), 184− 189. (48) Nakhleh, M. B.; Mitchell, R. C. Concept Learning Versus Problem Solving. J. Chem. Educ. 1993, 70 (3), 190−192. (49) Domin, D.; Bodner, G. Using Students’ Representations Constructed During Problem Solving to Infer Conceptual Understanding. J. Chem. Educ. 2012, 89 (7), 837−843. (50) Staver, J. R. Two Investigations of Students’ Understanding of the Mole Concept and Its Use in Problem solving. J. Res. Sci. Teach. 1995, 32 (2), 177−193.

640


Evaluating the Content and Response Process ... - ACS Publications

Recommend Documents