Assessing Student Knowledge of Chemistry and Climate Science

Article pubs.acs.org/jchemeduc

Assessing Student Knowledge of Chemistry and Climate Science Concepts Associated with Climate Change: Resources To Inform Teaching and Learning Ashley Versprille,† Adam Zabih,† Thomas A. Holme,‡ Lallie McKenzie,§ Peter Mahaffy,∥ Brian Martin,⊥ and Marcy Towns*,† †

Department of Chemistry, Purdue University, West Lafayette, Indiana 47907, United States Department of Chemistry, Iowa State University, Ames, Iowa 50011, United States § Chem11, Eugene, Oregon 97403, United States ∥ Department of Chemistry, The King’s University, Edmonton, Alberta T6B 2H3, Canada ⊥ Department of Physics, The King’s University, Edmonton, Alberta T6B 2H3, Canada ‡

S Supporting Information *

ABSTRACT: Climate change is one of the most critical problems facing citizens today. Chemistry faculty are presented with the problem of making general chemistry content simultaneously relevant and interesting. Using climate science to teach chemistry allows faculty to help students learn chemistry content in a rich context. Concepts related to electromagnetic radiation and gases can be taught using an understanding of climate change and how greenhouse gases work. However, it would be important to know the level of prior knowledge that the students bring to the course and their confidence in that knowledge. Thus, a two-tiered instrument was developed to measure student understanding of climate change, the behavior of gases, and the mechanism of radiative forcing by greenhouse gases. The instrument was implemented iteratively at two institutions to allow for revision and replication. The final form of the instrument may be used in general chemistry classes or interdisciplinary courses to shape and guide instruction. KEYWORDS: First-Year Undergraduate/General, Second-Year Undergraduate, Upper-Division Undergraduate, Chemical Education Research, Interdisciplinary/Multidisciplinary, Misconceptions/Discrepant Events, Testing/Assessment, Atmospheric Chemistry FEATURE: Chemical Education Research

R

ozone” and often conflate them. Therefore, it comes as no surprise that many students who are taking general chemistry courses in college have already developed alternative conceptions and fragmented understandings about climate science from their prior experiences. Global climate change is one of the most pressing environmental issues facing the world today. Climate change is one of two planetary boundaries listed as a core planetary boundary, as the regulation of the amount, distribution, and net balance of energy at the earth’s surface is so fundamentally important for the earth’s systems.9 Americans remain underinformed about the consensus in the scientific community, knowledge of climate change, and impacts and proposed solutions.10−12 Society as a whole, but particularly today’s youth, will face choices in the future about energy use and policies to reduce greenhouse gas

esearch across the disciplines has demonstrated that students have alternative conceptions and fragmented understandings about climate science due to the complexity of climate change models, and the multidisciplinary content knowledge (e.g., in areas such as earth and atmospheric science, physics, chemistry, and biology) needed to be able to understand and articulate key concepts in climate science.1−5 The term alternative conception is defined as a conception held by a student that is not in agreement with the scientifically accepted conception.6 Kerr and Walz emphasized that alternative conceptions about climate science often occur as a result of inaccurate information provided by the media and lack of climate science curriculum at both K−12 and college levels.1 Research has demonstrated that students across all levels seem to hold many of the same alternative conceptions about how greenhouse gases work, global warming, and climate change.7,8 For example, one of the most common alternative conceptions among high school students is evident in how they interchangeably use the phrases “greenhouse effect”, “global warming”, and “depletion of the © XXXX American Chemical Society and Division of Chemical Education, Inc.

Received: October 11, 2016 Revised: January 9, 2017

A

DOI: 10.1021/acs.jchemed.6b00759 J. Chem. Educ. XXXX, XXX, XXX−XXX

Journal of Chemical Education

Article

the atmosphere interact with each other, and mechanisms and processes that lead to both positive and negative radiative forcing.

emissions in an effort to reduce or manage the anthropogenic impact on climate. A synthesis of research findings across elementary, secondary, and college levels about student understanding of climate change has established that five broad scientific areas of confusion emerge (see Supporting Information for references associated with each area).2−5,7,8,13−44

■

PURPOSE OF THE STUDY AND RESEARCH QUESTION Multiple-choice exams have been used for measuring students’ understanding of science concepts as they allow professors to sample a large number of students in a short amount of time, so it is an efficient method to assess large populations.54,55 Multiplechoice exams are also easy to administer and grade, and the results obtained are also easily processed and analyzed. Researchers such as Halloun and Hestenses,56 McClary and Bretz,57 Tamir,58, and Taber59 have used multiple-choice exams to explore students’ conceptions in science. Taber59 cautioned that only the most common alternative conceptions were likely to be diagnosed as the exam writer would have to leave out the other less common alternative conceptions to avoid too many distractors. Another disadvantage to using multiple-choice exams to assess students’ understanding of scientific conceptions is that when students do not know the answer, they often guess.59 Thus, Tamir58 developed an alternative approach to the construction of previous multiple-choice test items wherein the distractors for the multiple-choice items were based on students’ responses to open-ended essay questions or student interviews rather than the professor’s words.58 The multiple-choice questions also addressed underlying conceptual knowledge related to a specific content area. Thus, a two-tiered multiple-choice diagnostic instrument was targeted for development in this study in which the first tier is a chemistry or climate science question based upon our prior work44 as described in the Methods section, and the second tier is a confidence question (e.g., How confident are you in your response to the previous question?). Confidence questions allow researchers and practitioners to identify how strongly students believe in the veracity of a particular response. The results can be used to identify content where student knowledge is correct and strongly held, and areas where students are less confident in their knowledge and could be correct or incorrect, and ideas that are not in agreement with those that are correct. This study distinguishes itself from prior research in that it focuses on the students’ conceptual understandings of core chemistry concepts that are related to climate change including the particulate nature of matter, covalent bonding, and absorption of infrared radiation by gases, whereas other studies focused on students’ conceptual understandings of the carbon cycle, ocean acidification, environmental issues, and historical climate data and analyses. The research question guiding this study follows: 1. What are the characteristics of the reliability and validity of the data collected with an instrument designed to probe students’ understanding of how greenhouse gases work, climate change, and the chemistry concepts associated with climate science?

Electromagnetic Radiation

Students are unsure of which portion of the electromagnetic spectrum reaches the earth’s atmosphere and its surface, and which portion interacts with greenhouse gases, and how that interaction occurs. Greenhouse Effect (How Greenhouse Gases Work)

Although many have heard of the term “the greenhouse effect”, an understanding of how greenhouse gases regulate earth’s energy balance, and that this regulation is a natural part of the earth’s system, is not evident. Even the use of the term “the greenhouse effect” often creates alternative conceptions, as the metaphor incorrectly suggests that atmospheric warming by greenhouse gases takes place via the same mechanism (glass windows blocking convection) that keeps plants warm in a greenhouse. Some students describe the mechanism of action by greenhouse gases at the macroscopic level invoking a layer or blanket of gases around the earth trapping heat. Many students can identify carbon dioxide as a greenhouse gas, but few name water vapor, methane, CFCs, or nitrous oxide. Across studies students have little understanding of what physical characteristics and processes at the particulate level define a gas as a greenhouse gas. Additionally, some students use the terms “global warming” and “the greenhouse effect” interchangeably. Ozone

Students confuse the stratospheric depletion of ozone (including the ozone hole phenomenon) with tropospheric warming by greenhouse gases and global warming (for example some indicate that ozone depletion causes global warming). Environmental Issues

Students have been found to conflate and confuse terms or components of the earth’s climate such as climate change, the greenhouse effect, and global warming with the following: general pollution, glaciers melting, oceans rising, the frequency of tornados/hurricanes, resource management, and acid rain. Proposed Solutions To Mitigate the Effects of Climate Change

Students are confused about possible solutions and their impact. The first three areas require mental models that are built on a fundamental understanding of chemistry.5,45,46 However, the majority of research on student understanding of climate change concepts has focused on geoscience,4,8,26,27,47 biology,5 and physics29 rather than chemistry concepts. This concern is exacerbated by the observation that secondary and undergraduate students struggle to build robust mental models of the particulate nature of matter which is essential for learning chemistry and understanding the chemical concepts underlying climate science.44,48−53 An understanding of the particulate nature of matter is critical for understanding the behavior of molecules and how molecules interact with each other.48−51 In order for students to fully understand the causes and impacts of climate change, they must have robust particulate level mental models of gas phase molecules; for example, the physical characteristics that make a gas a greenhouse gas, how and why greenhouse gases absorb IR radiation, how gases in

■

METHODS The development of an instrument is part of a larger project titled Visualizing the Chemistry of Climate Change, or VC3Chem. We have published the results of our qualitative study that included 24 interviews with general chemistry students about how greenhouse gases and climate change operate.44 These findings were used to ground the development of a multiplechoice instrument presented in this work. This method was B



Article

and adding items 19, 21, and 23. In fall 2016, item 7 was revised, and item 25 was added to probe student understanding of the interaction of electromagnetic radiation and greenhouse gases. Item 13, which proved to be particularly problematic, was revised once again. The revisions are discussed in more detail in the Results and Discussion sections. The instructor at institution B used climate science as a rich context in which to embed general chemistry concepts.45,46 The instrument was administered prior to instruction on gases and the atmosphere and at the end of this instructional sequence to capture changes in students’ ability to respond to topics assessed. The items are designed to query students’ understanding pertaining to (a) radiative forcing by greenhouse gases and climate change as represented by odd numbered items 1−7, 11, 13, 17, 19, and 25; and (b) learning outcomes usually covered in general chemistry62,63 via items 9, 15, 21, and 23. The final version of the instrument implemented in fall 2016 is shown below and is in the Supporting Information. Since every even numbered item is a confidence scale, it is shown only once. 1. The greenhouse effect can best be described as... (A) The same thing as global warming (B) Pollution related to acid rain (C) An increasing of the temperature of the planet (D) Damage to the ozone layer 2. How confident are you in your response to question 1 (successive odd number questions)? (A) Very high confidence (80−100%) (B) High confidence (60−80%) (C) Reasonable confidence (40−60%) (D) Little confidence (20−40%) (E) No confidence/just guessing (0−20%) 3. Which if the following is NOT a greenhouse gas? (A) Carbon dioxide (CO2) (B) Chlorofluorocarbons (CFCs) (C) Water vapor (H2O) (D) Oxygen (O2) (E) Methane (CH4) 5. Which is the most abundant greenhouse gas? (A) Carbon dioxide (CO2) (B) Hydrogen (H2) (C) Methane (CH4) (D) Nitrogen (N2) (E) Water vapor (H2O) 7. All greenhouse gas molecules interact with what type of electromagnetic radiation? (A) Visible (B) Ultraviolet (C) Shortwave radio (D) Infrared 9. Which statement best describes covalent bonds in molecules? (A) They are static, unmoving, and of fixed length. (B) They vibrate allowing them to lengthen and shorten. 11. What is the primary source of CO2 contributed by humans? (A) Human respiration (B) Driving cars (C) Pollution from factories

used to ensure that the multiple-choice items, responses, and distractors were in the students’ words. Second, the distractors used were designed to reflect the alternative conceptions that students hold based on the participants’ responses in the qualitative study to interview questions. The overall goal within the frame of the larger project was to develop an instrument to provide faculty with a way to elucidate their students’ prior knowledge and alternative conceptions of the chemistry underlying climate change, which may in turn inform their design of instruction and teaching. We chose to develop a two-tiered diagnostic instrument which consists of the first question being a content question about climate science or the chemistry associated with climate science, and the second question being a confidence question (e.g., How confident are you in your response to the previous question?).57,60,61 The confidence question has five choices which include: (A) very high confidence, (B) high confidence, (C) reasonable confidence, (D) little confidence, and (E) no confidence/just guessing. The confidence tiered questions allow for faculty to examine how strongly entrenched their students’ alternative conceptions are. For example, if a student is very confident in his or her response to a question, but the response is incorrect, then the professor knows he or she will have to consider how to help the student reorganize his or her conceptual understandings to move toward more scientifically accurate claims. The multiple-choice questions cover topics in climate science (e.g., identifying greenhouse gases, radiative forcing, and impacts of climate change) and chemistry (e.g., gas behavior, bonding, and electromagnetic spectrum). Instrument Development

The initial version of the instrument was piloted in the summer of 2013 at a large public midwestern research institution in the United States with 36 College of Science and College of Engineering students in their first-semester of general chemistry. Upon analysis of the results two questions were modified to clarify the responses. After the pilot the instrument consisted of 20 items, 10 two-tiered questions. The instrument was implemented in 2013 and implemented and revised in fall 2015 and 2016 as shown in Table 1 at two Table 1. Data Collection Noting the Number of Respondents, the Total Number of Students in the Course, and the Response Rate Year

Institution

2013

A

2015 2016

B B

Respondents Implemented in two courses 600, 692 275 128

Response Rate (%) 27.6, 77.8 57.3 95.5

different midwestern institutions. The instrument required approximately 10 min for the students to complete. In fall 2015 and 2016 students received points for completing the survey although no points were given to those who repeatedly chose the same response across the survey. Students were not able to revise a response once they had submitted it and could not return to previous questions. The modifications to the instrument were based upon an analysis of the results of the student responses and discussions within the VC3Chem team. In fall 2015 the instrument was modified to better situate the climate content within the chemical treatment of gases and the atmosphere by modifying item 13 C



13.

15.

17.

19.

21.

23.

25.

Article

relevance. To further ensure validity, interviews were carried out with two students from each general chemistry course in the fall of 2013, asking each student to solve the instrument questions aloud. This allowed the researchers to ensure that the items functioned as designed and probed student understanding in alignment with the item. It is often the case that Cronbach’s alpha (α) is used to denote that an instrument consistently and reliably measures the same construct. However, this instrument developed in this study pertains to multiple constructs; thus, Cronbach’s α is inappropriate for evaluating the reliability.64 In this study, item difficulty index and item discrimination index were used to analyze and describe items.64,65 Item difficulty is the proportion of students who respond to an item correctly, and it ranges from 0 to 1.00. A higher value means more students answered the item correctly. Item discrimination measures an association between those answering the item correctly with how they scored on the instrument overall and ranges between 1.00 and −1.00.

(D) Burning of fossil fuels to generate electricity The peak wavelength for radiation given off by the sun occurs in which region and the peak wavelength for radiation given off by the earth occurs in which region? (A) Visible (sun), visible (earth) (B) Infrared (sun), visible (earth) (C) Visible (sun), infrared (earth) (D) Infrared (sun), infrared (earth) Which change in frequency and wavelength would decrease the energy of light? (A) Increase frequency and decrease wavelength (B) Increase frequency and increase wavelength (C) Decrease frequency and decrease wavelength (D) Decrease frequency and increase wavelength Complete the following statement by choosing one response: The rate of climate change can best be slowed by ________. (A) “going green” and recycling more (B) reducing pollution (C) driving less or driving smaller vehicles (D) switching from fossil fuels to renewable energy sources Starting at the surface of earth and moving up in altitude in the atmosphere, what happens to the average temperature? (A) It becomes cooler as altitude increases. (B) It becomes cooler, but warms up towards the edge of the atmosphere. (C) It becomes cooler, then warms, finally cooling towards the edge of the atmosphere. (D) It cools quickly near the surface, but then cools very slowly towards the edge of the atmosphere. What happens to gas inside an aerosol can that is accidentally dropped into a camp fire? (A) The volume of the gas decreases. (B) The density of the gas increases. (C) The pressure of the gas increases. (D) The gas inside the can condenses to a liquid. If three balloons are filled with either CO2, N2, or O2, so that they have the same volume, pressure, and temperature, which balloon would be the most dense? (A) The one filled with CO2. (B) The one filled with N2. (C) The one filled with O2. (D) All balloons have the same density because they have the same volume. After a molecule of greenhouse gas absorbs electromagnetic radiation it ____. (A) rises into the ozone layer. (B) creates other greenhouse gas molecules. (C) releases energy by interacting with other molecules. (D) generates a layer of greenhouse gas molecules.

■

RESULTS Table 2 identifies the topic of each question, the item difficulty, and item discrimination for each administration of the instrument in fall 2015. The post item difficulty values are larger for all items except 13 and 15, indicating that a greater percentage of students chose the correct response upon instruction. Items 13 and 15 neither increased nor decreased in difficulty. Several items will be discussed below, particularly those that have a discrimination index of below 0.20. Table 3 displays the item difficulty and discrimination for the implementation in fall 2016. Item numbers 7 and 13 were modified, and item 25 was new and related to number 7. For every item the item difficulty increased between the pre- and postadministration.

■

DISCUSSION The item difficulty results displayed in Tables 2 and 3 show that students improved their knowledge of climate science and the behavior of gases, supporting the notion that teaching chemistry concepts using rich context does no harm (at the least) to developing student understanding of traditional general chemistry concepts pertaining to gases. Beyond this demonstration, it is important to consider student performance on items where the item difficulty and discrimination warrant discussion. In both years, for item 9 on the postadministration and 21 on the pre- and postadministration, the item difficulty scores are above 0.90, indicating that over 90% of the students chose the correct response. In a case such as this, the discrimination index decreases because the item is relatively easy for the entire group of students and does not discriminate well between high and low scorers. Item 21 is particularly interesting in this regard, as it represents a relatively standard, conceptual question about pressure, volume, and temperature (P, V, T). Students in this sample appear to be well-versed in these basic relationships even before instruction. This serves to accentuate the important challenge of situating instruction about core content like gases and gas laws within a rich context. While performance of this group on a “standard” test item might lead an instructor to conclude students have thorough knowledge of gases, putting a focus on the context of climate change reveals opportunities to enrich and expand that knowledge.

Data Analysis

The primary goal of analysis presented here is to adjudicate if the proposed instrument is behaving with appropriate levels of validity and reliability. This analysis is bolstered by the replication of administrations in more than one university setting. Six expert chemists validated the wording and content of the versions of the instrument for clarity, accuracy, difficulty, and D


0.62 0.57

Item difficulty Discrimination index

0.82 0.42

0.26 0.46

3; Not GHG

0.71 0.58

0.16 0.38

E

0.78 0.56

0.21 0.46

3; Not GHG

0.61 0.81

0.078 0.077

5; Most Abundant GHG

Items 7 and 13 were modified, and 25 is new.

0.75 0.37

Item difficulty Discrimination index

a

0.53 0.73

1; GH Effect

Item difficulty Discrimination index

Topic

0.88 0.30

0.21 0.31

7*; Absorb IR

0.95 0.11

0.78 0.23

9; Bonds

0.79 0.47

0.37 0.55

7; Absorb IR

Table 3. Fall 2016 Implementation with 128 Respondentsa

0.53 0.49

Item difficulty Discrimination index

1; GH Effect

5; Most Abundant GHG

Table 2. Fall 2015 Implementation with 275 Respondents

0.73 0.33

0.59 0.58

11; Source CO2

0.91 0.12

0.76 0.34

9; Bonds

0.04 0.07

Postadministration 0.69 0.65

15; ν, λ, Energy 0.41 0.19

0.52 0.44

13*; Sun Earth Peak λ Preadministration 0.21 0.19 Postadministration 0.55 0.67

Item; Topic

0.04 0.09

13; Max Solar Radiation

Item; Topic

Preadministration 0.53 0.62

11; Source CO2

0.82 0.33

0.75 0.46

17; Climate Mitigation

0.46 0.57

0.46 0.42

15; ν, λ, Energy

0.50 0.56

0.21 0.42

19; T Atmosphere

0.79 0.46

0.73 0.55

17; Climate Mitigation

0.95 0.11

0.90 0.31

21; P, V, T

0.46 0.53

0.21 0.28

0.59 0.52

0.59 0.62

23; Gas Density

19; T Atmosphere

0.81 0.32

0.57 0.49

23; Gas Density

0.83 0.26

0.50 0.46

25*; GHG Molecules Interact

0.95 0.04

0.92 0.03

21; P, V, T

Journal of Chemical Education Article



Article

Item 13: Electromagnetic Radiation from the Sun and Emitted from the Earth

chose the correct response. In Jarrett’s study, college students were able to respond to a similar question (Jarrett-21) that asked what type of energy all greenhouse gases interacted with (50% correctly chose IR). However, her study went further and cleverly asked what happens “when a molecule of greenhouse gas absorbs heat” (Jarrett-23). For high school students 22% chose the correct response, but 34% chose that the molecule would rise into the ozone layer. For undergraduates 50% of the students chose the correct response, and the discrimination index was 0.20. Across the studies it suggests that students may understand that greenhouse gases absorb IR light, but many possess faulty models of what happens next perhaps due to a lack of a (or an incomplete) particulate level model for the interaction of gases. Given the results for item 7 in fall 2015 and the results of Jarrett’s study, in fall 2016 item 7 was revised as shown in the Methods section and was inspired by Jarrett-21. Item 25 was added to probe student understanding of the fate of the absorbed IR radiation and has a similar stem to Jarrett-23 (one response is identical to Jarrett-23 and the remaining three are different). The postadministration results indicate over 80% of the students correctly answered both questions suggesting that students have begun to build models of greenhouses gases that include the type of radiation absorbed and the subsequent processes that take place.

The results for item 13 shown in Table 2 are thought provoking when compared to other research studies. In 2015 the item read as follows: The maximum intensity of solar energy arriving from the sun and striking the earth comes in which region of the electromagnetic spectrum? The responses were gamma (γ), infrared, ultraviolet, and visible. The item difficulty and item discrimination scores for the pre/postadministrations indicate that students do not choose the correct response, and the item does not discriminate between high and low scorers. Jarrett published a thesis in 2013 and a paper in 2012 that reported the development and analysis of a climate change concept inventory.26,27 Item six (Jarrett-6) evaluates content quite similar to item 13. The subjects in Jarrett’s study were high school students and undergraduates in Australia. For item Jarrett-6, student performance was also rather weak, and the item discrimination score was below 0.03 for high school students (n = 221) and 0.06 for undergraduates (n = 68). Thus, the results of our study are in accord with Jarrett’s and with those of other researchers who have found students at the high school level and at the undergraduate level have difficulty conceptualizing what types of electromagnetic radiation are emitted from the sun and in what intensity, which reach the earth’s atmosphere and are absorbed by different layers, and which finally reach the surface of the earth. The role of specific terms, such as “amount” versus “intensity” appears to be particularly vexing in devising an assessment item related to this content. Thus, in 2016 the item was revised to read as shown in the Methods section, focusing on the peak wavelength given off by the sun and emitted by the earth. Student performance on this revised item is improved in terms of item difficulty with 55% of the students choosing the correct response on the postadministration. Additionally, the item discrimination was improved from the fall 2015 administration. Orienting the item about peak wavelength proved to be more effective than intensity or amount of electromagnetic radiation.

Items 3 and 5: Improvement in Student Understanding

It is also helpful to consider some items in terms of their sensitivity to instructionally related gains in student understanding. Consider, for example, the pair of items (3 and 5) that query student ability to identify greenhouse gases in the atmosphere. The ability to identify greenhouse gases is prompted by having students choose the gas that is not a greenhouse gas in item 3 as displayed in Figure 1. It is quite clear that a shift in responses to the correct answer of O2 is evident on the basis of instruction. The second component of this item pair queries student understanding of both ability to serve as a greenhouse gas and amount in the atmosphere. Responses to item 5 supply insight into students’ conceptions of greenhouse gases and their abundance as shown in Figure 2 with the pre/post designation. Initially 63% of the students chose carbon dioxide as the most abundant greenhouse gas which is not surprising given that prior research has shown that students easily name carbon dioxide as a greenhouse gas. However, prior

Item 7: Interaction of Greenhouse Gases with Electromagnetic Radiation

In fall 2015, item 7 asked students to identify the impact of greenhouse gases. On the postadministration, 79% of students

Figure 1. Student preadministration and postadministration responses to item 3 from fall 2015. F



Article

Figure 2. Responses for item 5 for the pre- and postadministration in fall 2015 showing how student responses change.

Figure 3. Responses to item 5 from fall 2015 parsed by confidence response on item 6.

Figure 3 shows the percentage of students choosing each response by confidence level from “very high” to “no confidence”. On the pretest, over 60% of the students chose CO2 as the most abundant greenhouse gas, and 53% possessed reasonable to very high confidence in their answers. During the post administration the responses drastically changed to the correct response, water vapor, with 49% expressing very high confidence in their response.

research has also demonstrated that students often are unaware of other greenhouse gases, their abundance, origin, and lifetime in the environment.2,15,16,23,44 However, this instrument is able to capture through a pre/postadministration that student conception of the identity of greenhouse gases and their abundance can change during instruction. Item 5 also nicely illustrates the information that can be gained by plotting the confidence data with the student responses. G



Article

Responses to items 3 and 5 emphasize the importance of a robust model of matter at the particulate level. This would allow students to consider which molecules can absorb IR radiation and thus be greenhouse gases, knowledge which could help a student correctly respond to the question. To absorb IR radiation the molecule must have a vibrational mode that causes a fluctuation in the dipole moment of the molecule allowing the oscillating electrical and magnetic field to interact with the fluctuations of the dipole moment. Research by Cooper has demonstrated that the string of inferences from a chemical formula to a Lewis structure to consideration of physical properties of a molecule (e.g., bond dipoles leading to net polarity via vector addition) is a challenging one for students.48,49 However, if students do not use some sort of criteria such as physical properties to determine which molecules are greenhouse gases, then they are likely reduced to simply memorizing a list of compounds. Memorizing will not lead to the depth of understanding required to understand how greenhouse gases heat the atmosphere through collisional de-excitation. Particulate Nature of Matter, Animations, and VC3Chem

While the gains in student knowledge and confidence related to identification of greenhouse gases is positive in both administrations in fall 2015 and 2016, there are opportunities within the context of climate change to enhance student understanding of the molecular level. For example, there is evidence that animations help students build a stronger understanding of the particulate nature of matter.66−70 As part of the larger Visualizing the Chemistry of Climate Change or VC3Chem project in which this study is situated, the team developed and implemented multimedia modules that help students learn general chemistry concepts and techniques through the rich context of climate science.71 One of the modules is titled “Gases”, and it includes particulate level animations that help build strong particulate models related to the characteristics of greenhouse gases and the role of greenhouse gases in causing radiative forcing. One component of a particulate level understanding of radiative forcing by greenhouse gases is depicted in Figure 4a, which is a screen shot of the collisional heating learning tool. The image captures the depiction of a CO2 molecule about to absorb an infrared photon (the green squiggle) that will change its vibrational motion. Figure 4b shows the same tool with the atmosphere displayed where the blue spheres represent nitrogen gas molecules and the red spheres represent oxygen. As the animation runs the CO2 molecule absorbs photons of light causing its vibrational motion to change when the energy of the incoming photon corresponds to an absorption peak. When nitrogen or oxygen gas particles collide with the vibrationally excited CO2, energy is transferred, and the CO2 vibrates more slowly while the nitrogen or oxygen molecules speed up.

Figure 4. Screen shots of the collisional heating tool in the VC3Chem gases module. The location of the purple bar corresponds to the wavelength of the incoming IR radiation as well as absorption peaks for CO2. Part a shows a CO2 molecule about to absorb a photon of the appropriate energy to change its vibrational motion. Part b displays collisional de-excitation, whereby the most abundant atmospheric gases (N2 is blue and O2 is red) collide with vibrationally excited CO2 causing the temperature to rise as shown by the depiction of the thermometer and the increasing speed of the atmospheric gases in the simulation.71

First, it can be administered as a pre/postadministered instrument to measure changes in student understanding of both climate science concepts and chemistry concepts and knowledge. However, it could be administered once as a pretest to inform faculty on the existing conceptual knowledge of the students in the course. This would allow faculty to design pedagogical approaches that leverage what is revealed by implementing the survey. Second, the confidence ratings can be used to help faculty understand which concepts are strongly held (e.g., CO2 is the most abundant greenhouse gas) and which may be more uncertain. This may help faculty to design instruction to drive students toward more scientifically correct concepts and understandings. Limitations

Implementation in Classrooms

There is a tension between developing a compact instrument that can be administered in a relatively short amount of time and revealing enough about student understanding to inform and guide teaching practices. In a fundamental sense we acknowledge the challenge of exploring understanding of the “greenhouse effect” when the term itself engenders alternative conceptions about the mechanism of radiative forcing. Others may wish to modify this instrument to discover more about student understanding of the radiative forcing by greenhouse gases and climate change processes and effects. Additionally, the alternative conceptions between radiative forcing by greenhouse gases and ozone depletion could also be explored by adding more items. As with any instrument, it is important to explore the reliability and

As we developed, refined, and implemented this instrument, it was our intention to design it such that faculty could use it to adapt and modify their instructional practices based upon student responses. Analysis has shown that climate topics are integrated into a wide range of courses and faculty are looking for effective pedagogical strategies to guide their teaching.47 Kirk et al. wrote “undergraduate faculty members place a high priority on teaching climate science because of its relevance to the students with respect to their community, lives, and potential careers” (p 538).47 Given this backdrop and the results of our research, we offer the following recommendations for using this instrument. H



■

validity of the data generated when it is used in new contexts and/or if new items are added.

■

ASSOCIATED CONTENT

S Supporting Information *

The is included in the Supporting Information. A is also included. The Supporting Information is available on the ACS Publications website at DOI: 10.1021/acs.jchemed.6b00759. Final form of the instrument and table of the references parsed by the five areas of scientific confusion (PDF, DOCX)

■

REFERENCES

(1) Walz, K. A.; Kerr, S. C. Holes” in Student Understanding: Addressing Prevalent Misconceptions Regarding Atmospheric Environmental Chemistry. J. Chem. Educ. 2007, 84 (10), 1693−1696. (2) Shepardson, D. P.; Choi, S.; Niyogi, D.; Charusombat, U. Seventh grade students’ mental models of the greenhouse effect. Environ. Educ. Res. 2011, 17 (1), 1−17. (3) Shepardson, D. P.; Niyogi, D.; Choi, S.; Charusombat, U. Seventh grade students’ conceptions of greenhouse effect, global warming, and climate change. Clim. Change 2011, 104 (3-4), 481−507. (4) McNeal, K. S.; Spry, J. M.; Mitra, R.; Tipton, J. L. Measuring Student Engagement, Knowledge, and Perceptions of Climate Change in an Introductory Environmental Geology Course. J. Geosci. Educ. 2014, 62 (4), 655−667. (5) Danielson, K.; Tanner, K. Investigating undergraduate science students’ conceptions and misconceptions of ocean acidification. CBE Life Sci. Educ. 2015, 14, ar29. (6) Gilbert, J. K.; Watts, D. M. Concepts, misconceptions and alternative conceptions: Changing perspectives in science education. Stud. Sci. Educ. 1983, 10, 61−98. (7) Shepardson, D. P.; Niyogi, D.; Choi, S.; Charusombat, U. Seventh grade students’ conceptions of global warming and climate change. Environmental Educ. Res. 2009, 15 (5), 549−70. (8) McCuin, J. L.; Hayhoe, K.; Hayhoe, D. Comparing the Effects of Traditional vs Misconceptions-Based Instruction on Student Understanding of the Greenhouse Effect. J. Geosci. Educ. 2014, 62, 445−459. (9) Steffen, W.; Richardson, K.; Rockström, J.; Cornell, S. E.; Fetzer, I.; Bennett, E. M.; Biggs, R.; Carpenter, S. R.; de Vries, W.; de Wit, C.A.; Folke, C.; Gerten, D.; Heinke, J.; Mace, G. M.; Persson, L. M.; Ramanathan, V.; Reyers, B.; Sörlin, S. Planetary boundaries: Guiding human development on a changing planet. Science 2015, 347, 1259855. (10) Leiserowitz, A.; Maibach, E.; Roser-Renouf, C. Global Warming’s Six Americas 2009: An Audience Segmentation; Yale Project on Climate Change Communication; Yale University and George Mason University: New Haven, CT, 2008. http://climatecommunication. yale.edu/publications/?_sfm_topics=52 (Accessed October 2016). (11) Leiserowitz, A.; Smith, N.; Marlon, J. R. Americans’ Knowledge of Climate Change; Yale Project on Climate Change Communication; Yale University and George Mason University: New Haven, CT, 2010. http://environment.yale.edu/climate/files/ClimateChangeKnowledge2010.pdf\npapers3://publication/uuid/230CA706-26C2-41568133-1AC9E7701E6A (Accessed October 2016). (12) Leiserowitz, A.; Maibach, E.; Roser-Renouf, C.; Hmielowski, J. Global Warming’s Six Americas, March 2012 & November 2011; Yale Project on Climate Change Communication; Yale University and George Mason University: New Haven, CT, 2012. http:// climatecommunication.yale.edu/publications/?_sfm_topics=52 (Accessed October 2016). (13) Arslan, H. O.; Cigdemoglu, C.; Moseley, C. A three-tier diagnostic test to assess pre-service teachers’ misconceptions about global warming, greenhouse effect, ozone layer depletion, and acid rain. Int. J. Sci. Educ. 2012, 34 (11), 1667−1686. (14) Andersson, B.; Wallin, A. Students’ understanding of the greenhouse effect, the societal consequences of reducing CO2 emissions and the problem of ozone layer depletion. J. Res. Sci. Teach. 2000, 37 (10), 1096−1111. (15) Boyes, E.; Chuckran, D.; Stanisstreet, M. How do high school students perceive global climate change: What are its manifestations? What are its origins? What corrective action can be taken? J. Sci. Educ. Technol. 1993, 2 (4), 541−557. (16) Boyes, E.; Stanisstreet, M. The greenhouse effect − children’s perception of causes, consequences and cures. Int. J. Sci. Educ. 1993, 15 (5), 531−552. (17) Boyes, E.; Stanisstreet, M. The idea of secondary school children concerning ozone layer damage. Glob. Environ. Chang. 1994, 4 (4), 311− 324. (18) Boyes, E.; Stanisstreet, M. Children’s models of understanding of two major global environmental issues (ozone layer and greenhouse effect). Res. Sci. Technol. Educ. 1997, 15 (1), 19−28.

CONCLUSIONS We have developed and implemented a multiple-choice instrument that can be used to inform and shape instruction in classes where climate science is a focus, including capstone, current issue courses, or multidisciplinary courses that focus on earth system processes and climate change. The instrument contains items that pertain to chemistry concepts and that link chemistry concepts to climate science, and so the instrument may also be helpful in general chemistry and other chemistry classes where instructors wish to help students apply their conceptual understanding in chemistry to sustainability contexts such as climate change. We encourage faculty to use animations such as those found in VC3Chem to help students build mental models of the particulate nature of matter that can be used to describe radiative forcing by greenhouse gases with greater explanatory power.71 Additionally, the Intergovernmental Panel on Climate Change reports, the ACS Climate Science Toolkit, and the IUPAC ExplainingClimateChange resources are authoritative peerreviewed tools that can be used to help students learn more about climate science.72−74 On the basis of our review of the current research probing students’ understanding of climate science, the particulate nature of matter is overlooked as a crucial disciplinary component for understanding the chemistry of climate science.

■

Article

AUTHOR INFORMATION

Corresponding Author

*E-mail: [email protected]. ORCID

Thomas A. Holme: 0000-0003-0590-5848 Peter Mahaffy: 0000-0002-0650-7414 Marcy Towns: 0000-0002-8422-4874 Notes

The authors declare no competing financial interest.

■

ACKNOWLEDGMENTS This work was supported by the National Science Foundation under Grant 1022992. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation. The VC3Chem learning materials and simulations were created by the undergraduate student research team at the King’s Centre for Visualization of Science (www.kcvs.ca). Mary Kirchhoff and the American Chemical Society provided helpful support for the VC3Chem project. I



Article

(19) Boyes, E.; Stanisstreet, M. High school students’ perceptions of how major global environmental effects might cause skin cancer. J. Environ. Educ. 1998, 29 (2), 31−36. (20) Boyes, E.; Stanisstreet, M.; Papantoniou, V. S. The ideas of Greek high school students about the ozone layer. Sci. Educ. 1999, 83 (6), 724− 737. (21) Ç imer, S.; Ç imer, A.; Ursavas, N. Student teachers’ conceptions about global warming and changes in their conceptions during preservice education: A cross-sectional study. Educ. Res. Rev. 2011, 6 (8), 592−597. (22) Dove, J. Student teacher understanding of the greenhouse effect, ozone layer depletion, and acid rain. Environ. Educ. Res. 1996, 2 (1), 89− 100. (23) Fisher, B. Australian students’ appreciation of the greenhouse effect and the ozone hole. Aust. Sci. J. 1998, 44 (3), 46−55. (24) Gowda, M. V. R.; Fox, J. C.; Magelky, R. D. Students’ understanding of climate change: Insights for scientists and educators. Bull. Am. Meteorol. Soc. 1997, 78 (1), 2232−2240. (25) Groves, F. H.; Pugh, A. F. Elementary pre-service teacher perceptions of the greenhouse effect. J. Sci. Educ. Technol. 1999, 8 (1), 75−81. (26) Jarrett, L. E. Investigating Secondary School Students’ Understanding of Climate Change. Ph.D. Thesis, University of Wollongong, 2013. (27) Jarrett, L.; Ferry, B.; Takacs, G. Development and validation of a concept inventory for introductory-level climate change science. Int. J. Innov. Sci. Math. Educ. 2012, 20 (2), 25−41. (28) Jeffries, H.; Stanisstreet, M.; Boyes, E. Knowledge about the “Greenhouse Effect”: Have college students improved? Res. Sci. Technol. Educ. 2001, 19 (2), 205−221. (29) Keller, J. M. Development of a Concept Inventory Addressing Students’ Beliefs and Reasoning Difficulties Regarding the Greenhouse Effect. Ph.D. Thesis, University of Arizona, 2006. (30) Kilinc, A.; Stanisstreet, M.; Boyes, E. Turkish students’ ideas about global warming. Int. J. Environ. Sci. Educ. 2008, 3 (2), 89−98. (31) Koulaidis, V.; Christidou, V. Models of students’ thinking concerning the greenhouse effect and teaching implications. Sci. Educ. 1999, 83 (5), 559−76. (32) Morgan, M. D.; Moran, J. M. Understanding the greenhouse effect and the ozone shield: An index of scientific literacy among university students. Bull. Am. Meteorol. Soc. 1995, 76 (7), 1185−1190. (33) Myers, G.; Boyes, E.; Stanisstreet, M. School students’ ideas about air pollution: Knowledge and attitudes. Res. Sci. Technol. Educ. 2004, 22 (2), 133−152. (34) Ö sterlind, K. Concept formation in environmental education: 14year olds’ work on the intensified greenhouse effect and the depletion of the ozone layer. Int. J. Sci. Educ. 2005, 27 (8), 891−908. (35) Papadimitriou, V. Prospective primary teachers’ understanding of climate change, greenhouse effect, and ozone layer depletion. J. Sci. Educ. Technol. 2004, 13 (2), 299−307. (36) Pruneau, D.; Moncton, U.; Liboiron, L.; Vrain, E. People’s idea about climate change: A source of inspiration for the creation of educational programs. Can. J. Environ. Educ. 2001, 6 (1), 58−76. (37) Pruneau, D.; Gravel, H.; Bourque, W.; Langis, J. Experimentation with a socio-constructivist process for climate change education. Environ. Educ. Res. 2003, 9 (4), 429−446. (38) Ratinen, I. J. Primary student-teachers’ conceptual understanding of the greenhouse effect: A mixed method study. Int. J. Sci. Educ. 2013, 35, 929−955. (39) Rye, J.; Rubba, P.; Wiesenmayer, R. An investigation of middle school students’ alternative conceptions of global warming. Int. J. Sci. Educ. 1997, 19, 527−551. (40) Shepardson, D. P.; Niyogi, D.; Roychoudhury, A.; Hirsch, A. Conceptualizing climate change in the context of a climate system: Implications for climate and environmental education. Environ. Educ. Res. 2012, 18 (3), 323−352. (41) McCaffrey, M. S.; Buhr, S. M. Clarifying climate confusion: Addressing systemic holes, cognitive gaps, and misconceptions through climate literacy. J. Phys. Geograph. 2008, 29 (6), 512−528.

(42) Cook, J.; Bedford, D.; Mandia, S. Raising Climate Literacy Through Addressing Misinformation: Case Studies in AgnotologyBased Learning. J. Geosci. Educ. 2014, 62 (3), 296−306. (43) Huxster, J. K.; Uribe-Zarain, X.; Kempton, W. (2015). Undergraduate Understanding of Climate Change: The Influences of College Major and Environmental Group Membership on Survey Knowledge Scores. J. Environ. Educ. 2015, 46, 149−165. (44) Versprille, A.; Towns, M. General chemistry students’ understanding of climate change and the chemistry related to climate change. J. Chem. Educ. 2015, 92 (4), 603−609. (45) Mahaffy, P. G.; Martin, B. E.; Kirchhoff, M.; McKenzie, L.; Holme, T.; Versprille, A.; Towns, M. Infusing Sustainability Science Literacy Through Chemistry Education: Climate Science as a Rich Context for Learning Chemistry. ACS Sustainable Chem. Eng. 2014, 2 (11), 2488− 2622. (46) Mahaffy, P. Telling time: Chemistry education in the anthropocene epoch. J. Chem. Educ. 2014, 91, 463−465. (47) Kirk, K. B.; Gold, A. U.; Ledley, T. S.; Sullivan, S. B.; Manduca, C. A.; Mogk, D. W.; Wiese, K. Undergraduate Climate Education: Motivations, Strategies, Successes, and Support. J. Geosci. Educ. 2014, 62, 538−549. (48) Cooper, M.; Grove, N.; Underwood, S. M.; Klymkowsky, M. W. Lost in Lewis structures: An investigation of student difficulties in developing representational competence. J. Chem. Educ. 2010, 87, 869− 874. (49) Cooper, M. M.; Underwood, S. M.; Hilley, C. Z. Development and validation of the implicit information from Lewis structures instrument (IILSI): Do students connect structures with properties? Chem. Educ. Res. Pract. 2012, 13, 195−200. (50) Cooper, M. M.; Williams, L. C.; Underwood, S. M. Student understanding of intermolecular forces: A multimodal study. J. Chem. Educ. 2015, 92 (8), 1288−1298. (51) Nakhleh, M. B. Why some students don’t learn chemistry: Chemical misconceptions. J. Chem. Educ. 1992, 69 (3), 191−196. (52) Novick, S.; Nussbaum, J. Junior high school pupils’ understanding of the particulate nature of matter: An interview study. Sci. Educ. 1978, 62, 273−281. (53) Novick, S.; Nussbaum, J. Pupils’ understanding of the particulate nature of matter: A cross-age study. Sci. Educ. 1981, 65, 187−196. (54) Swartz, S. M. Acceptance and accuracy of multiple choice, confidence-level, and essay question formats for graduate students. J. Educ. Bus. 2006, 81 (4), 215−220. (55) Holtzman, M. Demystifying application-based multiple-choice questions. Coll. Teach. 2008, 56 (2), 114−120. (56) Halloun, I. A.; Hestenes, D. The initial knowledge state of college physics students. Am. J. Phys. 1985, 53, 1043−1055. (57) McClary, L. M.; Bretz, S. L. Development and assessment of a diagnostic tool to identify organic chemistry students’ alternative conceptions related to acid strength. Int. J. Sci. Educ. 2012, 34 (5), 2317− 2341. (58) Tamir, P. An alternative approach to the construction of multiple choice test items. J. Biol. Educ. 1971, 5, 305−307. (59) Taber, K. S. Ideas about ionization energy: a diagnostic instrument. School Science Review 1999, 81 (295), 97−104. (60) Treagust, D. F. Evaluating students’ alternative conceptions by means of diagnostic multiple-choice items. Res. Sci. Educ. 1986, 16, 199− 207. (61) Treagust, D. F. The development and use of diagnostic instruments to evaluate students’ alternative conceptions in science. Int. J. Sci. Educ. 1988, 10 (2), 159−169. (62) Holme, T.; Murphy, K. The ACS Exams Institute undergraduate chemistry anchoring concepts content map I: General Chemistry. J. Chem. Educ. 2012, 89, 721−723. (63) Holme, T.; Luxford, C.; Murphy, K. Updating the general chemistry anchoring concepts content map. J. Chem. Educ. 2015, 92, 1115−1116. (64) Adams, W. K.; Wieman, C. E. Development and validation of instruments to measure learning of expert-like thinking. Int. J. Sci. Educ. 2011, 33 (9), 1289−1312. J



Article

(65) Towns, M. H. Guide to developing high-quality, reliable, and valid multiple-choice assessments. J. Chem. Educ. 2014, 91, 1426−1431. (66) Yezierski, E.; Birk, J. Misconceptions about the particulate nature of matter: Using animations to close the gender gap. J. Chem. Educ. 2006, 83, 954−960. (67) Tang, H.; Abraham, M. Effect of computer simulations at the particulate and macroscopic levels of students’ understanding of the particulate nature of matter. J. Chem. Educ. 2016, 93 (1), 31−38. (68) Kelly, R. M.; Jones, L. L. Exploring how different features of animations of sodium chloride dissolution affect students’ explanations. J. Sci. Educ. Technol. 2007, 16, 413−429. (69) Kelly, R. M.; Jones, L. L. Investigating students’ ability to transfer ideas learned from molecular animations of the dissolution process. J. Chem. Educ. 2008, 85 (2), 303−309. (70) Nyachwaya, J. M.; Mohamed, A. R.; Roehrig, G. H.; Wood, N. B.; Kern, A. L.; Schneider, J. L. The development of an open-ended drawing tool: An alternative diagnostic tool for assessing students’ understanding of the particulate nature of matter. Chem. Educ. Res. Pract. 2011, 12, 121−132. (71) Mahaffy, P.; Martin, B.; Towns, M.; Kirchhoff, M.; McKenzie, L.; Versprille, A. Visualizing the Chemistry of Climate Change. http://www. vc3chem.com/ (Accessed September 2016). (72) ACS Climate Science Toolkit. https://www.acs.org/content/ acs/en/climatescience.html (Accessed December 2016). (73) Intergovernmental Panel on Climate Change. https://www.ipcc. ch/report/ar5/, (Accessed January 2017). (74) King’s Centre for Visualization in Science. Visualizing and Understanding the Science of Climate Change; www. explainingclimatechange.com (Accessed January 2017).

K


Assessing Student Knowledge of Chemistry and Climate Science

Recommend Documents