Arguing from Spectroscopic Evidence - ACS Publications - American

Jul 19, 2019 - minimize rote reproduction of algorithms. In pursuit of de- emphasizing the “plug and chug” focus of traditional mathematics course...
0 downloads 0 Views 2MB Size
Download PDF
This is an open access article published under an ACS AuthorChoice License, which permits copying and redistribution of the article or any adaptations for non-commercial purposes.

Article Cite This: J. Chem. Educ. XXXX, XXX, XXX−XXX

pubs.acs.org/jchemeduc

Arguing from Spectroscopic Evidence Ryan L. Stowe*,† and Melanie M. Cooper Department of Chemistry, Michigan State University, 578 South Shaw Lane, East Lansing, Michigan 48824, United States

Downloaded via 178.159.97.141 on August 21, 2019 at 23:41:44 (UTC). See https://pubs.acs.org/sharingguidelines for options on how to legitimately share published articles.

S Supporting Information *

ABSTRACT: Constructing and critiquing evidence-based claims is centrally important to aspiring medical professionals and to scientists more generally. Accordingly, The National Academy of Science’s Framework for K−12 Science Education describes engaging in argument from evidence as one of the practices that characterize work in science. However, despite the central role argumentation plays in construction and refinement of evidence-based explanations and models, it is very often absent from K−16 science learning environments. Here, we frame argumentation from spectroscopic evidence in terms of flexible application of a series of procedures to pull information from spectroscopic traces and use this evidence to inform claims as to the structure of an unknown molecule. Through analysis of responses to several multipart assessment items, we examined how students analyzed, interpreted and used spectroscopic evidence to inform their claims. We found that students were fairly adept at analyzing and interpreting data from infrared and 13C NMR traces as well as indicating correspondence between proton environments in their structural prediction and appropriate 1H NMR peaks. Unfortunately, none of these tasks were significantly associated with student success in proposing a claim consistent with an entire corpus of spectroscopic data. Further, scaffolding of the task prompt had no impact on student ability to successfully construct evidencebased structural predictions. Our findings indicate that students will require significant support to use their procedural knowledge flexibly in order to iteratively construct and critique claims supported by spectroscopic evidence. KEYWORDS: Second-Year Undergraduate, Organic Chemistry, Problem Solving/Decision Making, Spectroscopy, Chemical Education Research FEATURE: Chemical Education Research

■

supported by sound interpretation of evidence.4 The process of pulling information from various strands of data, itself the SEP “analyze and interpret data”, to gather evidence that supports or refutes a diagnosis is heavily reliant upon the use of procedural knowledge (i.e., the “rules of the game” for solving a given problem). For example, interpretation of an electrocardiogram (EKG) requires awareness of the parameters of a normal rhythm (including the normal amplitude, deflection, and duration of each component) and that one must make comparisons between these ideal parameters and a patient’s observed EKG trace. There may be a number of procedures that might characterize these comparisons, with some being more or less efficient for a given scenario. Flexible command of many such approaches coupled with knowledge of when a given procedure is most appropriate has been characterized as “deep procedural knowledge” by researchers in mathematics education.5−7 Such knowledge is essential to efficient and accurate interpretation

INTRODUCTION A significant portion of students enrolled in introductory organic chemistry aspire to careers in the healthcare sector. Accordingly, much of the rhetoric justifying organic chemistry as a medical school and dental school prerequisite focuses on analogies between competencies desired in medical professionals and those alleged to be fostered by the course.1,2 We have previously argued that these competencies can be precisely defined and operationalized as the Science and Engineering Practices (SEPs) defined by A Framework for K−12 Science Education (the Framework).3 The Practices describe processes characteristic of work in science and encompass both ways of thinking inherent in experimental design (e.g., “design and carry out investigations” and “analyze and interpret data”) and ways of communicating and refining scientific ideas in communities (e.g., “engage in argument from evidence” and “evaluate and communicate information”). Diagnosis, which is the construction of a claim as to the cause of a collection of symptoms, is centrally important to nearly all healthcare practitioners. Diagnosis may be regarded as the SEP “engaging in argument from evidence”, as it requires medical professionals to construct tentative causal accounts of symptoms © XXXX American Chemical Society and Division of Chemical Education, Inc.

Received: June 12, 2019 Revised: July 19, 2019

A

DOI: 10.1021/acs.jchemed.9b00550 J. Chem. Educ. XXXX, XXX, XXX−XXX

Journal of Chemical Education

Article

“conceptual” mean with regard to knowledge.23 At present, there is significant consensus that mathematics education should minimize rote reproduction of algorithms. In pursuit of deemphasizing the “plug and chug” focus of traditional mathematics courses, there have been several high profile initiatives that advocate teaching the conceptual basis for a particular procedure before the procedure itself.24−26 We will focus here on what is meant by “procedural knowledge” as well as whether such knowledge can be anything other than rote application of memorized rules. Hiebert and Lefevre defined procedural knowledge as “chains of prescriptions for manipulating symbols” in the mid-1980s. They painted a contrast between procedural and conceptual knowledge, which was said to be “rich in relationships. It can be thought of as a connected web of knowledge.”27 Star has taken issue with Hiebert and Lefevre’s characterization of procedural and conceptual knowledge in mathematics, claiming that they have conflated knowledge type (procedural−conceptual) and quality (novice−expert).6,7 He has noted that flexible, contextsensitive application of procedures represents a deep (that is, expert-like) understanding of both the procedures themselves and what constitutes an appropriate procedure for a given scenario. It should be noted that procedure “appropriateness” is not always defined in a clear-cut manner. Star has defined the most appropriate use of procedures as that which is most efficient (i.e., requires the fewest steps to reach a solution).28 This definition of “appropriateness” does not map well onto the use of procedural knowledge to construct and critique claims informed by spectroscopic data, where there is no defined “most step efficient solution”. Context sensitivity, that is, recognition of when information from a particular data source would be helpful in refining a claim, is more germane to discerning when a procedure would be most appropriate for the data analysis that informs argumentation from spectroscopic evidence. Here, we describe application of a chain of procedures to solve a problem as activation of a series of intellectual resources. Such “resources” were first proposed by Hammer as an extension of diSessa’s “knowledge in pieces” approach to conceptual change.29,30 These resources represent small-grain knowledge elements of a variety of types, including ideas extrapolated from experience, notions about the nature and appropriate use of knowledge, and knowledge of algorithms and heuristics.31−34 They are not “correct” or “incorrect” in and of themselves, but they may be activated in ways that are not appropriate for a particular scenario. For example, the resource that “more effort begets more result” has been activated in all manner of idiosyncratic, incorrect student explanations for differences in boiling point.35 As students gain facility with patterns of resource activation, “locally coherent sets of resources may... become established as resources in their own right.”34 Thus, the term “resource” alone does not necessarily imply a fixed grain size; some resources are smaller grain than others.

and use of evidence and therefore a vital component of the overall practice of diagnosis. The dialogic process of construction and critique of claims that characterizes diagnosis is analogous to student engagement in argumentation in the context of spectroscopy. That is, when asked to determine the identity of an unknown from analysis of spectroscopic data, students must pull evidence from various spectra, construct a claim informed by this evidence, and analyze whether that tentative claim is in fact consistent with all of the evidence assembled. Students do not begin this process by “knowing the answer”, unless the problem is an exact reproduction of one discussed previously, and so their relation of evidence to claim is meant to persuade the instructor or their peers of the reasonableness of the claim itself. Importantly, the structural hallmarks of argumentation (i.e., construction and critique of evidence-based claims) may occur via dialogue among members of a group or internally, via an individual using different mental “voices” for claim building and vetting.8 Here, we are focused on tasks designed to require argumentative discourse in the minds of individual students. We are aware of no existing peer-reviewed literature that examines spectroscopic analysis through the lens of argumentation. Prior reports instead focus on strategies believed to foster “problem solving skills”,9−14 describing the differences between expert and novice analyses of spectroscopic data,15−18 or characterizing invalid chemical assumptions and heuristics that are believe to constrain student reasoning during IR and 1H NMR spectroscopic interpretation.19 We can find no published strategies that contain strong evidence that a particular intervention improves student ability to argue from spectroscopic evidence; that is, extant literature does not compare the success of large student cohorts engaged in the described intervention with those not so engaged. Despite the significant role argumentation plays in refining knowledge claims within the scientific community,20 it is very often absent from instruction in both K−12 and higher education spaces.21 Our work here is an attempt to characterize student engagement in argumentation from spectroscopic evidence in the context of scaffolded assessment items.22 In particular, we are interested in the following questions: 1. How successful are students at deploying procedural resources to analyze and interpret data from a variety of spectroscopic sources? 2. What associations exist between student analysis and interpretation of spectroscopic data and their use of this evidence to inform claims as to the identity of an unknown? 3. To what degree do analysis, interpretation, and use of different strands of spectroscopic data predict student success at proposing a structure consistent with all assembled evidence? 4. How does varying the structure of the prompt impact student success in constructing a claim consistent with provided spectroscopic evidence?

Analysis and Interpretation of Spectroscopic Data as Activation of Resources

■

To analyze and interpret data from the spectroscopic traces commonly examined in organic chemistry (e.g., infrared, 13C NMR, and 1H NMR), one must be familiar with a range of algorithms as well as when those algorithms could be productively used. Our use of “algorithm” here is meant to indicate that, if one follows a set of well-behaved rules, one can successfully pull relevant information from a spectrum. In other words, analysis and interpretation of spectroscopic data requires

ON PROCEDURAL KNOWLEDGE Most work defining and exemplifying various aspects of procedural knowledge derives from the mathematics education literature. Indeed, there has been vigorous discussion for decades among mathematics education scholars about appropriate learning environment emphasis on “procedural” versus “conceptual” knowledge, as well as what “procedural” and B

DOI: 10.1021/acs.jchemed.9b00550 J. Chem. Educ. XXXX, XXX, XXX−XXX

Journal of Chemical Education

Article

in the context of a particular problem. For example, if one does not know the “n + 1” rule, then they will be unable to say anything about the number of protons on carbons adjacent to a given proton environment. However, knowledge of this “rule” does not guarantee that students will consistently and methodically apply it when critiquing structural predictions. Cartrette and Bodner characterized individuals that were “more successful” in relating structural claims to spectroscopic evidence as having a methodical strategy that involved effective application of algorithms such as the “n + 1” rule to depict fragments indicated by portions of a spectroscopic trace en route to proposing a complete structural prediction.15 “More successful” problem solvers also, without exception, carefully checked consistency between their prediction and assembled spectroscopic evidence by again employing a variety of procedural resources. Here, we will examine student success at relating evidence pulled from spectroscopic data to structural predictions in the context of formative and summative assessment prompts.

procedural knowledge. In characterizing data analysis and interpretation as “procedural”, we are assuming that the spectra analyzed are fairly unambiguous and that more advanced conceptual knowledge is not required for interpretation, as would be the case if a spectrum were to show, for instance, longrange proton coupling. All of the spectroscopic traces examined in the course context for this study were unambiguous. Each procedure is likely learned as activation of a series locally coherent resources that, over time, become compiled into resources themselves. For the purposes of the analyses that follow, we define several algorithms elementary to spectroscopic data analysis. These include • Determining the number of carbon environments by counting the peaks in a 13C NMR spectrum • Recognizing the presence or absence of functional groups (e.g., hydroxyl and carbonyl functionality) in an infrared spectrum • Determining the number of protons on carbons adjacent to a given environment from the splitting of a particular 1 H NMR peak • Determining the number of protons in a given environment from the integration of a particular 1H NMR peak Each of these procedures are almost certainly composed of smaller-grain knowledge elements for many students. Indeed, helping students compile the locally coherent resources that make up each algorithm into a larger-grain resource that can be deployed with little mental effort is a central goal of our treatment of spectroscopy. However, a finely grained analysis of the character of students’ procedural knowledge is beyond the scope of this study. We cannot know, from written responses to prompts, whether students have compiled the procedures noted above into resources. It should be mentioned that the procedures we have described are by no means exhaustive and students almost certainly bring other resources to bear on argumentation problems beyond what we noted. For example, helpful heuristics such as “a 6H doublet around 1 ppm in a proton NMR typically corresponds to two methyl groups in an isopropyl functionality” are undoubtedly cultivated among the student populace as they work through many spectroscopy problems. The reader may note that we have not made mention of the physical basis for spectroscopic techniques when describing the procedures needed to analyze and interpret spectroscopic evidence; this was intentional. One need not understand the impact of nearby magnetic environments on proton NMR peak splitting to learn that the multiplicity of a particular peak corresponds to one more than the number of protons on carbons adjacent to a given proton environment (the “n + 1” rule). For the purposes of our analysis, we have only addressed these four algorithms because (as the reader will see) even such relatively simple resources are difficult for students to coordinate. It is an open question whether possessing conceptual knowledge improves one’s ability to argue from spectroscopic evidence.

■

METHODS

Course Context

Our work occurred in the context of a transformed, largeenrollment organic chemistry course known as Organic Chemistry, Life, the Universe and Everything (OCLUE).22 This a two-semester course and has enrolled between 350 and 700 students since the fall of 2016. Students meet as a large group for approximately 3 h/week and attend a weekly 50 min recitation section, where they work in groups on tasks that require the use of SEPs under the guidance of a graduate teaching assistant. The data presented here derives from OCLUE enactments from fall 2016 to fall 2018. Like the transformed general chemistry course Chemistry, Life, the Universe, and Everything (or CLUE),36 OCLUE is structured around scaffolded progressions of core ideas37 that build in complexity as students predict, explain, and model ever more complex phenomena. These core ideas are not discrete topics but rather large-grain ideas that can be taught at various levels of sophistication, have significant power to explain phenomena, and undergird all topics in the course.3,37 In both CLUE and OCLUE these ideas are “energy”, “atomic/molecular structure and properties”, “stability and change in chemical systems”, and “electrostatic and bonding interactions.” Connections between core ideas and phenomena are explicit in instruction in order to help students develop and organize their knowledge as they progress. The physical basis for spectroscopic techniques as well as analysis of spectroscopic data are introduced early in the first semester of OCLUE and build in complexity throughout the course. Infrared (IR) spectroscopy is introduced first, with emphasis on recognizing peaks that correspond to carbonyl and hydroxyl functional groups. Carbon nuclear magnetic resonance spectroscopy (13C NMR) is discussed following IR spectroscopy and before proton NMR, as 13C NMR is less information-rich than proton NMR and should therefore impose less cognitive load on students. Finally, proton NMR is introduced and unpacked. Spectroscopy is introduced early in the first semester to provide students with evidence about molecular structure; the conceptual basis for various spectroscopic techniques precedes student use of these techniques to make sense of data. For example, students learn why electron density affects NMR signals, and why carbons with electronegative substituents

Argumentation from Spectroscopic Evidence through a Resources Lens

The procedures we have characterized above can be used in more or less efficient ways to support the construction and critique of a variety of claims. As with “deep procedural knowledge” more generally, facility with argumentation from spectroscopic evidence relies upon students possessing the requisite resources and also being strategic with their application C

DOI: 10.1021/acs.jchemed.9b00550 J. Chem. Educ. XXXX, XXX, XXX−XXX

Journal of Chemical Education

Article

appear downfield prior to analyzing and interpreting 13C NMR. As new functionality is explored throughout both semesters of the course, the spectroscopic signatures of that functionality are introduced and discussed. At the same time, the emphasis shifts to the use of spectroscopic evidence to support claims about molecular structure. In this report, we focus on student engagement in argumentation from spectroscopic evidence, rather than student understanding of the physical basis for that evidence.

Box 1. 3D-LAP Criteria for Prompts with the Potential to Engage Students in Argumentation from Evidence 1. Question gives an event, observation, or phenomenon 2. Question gives or asks student to make a claim based on a given event, observation, or phenomenon 3. Question asks student to provide evidence in the form of data or observations to support the claim 4. Question asks student to provide reasoning about why the evidence supports the claim

Student Participants

These studies were carried out at a large, midwestern research university. All student participants were informed of their rights as research subjects, and all data was obtained and handled in accordance with the Institutional Review Board. A total of 300 students participated in this study. All participants were enrolled in both semesters of OCLUE within a single academic year. Our sample is made up of students from three cohorts, with each cohort consisting of 100 randomly selected students who were enrolled in a two-semester OCLUE enactment; 100 students were enrolled in the course from 2016 to 2017, 100 were enrolled from 2017 to 2018, and 100 were enrolled from 2018 to 2019. To determine whether significant differences existed in the demographic and academic measures for our three cohorts, we ran a series of Mann−Whitney U tests comparing, in a pairwise fashion, cohort ACT scores, grades in the first semester of OCLUE, and overall GPA prior to enrolling in organic chemistry. In cases where students had only taken the SAT, SAT scores were converted to ACT scores using concordance tables published by the ACT.38 Apart from the instance mentioned below, no significant differences in the demographic and academic measures of our cohorts were found. The cohort enrolled in OCLUE during the 2017−2018 academic year had somewhat higher first-semester OCLUE grades than the cohort enrolled during the 2018−2019 academic year (mean of 3.6 vs 3.4, U = 4023.5, z = −2.655, p = 0.008, r = 0.18; small effect size). All statistical tests reported in this contribution were conducted using version 24 of SPSS Statistics for Mac.39 Our threshold for significance in this study was p ≤ 0.01. Summaries of all statistical analyses of demographic and academic measures are reported in Table S1, contained in the Supporting Information.

Each prompt type disclosed below will be annotated to denote the criteria they fulfill. Note that not all assessments used fulfill all of the criteria given; these assessments are best categorized as not having the potential to elicit evidence of student engagement in argumentation. All infrared spectra used for our prompts were derived from the Spectral Database for Organic Compounds.44 All 13C NMR and 1H NMR spectra were created using the NMR prediction tools found in ChemDraw Professional 15.1 for Mac. Following discussion of our different prompt varieties, we will discuss our coding of these assessment tasks with focus on analysis and interpretation of particular strands of spectroscopic data, the correlation between pulling information from spectra and leveraging this information in constructing claims, and overall success at proposing claims consistent with provided evidence. Prompt Type 1: Low Structure

Assessment items of this variety were the least structured of the prompts we administered to students. Students were shown infrared, 1H NMR, and 13C NMR spectra; given a molecular formula; and asked to construct a claim consistent with assembled evidence, as shown in Figure 1. “Low structure”

Instruments

Our data corpus consisted of responses to four types of prompts that were designed to engage students in analysis and interpretation of spectroscopic data and construction of claims consistent with assembled evidence (i.e., argumentation). Prompt scaffolding varied in order to discern how students could and should be supported in arguing from spectroscopic evidence on written assessments. Assessment items were given an summative assessments on midterm exams and as formative assessments to be completed on the online homework system beSocratic. beSocratic enables students to construct explanations, predictions, and models in response to open-ended questions.40−42 Below we will discuss each prompt variety with focus on why each was scaffolded in a particular manner. Our scaffolding strategies were informed by the criteria Laverty et al. put forth in the 3-Dimensional Learning Assessment Protocol (3D-LAP) for assessments that have the potential to elicit evidence of student engagement in science and engineering practices (SEPs).43 In particular, we focused on the criteria the assessments had to fulfill to potentially elicit evidence of student engagement in argument from evidence (Box 1).

Figure 1. Unstructured prompt asking students to propose a structure consistent with several strands of spectroscopic evidence. This prompt does not have the potential to engage students in argumentation from evidence as defined by the 3D-LAP.

prompts presented students with several spectroscopic traces that were collected by bombarding an unknown compound with electromagnetic radiation (EMR) of different wavelengths and detecting EMR absorbed or emitted (e.g., “an observation”). Following this, respondents were asked to construct a claim consistent with assembled spectroscopic data. Prompts of this type never explicitly asked students to provide evidence that D

DOI: 10.1021/acs.jchemed.9b00550 J. Chem. Educ. XXXX, XXX, XXX−XXX

Journal of Chemical Education

Article

supported their claim or to link evidence to claim by way of sound reasoning. Accordingly, they do not fulfill the 3D-LAP criteria for having the potential to elicit evidence of student engagement in argumentation from spectroscopic evidence (Box 1). The most that can be concluded from responses to a low structure prompt is whether students constructed a consistent claim. These prompts were used as controls to address RQ4 but could not provide information useful in addressing RQs 1−3. These prompts were only given as part of formative assessments. Prompt Type 2: Moderate Structure

The second type of prompt embodies the minimum amount of scaffolding we believe is required to meet the criteria laid forth in the 3D-LAP for having the potential to elicit evidence of engagement in argumentation. As shown in Figure 2, prompts of

Figure 3. Highly structured prompt designed to elicit evidence of student engagement in argumentation from spectroscopic evidence. The prompt spacing is condensed from that given to students. Adapted with permission from ref 2. Copyright 2017 American Chemical Society.

evidence they should pull from assembled traces (e.g., functional groups indicated by the IR and number of carbon environments). Further, they were asked to indicate correspondence between peaks in the 1H NMR and their prediction, that is, directly relate aspects of their claim to the evidence, which we consider as meeting the “reasoning” criteria of the 3D-LAP (Figure 3). Prompts of this type are useful for addressing whether students can analyze and interpret various strands of spectroscopic data given explicit guidance to do so. “High structure” prompts were given on a formative assessment administered in the spring of 2018, as well as on midterm exams given during the springs of 2017 and 2018. The data we will use to answer RQs 1−3 are derived from responses to these prompts.

Figure 2. Prompt designed to engage students in argumentation from spectroscopic evidence. This represents an assessment with a moderate amount of scaffolding.

this type provide students with an observation analogous to that shown in Figure 1. Additionally, students are asked to propose a claim consistent with the spectroscopic traces before them. However, unlike low structure prompts, “moderate structure” prompts ask students to justify their prediction by citing specific strands of evidence; that is, they are being asked to relate evidence to their claim by way of reasoning. There existed minimal cuing as to what evidence would be appropriate to mention nor how that should be related, and so this should still be considered a fairly unstructured prompt. Moderate structure prompts were used to examine the impact of prompt structure on students’ ability to construct evidence-based claims as part of addressing RQ4.

Prompt Type 4: Critique Centered

The final prompt type we will discuss here was meant to elicit evidence of student construction and critique of evidence-based claims (Figure 4). Accordingly, students were given two sources of spectroscopic data obtained from an unknown (infrared and 13 C NMR spectra), asked to pull information from each source, and asked to construct two claims consistent with assembled evidence. Following this, students were given an 1H NMR spectrum corresponding to that same unknown, asked to analyze and interpret an aspect of this spectrum, and subsequently asked to rule out one or both of their prior claims. The final part of this question asked for the construction of a claim consistent with all data sources. A critique centered prompt was given as a formative assessment during the fall 2018 semester. A critiquecentered prompt was used to examine the impact of prompt scaffolding on students’ success in constructing evidence-based claims as part of addressing RQ4.

Prompt Type 3: High Structure

Our third prompt variety should be considered “highly structured” relative to the first and second prompt types shown in Figures 2 and 3. As with previously mentioned prompts, students were called to propose a claim consistent with spectroscopic evidence and relate specific strands of evidence to their claim. However, unlike with moderate structure prompts, students were explicitly cued to focus on particular pieces of

Coding Protocol

The descriptors of student responses that formed the basis for our analyses were binary in nature. That is, students either E

DOI: 10.1021/acs.jchemed.9b00550 J. Chem. Educ. XXXX, XXX, XXX−XXX

Journal of Chemical Education

Article

one of the authors, not disagreement about the meaning of a code. Both authors met following joint coding and reached consensus as to the proper code each of the 30 responses merited for each question subpart. RQ1: How successful are students at deploying procedural resources to analyze and interpret data from a variety of spectroscopic sources?

Student ability to deploy the algorithms mentioned earlier to pull information from different spectroscopic sources was analyzed using responses to portions of high structure prompts. Such prompts explicitly asked students to describe the functional groups indicated by an infrared spectrum and determine the number of chemically distinct carbons shown by a 13C NMR spectrum. Assessments structured in this manner also asked students to indicate which proton environment in their prediction corresponded to which 1H NMR peak (Figure 3). In order to address RQ1, we noted whether student answers to appropriate question subparts indicated appropriate or inappropriate resource deployment. For example, we coded whether students correctly indicated the number of carbon environments when asked to do so. In analyzing the use of algorithms relevant to 1H NMR, it was difficult to deconvolute the two algorithms mentioned previously (i.e., use of the “n + 1” rule and peak integration). Accordingly, they were taken together for the purposes of RQs 1−3. That is, student ability to indicate reasonable correspondence between a proton environment in their prediction and a 1H NMR peak was read to indicate facility with interpreting information from both peak splitting and integration. Additionally, although student analysis and interpretation of 13C NMR and IR data was decoupled from claim construction in high structure items, students were required to use their interpretation of 1H NMR data in the context of their structural prediction on part (e) of these assessments. Accordingly, we describe student ability to reasonably label the proton environments in their prediction as “analysis, interpretation, and use” of proton NMR data. All coding conducted as part of this work was binary; students either correctly analyzed and interpreted particular information from spectra, or they did not. Importantly, we did not and do not assume that a correct or incorrect response indicates the best a student is capable of; responses merely indicate those resources activated in a particular moment in response to a given prompt.

Figure 4. Assessment prompt meant to engage students in construction and critique of claims grounded in spectroscopic evidence. The prompt spacing is condensed from that given to students.

indicated the correct response to part of a prompt or they did not. Accordingly, there was no nuanced interpretation of responses to negotiate among raters. As a result of this, we did not see need for extensive testing of the reliability of our coding. The first author coded all the data analyzed for this contribution. To spot-check the first author’s description of student responses, the second author coded a random sample of 30 responses to high structure exam items, and their codes were compared against those assigned by the first author. In particular, both authors coded (1) whether students indicated the number of carbon environments shown in the 13C NMR, (2) whether students noted the functionality shown in the infrared spectrum, (3) whether students constructed a structural claim with an appropriate number of carbon environments, (4) whether students constructed a structural claim with the functionality shown in the infrared spectrum, (5) whether students indicated reasonable correlation between one or more peaks in the 1H NMR spectrum and their claim, and (6) whether students constructed a claim consistent with all assembled evidence. These codes correspond to responses to different parts of the high structure prompts. For example, part A (Figure 3) asks students to describe the functionality indicated by the infrared spectrum. Responses to part A provided evidence as to whether students pulled reasonable information from the IR spectrum regarding functional groups present in a molecule. Agreement between the authors on the codes appropriate for student responses to each prompt portion ranged from 83 to 100%. All discrepancies that arose were the result of an error on the part of

RQ2: What associations exist between student analysis and interpretation of spectroscopic data and their use of this evidence to inform claims as to the identity of an unknown?

Engagement in argumentation from spectroscopic evidence requires that students use evidence from analyzed data to inform the construction and critique of claims. Accordingly, we were interested in what associations (if any) existed between student ability to pull information from spectroscopic data (coded as described under RQ2) and incorporation of this information into their structural prediction. We coded whether the claims students constructed in response to high structure prompts were consistent with particular strands of spectral data. For example, does the structural prediction have six carbon environments as the 13C NMR suggests? Does the structural prediction have a hydroxyl group as the infrared spectrum would indicate? Following this, we analyzed association between analysis and interpretation of data and the construction of claims consistent with particular data sources via a series of χ2 tests. For results that showed a significant association, the strength of the relationship F

DOI: 10.1021/acs.jchemed.9b00550 J. Chem. Educ. XXXX, XXX, XXX−XXX

Journal of Chemical Education

Article

Table 1. Three Experiments Conducted to Determine the Impact of Prompt Structure on Students’ Construction of Claims Consistent with Spectroscopic Evidence Experiment

Comparison Prompt

Experimental Prompt

Experiment 1

Low structure

High structure

Experiment 2

Low structure

Critique centered

Experiment 3

Moderate structure, two spectroscopic traces

Moderate structure, three spectroscopic traces

Purpose Determine the impact of explicitly prompting for data analysis and argument justification Determine the impact of explicitly prompting students to construct and critique claims Determine the impact of varying the amount of data students had to consider when constructing claims

Figure 5. Percentage of students who analyzed, interpreted, or used spectroscopic data from several sources as prompted by highly structured assessment items. “Exam 1” prompts were present on summative assessments, whereas “HW” prompts represent assessments given as part of a homework assignment. The portions of multipart assessment items germane to analysis, interpretation, or use of spectroscopic data are found in the “Prompts” box in the bottom right of the figure.

was calculated using Cramer’s V and interpreted using guidelines published by Cohen.45 Accordingly, a small effect would have a Cramer’s V of 0.1, a medium effect would have a Cramer’s V of 0.3, and a large effect would have a Cramer’s V of 0.5.

because of their very explicit prompting for invocation of the algorithms mentioned previously. RQ4: How does varying prompt structure impact student success in constructing a claim consistent with provided spectroscopic evidence?

RQ3: To what degree do analysis, interpretation, and use of different strands of spectroscopic data predict student success at proposing a structure consistent with all assembled evidence?

Three experiments were conducted to determine whether different prompt structures might impact student success in constructing a claim consistent with spectroscopic evidence (Table 1). Past studies indicate that the manner in which a prompt is structured can powerfully affect the resources students bring to bear in their responses.46 If students do not know what we are asking, they might not construct the best explanation, prediction, or model they are capable of generating. Accordingly, we hypothesized that differently structured spectroscopic argumentation prompts might impact student success in proposing appropriate evidence-based claims. Two of the three experiments conducted to address our first research question focused on determining whether explicitly requiring students to analyze, interpret, and use spectroscopic data improved the percentage of our sample that constructed appropriate claims. In both of these cases, the class was randomly assigned to either an experimental or control condition. The control condition was given a low structure prompt, whereas the experimental condition was given a high structure or “critique focused” prompt involving use of the same

Binary logistic regressions were performed to determine whether student success in proposing claims consistent with spectroscopic evidence could be predicted in part by their success analyzing and interpreting particular strands of spectroscopic data. Models were constructed for high structure prompts with the outcome variable being “correct prediction” in all instances and the predictor variables being “indicate the correct number of carbon environments shown by the 13C NMR”, “indicate the correct functionality shown by the infrared spectrum”, and “indicate reasonable correspondence between some of the proton environments in your prediction and some of the 1H NMR peaks”. For this analysis, we will report whether any predictor variable significantly increased the model’s predictive validity relative to the intercept-only “null model”. High structure prompts were chosen as the basis for our models G

DOI: 10.1021/acs.jchemed.9b00550 J. Chem. Educ. XXXX, XXX, XXX−XXX

Journal of Chemical Education

Article

Figure 6. Percentage of students who analyzed and interpreted data from infrared and 13C NMR spectra as well as the percentage of students who constructed a structural prediction consistent with these two types of spectroscopic data. “Exam 1” prompts were present on summative assessments, whereas “HW” prompts represent assessments given as part of a homework assignment.

infrared, 13C NMR, and 1H NMR traces. Accordingly, we will unpack student responses to high structure prompts that explicitly require use of relevant procedural resources (see Figure 3 for an example of one such prompt). The percentage of students from our sample who deployed these resources appropriately is shown in Figure 5. Recall that, for the purposes of the analyses encapsulated by Figure 5, we defined “analyze and interpret spectroscopic data” as application of a set of four procedures. Thus, analysis and interpretation of data from an infrared spectrum signified that students could correctly describe functional groups indicated by a particular spectrum. As this contribution is focused on productive activation of resources rather than describing difficulties, students who described a functionality shown by an IR trace and also described a functionality not indicated by that same data source were counted as “analyzing and interpreting data” here. Analysis and interpretation of 13C NMR data focused on whether students could correctly describe the number of carbon environments indicated by a spectrum. It should be noted that analysis and interpretation of data from 13C NMR and IR spectra were explicitly prompted for by the high structure prompts, whereas students were not specifically prompted to extract information from 1H NMR spectra. Instead, they were asked to use evidence from a proton NMR to justify their claims (see the portion of Figure 5 that reads “Prompts”). For this reason, we consider indicating a reasonable correspondence between at least some of the proton environments in their structural prediction and the 1H NMR spectrum as evidence students were “analyzing, interpreting, and using” data from a proton NMR. As such correspondence had to take into account information from peak splitting and multiplicity, we considered this an indicator that students could leverage both of the proton-NMR-relevant procedures previously described to interpret 1H NMR data and use that data to justify their prediction. Interestingly, no association existed between the type of assessment (i.e., formative or summative) and student success in analyzing, interpreting, and using data from IR and 1H NMR spectra as prompted for by the high structure assessments (p > 0.01 in all instances). No measures of association could be calculated relating the type of assessment to

data set. To analyze whether prompt structure impacted the reasonableness of student claims, we coded whether each student constructed a claim consistent with assembled evidence and compared the percentage of successful students in the control versus experimental conditions. Association between condition and constructing a reasonable claim was analyzed via a Pearson’s χ2 test. The third experiment was meant to shed light on whether asking students to examine fewer strands of spectroscopic evidence would impact their success in constructing a claim consistent with that evidence. It has been well-established that humans have a limited capacity to process information.47−50 In light of this, one might guess that difficulties with engaging in argumentation from spectroscopic evidence could emerge in part from the various procedures that make up this practice, overwhelming students’ working memory. Requiring fewer processing tasks, by reducing the amount of data to analyze, interpret, and use, might therefore positively impact student ability to construct evidence-based claims. To test this, we randomly assigned students to one of two groups. Both groups were assigned a set of two moderate structure prompts with one prompt asking for a claim consistent with two spectroscopic traces (infrared and 13C NMR) and the other prompt asking for a structural prediction consistent with three spectroscopic traces (infrared, 13C NMR, and 1H NMR). Group 1 received all three traces for Unknown A and two traces for Unknown B. Group 2 received two traces for Unknown A and three for Unknown B. Each assessment response was coded according to whether the claim presented was completely consistent with the assembled evidence. We examined whether there was an association between the percentage of students who proposed a reasonable claim and the number of strands of data students were required to analyze via a series of χ2 tests.

■

RESULTS AND DISCUSSION

RQ 1: Success in Analyzing and Interpreting Particular Strands of Spectroscopic Data

This research question focuses squarely on student facility with the four algorithms previously highlighted as integral to the analysis, interpretation, and use of spectroscopic data from H

DOI: 10.1021/acs.jchemed.9b00550 J. Chem. Educ. XXXX, XXX, XXX−XXX

Journal of Chemical Education

Article

success analyzing and interpreting 13C NMR data, because at least one variable in each two-way table was a constant. Lack of association between type of assessment and success in prompted performances indicates that students were just as successful on homework problems as they were on similarly structured exam prompts. From examination of Figure 5, it is readily apparent that most student respondents proved adept at deploying the procedural resources in the context of highly structured prompts on exams and homework. In particular, nearly everyone could pull information on functional groups from the infrared spectrum and discern the number of carbon environments from the 13C NMR. Both of these tasks are fairly straightforward and practiced a great deal during lecture and on homework. Students were similarly successful in labeling correspondence between 1H NMR peaks and their claims, which required that two related procedures (i.e., the “n + 1” rule and interpretation of peak integration values) were simultaneously leveraged.

One could persuasively argue that analysis, interpretation, and use of data from an infrared spectrum is the simplest example of argumentation from spectroscopic evidence encountered in organic chemistry. Indeed, one need only recognize the signature of a functional group and draw some sort of molecule incorporating that functional group to have a claim consistent with IR evidence. For example, one could interpret a strong infrared peak around 1700 cm−1 as signifying the presence of a carbonyl and then draw near infinite permutations of a carbonylcontaining compound. It is perhaps a function of the relatively simple nature of these tasks that most students were successful at analyzing, interpreting, and using data from infrared spectra on the formative and summative items we examined. As mentioned under RQ1, most students could also correctly state the number of chemically distinct carbons indicated by a carbon NMR, which required only that they count the peaks in the spectrum. However, construction of a structural prediction with an appropriate number of different carbon environments proved to be more challenging than assembling a prediction consistent with IR evidence. This may be due to the fact that an actionable understanding of “different carbon environments” requires students to examine their claim as a whole and be sensitive to any symmetry elements that might be present in the molecule they drew.

RQ 2: Association between Analysis and Interpretation of Spectroscopic Data and Use of that Evidence to Inform Claims

It is clear that most students can deploy algorithms to extract information from spectra. What is less clear is whether this evidence actually informs structural predictions. Here, we explore whether there exist associations between analysis and interpretation of data and use of this evidence in claim construction. Our analysis leverages student responses to the three high structure prompts previously discussed because these prompts allow us to ascertain whether appropriate information is being derived from the spectra. In particular, we focused on association between the information students obtain from IR and 13C NMR data and their predictions. The percentage of students who analyzed and interpreted data from infrared and carbon NMR spectra is presented alongside the percentage of students whose structural predictions were consistent with this data in Figure 6. As an example, a student who correctly stated that a 13C NMR indicated that there were six different kinds of carbon would be counted as “analyzing and interpreting” carbon NMR data appropriately, whereas a student who constructed a claim that had six carbon environments would be categorized as using the aforementioned evidence in their argument. For the purposes of this analysis, claims were not required to be consistent with all assembled data, just the data that are relevant to the claim. For example, if a student constructed a prediction with the proper number of carbon environments that was inconsistent with the given 1H NMR, they would still be counted as using evidence from the 13C NMR spectrum. The majority of students who responded to our three highly structured prompts were able to analyze, interpret and use data from infrared and 13C NMR spectra. This may indicate that analysis and interpretation are perceived as part of an ensemble of activities that should culminate in construction of an evidence-based claim. Because almost all students could pull information from IR and carbon NMR spectra, the association between data analysis and interpretation and structural prediction could not be analyzed via a χ2 test because 25% or more of the cells in the contingency tables have expected counts lower than 5, violating the assumptions of the test. That is, the number of students who were unable to pull appropriate information from infrared and carbon NMR spectra and also unable to construct claims incorporating this information was too few to run a χ2 test.

RQ3: Degree to Which Facility with Particular Procedures Predicts Student Success in Proposing a Structure Consistent with All Assembled Spectroscopic Evidence

To determine whether analysis, interpretation, or use of particular spectroscopic data sources predicted student success in proposing a structure consistent with all assembled evidence in the context of three highly structured prompts, we constructed binary logistic regression models described by the formula below: logit(Y ) = B0 + B1X1 + B2 X 2 + B3X3

The outcome variable Y represents student construction of a structural prediction consistent with all assembled evidence. Predictor variables X1, X2, and X3 represent analysis and interpretation of data from infrared and 13C NMR spectra (X1, and X2, respectively) as well as indication of correspondence between 1H NMR peaks and some part of a structural claim (X3). The outcome variable and all predictor variables are dichotomous. B0 conveys the likelihood students will construct evidence-based claims if no predictor variables improve model predictive validity. A model in which logit(Y) = B0 may be thought of as a “null model”. The regression coefficients B1, B2, and B3 describe the direction and magnitude of the relationship between the relevant predictor variables and the logit of Y.51 In all three models examined for this study, no predictor variable significantly improved the predictive validity of a model relative to the analogous null model. These three logistic regressions suggest that, for our data set, success on tasks that required analysis, interpretation, or use of spectroscopic data did not predict student success in assembling a structural prediction consistent with the full corpus of spectroscopic data. RQ 4: Association between Prompt Structure and Construction of Reasonable Claims

Thus far, we have seen that students can very often engage in the algorithms needed to pull particular information from spectroscopic data as well as use this information to inform their predictions. We have also seen that, for high structure prompts, facility with particular procedures does not predict I

DOI: 10.1021/acs.jchemed.9b00550 J. Chem. Educ. XXXX, XXX, XXX−XXX

Journal of Chemical Education

Article

Figure 7. Percentage of students who constructed claims consistent with spectroscopic evidence as part of experiments designed to show the impact of prompt structure on student success.

success in constructing a claim consistent with all assembled data. For our fourth research question, we sought to examine the impact of different prompt structures on student construction of reasonable claims. It is possible that assessment items that differ from our high structure variant might be more successful in cuing students into the processes of construction and critique essential to creating a reasonable argument. To get at the impact of varying prompt structure on student success in constructing evidence-based claims, we analyzed the results of three experiments. Two were designed to enable inferences into the impact of different prompt scaffolds on the percentage of students who successfully constructed evidence-based claims, and the third was meant to shed light onto whether student success might be improved by lowering the amount of data they were required to analyze. The results of these three experiments are summarized in Figure 7. The percentage of students who constructed claims consistent with the data set they were provided was approximately the same regardless of whether they were given a high structure or low structure prompt as part of experiment 1 (Figure 7). Therefore, there was no significant association between prompt structure and likelihood of constructing an evidence-based claim (p > 0.01). Our scaffolding strategy for high structure prompts was to explicitly ask for focus on various facets of the assembled data set with the hope that students would consider the evidence they pulled (and wrote) when constructing and critiquing their structural predictions. The results of this experiment would indicate that this approach was not successful in realizing our

goal. As this is a quantitative study, we cannot do more than speculate as to the cause for the results of this first experiment. The results of experiment 2 mirror those from experiment 1. That is, the percentage of students who successfully constructed an evidence-based claim in response to a “critique-centered” prompt did not differ significantly from what would be expected by chance. Accordingly, there was no significant association noted between prompt structure and percentage of students who proposed reasonable structures (p > 0.01). These results indicate that our attempt to explicitly require construction and critique of structural claims was not successful in the context of this experiment. With regard to the results of experiment 1 and experiment 2, it may be that each subpart of highly scaffolded questions was perceived as a separate, stand-alone entity to be answered and forgotten rather than considered in the context of claim construction and critique. It would be interesting to explore the impact of scaffolding from a students’ perspective through think-aloud interviews conducted in the context of different prompt types. The third experiment differed from the first two in that the aspect of prompt structure that changed between the experimental and control groups was not scaffolding but the amount of data to be analyzed. For this experiment, students were randomly placed into two groups with one group receiving three spectroscopic traces corresponding to Unknown A (1H NMR, 13C NMR, and IR) and two traces corresponding to Unknown B (13C NMR and IR) and the second group receiving two spectroscopic traces corresponding to Unknown A (13C NMR and IR) and three traces corresponding to Unknown B J

DOI: 10.1021/acs.jchemed.9b00550 J. Chem. Educ. XXXX, XXX, XXX−XXX

Journal of Chemical Education

Article

(1H NMR, 13C NMR, and IR). The three-trace variant of the prompts used for experiment 3 have been reproduced in the Supporting Information for this manuscript (Figures S1 and S2). In the context of spectroscopic traces derived from Unknown A, we can see that students given more data to analyze were more successful than those given less. Indeed, in this case, there is a significant association between the amount of data to be analyzed and successful construction of evidence-based claims: χ2(1) = 10.4, p = 0.001, Cramer’s V = 0.33. More students given three spectroscopic traces to analyze constructed successful claims than would be expected by chance. This runs counter to our hypothesis that student success should be improved by reducing the cognitive load of the problem via decreasing the data that must be considered during claim construction. However, this result was not replicated in the context of traces derived from Unknown B. That is, no significant association was found between the richness of the data set presented to students and their success in constructing evidence-based claims (p > 0.01). We cannot account for the higher percentage of students who successfully proposed the structure of Unknown A relative to those who constructed a reasonable claim as to the structure of Unknown B. Unknown A was 2-methylpropan-1-ol, whereas Unknown B was 3-methylbutan-2-one; both produce spectroscopic traces of similar complexity. The results of both parts of experiment 3 stand in opposition to our notion that argumentation from spectroscopic evidence might be overwhelming students’ processing capacity. We have no evidence that students are more successful in constructing evidence-based claims when presented with less data-rich contexts. Indeed, given that the 1H NMR spectra were the most information-rich data sources our students encountered, it may be that omitting a proton NMR spectrum from a prompt prevented students from using familiar procedural resources and forced reliance on less familiar strategies. This may have negatively impacted student ability to analyze, interpret, and use spectroscopic evidence to construct a reasonable claim as to the identity of Unknown A.

Deep versus Shallow Procedural Knowledge

We framed our initial discussion of analysis, interpretation, and use of spectroscopic data in terms of activation of a series of procedural resources. Recall that facility with such resources can be characterized as “deep procedural knowledge” (i.e., flexible, context-sensitive deployment of resources), or “shallow procedural knowledge” (i.e., inability to deploy resources reasonably and efficiently, despite knowing a procedure). Tasks that simply ask students to analyze and interpret spectroscopic data to provide straightforward information can be readily answered by individuals who possess shallow procedural knowledge. For example, one can recognize a broad peak around 3400 cm−1 in an infrared spectrum as the signature of a hydroxyl group without knowing anything about how to flexibly use the trove of information found within proton NMR spectra. Argumentation, that is, construction and critique of evidence-based claims, requires one to both pull evidence from all available data sources and iterate between claim construction and evaluation of whether a certain claim is consistent with assembled evidence. Facility with argumentation from spectroscopic evidence, we argue, is predicated on deep rather than shallow procedural knowledge. Given the relatively modest success of our student sample at constructing a claim consistent with spectroscopic data, one might think that the students surveyed lack deep procedural knowledge. Although this may be true for a subset of students, we would argue focusing on what students lack is not terribly productive; it would be more fruitful to focus on what these students brought to the table (e.g., an ability to analyze and interpret data from several sources) and how those resources might be woven together to better enable successful engagement in internal and external argumentative discourse. Importance of Framing

Every assessment prompt discussed here has the potential to engage students in an internal dialogic process of construction and critique that bears the structural hallmarks of an argument.8,52 That is, to address each item, individuals should pull evidence from spectroscopic data and iteratively construct and critique claims until they are satisfied that said claims are consistent with the full data corpus. Indeed, methodically double-checking the consistency of claims with evidence was done by all of the “more successful” solvers of spectroscopy problems in Cartrette and Bodner’s earlier work.15 However, the fact that a task asks for students to support claims with evidence does not at all guarantee that students will perceive said task as serving a persuasive function. Literature on student understandings of the type of activity they are doing, that is, their framing of that activity, suggests that students very often perceive tasks or scenarios meant to provoke argumentation as a classroom performance rather than a purposeful endeavor.53−55 Berland and Hammer characterize engaging in an argument to please the teacher (i.e., “doing the lesson”) rather than as part of figuring out a problem of interest to the classroom community as “pseudoargumentation”.54,55 As the activities we have created ask for students to figure out the identity of an unknown without any reason for why they might want to do so; it is possible that these activities were perceived as a rote performance rather than an opportunity to figure out something meaningful. Our tasks are very similar (though perhaps more scaffolded) than those which appear on many organic chemistry tests across the nation.2 Thus, if we ended up cuing many students into a frame at odds with the scientific practice of argumentation, many

■

CONCLUSIONS Analysis of the data presented here indicates that many students can successfully engage with the parts of homework and exam prompts that ask them to pull specific information from spectroscopic data. Virtually everyone whose responses were examined was quite adept at recognizing functionality in an infrared spectrum, discerning the number of chemically distinct carbons from a 13C NMR spectrum, and correctly labeling correspondence between proton environments in their structural prediction and peaks in an 1H NMR spectrum. It is possible that the significant practice OCLUE-enrolled students have had with spectroscopic analysis paid dividends in terms of ready and appropriate deployment of certain procedural resources on homework and exams. Additionally, the majority of students were able to construct structural predictions that included functionality indicated by the infrared spectrum and had an appropriate number of carbon environments. However, logistic regression models built from student responses to three multipart prompts found no significant association between successfully pulling data from a spectrum and overall model predictive validity. Further, different prompt structures had no impact on the success of students in constructing evidencebased claims. In summary, students struggled mightily with constructing a reasonable claim to their argument, despite being able to analyze and interpret data from several sources. K

DOI: 10.1021/acs.jchemed.9b00550 J. Chem. Educ. XXXX, XXX, XXX−XXX

Journal of Chemical Education

Article

We can only say what resources appeared to be used in the context of the prompts we have discussed. We cannot speculate as to whether students possess or know they possess the intellectual resources relevant to these problems; it is possible students would have been more successful on prompts worded differently or if they had more time to address summative items. Second, all students surveyed as part of this study enrolled in two semesters of a transformed undergraduate organic chemistry curriculum that placed substantial emphasis on argumentation from spectroscopic evidence. As such, they had ample practice cultivating the procedural resources needed to analyze and interpret spectroscopic data. Our results cannot generalize beyond this context; we do not know how students taught using a more traditional curriculum might have fared on the prompts discussed.

others may do likewise. Studies in the K−12 science education space indicate that students tend to draw on resources useful for argumentation to a greater degree when they see a need for argumentative discourse (e.g., they want to persuade their peers of the reasonableness of a claim to make sense of a phenomenon).55,56 Stated differently, students can very often reasonably link claims to evidence and attend to alternative arguments when they see a reason to do so, at least in the K−12 contexts studied.54 Thus, it is possible that students might be more successful at engaging in argument from spectroscopic evidence if they perceived a need to construct and critique evidence-based claims. Implications for Teaching

Argumentation from spectroscopic evidence, as with argumentation more generally, is extremely challenging. To be successful, students must pull information from various strands of data, itself a scientific practice requiring facility with a range of resources, and then iteratively construct and critique claims until they arrive at one consistent with the full data corpus. Given that so few students in our sample were able to arrive at reasonable structural predictions on exams and homework, it may not be reasonable to ask students to predict an unknown consistent with spectroscopic data as one part of a larger, timed summative exam. Ideally, we would want students (and future healthcare professionals) to carefully reflect on consistency between predictions and evidence, not quickly write down their “best guess” and then move on because of time constraints. This may mean that argumentation from spectroscopic evidence is better suited to laboratory environments than lecture. The broader focus of our analytical efforts also bears mention−that is, emphasis on student resources rather than constraints, deficiencies, or misconceptions. We see little value in focusing substantial effort on parsing out all of the struggles students have with engaging in a practice such as argumentation. It is very likely that students have a range of small-grain knowledge elements that can be called on in more or less productive ways to address an assessment item or problem posed during instruction.30,32,34,35,57−59 Characterizing resources strung together in less-than-optimal ways only serves to tell the community that certain students at a certain point in time had particular troubles with a particular task. A more fruitful course of action would be to examine what students know and can do across several related contexts to get a sense of the stability of their patterns of resource activation. From there, one can work toward designing learning environments that support productive weaving of intuitive resources with those developed by instruction to engage in the performances we value (such as argumentation). Here, we have found that consistent integration of spectroscopic analysis throughout OCLUE has equipped our students to analyze and interpret spectroscopic data from various sources; that is, our students can leverage several procedural resources that will be needed for engaging in argument from spectroscopic evidence. Our task going forward is to consider how we might help students flexibly deploy their ability to use data to construct and critique evidence-based claims.

■

■

ASSOCIATED CONTENT

S Supporting Information *

The Supporting Information is available on the ACS Publications website at DOI: 10.1021/acs.jchemed.9b00550.

■

Demographic and academic measures as well as example prompts (PDF, DOCX)

AUTHOR INFORMATION

Corresponding Author

*E-mail: [email protected]. ORCID

Ryan L. Stowe: 0000-0002-5548-495X Melanie M. Cooper: 0000-0002-7050-8649 Present Address †

R.L.S.: Department of Chemistry, University of Wisconsin Madison, 1101 University Avenue, Madison, Wisconsin 53706, United States Notes

The authors declare no competing financial interest.

■

ACKNOWLEDGMENTS This work was supported in part by the National Science Foundation under DUE:1502552. We would also like to thank Elizabeth Day for her assistance editing this manuscript and advice on statistical analyses.

■

REFERENCES

(1) Moran, B. How to Get an A− in Organic Chemistry. The New York Times, Nov 1, 2013, p ED14. (2) Stowe, R. L.; Cooper, M. M. Practice What We Preach: Assessing “Critical Thinking” in Organic Chemistry. J. Chem. Educ. 2017, 94 (12), 1852−1859. (3) The National Research Council. A Framework for K−12 Science Education: Practices, Crosscutting Concepts, and Core Ideas; National Academies Press: Washington, DC, 2012. (4) Osborne, J. F.; Patterson, A. Scientific Argument and Explanation: A Necessary Distinction? Sci. Educ. 2011, 95 (4), 627−638. (5) Maciejewski, W.; Star, J. R. Developing Flexible Procedural Knowledge in Undergraduate Calculus. Res. Math. Educ. 2016, 18 (3), 299−316. (6) Star, J. R. Foregrounding Procedural Knowledge. J. Res. Math. Educ. 2007, 38 (2), 132−135. (7) Star, J. R. Reconceptualizing Procedural Knowledge. J. Res. Math. Educ. 2005, 36 (5), 404−411.

LIMITATIONS

As with all studies, the work presented here has several limitations. First and most fundamentally, this was a quantitative study in which a large number of student responses to several multipart spectroscopic interpretation prompts were analyzed. L

DOI: 10.1021/acs.jchemed.9b00550 J. Chem. Educ. XXXX, XXX, XXX−XXX

Journal of Chemical Education

Article

(32) Hammer, D.; Elby, A. On the Form of a Personal Epistemology. In Personal Epistemology: The Psychology of Beliefs about Knowledge and Knowing; Hofer, B. K., Pintrich, P. R., Eds.; Lawrence Erlbaum Associates: Mahwah, NJ, 2002; pp 169−190. (33) Hammer, D.; Elby, A. Tapping Epistemological Resources for Learning Physics. J. Learn. Sci. 2003, 12 (1), 53−90. (34) Hammer, D.; Elby, A.; Scherr, R. E.; Redish, E. F. Resources, Framing, and Transfer. In Transfer of Learning from a Modern Multidisciplinary Perspective; Mestre, J. P., Ed.; IAP: Greenwich, CT, 2005; pp 89−119. (35) Cooper, M. M.; Corley, L. M.; Underwood, S. M. An Investigation of College Chemistry Students’ Understanding of Structure−Property Relationships. J. Res. Sci. Teach. 2013, 50 (6), 699−721. (36) Cooper, M.; Klymkowsky, M. Chemistry, Life, the Universe, and Everything: A New Approach to General Chemistry, and a Model for Curriculum Reform. J. Chem. Educ. 2013, 90 (9), 1116−1122. (37) Cooper, M. M.; Posey, L. A.; Underwood, S. M. Core Ideas and Topics: Building Up or Drilling Down? J. Chem. Educ. 2017, 94 (5), 541−548. (38) ACT/SAT Concordance Tables. ACT Website. https://www.act. or g/con tent/ da m/a ct/u nsecur ed/documen ts/ACT- SATConcordance-Tables.pdf (accessed June 10, 2019). (39) SPSS Statistics for Mac; IBM Corp.: Armonk, NY, 2017. (40) Bryfczynski, S.; Pargas, R. P.; Cooper, M. M.; Klymkowsky, M.; Dean, B. C. Teaching Data Structures with BeSocratic. In ITiCSE ’13, Proceedings of the 18th ACM Conference on Innovation and Technology in Computer Science Education, Canterbury, England, July 1−3, 2013; ACM: New York, NY, 2013; pp 105−110, DOI: 10.1145/2462476.2465583. (41) Bryfczynski, S.; Pargas, R. P.; Cooper, M. M.; Klymkowsky, M.; Hester, J.; Grove, N. P. Classroom Uses for BeSocratic. In The impact of pen and touch technology on education; Hammond, T., Valintine, S., Adler, A., Payton, M., Eds.; Springer International Publishing: New York, 2015; pp 127−136. (42) Cooper, M. M.; Underwood, S. M.; Bryfczynski, S. P.; Klymkowsky, M. W. A Short History of the Use of Technology To Model and Analyze Student Data for Teaching and Research. In Tools of Chemistry Education Research; ACS Symposium Series; American Chemical Society: Washington, DC, 2014; Vol. 1166, pp 219−239, DOI: 10.1021/bk-2014-1166.ch012. (43) Laverty, J. T.; Underwood, S. M.; Matz, R. L.; Posey, L. A.; Carmel, J. H.; Caballero, M. D.; Fata-Hartley, C. L.; Ebert-May, D.; Jardeleza, S. E.; Cooper, M. M. Characterizing College Science Assessments: The Three-Dimensional Learning Assessment Protocol. PLoS One 2016, 11 (9), No. e0162333. (44) Spectral Database for Organic Compounds SDBSWeb; National Institute of Advanced Industrial Science and Technology, 2018. https://sdbs.db.aist.go.jp (accessed May 22, 2019). (45) Cohen, J. Statistical Power Analysis for the Behavioral Sciences, 2nd ed.; Routledge: Hillsdale, NJ, 1988. (46) Cooper, M. M.; Kouyoumdjian, H.; Underwood, S. M. Investigating Students’ Reasoning about Acid−Base Reactions. J. Chem. Educ. 2016, 93 (10), 1703−1712. (47) Miller, G. A. The Magical Number Seven, plus or Minus Two: Some Limits on Our Capacity for Processing Information. Psychol. Rev. 1956, 63 (2), 81−97. (48) De Ribaupierre, A.; Fagot, D.; Lecerf, T. Working Memory Capacity and Its Role in Cognitive Development: Are Age Differences Driven by the Same Processes Across the Lifespan? In Cognitive Development and Working Memory: A Dialogue between Neo-Piagetian Theories and Cognitive Approaches; Barrouillet, P., Gaillard, V., Eds.; Psychology Press: Hove, East Sussex, 2011; pp 105−133, DOI: 10.4324/9780203845837. (49) Cowan, N.; Chen, Z.; Rouder, J. N. Constant Capacity in an Immediate Serial-Recall Task: A Logical Sequel to Miller (1956). Psychol. Sci. 2004, 15 (9), 634−640.

(8) Ford, M. J. A Dialogic Account of Sense-Making in Scientific Argumentation and Reasoning. Cognition Instruct. 2012, 30 (3), 207− 245. (9) Pulliam, C. R.; Pfeiffer, W. F.; Thomas, A. C. Introducing NMR to a General Chemistry Audience: A Structural-Based Instrumental Laboratory Relating Lewis Structures, Molecular Models, and 13C NMR Data. J. Chem. Educ. 2015, 92 (8), 1378−1380. (10) Vosegaard, T. ISpec: A Web-Based Activity for Spectroscopy Teaching. J. Chem. Educ. 2018, 95, 97. (11) Debska, B.; Guzowska-Swider, B. Molecular Structures from 1H NMR Spectra: Education Aided by Internet Programs. J. Chem. Educ. 2007, 84 (3), 556. (12) Flynn, A. B. NMR Interpretation: Getting from Spectrum to Structure. J. Chem. Educ. 2012, 89 (9), 1210−1212. (13) Winschel, G. A.; Everett, R. K.; Coppola, B. P.; Shultz, G. V.; Lonn, S. Using Jigsaw-Style Spectroscopy Problem-Solving To Elucidate Molecular Structure through Online Cooperative Learning. J. Chem. Educ. 2015, 92 (7), 1188−1193. (14) Angawi, R. F. Using a Problem Solving-Cooperative Learning Approach To Improve Students’ Skills for Interpreting 1H NMR Spectra of Unknown Compounds in an Organic Spectroscopy Course. J. Chem. Educ. 2014, 91 (6), 823−829. (15) Cartrette, D. P.; Bodner, G. M. Non-Mathematical Problem Solving in Organic Chemistry. J. Res. Sci. Teach. 2010, 47 (6), 643−660. (16) Topczewski, J. J.; Topczewski, A. M.; Tang, H.; Kendhammer, L. K.; Pienta, N. J. NMR Spectra through the Eyes of a Student: Eye Tracking Applied to NMR Items. J. Chem. Educ. 2017, 94, 29. (17) Domin, D.; Bodner, G. Using Students’ Representations Constructed during Problem Solving To Infer Conceptual Understanding. J. Chem. Educ. 2012, 89 (7), 837−843. (18) Cullipher, S.; Sevian, H. Atoms versus Bonds: How Students Look at Spectra. J. Chem. Educ. 2015, 92 (12), 1996−2005. (19) Connor, M. C.; Finkenstaedt-Quinn, S. A.; Shultz, G. V. Constraints on Organic Chemistry Students’ Reasoning during IR and 1H NMR Spectral Interpretation. Chem. Educ. Res. Pract. 2019, 20, 522. (20) Ford, M. Disciplinary Authority and Accountability in Scientific Practice and Learning. Sci. Educ. 2008, 92 (3), 404−423. (21) Osborne, J. Arguing to Learn in Science: The Role of Collaborative, Critical Discourse. Science 2010, 328 (5977), 463−466. (22) Cooper, M. M.; Stowe, R. L.; Crandell, O. M.; Klymkowsky, M. W. Organic Chemistry, Life, the Universe and Everything (OCLUE): A Transformed Organic Chemistry Curriculum. J. Chem. Educ. 2019, under revision, DOI: 10.1021/acs.jchemed.9b00401 (23) Klein, D. A Quarter Century of US ‘Math Wars’ and Political Partisanship. BSHM Bulletin: Journal of the British Society for the History of Mathematics 2007, 22 (1), 22−33. (24) Principles and Standards for School Mathematics; National Council of Teachers of Mathematics: Reston, VA, 2000. (25) Principles to Actions: Ensuring Mathematical Success for All; National Council of Teachers of Mathematics: Reston, VA, 2014. (26) Research Advisory Committee of the National Council of Teachers of Mathematics. NCTM Curriculum and Evaluation Standards for School Mathematics: Responses from the Research Community. J. Res. Math. Educ. 1988, 19 (4), 338−344. (27) Hiebert, J.; Lefevre, P. Conceptual and Procedural Knowledge in Mathematics: An Introductory Analysis. In Conceptual and procedural knowledge: The case of mathematics; Hiebert, J., Ed.; Lawrence Erlbaum Associates: Hillsdale, NJ, 1986; pp 1−27. (28) Star, J. R.; Newton, K. J. The Nature and Development of Experts’ Strategy Flexibility for Solving Equations. ZDM Mathematics Education 2009, 41, 557−567. (29) Hammer, D. Student Resources for Learning Introductory Physics. Am. J. Phys. 2000, 68, S52−S59. (30) diSessa, A. A. Knowledge in Pieces. In Constructivism in the Computer Age; Forman, G. E., Pufall, P. B., Eds.; Lawrence Erlbaum Associates: Hillsdale, NJ, 1988; pp 49−70. (31) Hammer, D. Epistemological Beliefs in Introductory Physics. Cognition Instruct. 1994, 12 (2), 151−183. M

DOI: 10.1021/acs.jchemed.9b00550 J. Chem. Educ. XXXX, XXX, XXX−XXX

Journal of Chemical Education

Article

(50) Mathy, F.; Feldman, J. What’s Magic about Magic Numbers? Chunking and Data Compression in Short-Term Memory. Cognition 2012, 122 (3), 346−362. (51) Peng, C.-Y. J.; Lee, K. L.; Ingersoll, G. M. An Introduction to Logistic Regression Analysis and Reporting. J. Educ. Res. 2002, 96 (1), 3−14. (52) Ford, M. Grasp of Practice’ as a Reasoning Resource for Inquiry and Nature of Science Understanding. Sci. & Educ. 2008, 17 (2−3), 147−177. (53) Enyedy, N.; Goldberg, J. Inquiry in Interaction: How Local Adaptations of Curricula Shape Classroom Communities. J. Res. Sci. Teach. 2004, 41 (9), 905−935. (54) Berland, L. K.; Hammer, D. Students’ Framings and Their Participation in Scientific Argumentation. In Perspectives on Scientific Argumentation: Theory, Practice and Research; Khine, M. S., Ed.; Springer Netherlands: Dordrecht, 2012; pp 73−93, DOI: 10.1007/ 978-94-007-2470-9_5. (55) Berland, L. K.; Hammer, D. Framing for Scientific Argumentation. J. Res. Sci. Teach. 2012, 49 (1), 68−94. (56) Cavagnetto, A. R. Argument to Foster Scientific Literacy: A Review of Argument Interventions in K−12 Science Contexts. Rev. Educ. Res. 2010, 80 (3), 336−371. (57) Cooper, M. M.; Stowe, R. L. Chemistry Education Research From Personal Empiricism to Evidence, Theory, and Informed Practice. Chem. Rev. 2018, 118 (12), 6053−6087. (58) diSessa, A. Alternative Conceptions and P-Prims. In Encyclopedia of Science Education; Gunstone, R., Ed.; Springer: Netherlands, 2015; pp 34−37, DOI: 10.1007/978-94-007-2150-0_87. (59) diSessa, A. A. A History of Conceptual Change Research: Threads and Fault Lines. In The Cambridge Handbook of the Learning Sciences; Cambridge University Press: New York, NY, 2014; pp 88− 108.

N

DOI: 10.1021/acs.jchemed.9b00550 J. Chem. Educ. XXXX, XXX, XXX−XXX