Characterization of First-Semester Organic Chemistry Peer-Led Team

Nov 6, 2018 - Implications for instructors are suggested, including encouraging students to verbally explain their rationale while drawing mechanisms ...
0 downloads 0 Views 2MB Size
Download PDF
Article Cite This: J. Chem. Educ. 2019, 96, 25−34

pubs.acs.org/jchemeduc

Characterization of First-Semester Organic Chemistry Peer-Led Team Learning and Cyber Peer-Led Team Learning Students’ Use and Explanation of Electron-Pushing Formalism Sarah Beth Wilson† and Pratibha Varma-Nelson*,‡ †

Department of Chemistry, University of Evansville, Evansville, Indiana 47722, United States Department of Chemistry & Chemical Biology, Indiana University−Purdue University Indianapolis (IUPUI), Indianapolis, Indiana 46202, United States

Downloaded via TULANE UNIV on January 8, 2019 at 11:18:24 (UTC). See https://pubs.acs.org/sharingguidelines for options on how to legitimately share published articles.

‡

ABSTRACT: The purpose of this parallel convergent mixed methods study was to characterize organic chemistry students’ expression of electron-pushing formalism skills who had participated in peer-led team learning (PLTL) and cyber peer-led team learning (cPLTL), a synchronous online version of peer-led team learning (PLTL) workshops. A new electron-pushing formalism analytic framework was developed from a review of the literature in addition to analysis of students’ interview artifacts, using a constant-comparison process. Utilization of this new electron-pushing formalism analytic framework for coding student interview artifacts revealed that cPLTL students were significantly less likely to successfully draw the product suggested by the curved arrows than their PLTL classmates. Implications for instructors are suggested, including encouraging students to verbally explain their rationale while drawing mechanisms as well as optimizing graphical collaborative learning activities for online learners. KEYWORDS: First-Year Undergraduate/General, Second-Year Undergraduate, Chemical Education Research, Organic Chemistry, Collaborative/Cooperative Learning, Internet/Web-Based Learning, Problem Solving/Decision Making, Distance Learning/Self Instruction, Constructivism, Mechanisms of Reactions FEATURE: Chemical Education Research

■

INTRODUCTION

The chief advantage of a mechanistic approach, to the vast array of disparate information that makes up organic chemistry, is the way in which a relatively small number of guiding principles can be used, not only to explain and interrelate existing facts, but to forecast the outcome of changing the conditions under which already-known reactions are carried out, and to foretell the products that may be expected from new ones. Thus, EPF could be an alternative to copious rote memorization for organic chemistry students because mechanisms give students “a logical means to predict products”.25Klein’s Organic Chemistry textbook, used by this institution, provided explanations of the rules and assumptions of this formalism to the students.26 Regrettably, a number of studies of novices’ understanding of EPF have revealed that the symbolism often has limited meaning for students.3,6,7,27−32 This finding aligns with the proposition that the hardest part of students’ learning science is the requirement that students practice multilevel thought, connect-

1

The roots of electron-pushing formalism (EPF) extend from a paper by Kermack and Robinson,2 who described the movement of electron density from areas of high electron density to areas of low electron density in the conjugated π system of butadiene. The curved arrows of EPF are “a symbolic device for keeping track of electron pairs in chemical reactions... as covalent bonds are formed and broken.”1,3−5 Practicing organic chemists consider EPF to be fundamental for communication and problem-solving to predict the products of reactions,6,7 including the regio- or stereochemistry of products.3 Due to the centrality of EPF to organic chemistry, a wealth of EPF instructional strategy literature exists.4,5,8−19 A reaction mechanism is the complete description of a reaction pathway, including any reactive intermediates20 and the curved arrows which represent the flow of electrons at each step of the reaction pathway. Since the publication of Morrison and Boyd’s first organic chemistry textbook in 1959, “reaction mechanisms have become a mainstay of organic chemistry courses.”3,21−23 As written by Sykes on page one of his Guidebook to Mechanism in Organic Chemistry:24 © 2018 American Chemical Society and Division of Chemical Education, Inc.

Received: May 23, 2018 Revised: October 22, 2018 Published: November 6, 2018 25

DOI: 10.1021/acs.jchemed.8b00387 J. Chem. Educ. 2019, 96, 25−34

Journal of Chemical Education

Article

collaboratively solve problems aligned with the content they are learning in their lecture courses under the guidance of trained undergraduate peer leaders.39,40,49,50 Second, Skemp51 suggested that there are two kinds of learning: instrumental learning and relational learning. Instrumental learning “consists of recognizing a task as one of a particular class for which one already knows a rule”,51 while relational learning consists of relating a task to a network of connected concepts.51 Sykes’ description of applying guiding principles to solve organic chemistry problems mechanistically corresponds to relational learning, while rote memorization or searching the textbook for rules52 corresponds to instrumental learning.

ing macroscopic and submicroscopic phenomena to symbolic representations,33,34 such as using EPF to demonstrate mechanistic reasoning to rationalize the progression of chemical reactions. Likewise, the identification of compounds’ roles in reactions is critical for ascertaining the areas of high and low electron density which lead to reactions occurring, yet one study reported that students only consider nucleophiles and electrophiles or Brønsted−Lowry acids and bases when prompted to do so.35 Correspondingly, Anzovino and Bretz27 reported that students were unable to recognize nucleophile/electrophile or acid/base pairs unless shown a mechanism or product, which suggested that the students in their study were not utilizing mechanistic reasoning.36 Grove, Cooper, and Rush7 reported that only 60% of the students in their study showed mechanisms even when instructed to do so and 15−20% added the arrows after predicting the product. Therefore, supplying curved arrows after predicting a product of a reaction was “decorating with arrows”7 or an “academic exercise”,1 rather than having EPF be a means for students to deduce products by utilizing mechanistic reasoning. Finally, Rushton et al.37 reported that students did not consider reaction mechanisms to be essential for the process of predicting products of reactions, although Flynn and Featherstone postulated that the ability to use and interpret electron-pushing formalism is a key step toward understanding chemical reactions at the molecular level38 and practicing mechanistic reasoning. In 2010, the institution established first-semester organic chemistry peer-led team learning (PLTL) workshops39−41 as a means to provide an environment for students to discuss, debate, and collaboratively solve organic chemistry problems because several studies have reported significant improvement in organic students’ grades when they participated in PLTL.42−45 ANCOVA analysis revealed that there was a significant increase in students’ American Chemical Society Exam Z scores since implementing PLTL at this institution (p < 0.05, F = 8.31, df = 1), although the effect size was small (partial η2 = 0.01). Cyber peer-led team learning (cPLTL),46 the online version of PLTL, was implemented in first-semester organic chemistry at the institution in 2014 in order to provide additional workshop session choices for students. However, no studies had yet been reported that compared online and face-to-face students’ use and explanation of course content that relies on visual representations for communication of understanding, such as EPF in organic chemistry. Therefore, characterizing and comparing PLTL and cPLTL students’ utilization of EPF to predict the products of organic chemistry reactions would be an important contribution to (1) expanding the EPF literature, (2) learning in other disciplines which have an emphasis on understanding visual representations, and (3) exemplifying ways to compare content mastery in online versus face-to-face learning settings other than solely comparing course grades, retention, or performance on standardized assessments.

Research Question

In this study, we investigated the following research question related to Organic Chemistry I: What conceptual difficulties do high- and low-performing students from online and face-to-face PLTL workshop settings exhibit as they draw and explain the curved arrows of reaction mechanisms?

■

METHODOLOGY This study employed a convergent parallel mixed methods design, wherein both qualitative and quantitative data were collected and analyzed in parallel (Figure 1).53 This research

Figure 1. Convergent parallel mixed methods data collection.

approach was selected in order to enable the researchers to gain a more “in-depth perspective” in order to characterize the PLTL or cPLTL students’ explanation and use of electron-pushing formalism to solve organic chemistry problems, than either a quantitative analysis of course grades and final exam performance or a qualitative analysis of students’ use of EPF without categorizing the interviewees would have afforded in isolation.53−55 This research study tested the null hypothesis that there would be no differences in the characterization of highand low-performing PLTL and cPLTL organic chemistry students’ mechanistic reasoning (EPF skills).

■

SETTING Each of the first-semester organic chemistry students at the large midwestern research-intensive university where the study was conducted was required to participate in a 1 h and 50 min PLTL workshop once per week, a duration which is aligned with the recommended PLTL model.41,56 Multiple lecture sections each fall were treated as a single course because they used identical lecture slides, PLTL problem sets, and final exams. The course textbook was Organic Chemistry by Klein,26 while the PLTL/ cPLTL workshop problem sets were jointly written each semester by the workshop coordinator and the lecturers. Twothirds of the workshop problem sets included questions that required students to draw resonance structures and/or mechanisms, showing EPF. The PLTL sections of the workshops consisted of approximately 30 students, subdivided into three groups of 8−10 students, while the cPLTL sections of the workshops consisted of approximately 8 students due to Adobe Connect bandwidth limitations. Trained undergraduate peer leaders facilitated comparison groups of PLTL and cPLTL

■

THEORETICAL FRAMEWORK This study is grounded in two theoretical frameworks: social constructivism and Skemp’s learning characterizations. Social constructivism asserts that people develop common language and understanding of artifacts (objects or symbols) through social interactions within a group.47 As Vygotsky proposed, cognitive development progresses first amidst social interaction and then within the individual.48 PLTL and cPLTL are both social constructivist, workshop-based, supplemental instruction learning opportunities in which small groups of students 26

DOI: 10.1021/acs.jchemed.8b00387 J. Chem. Educ. 2019, 96, 25−34

Journal of Chemical Education

Article

the following: 35% over 23 years old; 45% female; 40% underrepresented minorities; and mean previous chemistry GPA of 2.52. The 24 comparison group cPLTL students’ demographics included the following: 33% over 23 years old; 63% female; 33% underrepresented minorities; and mean previous chemistry GPA of 2.57. χ2 analyses for the students’ age [χ2(1, N = 64) = 0.840, p = 0.041], gender [χ2(1, N = 65) = 1.385, p = 0.239], ethnicity [χ2(5, N = 65) = 1.409, p = 0.923], workshop attendance [χ2(1, N = 64) = 9.420, p = 0.399], and previous chemistry GPA [χ2(20, N = 65) = 16.363, p = 0.694] indicated that the subject populations were comparable except with regard to age. The overall PLTL population’s demographics included the following: 29% over 23 years old; 58% female; 32% underrepresented minorities; and mean previous chemistry GPA of 2.89. A maximum diversity, purposeful sampling58 approach was utilized for the selection of interviewees, including participants of different gender, ethnicity, and prefinal exam course performance. This approach is designed to elicit the greatest variety of responses from a small to medium sample size. There were 19 students from the comparison group peer leaders’ sections interviewed, including generally two students from each setting per semester from each peer leader, resulting in 19 interviewees (6 spring; 13 fall) (Table 1). This sample size was consistent with Creswell’s suggested sample size for a grounded theory study, which the constant-comparison process supports.59 Each of the participants was interviewed during the week preceding the final exam in order for students to have the maximum familiarity with the course material.

students to collaboratively solve problems, using identical workshop questions. No answer keys were provided to either PLTL or cPLTL students, in alignment with the PLTL/cPLTL emphasis on problem-solving.49

■

PARTICIPANTS Participants for the study were recruited from the first-semester organic chemistry lecture course which occurred in spring and fall 2014 semesters. All of the first-semester organic chemistry students self-enrolled into PLTL or cPLTL sections that best fit their schedules since a variety of days of week and time of day were available. The subjects of the study included a subset of the population who self-selected to enroll in the “comparison group” workshop sections, which were led by peer leaders who facilitated both PLTL and cPLTL workshop sections during the same semester (1 peer leader led a pair of comparison groups in spring 2014, and 3 peer leaders led pairs of comparison groups in fall 2014). All peer leaders, including the comparison group peer leaders, received weekly training, including the content emphasized by each week’s problem set and applicable collaborative learning techniques.57 All peer leaders were paid a salary for participating in the weekly training meeting and facilitating two weekly workshop sessions. Participants’ demographic and chemistry course grades (Table 1) were provided by the University’s Institutional Table 1. Comparison of Group Student Characteristics Student Characteristics Previous chemistry GPA range (0−4.0) PLTL participants, N cPLTL participants, N Male participants, % Female participants, % Age range, years Peer leaders’ names Interviewees’ names

Spring 2014, N = 12

■

Fall 2014, N = 52

2.0−4.0

1.3−3.9

8

32

4

20

42

52

58

48

19−33 Naji

18−41 Anya, Brody, Isaac

Ashley, Eli, Erin, Isaac, Kayla, Veronica

Andrew, Blake, Christopher, Debbie, Holly, Jenae, Joyce, Keith, Kenneth, Katherine, Matthew, Susan, Thomas

DATA COLLECTION

Course Performance

End-of-course grades were used to categorize students’ course performance as high (A, B, or C) or low (D or F). Student’s course grades were composed of 85% performance on semester/ final exams and 15% performance on workshop preparedness quiz grades, and then converted into the standard four-point GPA scale for analysis. Students’ PLTL or cPLTL workshop attendance was tracked by the presence or absence of the weekly beginning-of-workshop multiple choice preparedness quiz scores (blank for absent; 0−5 for present). Interviews

Semistructured 60 min student interviews were conducted during the final week of each semester by the first author. These interviews began with a series of open-ended questions about their perceptions of their experiences in their setting. Then, students were asked to solve a series of four organic chemistry

Research and Decision Support (IRDS) office via a secure online file transfer process, as approved by the Institutional Review Board, in order to ensure the confidentiality of the subjects. The 40 comparison group PLTL students’ demographics included

Figure 2. Stages of qualitative data collection and analysis. 27

DOI: 10.1021/acs.jchemed.8b00387 J. Chem. Educ. 2019, 96, 25−34

Journal of Chemical Education

Article

Figure 3. Electron-pushing formalism interview probes.

probes they had not seen before, using a think-aloud protocol.60 Audio recordings and participants’ writing/drawings on printed copies of the interview probes were collected during the interview. A video camera was used to record students’ gestures and notations in order to match verbalizations with the students’ writing/drawings. Each interviewee provided informed consent prior to the interview and received a $20 Barnes & Noble gift card as compensation for completing the interview.

students to draw curved arrows to show the movement of electron density between atoms to explain the transition from one resonance structure of a molecule to the next. This probe supports the pedagogical strategy of having students practice resonance structures before asking them to explain the movement of electrons in reactions38 in order to imbue the EPF symbolism with molecular phenomena conceptualization. The first probe is also aligned with Bhattacharya’s finding that the ability to draw resonance structures is a key characteristic of mechanistic reasoning for EPF.3 The second probe, which required students to draw the curved arrows to rationalize the transition between intermediates for a peptide bond formation reaction, corresponds to a typical workshop problem which would seem particularly relevant for the large proportion of preprofessional students who enroll in first-semester organic chemistry. The third probe required students to identify a substitution reaction type from the substrate classification and reaction conditions, and then draw the mechanism of the reaction to predict the product, including configuration of the stereocenter. This type of question is also typical of both workshop problem sets and exam questions. Finally, the fourth probe was an exam-style question in which the alkyne attack of bromine to form a bromonium ion and subsequent nucleophilic attack of water were reminiscent of halohydrin formation from an alkene reaction, which students had practiced in class and in PLTL/cPLTL workshops. Likewise, students had drawn mechanisms for keto−enol tautomerization reactions. There-

Electron-Pushing Formalism Probes

During stage one of the qualitative data collection (Figure 2), four interview probes were developed to assess students’ utilization of EPF. The probes were discussed with the course instructors and other faculty. On the basis of their feedback, the probes were modified before administering them to firstsemester organic chemistry students. Using a think-aloud protocol60 with audio and video recording to capture what was being said as the student drew,61−63 spring 2014 subjects’ responses to the probes were analyzed for proficiency with electron-pushing formalism components. Analysis of these subjects’ responses revealed a range of responses to each probe, including both correct applications of EPF as well as each of EPF error modes reported in the literature. Therefore, the probes were unchanged for the remaining student interviews of the study. The probes (Figure 3) were designed to progress in difficulty as well as provide a diminishing level of scaffolding from one question to the next. For example, the first probe required 28

DOI: 10.1021/acs.jchemed.8b00387 J. Chem. Educ. 2019, 96, 25−34

Journal of Chemical Education

Article

Table 2. Electron-Pushing Formalism Analytic Framework Salient Features Electron-rich attacks electrondeficient

Nonspecific curved arrow Electron-deficient attacks electron-rich Electron-rich attacks electronrich Repetitive arrowsa Octet rule violation for carbon Lack of acid−base or nucleophile−electrophile operational knowledge Ignoring pH of medium Missing arrows Skipped mechanism Lewis structure challenges Out-of-sequence arrowsa Wrong arrowa Memorization Noting bond attachment differences instead of applying mechanistic reasoning Next structure was not the product from the arrows drawna

Corresponding Electron-Pushing Formalism (EPF) Indicators Correct EPF Elements Discourse or artifact that indicates that a curved arrow was drawn (1) from an electron-rich species (nucleophile or base) to an electron-deficient (electrophile or acid) species in a chemical reaction or (2) from a negative charge, lone pair, or π bond to an appropriate electron-deficient site in a resonance structure Incorrect/Neglected EPF Elements Discourse or artifact which indicates that a student depicted unorthodox curved arrowhead or tail placement, such as a curved arrowhead to the middle of a bond Discourse or artifact which indicates that an electron-deficient species attacks electron-rich species Discourse or artifact which indicates that an electron-rich species attacks electron-rich species More than one curved arrow to depict the movement of a single pair of electrons Discourse or artifact which indicates that a student writes a resonance structure, intermediate, or product with five or more bonds to carbon Discourse or artifact which indicates a student is unable to identify Brønsted-Lowry acid/base or nucleophilic/electrophilic participants in a reaction sequence Discourse or artifact which indicates that a student proposes a mechanism that generates hydroxide in acidic reaction conditions or hydronium ions in basic reaction conditions Artifact which indicates that a bond has been broken during a mechanism, but the related curved arrows were not drawn in the previous step Discourse or artifact which indicates that a student proposed the product of a reaction without providing the mechanism by which that product would be produced Artifact which does not portray the correct formal charge on an atom, based on the curved arrows shown in the previous step Discourse or artifact which indicates that the curved arrows were not drawn in an appropriate sequence Artifact which indicates that a student drew an inappropriate arrow type for the application implied, such as a reaction arrow between resonance structures or single-headed arrow to communicate heterolytic cleavage Discourse or artifact which indicates that a student memorized reaction conditions and product structure, rather than employing a mechanism to determine a products’ structure or stereochemistry Student notes bond attachment differences between reactants and products to determine how to draw curved arrows instead of relying on acid/base or nucleophile/electrophile identifications Artifact which indicates a structure which would not result from the curved arrows drawn on the reactant, intermediate, or resonance structure of the previous step

a

These error categories emerged from analysis of the interview transcripts and artifacts.

a cumulative list of electron-pushing formalism errors was noted and defined, which led to the development of an electronpushing formalism analytic framework (Table 2). The first and second coders utilized this framework to code the students’ responses to the interview problem set. For instances in which the first coder identified a passage that the second coder did not code initially, the first coder provided the second coder a spreadsheet of filename and line numbers of passages to consider. Then, the second coder responded via email with the identification of EPF analytic framework categories that pertained to the given passages. Additionally, a third coder, a fifth-year doctoral candidate with more than two decades of industry experience as a synthetic organic chemist, was asked to code an interview transcript/artifacts for electron-pushing formalism of a randomly selected participant to demonstrate that the coding scheme was understandable to a practitioner, not just researchers. Three processes were an integral part of this study to ensure its reliability and validity, including the following: calculation of inter-rater reliability, triangulation, and member-checking.54,55,68 Cohen’s κ was calculated for the coding of the EPF analytic framework to measure inter-rater reliability between two coders (Cohen’s κ = 0.87),69 and then Light’s κ was calculated to assess the inter-rater reliability among three coders by calculating the average of the pairwise κ values since SPSS version 23 is not capable of calculating an inter-rater reliability statistic for three coders (Light’s κ = 0.65).70 The process of triangulation entailed corroborating evidence from different

fore, the fourth probe required students to identify substrates’ roles as nucleophile, electrophile, acid, or base at each step to deduce the overall reaction mechanism of a novel reaction, based on their understanding of EPF principles from known reactions.

■

DATA ANALYSIS Interviews were transcribed verbatim, and data were managed using NVivo for PC (version 10). Pseudonyms were assigned to refer to students involved in this study. Interview transcripts/ artifacts were coded for the following EPF errors previously reported in the literature:1,3,5,6,38,64 • An electron-deficient species attacks an electron-rich species • An electron-rich species attacks an electron-rich species • Drawing arrows which would result in the violation of the octet rule for carbon • Arrows for multiple reaction steps are drawn at once • Ignoring the pH of the medium, such as proposing an acid-based mechanism in a basic solvent • Formal charge errors • Stereochemical discrepancies, such as a three-dimensional drawing of an sp2-hybridized carbon Once the individual student’s interviews were coded using a list of known student difficulties from the literature,1,5,38,64 the remaining student errors were analyzed using a constantcomparison process65−67 to identify emergent themes. Thereby, 29

DOI: 10.1021/acs.jchemed.8b00387 J. Chem. Educ. 2019, 96, 25−34

Journal of Chemical Education

Article

assessed and reflected by the course grades. Thus, cPLTL students demonstrated lower ability to use or interpret electronpushing formalism in their problem-solving process than their PLTL classmates.

individuals, types of data, and methods of data collection to demonstrate the credibility of each proposed theme. Finally, every participant confirmed the accuracy of his or her interview transcript.

■

PLTL Students’ Use of Electron-Pushing Formalism

RESULTS

The low- and average-performing PLTL students exhibited mapping behavior during the end-of-semester interviews. Mapping, which consists of pointing from reactant to product repeatedly to match atoms in order to decide where to draw curved arrows for an organic chemistry mechanism, is a puzzlesolving behavior6 rather than a problem-solving behavior when the roles of the reactants (nucleophile, electrophile, etc.) are not identified. Moreover, each of these students who repeatedly pointed from reactant to product displayed a lack of connection between molecular understanding of the reaction and the graphical representation of the curved arrow. For example, Holly drew fishhook arrows to denote heterolytic cleavage of bonds and proposed that the direction of curved arrows is irrelevant “as long as all the arrows flow in the same direction”. Likewise, Susan’s uses of nonspecific and repetitive arrows for the movement of a single pair of electrons each indicate poorly developed concept−symbol alignment. Similarly, Debbie’s drawing the curved arrows out-of-sequence indicated a lack of connection between the underlying molecular phenomena and symbolic representation. Katherine’s drawing a curved arrow from an electron-poor species to an electron-rich species and Erin’s drawing curved arrows from electron-rich to electron-rich species in their mechanisms also indicated a lack of concept− symbol alignment. In contrast, the high-performing PLTL students exhibited EPF behaviors that suggested an alignment between molecular understanding of chemical reactions and the EPF representation, rather than utilization of pattern recognition. Namely, Matthew, Keith, Eli, and Veronica each drew mechanisms before predicting products, with curved arrows drawn in the same sequence that the reaction would progress. Keith and Eli revealed in their interviews that they specifically practiced explaining their reasoning while solving problems as well as encouraging classmates to do the same. Keith’s and Eli’s behavior of relating a task to a network of connected concepts is aligned with Skemp’s description of relational learning.51 However, Matthew reported that he commonly practiced drawing curved arrows of mechanisms repeatedly, but did not practice identifying the roles of reactants (i.e., nucleophile/ electrophile or acid/base): Interviewer: Do you practice mechanisms a lot? Matthew: I do. I have lots of fun with them. Interviewer: How do you practice them? Matthew: I just like drawing them and once I draw it out I like to go back and pick a random point and I’ll draw that structure and see where I can go from there. Matthew progressed through the next two interview problems with confidence and speed, and then halted when he encountered the fourth interview problem, a problem constructed to require mechanistic reasoning instead of recollection of problems presented in the course. At that point, he drew several curved arrows that moved toward, rather than from, high-electron-density areas. Matthew drew the dibromoalkene which would result from a bromide attacking a bromonium ion intermediate. Then, he did not persist in solving the problem, although simply changing the nucleophile from bromide to water would have poised the intermediate for a

Course Performance

χ-squared analysis indicated there was no statistically significant difference in student attendance in workshops. As reported in a separate publication, Mann−Whitney U-tests indicated that there were no significant differences in the distribution of course grades or performance of PLTL and cPLTL students on the ACS First-Semester Organic Chemistry Exam, but cPLTL students exhibited a higher proportion of DFW grades and lower proportion of C grades than the PLTL students.71 Therefore, the comparison of PLTL and cPLTL students’ use and explanations of EPF to solve problems, the essential element of organic chemistry reasoning, is critical for understanding the discrepancy in course performance. Coding with Electron-Pushing Formalism Analytic Framework

The inter-rater reliability of the first two coders, Cohen’s κ, was calculated to be 0.81, which corresponds to “Almost perfect” agreement.72 The inter-rater reliability calculation for all three raters was in the “Almost perfect” agreement range (Light’s Kappa = 0.91).72 A Mann−Whitney U-test indicated a significantly higher frequency of incorrect curved arrows drawn by cPLTL students for interview probe 4 (Figure 4).

Figure 4. Mean frequency of curved arrow usage from students in each setting (PLTL N = 9; cPLTL N = 10). The error bars indicate the standard deviation.

Furthermore, a Mann−Whitney U-test of the frequencies of the specific error categories of incorrect curved arrows by setting for the interview responses indicated that cPLTL students were significantly more likely to draw a product that was inconsistent with the curved arrows drawn (Table 3). Moreover, there was a significant correlation between the subjects’ overall course grade and percent correct curved arrows on both interview question one (Pearson correlation = 0.54, p < 0.05) and interview question four (Pearson correlation = 0.76, p < 0.05), which suggests that the ability to interpret the meaning communicated by curved arrows is a key component of the course being 30

DOI: 10.1021/acs.jchemed.8b00387 J. Chem. Educ. 2019, 96, 25−34

Journal of Chemical Education

Article

Table 3. Frequency of Occurrences of Interview Students’ Curved Arrows Error Categories by Setting

Interviewer: Why not? ... Jenae: Maybe I like just learning and memorizing and spitting it out. But this one [course] you had to do sort of like go beyond and analyze it. Each of the low-performing cPLTL students exhibited a poor connection between the symbolism of curved arrows and the underlying chemical principles. For example, Thomas, Joyce, Christopher, and Jenae drew products of reactions that were inconsistent with the curved arrows drawn. Ashley either drew repetitive arrows to indicate the movement of a single pair of electrons or nonspecific arrows. Thomas drew curved lines without arrowheads that would have indicated the directionality of electron movement, while Christopher drew half of his curved arrows from areas of low electron density to areas of high electron density. Finally, Jenae proposed the generation of hydroxide ion an acidic reaction milieu. The high-performing cPLTL students’ interviews revealed a range of instrumental to relational learning approaches. Kenneth depended heavily on a “make a bond, break a bond” heuristic, a guideline that several students from the course repeated during the field observations of the workshops whereby they ensured that carbon was never participating in more than four bonds at a time. This heuristic was said as a fundamental aspect of the reasoning component of Kenneth’s problem-solving process in lieu of identifying acid, base, nucleophile, or electrophile roles of

keto−enol tautomerization. This phenomenon gave the impression that Matthew’s emphasis on drawing and redrawing mechanisms without explaining the rationale of the reactions enabled him to draw familiar mechanisms quickly, but he did not use mechanistic reasoning as his problem-solving process. Perhaps Matthew’s semester exam grades seemed mismatched to both his final exam performance and his inability to solve interview probe four, therefore, because he was a well-practiced, instrumental learner rather than a relational learner. cPLTL Students’ Use of Electron-Pushing Formalism

In contrast to the behavior of their PLTL counterparts, a small minority of the cPLTL students exhibited mapping behavior: two high-performing students given the pseudonyms Kayla and Andrew exhibited mapping behavior with mechanistic analysis. Whenever low-performing cPLTL students generated an answer without drawing curved arrows that aligned with molecular understanding of the chemical reaction, they tended to cite dependence on text-based instrumental learning techniques instead of graphical-based techniques like mapping. For example, the low-performing student Ashley said that she tried to memorize reaction conditions instead of drawing mechanisms to predict products of reactions. Similarly, another lowperforming student Jenae expressed her preference for memorizing and reporting: Jenae: I liked 105 and 106 [the two-semester sequence for general chemistry], but organic chemistry I cannot get. 31

DOI: 10.1021/acs.jchemed.8b00387 J. Chem. Educ. 2019, 96, 25−34

Journal of Chemical Education

Article

the reactants. Kayla exhibited both mapping1 and “decorating with arrows”7 behaviors. Andrew displayed mapping, drew arrows out-of-sequence, and predicted products without drawing mechanisms. In contrast, Blake’s consistent role as explainer in his cPLTL group exemplified both how drawing is an integral part of Vygotskian social constructivism, where participants coconstruct the meaning of representations as they draw and discuss together,73 and his focus on relational learning.51 Isaac’s habitual written and verbal summarizations of concepts revealed his relational learning emphasis.51 Furthermore, the explanations and curved arrows provided by Blake and Isaac demonstrated “mechanistic reasoning”: a firm connection between molecular understanding the physical phenomena and the symbolism of the curved arrows of electron-pushing formalism. As Isaac expressed about his problem-solving process: “I kind of just start to doodle it out and then go back and usually see if those are plausible things that could happen.” Nevertheless, the notable adeptness of Blake and Isaac to both interpret and reason with the graphical representation of curved arrows to solve organic chemistry problems was not representative of the cPLTL student population in general. Rather, the majority of the cPLTL students seemed to struggle to draw and/or interpret curved arrows appropriately.

performing students exhibited mapping. These post-productprediction curved arrow drawing and mapping behaviors were more frequent among the high-performing cPLTL students, many of whom struggled with the fourth interview probe, which required solving problems by utilizing mechanistic reasoning rather than reproducing a known reaction or deducing curved arrows from identifying the differences between reactants and products. Furthermore, this finding suggests a limitation in the development of even some high-performing students’ connection between the EPF symbolism and the corresponding molecular phenomena. The first author has seen as the workshop coordinator of the organic chemistry PLTL workshop series that students are most engaged when encouraged to take turns drawing curved arrows on a small, portable whiteboard to supplement their discussion of mechanistic problems. Peer leaders in the PLTL workshops encouraged students to take turns drawing curved arrows for resonance structures or reactions on portable white boards as they discussed problems, but these same peer leaders did not have a comparable collaborative white board application in Adobe Connect at the time of the study. Instead, cPLTL students merely looked at small views of one another’s worksheets as provided by document camera images.

■

■

CONCLUSIONS The interview comments from PLTL and cPLTL students who struggled to solve the final interview probe exhibited verbalism and instrumental learning behaviors, while students who reported that they frequently explained the rationale of their mechanistic problem-solving process as part of their learning and practice strategies exhibited greater alignment between their explanations of reaction mechanisms and the curved arrows they drew. Additionally, students taking turns drawing and explaining curved arrows is not only a key component of practicing organic chemists’ communication among colleagues, but also a vital social constructivist activity among students, wherein students coconstruct the meaning of representations as they draw and discuss together.73 Since the comparison group populations were comparable, workshop attendance was comparable, the students attended the same instructor lectures, and the learning materials were the same, we propose that the lack of the dynamic, interactive, interdependent drawing experience may have contributed to cPLTL students’ having less practice collaboratively deducing reaction mechanisms and a greater likelihood of simply writing down what a classmate showed as a reaction mechanism. Consequently, the cPLTL students were less likely to demonstrate an ability to draw, interpret, and reason with EPF when encountering a novel problem like interview probe 4 (51% vs 20% incorrect curved arrows drawn for probe 4).

DISCUSSION The purpose of this study was to characterize PLTL and cPLTL students’ use and explanations of curved arrows to communicate their molecular understanding of the chemical reactions. Although the course grade distributions were comparable for students from the two settings,74 cPLTL students’ interview responses were more likely to exhibit incorrect curved arrows. In particular, cPLTL students were more likely to draw a product that was not suggested by the curved arrows drawn. Additionally, there was a significant correlation between the students’ course grade and percent correct arrows on the fourth interview probe, which was designed to assess students’ use of EPF to communicate mechanistic reasoning. Low-performing students from both settings revealed evidence of instrumental learning in both workshop discourse and interviews. For example, Susan said that she studied reactions with flashcards in lieu of drawing reaction mechanisms. Similarly, several of these students referred to a memorized table of SN1 and SN2 substitution reaction criteria instead of evaluating the reaction conditions holistically, which indicated instrumental learning. Every student interviewed provided a nearly verbatim definition of the curved arrow representation, but the low-performing students were progressively unable to correctly draw and interpret curved arrows from the first to the last interview probe. This phenomenon suggests that the lowperforming students exhibited a behavior that Vygotsky called verbalism, in which students repeat phrases in a parrot-like manner to hide a lack of understanding of the concept represented by a symbol.75 Clearly, these students had memorized the definition that a “curved arrow represents the movement of electrons”, but the symbol of the curved arrow did not actually hold meaning for the low-performing students that would enable them to solve problems. There were examples of high-performing learners from both settings who exhibited instrumental or relational learning.51 Several high-performing students drew correct mechanisms for the first three interview probes, yet drew the arrows only after predicting the product. Likewise, approximately half of the high-

■

LIMITATIONS First, we compared occurrences of incorrect curved arrows to course grades, but course grades are not objective measures of student learning. Therefore, we emphasized the analysis of the think-aloud problem-solving interviews rather than merely considering course grades. Second, although χ2 analysis revealed a significant difference in the ages of PLTL and cPLTL students, we recognize the appeal of “PLTL in pajamas”,76 wherein students could participate in workshops from anywhere with a reliable Internet connection rather than travel to campus. Finally, while the sample size of interviewees was relatively small, 32

DOI: 10.1021/acs.jchemed.8b00387 J. Chem. Educ. 2019, 96, 25−34

Journal of Chemical Education

Article

(7) Grove, N. P.; Cooper, M. M.; Rush, K. M. Decorating with arrows: Toward the development of representational competence in organic chemistry. J. Chem. Educ. 2012, 89, 844−849. (8) Wentland, S. A new approach to teaching organic chemical mechanisms. J. Chem. Educ. 1994, 71 (1), 3−8. (9) Simpson, P. Organic reaction mechanisms in the sixth form: Part 1. Sch. Sci. Rev. 1988, 70, 77−82. (10) Simpson, P. Organic reaction mechanisms in the sixth form: Part 2. Sch. Sci. Rev. 1989, 70, 31−38. (11) Simpson, P. Organic reaction mechanisms in the sixth form: Part 3. Sch. Sci. Rev. 1989, 70 (63−69), 63−69. (12) Brisbois, R. G. Davidsoniana Jones and the cult of the curved arrow. J. Chem. Educ. 1992, 69 (12), 971−973. (13) Buncel, E.; Wilson, H. The Reactivity Selectivity Principle: Should it ever be used? J. Chem. Educ. 1987, 64 (6), 475−480. (14) Caserio, M. C. Reaction mechanisms in organic chemistry: Concerted reactions. J. Chem. Educ. 1971, 48 (12), 782−790. (15) Friesen, J. B. Saying what you mean: Teaching mechanisms in organic chemistry. J. Chem. Educ. 2008, 85 (11), 1515. (16) Carson, S.; Miller, E. S. Introducing primary scientific literature to first-year undergraduate researchers. Counc. Undergrad. Res. 2013, 34 (4), 17−22. (17) Miller, A.; Solomon, P. H. Writing Reaction Mechanisms in Organic Chemistry, 2nd ed.; Academic Press: San Diego, CA, 2000. (18) Ryles, A. P. Teaching A-level organic chemistry mechanisms: Some suggestions. Sch. Sci. Rev. 1990, 72, 71−74. (19) Trost, B. M. The atom economy–a search for synthetic efficiency. Science 1991, 254 (5037), 1471−1477. (20) Loudon, M.; Parise, J. Organic Chemistry, 6th ed.; Roberts and Company Publishers, Inc.: Greenwood Village, CO, 2016. (21) Goldish, D. M. Let’s talk about the organic chemisty course. J. Chem. Educ. 1988, 65 (7), 603−604. (22) Morrison, R. T.; Boyd, R. N. Organic Chemistry; Allyn and Bacon: Boston, MA, 1959. (23) Wheeler, D. M. S.; Wheeler, M. M. Trends in the teaching of organic chemistry: A survey of some textbooks. J. Chem. Educ. 1982, 59 (10), 863−865. (24) Sykes, P. A Guidebook to Mechanism in Organic Chemistry, 6th ed.; John Wiley & Sons: New York, NY, 1986. (25) Ruder, S. M.; Straumanis, A. R. A method for writing open-ended curved arrow notation questions for multiple-choice exams and electronic-response systems. J. Chem. Educ. 2009, 86 (12), 1392−1396. (26) Klein, D. Organic Chemistry; John Wiley & Sons: Hoboken, NJ, 2012. (27) Anzovino, M. E.; Bretz, S. L. Organic chemistry students’ ideas about nucleophiles and electrophiles: The role of charges and mechanisms. Chem. Educ. Res. Pract. 2015, 16, 797−810. (28) Cartrette, D. P.; Dobberpuhl, M. R. The concept inventory as a diagnostic of understanding and achievement in a year-long organic chemistry course. Chem. Educ. 2009, 14 (4), 1−10. (29) Cooper, M. M.; Grove, N. P.; Pargas, R.; Bryfczynski, S. P.; Gatlin, T. OrganicPad : an interactive freehand drawing application for drawing Lewis structures and the development of skills in organic chemistry. Chem. Educ. Res. Pract. 2009, 10, 296−301. (30) Bodner, G. M.; Domin, D. S. Mental models: The role of representations in problem solving in chemistry. Univ. Chem. Educ. 2000, 4 (1), 24−30. (31) Domin, D.; Bodner, G. M. Using students’ representations constructed during problem solving to infer conceptual understanding. J. Chem. Educ. 2012, 89 (7), 837−843. (32) Kozma, R. The material features of multiple representations and their cognitive and social affordances for science understanding. Learn. Instr. 2003, 13 (2), 205−226. (33) Johnstone, A. H. Why is science difficult to learn? Things are seldom what they seem. J. Comput. Assist. Learn. 1991, 7, 75−83. (34) Gilbert, J. K.; Treagust, D. F. In Multiple Representations in Chemical Education; Gilbert, J. K., Treagust, D. F., Eds.; Springer: Dordrecht, Netherlands, 2009; pp 1−9.

the maximum diversity sampling approach enabled us to uncover critical differences in learning approaches and the meaningfulness of the EPF symbolism among the high-, average-, and low-performing PLTL and cPLTL participants that will inform instructors for their development of curricula, learning activities, and assessments.

■

IMPLICATIONS FOR RESEARCH AND PRACTICE Students who professed that they practiced explaining their mechanistic reasoning as they drew curved arrows demonstrated relational thinking and more adeptness at solving a novel EPF probe at the end of the semester. Additionally, provision of classroom opportunities for students to attempt mechanistic problems, fail, and try again is an important part of learning to participate in organic chemsitry.77 Therefore, we encourage instructors and peer leaders to emphasize that students explaining concepts to one another has a significant impact on their own learning78,79 and include time for students to practice verbalizing their mechanistic rationale during class in order to increase the likelihood of students to exhibit relational learning.51 Furthermore, we recommend that a collaborative whiteboard application be incorporated into future cPLTL implementations with a graphical component, such as organic chemistry, in order to equalize the learning environments between PLTL and cPLTL.

■

AUTHOR INFORMATION

Corresponding Author

*E-mail: [email protected]. ORCID

Sarah Beth Wilson: 0000-0002-0149-7480 Notes

The authors declare no competing financial interest.

■

ACKNOWLEDGMENTS We wish to thank Jordan Cagle for transcribing audio recordings and coding transcripts. We especially thank Anthony Chase for his critical review of the manuscript and suggestions. We also thank Grant Fore and Annwesa Dasgupta, for their helpful comments, and the reviewers for their constructive feedback. We gratefully acknowledge the support of the project’s sponsors: Department of Chemistry & Chemical Biology and IUPUI.

■

REFERENCES

(1) Ferguson, R.; Bodner, G. M. Making sense of the arrow-pushing formalism among chemistry majors enrolled in organic chemistry. Chem. Educ. Res. Pract. 2008, 9 (2), 102−113. (2) Kermack, W. O.; Robinson, R. An explanation of the property of induced polarity of atoms and an interpretation of the theory of partial valencies on an electronic basis. J. Chem. Soc., Trans. 1922, 121, 427− 440. (3) Bhattacharyya, G. From source to sink: mechanistic Reasoning using the electron-pushing formalism. J. Chem. Educ. 2013, 90 (10), 1282−1289. (4) Grossman, R. B. The Art of Writing Reasonable Organic Reaction Mechanisms, 2nd ed.; Springer: New York, 2003. (5) Scudder, P. H. Electron Flow in Organic Chemistry; John Wiley & Sons: New York, 1992. (6) Bhattacharyya, G.; Bodner, G. M. It gets me to the product ”: How students propose organic mechanisms. J. Chem. Educ. 2005, 82 (9), 1402−1407. 33

DOI: 10.1021/acs.jchemed.8b00387 J. Chem. Educ. 2019, 96, 25−34

Journal of Chemical Education

Article

(58) Patton, M. Q. Qualitative Evaluation and Research Methods, 3rd ed.; Sage: Newbury Park, CA, 2002. (59) Creswell, J. W. Educational Research: Planning, Conducting, and Evaluating Quantitative and Qualitative Research; Prentice Hall: Upper Saddle River, NJ, 2002. (60) Ericsson, K. A.; Simon, H. A. Protocol Analysis: Verbal Reports as Data; MIT Press: Cambridge, MA, 1984. (61) Cooper, M. M.; Corley, L. M.; Underwood, S. M. An Investigation of College Chemistry Students ’ Understanding of Structure − Property Relationships. J. Res. Sci. Teach. 2013, 50 (6), 699−721. (62) Harle, M.; Towns, M. H. Students’ understanding of external representations of the potassium ion channel protein part II: structurefunction relationships and fragmented knowledge. Biochem. Mol. Biol. Educ. 2012, 40 (6), 357−363. (63) Linenberger, K. J.; Bretz, S. L. A Novel Technology to Investigate Students’ Understandings of Enzyme Representations. J. Coll. Sci. Teach. 2012, 42 (1), 45−49. (64) Grove, N. P.; Cooper, M. M.; Cox, E. L. Does Mechanistic Thinking Improve Student Success in Organic Chemistry? J. Chem. Educ. 2012, 89 (7), 850−853. (65) Attride-Stirling, J. Thematic networks: an analytic tool for qualitative research. Qual. Res. 2001, 1 (3), 385−405. (66) Braun, V.; Clarke, V. Using thematic analysis in psychology. Qual. Res. Psychol. 2006, 3, 77−101. (67) Strauss, A. L.; Corbin, J. Basics of Qualitative Research: Grounded Theory, Procedures, and Techniques; Sage: Newbury Park, CA, 1990. (68) Mertens, D. M. Research and Evaluation in Education and Psychology: Integrating Diversity with Quantitative, Qualitative, and Mixed Methods; Sage: Thousand Oaks, CA, 2010. (69) Finn, B. K.; Campisi, J. Implementing and evaluating a peer-led team learning approach in undergraduate anatomy and physiology. J. Coll. Sci. Teach. 2015, 44 (6), 38−44. (70) Hallgren, K. A. Computing inter-rater reliability for observational data: An overview and tutorial. Tutor. Quant. Methods Psychol. 2012, 8 (1), 23−34. (71) Wilson, S. B.; Varma-Nelson, P. Social constructivism in cyberspace. 2018, under revision. (72) Landis, J.; Koch, G. The measurement of observer agreement for categorical data. Biometrics 1977, 33 (1), 159−174. (73) Brooks, M. Drawing, visualization, and young children’s exploraton of “big ideas. Int. J. Sci. Educ. 2009, 31 (3), 319−341. (74) Wilson, S. B. A Comparison of First-Semester Organic Chemistry Students’ Experiences and Mastery of Curved-Arrow Formalism in Face-toFace and Cyber Peer-Led Team Learning; Purdue University, 2015. (75) Vygotsky, L. S. Thought and Language; The MIT Press: Cambridge, MA, 1986. (76) Alberte, J. L.; Cruz, A.; Rodriguez, N.; Pitzer, T. In Proceedings of the Inaugural Conference; Peer-Led Team Learning International Society: Brooklyn, NY, 2013; pp 1−5. (77) Webber, D. M.; Flynn, A. B. How are students solving familiar and unfamiliar Organic Chemistry mechanism questions in a new curriculum? J. Chem. Educ. 2018, 95, 1451−1467. (78) Coleman, E. B. Using explanatory knowledge during collaborative problem solving in science. J. Learn. Sci. 1998, 7 (3-4), 387−427. (79) Palinscar, A.; Brown, A. Reciprocal teaching of comprehensionfostering and comprehension-monitoring activities. Cogn. Instr. 1984, 1 (2), 117−175.

(35) Cartrette, D. P.; Mayo, P. M. Students’ understanding of acids/ bases in organic chemistry contexts. Chem. Educ. Res. Pract. 2011, 12 (1), 29−39. (36) Strickland, A. M.; Kraft, A.; Bhattacharyya, G. What happens when representations fail to represent? Graduate students’ mental models of organic chemistry diagrams. Chem. Educ. Res. Pract. 2010, 11 (4), 293−301. (37) Rushton, G. T.; Hardy, R. C.; Gwaltney, K. P.; Lewis, S. E. Alternative conceptions of organic chemistry topics among fourth year chemistry students. Chem. Educ. Res. Pract. 2008, 9 (2), 122−130. (38) Flynn, A. B.; Featherstone, R. B. Language of mechanisms: exam analysis reveals students’ strengths, strategies, and errors when using the electron-pushing formalism (curved arrows) in new reactions. Chem. Educ. Res. Pract. 2017, 18 (1), 64−77. (39) Gosser, D.; Roth, V.; Gafney, L.; Kampmeier, J.; Strozak, V.; Varma-Nelson, P.; Radel, S.; Weiner, M. Workshop Chemistry: Overcoming the barriers to student success. Chem. Educ. 1996, 1 (1), 1−17. (40) Gosser, D. K., Jr.; Kampmeier, J. A.; Varma-Nelson, P. Peer-Led Team Learning: 2008 James Flack Norris Award Address. J. Chem. Educ. 2010, 87 (4), 374−380. (41) Wilson, S. B.; Varma-Nelson, P. Small Groups, Significant Impact: A Review of Peer-Led Team Learning Research with Implications for STEM Education Researchers and Faculty. J. Chem. Educ. 2016, 93 (10), 1686−1702. (42) Rein, K. S.; Brookes, D. T. Student response to a partial inversion of an organic chemistry course for non-chemistry majors. J. Chem. Educ. 2015, 92 (5), 797−802. (43) Tien, L. T.; Roth, V.; Kampmeier, J. A. Implementation of a peerled team learning instructional approach in an undergraduate organic chemistry course. J. Res. Sci. Teach. 2002, 39 (7), 606−632. (44) Wamser, C. C. Peer-led team learning in organic chemistry: Effects on student performance, success, and persistence in the course. J. Chem. Educ. 2006, 83 (10), 1562−1566. (45) Lyle, K. S.; Robinson, W. R. A statistical evaluation: Peer-led team learning in an organic chemistry course. J. Chem. Educ. 2003, 80 (2), 132. (46) Mauser, K.; Sours, J.; Banks, J.; Newbrough, J. R.; Janke, T.; Shuck, L.; Zhu, L. Cyber Peer-Led Team Learning (cPLTL): Development and implementation. Educ. Rev. Online 2011, 1−17. (47) Berger, P. L.; Luckmann, T. The Social Construction of Reality; Anchor: New York, 1967. (48) Vygotsky, L. S. Minds in Society: The Development of Higher Psychological Processes; Harvard University Press: Cambridge, MA, 1978. (49) Gosser, D. K.; Cracolice, M. S.; Kampmeier, J. A.; Roth, V.; Strozak, V.; Varma-Nelson, P. Peer-Led Team Learning: A Guidebook; Prentice Hall: Upper Saddle River, NJ, 2001. (50) Woodward, A. E.; Weiner, M.; Gosser, D. Problem solving workshops in general chemistry. J. Chem. Educ. 1993, 70 (8), 651. (51) Skemp, R. R. Intelligence, Learning, and Action: A Foundation for Theory and Practice in Education; John Wiley & Sons: New York, 1979. (52) Anderson, T. L.; Bodner, G. M. What can we do about ‘Parker’? A case study of a good student who didn’t ‘get’ organic chemistry. Chem. Educ. Res. Pract. 2008, 9, 93−101. (53) Creswell, J. W. Research Design: Qualitative, Quantitative, and Mixed Methods Approaches, 4th ed.; Sage: Los Angeles, CA, 2014. (54) Creswell, J. W. Educational Research: Planning, Conducting, and Evaluating Quantitative and Qualitative Research; Pearson Education, Inc.: Upper Saddle River, NJ, 2012. (55) Jick, T. D. Mixing Qualitative and Quantitative Methods: Triangulation in Action. Adm. Sci. Q. 1979, 24 (4), 602−611. (56) Gafney, L.; Varma-Nelson, P. Peer-Led Team Learning: Evaluation, Dissemination and Institutionalization of a College Level Initiative; Springer: Dordrecht, Netherlands, 2008. (57) Barkley, E. F.; Cross, K. P.; Major, C. H. Collaborative Learning Techniques: A Handbook for College Faculty; Jossey-Bass: San Francisco, CA, 2004. 34

DOI: 10.1021/acs.jchemed.8b00387 J. Chem. Educ. 2019, 96, 25−34