A Comparison of How Undergraduates, Graduate Students, and

Article Cite This: J. Chem. Educ. XXXX, XXX, XXX−XXX

pubs.acs.org/jchemeduc

A Comparison of How Undergraduates, Graduate Students, and Professors Organize Organic Chemistry Reactions Kelli R. Galloway, Min Wah Leung, and Alison B. Flynn* Department of Chemistry & Biomolecular Sciences, University of Ottawa, Ottawa, Ontario K1N 6N5, Canada S Supporting Information *

ABSTRACT: To explore the differences between how organic chemistry students and organic chemistry professors think about organic chemistry reactions, we administered a card sort task to participants with a range of knowledge and experience levels. Beginning students created a variety of categories ranging from structural similarities to process oriented categories. Professors and more experienced graduate students created their categories only for process oriented reasons. Professors discussed different features of the reactions than the students did, suggesting that students need guidance and opportunities to develop skills to identify mechanistically relevant features in a reaction. More specifically, at the University of Ottawa, a transformed organic chemistry curriculum has been designed and implemented where students are first taught the language of mechanisms before learning about specific reactions. Then, students are taught reactions in order of their governing pattern of mechanism, rather than by functional group. We developed a card sort task to investigate how students perceive the organization of the reactions in the curriculum as well as to explore how graduate students and professors think about organic chemistry reactions. There were 25 cards designed with reactants and reagents for reactions taught within the first two semesters of organic chemistry. The card sort task is composed of two parts: first, participants are asked to sort 15 cards into categories; second, the participants are given 10 additional cards and asked to incorporate them into their existing categories. During the fall 2017 semester, second semester organic chemistry students (N = 16), organic chemistry graduate students (N = 10), and professors who either teach and/or conduct research in organic chemistry (N = 7) were interviewed using the card sort task. We analyzed the participants’ categories for cards that were frequently sorted together and the reasons they gave for creating the categories and then compared the findings across the different participant groups. KEYWORDS: Second-Year Undergraduate, Graduate Education/Research, Chemical Education Research, Organic Chemistry, Reactions, Mechanisms of Reactions FEATURE: Chemical Education Research

■

INTRODUCTION

and unfamiliar questions, and a report detailing the comparison is currently being prepared. The evaluation has also investigated how students use and interpret the mechanistic language of chemistry because the language is taught prior to specific reactions.12,13 A quantitative analysis of exam questions found that students drew correct arrows more often than reversed or illogical arrows and that students tended to be more successful at drawing arrows for symbolism questions than for drawing products.12 The findings also included that students struggled with intramolecular reactions, implicit atoms, and rearrangements. To follow up, 29 students in their first semester of organic chemistry were interviewed and asked to work through symbolism questions for reactions they had not yet been taught.13 The study found that the method by which students integrated electron movement into their discussions of the

Evaluation of a Transformed Curriculum

At the University of Ottawa, a transformed organic chemistry (OC) curriculum was implemented in 2012.1 Within this curriculum, students are taught the electron-pushing formalism (the “language of mechanisms”) before they learn specific reactions. Then, the reactions are organized by governing pattern of mechanism rather than by functional group. The design of this curriculum was based on literature on students’ difficulties with the electron-pushing formalism2−5 and students’ lack of depth of understanding of organic chemistry concepts6−10 with the goal to reflect how modern expert chemists think and talk about organic chemistry reactions. An evaluation of the curriculum has been underway using a design-based research approach.11 Using a historical control, students’ answers to exam questions have been analyzed to compare the performance in the new curriculum. The selected exam questions examined students’ proposed mechanisms on familiar © XXXX American Chemical Society and Division of Chemical Education, Inc.

Received: September 23, 2017 Revised: December 19, 2017

A

DOI: 10.1021/acs.jchemed.7b00743 J. Chem. Educ. XXXX, XXX, XXX−XXX

Journal of Chemical Education

Article

Research Using Card Sort Tasks to Measure Differences in Novices and Experts

symbolism contributed to successfully working through the problems. In addition to the studies mentioned, several other investigations are underway to evaluate the effect of the transformed curriculum on how students learn organic chemistry. One of these studies has involved exploring how students think about and perceive organic chemistry reactions. The curriculum has been designed to provide opportunities for students to make connections across reactions. The idea is that if students can see patterns and make connections across reactions in a mechanistic way, then the need to memorize reactions is diminished. Within the current structure of the curriculum, similarities between reactions with similar mechanisms are acknowledged for the students, such as identifying the most nucleophilic and electrophilic sites, but there is not an emphasis on students developing skills to see the patterns. Before making changes to the pedagogy or curriculum, we designed a study to investigate how students are seeing the patterns of mechanisms (or not) to make evidence-based changes.14 At the same time, we decided to also explore how organic chemistry professors and organic chemistry graduate students think about organic chemistry reactions to learn ways in which the curriculum reflects how expert chemists organize their thinking about reactions. For our context, the organic chemistry students are considered the “novices” and the organic chemistry professors are referred to as the “experts” with the graduate students as “intermediates” between novices and experts.

Card sort tasks have been used to explore differences between novices and experts in psychology,30 physics,28,31−35 biology,29,36 and chemistry.19,21,22,24,37 A majority of these studies have asked participants to sort discipline-specific problems. The purpose of sorting problems has been to learn about the different approaches to solving the problems. Cards were developed with each depicting a discipline-specific problem, and participants were asked to sort into categories. Often, the cards were designed with different deep and surface features to explore how the different participants interacted with the information. The studies in physics, biology, and chemistry have found that beginning students tended to sort the problems based on surface features and the experts tended to create categories for problems with similar deep features.24,28,29,31−36 Other studies in chemistry have used card sort tasks to investigate differences in thinking about visualizations of chemical phenomena. Here, cards were developed with varying representations including graphs, still images of a video or animation, or chemical equations in macroscopic, symbolic, or submicroscopic form.21,22,38 These studies found that experts tended to create categories for similar chemical principles across a variety of different representations, but novices created categories in more diverse ways, either sorting by type of representation or media or making more unexpected or miscellaneous sorts. Another study used a categorization task to compare how general chemistry students, organic chemistry students, and professors thought about organic chemistry structures.37 Participants were asked to sort eight organic chemistry line structures into two categories four times over an academic year. The researchers coded the categories as to whether they were based on structure, function, or stereochemistry. The students’ categories were mainly structural or functional based, but the professors’ categories were majority functional with no categories created for structural similarities. These studies show that differences in expert and novice thinking can be explored using card sort tasks with a variety of methods.

Measuring the Differences Between Novices and Experts

A novice is classified as a person with “minimal exposure to a domain”, and an expert is defined as someone who has “acquired more knowledge in a domain” through some experience and that knowledge is “organized or structured”.15 Using what Chi calls a “relative approach”,15 research in the last 30 years has worked to characterize expert-like thinking and has explored how experts came to be that way to help others gain such knowledge and skill.15−18 A few of the differences that have been identified between novices and experts are the following:15,17 • Experts can generate the best solution to a problem with little effort • Experts see features and patterns differently than novices • Experts not only have a breadth of knowledge but their knowledge is organized to reflect deep understanding • Experts use different strategies to solve problems than novices The research characterizing the differences between experts and novices has sampled experts in discipline-specific areas including chemistry, biology, and physics. In chemistry, the studies investigating differences in expert and novice thinking in chemistry have included classifying chemical reactions,19 solving chemical equilibrium problems,20 visualizing chemical representations,21,22 and approaching solving open-ended problems.23,24 A range of different tools and approaches have been used to measure the differences between experts and novices. These different measurements have included concept-inventory tests,25−27 think-aloud interviews for problem solving,20,23 and card sort tasks administered either individually or in a classroom setting.19,22,24,28,29 Each of these approaches has its advantages and disadvantages depending on the purpose of the research. Because our research was a first step to explore the differences in how undergraduates and professors think about organic chemistry reactions, we decided to develop a card sort task to elicit conversation in an individual interview setting.

■

THEORETICAL LEARNING FRAMEWORKS Our research was guided by the theoretical frameworks of meaningful learning and human constructivism39−42 and information processing.43−48 Together, these theories support that meaningful learning takes place when there is a positive integration of cognitive, affective, and psychomotor learning experiences. Ausubel’s three conditions for meaningful learning are (1) the learner must have relevant prior knowledge, (2) the new material must be presented in a meaningful way, and (3) the learner must choose to nonarbitrarily integrate the new material into the prior knowledge.39,40,42 These three conditions correspond to how learners receive and process information. A learner’s prior knowledge is related to their long-term memory storage, whether the information is present and retrievable (meaningful learning condition 1). A meaningful presentation of the new material depends on the learner’s working memory (meaningful learning condition 2). If a learner is holding many ideas in their working memory, then their working memory may become overloaded, hindering the learner from taking in the new information. However, if the learner has a more organized long-term memory, then the learner may be able to process the new information in a more meaningful way. Lastly, the learner can make a meaningful connection between new B



Article

Figure 1. A selection of the reaction cards developed for the card sort tasks. Each card listed a reactant and reagents with a letter label in the corner. Reactions were selected across the different mechanism categories that progress through the curriculum. The number in parentheses indicates the number of cards per mechanism category.

by examining any similarities and differences in how they organize organic reactions.

experiences and prior knowledge when the new information is connected to relevant information in the long-term memory in a nonrandom way (meaningful learning condition 3). For this study, we were interested in the organization of the prior knowledge and the integration of new knowledge into the old. We wanted to know what students deem relevant when looking at organic chemistry reactions and the similarities they see across reactions, and how what students see is different from graduate students and professors who have presumably more developed knowledge structures. These theories provide a lens to design the methods and interpret the findings to explore these ideas.

■

METHODS

Card Sort and Interview Guide

A card sort activity was developed and administered during one-on-one interviews to initiate conversation about how participants think about and organize organic reactions. The card sort consisted of 25 reaction cards (Figure 1, a complete list of the cards is included in the Supporting Information) and two tasks. The 25 reactions for the cards were chosen and adapted from course materials, exams, and previous research in our group.12−14 Each card listed the reactants, reagents, and reaction arrow (Figure 1). We chose this format without electronpushing arrows or products to mimic how undergraduate students might see these reactions on course exams. The cards were labeled with upper-case letters for identification purposes. A detailed description of the development of the card sort is published elsewhere.14 Two expert chemists (not part of the research team) reviewed the cards for reaction feasibility and formatting, and revisions were made based on the feedback. A pilot study was conducted to ensure adequate collection of data with the card sort. For the pilot interviews, the participant was given all 25 cards to start, asked to sort into categories, and then asked to sort a different way for the second task. Based on the pilot interviews, the card sort interview was revised. For the first task, the participant was given 15 of the 25 cards and asked to sort them into categories however they would like. Then, the participant was asked to describe the

■

RESEARCH QUESTIONS This study focused on how people with different experiences and knowledge think about organic chemistry reactions outside the context of being asked to draw mechanisms or predict products. We had three guiding research questions for this study: 1. How do OCII students, OC graduate students, and OC professors organize organic chemistry reactions? 2. What features do OCII students, OC graduate students, and OC professors attend to when organizing the reactions? 3. How do OCII students’ categorizations of organic reactions compare with categorizations made by OC grad students and OC professors? The first two questions set up an exploration of the data to gather evidence for how people with different experiences and expertise think about organic reactions. The third question guided the comparison between different experience and levels of expertise C



Article

students. One PhD student was interviewed whose research was in biochemistry, not organic chemistry. The results from this student will be discussed separately from those of the other graduate students. The sample professors included both men and women and a range of experience levels from a few years to multiple decades (the exact distribution will be kept confidential to protect their identities). The majority of the professors were Caucasian. Four listed their first language as English and two as French.

categories created and explain the sorting process. For the second task, the participant was given the remaining 10 cards with instructions to incorporate them into the categories from the first task by creating new categories, rearranging the existing categories, or keeping the original categories. Again, the participant was asked to describe the categories created and explain the sorting process. The tasks were not timed, and the participants were allowed as much time as they wanted to sort the cards. The interviews were semistructured allowing the interviewer to ask the participant follow-up questions to gain an understanding of the participant’s choice of categories. The cards were laminated, and the participants were given dry erase pens to write on the cards. Also, the participants were given regular pens, paper, a textbook, and a model kit to use at their leisure. Each interview was audio and video recorded to capture an accurate account of what was said as well as how the participant sorted the cards. The length of the interviews ranged from 27 min to 1 h 20 min with an average length of 50 min.

Data Analysis

For a composite view of the data, descriptive statistics were calculated for the number of categories and the average number of cards per category for each sample. Then, we plotted each samples’ sorts in Gephi,49 a data visualization software designed for social network analysis, and formatted the graph using the Force Atlas algorithm. In Gephi, each card is represented as a node, and lines are plotted between nodes (called edges) to indicate when two cards were placed in the same category. The Gephi visualizations allowed for viewing the participants’ categories simultaneously to look for patterns within and between the samples. The participants’ category descriptions were analyzed qualitatively using NVivo 11 for Mac. The interview audio recordings were transcribed verbatim, and the video recordings were transcribed for the sorting processes and what the participants would point at during the interviews. Analysis began with open coding to allow the codes to come from the participants’ descriptions. We approached each sample with fresh eyes, allowing new codes to be developed as necessary. We used constant comparative analysis to refine the codes and compare with participants, across participants, and then within and across samples.50,51 Additional analyses were conducted, such as comparing cards frequently sorted together and examining the features the participants identified while sorting, to explore the similarities and differences in how the participants with different experience and expertise sorted the reaction cards.

Participant Recruitment

Prior to participant recruitment and data collection, we applied for and received approval from the institution’s Ethics Review Board to conduct this research. Participants received a $10 gift card to Starbucks. Throughout this article, participants are referred to using pseudonyms. Participants from three different populations were recruited for this study. Undergraduate students were invited to participate from two sections of OCII (one English and one French). An announcement was made in class and posted to the course management site with information on how to participate. Sixteen students scheduled interviews within a five week period during the semester. Within this time-span, the students had been taught reactions of σ electrophiles and spectroscopy and were beginning to learn π electrophilic reactions with leaving groups, in addition to having taken the OCI course.1 Graduate students and professors were recruited through e-mail. Graduate students in organic chemistry were recruited by sending an e-mail through the Department of Chemistry & Biomolecular Sciences graduate student listserv. The e-mail described the project and the request for participation from graduate students conducting research in organic chemistry. Initially, only four students volunteered, but they informed their lab mates about the project, and additional graduate students volunteered. Individual e-mails were sent to professors who teach, have taught, and/or conduct research in organic chemistry with an invitation to participate in the study. Ultimately, 10 graduate students and 7 professors were interviewed over a two month period.

Establishing Trustworthiness

To establish the trustworthiness of our study, we used guidelines put forth by Lincoln and Guba.52 We developed our credibility by engaging in peer debriefing and negative case analysis. For peer debriefing, the first author would dialogue with two other researchers with less knowledge of the project. These conversations were used to explore meanings and interpretations of the data and give the researcher a space to realize any biases that may be in the way of moving the project forward. For negative case analysis, when categories emerged and themes were being formed, they were held up against the raw data. We searched for disconfirming cases and revised the emerging findings as negative evidence was found. To aid in the transferability of our research, we have provided a thick description of the participants and the setting, and we provide evidence below in the form of participant quotations to support the findings. With this context, we set forth the information and data necessary to make transferrable judgements. In addition, we performed an intercoder analysis. The goal for the intercoder analysis was to ensure that a subset of codes supporting the emerging findings could be consistently used and that the findings accurately represented what the participants said. The first author coded all of the interview transcripts. A chemistry education researcher not associated with the project coded ∼10% of the interviews (1 OCII student, 1 MSc student, 1 PhD student, and 1 professor) using a subset of the

Sample Descriptions

The OCII students were fairly representative of the students who enroll in the course. Of the 16 students, 11 were in the second year of university, and 5 were in their third year. There were 10 females and 6 males. Half of the sample listed their first language as English, one-fourth as French, and one-fourth as Other. The students represented a variety of ethnicities which is representative of the course and the university. Of the 10 graduate students, 4 were Masters’ students (MSc) in their first or second years of graduate school, and 6 had completed their comprehensive (doctoral candidacy) exams ranging from third to sixth years and will be referred to as PhD students (PhD). Three of the students completed their undergraduate degrees at uOttawa. Three of the MSc students were male, and one was female. There were two male and four female PhD D



Article

students had the largest variation and range of number of categories indicating greater diversity among the categories they created. Also, the average number of cards per category is similar between the samples for both tasks. Results from inferential statistics comparing the number of categories created for each sample from Task 1 to Task 2 and a comparison between the samples for Task 1 and Task 2 are included in the Supporting Information. With little to no difference in the number of categories created by the participants with different knowledge and experiences, analysis of the content of the categories is necessary to distinguish the groups.

codes. After the coding was completed, differences in coding were discussed and consensus was reached. The discussion also yielded more precise code names and definitions which led to more consistent usage of codes. Following the intercoder discussion, the analysis continued looking for disconfirming cases and evidence for the emerging themes.

■

RESULTS AND DISCUSSION

Descriptive Statistics

An overview of the number of categories each group created and the average number of cards per category is listed in Table 1.

Visualizing the Data in Gephi

Table 1. Descriptive Statistics for the Number of Categories and Average Number of Cards Per Category for Task 1 and Task 2 for OCII Students, MSc Students, PhD Students, and Professors Number of Cards per Category

Number of Categories Participant Group OCII MSc PhD Prof OCII MSc PhD Prof

Mean (Stdev) 7.7 8.0 7.8 7.3

(2.2) (1.4) (0.8) (1.1)

9.1(2.7) 10.0 (1.4) 10.6 (1.1) 8.9 (1.2)

Median Task 1 7 8.5 8 7 Task 2 8 9.5 11 8

The participants’ sorts were visualized in Gephi to get a bigpicture view of the data (Figure 2). Nodes represent the reaction cards using the letter labels. Cards that were sorted in the same category are connected by edges (i.e., lines). The thickness of the edge corresponds to the number of participants who sorted a card pair together. Thicker, darker lines indicate that a card pair was sorted into the same category with high frequency, and a thinner, lighter line indicates a card pair that was sorted together with less frequency. The Gephi visualizations were qualitatively compared for the groups of participants for Task 1 and Task 2. Gephi has capabilities to calculate network statistics, but we chose to make qualitative comparisons because of our small sample sizes. OCII students have the greatest number of edges for both Task 1 and Task 2. The plots for the OCII students show another side to the diversity in their sorting of the cards than we saw in the descriptive statistics previously. Not only did the students vary in the number of categories they created, the contents of the categories varied as well. Despite this diversity in their categories, there were cards sorted more frequently together across the students, as indicated by card pairs with thicker edges. The plots for the MSc students are more ordered than OCII for Task 1 but are a web of edges for Task 2 similar to the OCII students. For both Task 1 and Task 2, all four of the MSc

Range

Mean (Stdev)

Range

5, 6, 6, 6,

13 9 9 9

2.1 2.1 1.9 2.1

(0.60) (0.36) (0.22) (0.30)

1, 1, 1, 1,

5 5 5 6

5, 9, 6, 8,

15 12 12 11

2.9 2.5 2.4 2.9

(1.0) (0.33) (0.27) (0.40)

1, 1, 1, 1,

9 7 11 10

If a participant created a category for “miscellaneous” or “unknown” reactions, those cards were split into individual categories so as not to assume the participant perceived similarities between those reactions. The average number of categories created for each task is similar between the samples. The OCII

Figure 2. Gephi visualizations for OCII (N = 16), MSc (N = 4), PhD (N = 6), and professors (N = 7). E



Article

students placed Card J (alkene epoxidation using a peracid) in its own individual category as seen by the node for J being separate and unattached from the rest of the network. Also, there are fewer thicker edges in the MSc students’ plots, which could be a result of the smaller sample size. The plots for the PhD students and the professors appear to exhibit a pattern and order where the plots did not for the OCII and MSc students. For Task 1, both the PhD students and the professors have two networks within the plot indicating they consistently sorted some cards separately from other cards. The PhD students’ plot shows a separate network for cards EHN (σ electrophile reactions/substitution and elimination reactions), and the professors’ plot shows a separate network for ACGJL (reactions of π nucleophiles/alkenes and aromatic reactions). These separate networks from Task 1 are connected as one large network in Task 2 but remain distinct. The PhD students and the professors appeared to have sorted the cards in more similar ways and with less diversity than the OCII students. Using the Gephi diagrams and the card pair frequencies, we can identify card pairs that were frequently sorted together by all the participant groups. For Task 1, a majority of participants in all samples sorted together EN (σ electrophilic reactions/ substitution and elimination reactions), FI (π electrophilic/ reactions of anhydrides), and DK (π electrophilic/organometallic reactions with ketones). For Task 2, EN and FI were still frequently sorted together by all samples but DK was no longer a frequently sorted pair for the MSc students. When looking for similarly sorted card pairs between the groups, the PhD students and the professors had six pairs in common, more than any other group. The PhD students and the professors commonly sorted together BR, BX, FW, HN, MY, and NU. These are card pairs that were not commonly sorted together by the OCII or MSc students. The similar sorting between the PhD students and professors suggests similar ways of thinking about reactions between these two groups. The qualitative analysis is needed to provide further evidence of similar thinking about reactions between the PhD students and the professors.

may have also paid attention and seen similar structural features, similar properties of the structures, and similar reaction characteristics, but the sorting decisions they gave were based upon molecular-level characteristics. We found that participants who created categories for identical structure or properties talked about the reactions in a static way, meaning they discussed only features present on the card. The participants who created categories for the type of reaction and type of mechanism referred to the process oriented features of a reaction, including discussing how the features on the card would interact. We continued with open coding for the analysis of the MSc students, PhD students, and professors using previously created codes and allowing for new codes to arise from the new data being analyzed. Despite this fresh look at each set of interviews, the coding categories developed from the OCII analysis remained useful ways to characterize the participants’ descriptions for the categories they created. Each category that the participants created was coded using the four levels of interpreting the reactions along with a coding category “Unknown” if the participant expressed not knowing the reaction on the card. Examples from each group for the five coding categories are given in Table 2. We want to point out that while a participant’s card category might be coded as “Type of Mechanism” or “Type of Reaction,” participants could have used other reasoning patterns not captured by the words they chose to speak in the interview. The codes were applied using strict definitions. For instance, a participant would have needed to mention something about electron movement, bonds breaking or forming, electron density, or nucleophiles and electrophiles for the card category to be coded as “Type of Mechanism.” Our reasoning behind these strict definitions was that we cannot infer what a student is thinking: they must tell us because we can only use the words we hear from them. However, these strict definitions could have limited the coding of participants’ card categories. For example, some professors gave detailed descriptions for their categories, but others were more general. The professors who sorted in a general way could have been seasoned professors who oscillate between mechanistic explanations and simpler explanations, using the simplest explanation to suit their needs. Alternatively, the professors could have been thinking about mechanisms, but without voicing it, we would have no evidence for the coding of their card categories. The coding system and subsequent results represent what the participants said in the interview without inferring beyond their own words. We counted the number of categories that the participants created across the five coding categories. Then we graphed these distributions using percentages (Figure 3). There is a small shift in the distribution of the coding categories from Task 1 to Task 2, but the overall interpretations are similar. The predominant category for both OCII and MSc students was Type of Reaction representing about 50% of the total categories created for those samples. The most common category for PhD students and professors was Type of Mechanism with about 60% of the categories for PhD students and close to 80% of the categories created by the professors. OCII and MSc students created 20−30% of their categories for structural similarities, either identical structure or properties of structure, but neither PhD students nor professors created categories for structural similarities. Each group of participants created at least a small percentage of categories for reactions that were unknown to them. The raw frequency counts for these distributions are included in the Supporting Information.

Reasons for Creating Categories

After sorting their categories, participants were asked to describe the categories they made and how they decided to sort them into these categories. These descriptions were analyzed qualitatively to explore how and why the participants created the categories that they did. From the analysis with the OCII students, four levels of criteria were observed for how the students interpreted the reactions to create their categories. The levels observed for creating categories were the following: Identical Structure, Properties of Structure, Type of Reaction, and Type of Mechanism. We discuss these criteria for creating categories as levels because they can represent different ways that the students chose to discuss the reactions on the cards. Students who created categories for Identical Structure used only the structural features shown on the card to make their sorting decisions without conversation about unseen elements of the reaction. Students who created categories for Properties of Structure used knowledge beyond what was given on the card to interpret physical and/or chemical properties of a given structure. Students who created categories for Type of Reaction spoke about what happens in the reaction, how the atoms move, what the product will look like, etc. Students who created categories for Type of Mechanism incorporated language regarding how the electrons would move and other submicroscopic details during the reaction process. The participants who sorted for Type of Mechanism F



Article

Table 2. Examples for the Reasons for Creating Categories from Each Sample Examples of Reasons for Responses in Each Sample Interpretation Level

OCII

MSc

PhD

Prof

Type of Mechanism

“Yeah, so these (FIKOTVY) were ketones that acted as electrophiles and on the other if the oxygen part of the ketone or aldehyde is the thing that initiated the reaction, it acts as a nucleophile (BDRX).” Madison

“I have some of like nucleophilic attack kind of thing happens with like hard to hard here (K) and I think it’s the same happening here (B).” Brianna

“[T]hese reactions (BFIKO) are all reactions in which a nucleophile attacks a carbonyl and forms a tetrahedral intermediate.” Noah

“[T]he easy one is any kind of π bond nucleophile reacting with an electrophile (ACGJL).” Dr. Brown

Type of Reaction

“[T]his one (AGM) is an addition of halogen.” Hannah

“So this one I put them (GU) together because the water the solvent helps with the reaction.” Aiden

“I made an intramolecular cyclization category (BQVX).” Emma

“Carbonyl chemistry over there (BDFI).” Dr. Wilson

Properties of Structure

“[T]hese ones (DEN) all have like pretty strong leaving groups. And they all have stereochemistry to them.” Riley

“Yeah so I kinda started splitting, this sides more bases, nucleophiles, that type of chemistry (EFINO). And over here we’ve got acids (ABDKLM).” Jack

Not observed

Not observed

Identical Structure

“All the [benzene groups] that didn’t have either uh a carbonyl of something nitrogen or uh acid I put it in (EGP) and then the ones with carbonyls on the substituents I put as one category (KTV).” Abigail

“[T]his is just metal based reactions (DKOP).” Jacob

Not observed

Not observed

Unknown

“[A]s for this (F), I truly just I made like a list of things that I don’t know.” Isabella

“[T]hese two guys (CL) uh I feel like I put them sort of separate but I put them together because uh I both don’t know how these would proceed.” Aiden

“This is a question mark. This is a leftover (L).” Madeline

“And this one (V) here initially I think that this might be um some kind of uh like a Friedel− Crafts type of thing but to be honest I’m not sure. I don’t really know.” Dr. Smith

Figure 3. Distribution of reaction card categories created for each sample across the different reasons for creating categories.

Features Attended To

with a carbonyl present. OCII students also frequently mentioned “halogen”, “chlorine”, “metal”, and “substitution”, none of which were frequently mentioned by the professors. There was not a single feature that all OCII students mentioned while discussing the reactions. The students participants held diverse ideas about organic reactions the student participants held diverse ideas about organic reactions and brought different perspectives to how they thought about the reactions presented on the cards.14 There were features identified by all participants among the other three groups of participants. Lists of the most commonly mentioned features by sample are included in the Supporting Information. In addition to carbonyl, all the professors mentioned nucleophiles and electrophiles, acid−base

The features of the reactions that the participants discussed indicated what they were paying attention to as they looked at the reactions. The participants identified structural features (e.g., functional groups), reaction related features (e.g., name of reaction, name of product, and reaction intermediate), and chemical properties (e.g., acid, base, and electronegativity). We compared the features discussed between the samples of participants to identify distinguishing factors in how they discussed the reactions (Figure 4). The feature that was commonly mentioned (by 88% of the sample or greater) among all the participants was “carbonyl” which could be attributed to the fact that there were 15 cards G



Article

Figure 4. Selection of features that were mentioned by the majority of participants and some that distinguished the professors from the other participant groups.

Andrew (OCII): “This one [S] is a new one. I saw NaNH2 which would dissolve into NH2 minus which is a strong uh strong charged and small nucleophile which would probably be SN2 because it doesn’t look like there is room for elimination in that molecule.” Kaitlyn and Andrew used their prior knowledge about nucleophiles to work through how the reactions would proceed. Even though Andrew incorrectly categorized reaction S as SN2, he reasoned through the reaction with his chemistry knowledge. The professors differed from the students in the consistency with which they incorporated discussion of nucleophiles and electrophiles throughout the interview and the depth of the conversion of nucleophiles and electrophiles. Consider the examples from Dr. Brown and Dr. Martin: Dr. Brown: “I guess once you recognize that you have a nucleophile attacking a π* that’s probably the harder and usually the rate-determining step of the reaction.” Dr. Martin: “We’ve got examples of aromatic rings being nucleophiles [CLV]. They’re weak nucleophiles. They’re not great nucleophiles. But as long as you’ve got a really great electrophile they can do chemistry, so that’s electrophilic aromatic substitution. But the π systems don’t often behave as electrophiles to be attacked by nucleophiles, right [S]? Um this one’s [S] a very specific a special case for that because we have uh a highly electron poor aromatic ring because of the two nitro groups so it’s totally different than all the other examples.” Second, the professors talked about “activation steps” in a reaction involving proton transfer as a necessary step but not the key bonding step of the reaction. As Dr. Smith described: Dr. Smith: “[T]here’s uh a couple of things in here that involve acids as part of the reaction but to me I don’t see that as being the main you know like so just quick examples. So these guys [BRTX] all involve acids or bases and uh but to me they’re not acid−base reactions. The primary reaction is something else. ...The acid−base part to me is just a I guess, the term just a little pre-reaction or a reaction condition I guess that allows it to happen so.”

reactions, enolate, and intramolecular reactions. The PhD and MSc students had frequently mentioned features in common such as “acid”, “mechanism”, and “substitution”. The most commonly mentioned features by the PhD students overlapped with the professors’ including “nucleophiles and electrophiles” and “aromatic”, and the MSc students’ frequently identified features shared “metal” in common with the OCII students. To further investigate differences in how the professors viewed the reactions compared to the students, we identified codes for features that were created during the analysis of the graduate students and professors’ interviews. Since these analyses were conducted after the OCII students’ analyses were complete, the codes represent features that the OCII students rarely identified. A selection of the codes is included in Figure 4 along with the features that were mentioned frequently by all participant groups. The first three features shown in Figure 4 were commonly mentioned by participants. The other five features listed show the disparity in how the professors described the reactions compared to the OCII students (or MSc students). The professors mentioned aspects about the reactions that they saw and deemed necessary for talking about the reactions listed on the cards. The absence of mentions of these features by the OCII students suggests that either the students did not know these features, did not see them, or chose not to mention them. Similar to the OCII students, the MSc students did not mention these features either. The PhD students did identify a few of the listed features but not to the same extent as the professors. We will discuss a few of the features that distinguish OCII students and professors with examples. First, OCII students and professors both mentioned “nucleophiles and “electrophiles.” The examples from the students showed that they were trying to incorporate the knowledge they were learning in class with what they were seeing on the cards. For instance: Kaitlyn (OCII): “I know the lithium [K] I don’t know there’s a name for it it’s lithium is attached to an alkyl group then it can act as a nucleophilic group.” H



Article

categories of their choosing. Then, the participants were given 10 additional cards to integrate into their categories or rearrange how they prefer. All the participants created categories for chemically relevant information; there were no nonsensical categories. We observed four main levels of interpreting reactions: Identical Structure, Properties of Structure, Type of Reaction, and Type of Mechanism. The OCII students created categories across the observed levels of interpreting the reactions with the most common being Type of Reaction. The MSc students sorted similarly to the OCII students in that they also created categories spanning the four observed levels with the most common being Type of Reaction. Still, the MSc students were unique among themselves, sometimes with similarities to the OCII students and sometimes with similarities to the PhD students. The PhD students and professors created categories with some similarity to each other. Both the PhD students and the professors only created categories for Type of Reaction and Type of Mechanism with Type of Mechanism being the most common. No PhD student or professor created a category for structural similarities. As the PhD students are developing expertlike thinking, perhaps they are learning how their professors think through discussions in groups meetings or questions posed at department seminars. The OCII students were nearing the end of their second semester of OC. Their limited experience is manifested in different ways for these students as they try to make sense of what they are learning in class without the depth of understanding to yet make connections like an expert. Although the MSc students have completed the undergraduate education, they have not yet gained the experience of the PhD students and so still have limited depth of understanding.

Conversely, OCII students Madison, Kaitlyn, and Zachary distinguished reactions of carbonyls that were in base versus in acid. Zachary described: Zachary (OCII): “[T]hat one was like the π bond getting attacked (K), so, and this one [B] um this one was... this one [B] was almost like that one [K], so like the π bond got er no wait no, no, this one is a little different because the lone pairs here [oxygen of carbonyl] would here I think the lone pairs would attack that [H2SO4].” Both reaction B and K involved nucleophilic additions to an electrophilic carbonyl, but Zachary saw the addition of the proton transfer in reaction B as a distinguishing characteristic from reaction K and not the “pre-reaction” that Dr. Smith mentioned. Last, the professors talked about reaction mechanisms being ambiguous in how they would proceed. Dr. Tremblay discussed the nuanced possibilities of the reactions as depending on the reaction conditions: Dr. Tremblay: “This is alkylations [H], and the question which is behind these is the selectivity of the alkylation, side selectivity of the alkylation depending on uh concentration of electrophile, nucleophile.” Dr. Tremblay wanted to know the concentrations of the reacting species to be able to suggest a proposed reaction pathway. Dr. Smith described the mechanistic ambiguity of reaction E: Dr. Smith: “This one [E] too um I always have to think about these kinds of reaction conditions are always tricky and they can give you multiple things so these ones I always kinda take my time because I’ve never kinda 100% sure what’s supposed to happen.” Dr. Smith acknowledged the variety of mechanistic possibilities. Similarly, Madison mentioned the possibility of multiple reaction pathways for reaction U, but she was the only OCII student to suggest the possibility of mechanistic ambiguity.

Limitations

This study was qualitative in nature and makes no claims on how all OCII students, OC graduate students, and OC professors think about organic reactions. In addition, the participants were a convenience sample and do not represent the population of OCII students, OC graduate students, and OC professors at this university. The goal of this study was to explore how people with varying experience and knowledge of organic chemistry think about organic reactions using the card sort task. The card sort task was a useful tool to prompt the conversations with the participants to learn about their ideas. Future research can use the card sort task to make quantitative comparisons among people with different levels of expertise.

A Contrasting Case

During the participant recruitment process, a PhD student volunteered who conducted research in biochemistry, not organic chemistry. This student (called Olivia here) provided insight into how graduate students earning a degree in chemistry but not specializing in organic chemistry think about organic reactions. From the beginning of her interview, Olivia made clear that she was not an organic chemist. Unlike the organic PhD students, Olivia created categories for Identical Structure and Properties of Structure for both Tasks 1 and 2. Her attention was not on how the reactions would proceed but on the surface level similarities between the reactions. Although Olivia was earning a PhD in chemistry, she did not discuss any aspects of the reactivity of the species on the cards. One assertion could be made that people who earn an advanced degree in chemistry ought to have a breadth of knowledge across all of the subdisciplines, including basic abilities of mechanistic thinking, instead of leaving with highly specialized knowledge. Although Olivia might be an outlier, her interview was one example of how a concentration in graduate school can hinder developing breadth of chemistry knowledge.

■

Implications for Teaching and Learning

In support with previous literature,21,22,24,28,29,31−37 the results from this study continue to show that undergraduate students and professors do not think about chemical phenomena in the same way. We cannot make assumptions about what we think our students see when looking at an organic reaction. Instead, we can (1) intentionally model how we see and interpret organic reactions and (2) provide opportunities for students to develop the skills to identify mechanistically useful features in a reaction. Students will not know what is useful or where to focus their thinking without guidance. That said, simply telling students what features to pay attention to could lead to memorization. Instead, students can learn to identify features for the information the features yield about the reaction. Dr. Tremblay, one OC professor interviewed in this study, talked about reading organic reactions like reading comics: the pictures tell the story. Students need to be taught how to read organic reactions, what to pay attention and why, with concomitant opportunities to practice and receive feedback. For example, if we want students

CONCLUSIONS

We explored the differences in how OCII students, MSc students, PhD students, and OC professors think about organic chemistry reactions using a card sort task. First, the participants were given 15 cards with instructions to sort the cards into I


Journal of Chemical Education to learn to think mechanistically and demonstrate such thinking, then they need to be able to identify areas of high and low electron density. In this example, students need to know why thinking mechanistically is a useful long-term goal and why identifying areas of high and low electron density will assist in mechanistic thinking. Conversely, students need to experience the cognitive dissonance of identifying features that will not be useful to think about a reaction mechanistically. In this study, Abigail created separate categories for reactants of a benzene and carbonyl or nitrogen, and Riley created categories for reactants with leaving groups. Both of these categorization schemes could be useful if the goal was functional group transformation on a benzene or the removal of a substituent, but the strategy is not directly transferrable to mechanistic thinking. Riley needs to know why identifying leaving groups could be useful, and Abigail needs to think about the reactivity of the benzene substituents.

■

REFERENCES

(1) Flynn, A. B.; Ogilvie, W. W. Mechanisms before Reactions: A Mechanistic Approach to the Organic Chemistry Curriculum Based on Patterns of Electron Flow. J. Chem. Educ. 2015, 92 (5), 803−810. (2) Bhattacharyya, G.; Bodner, G. M. It Gets Me to the Product”: How Students Propose Organic Mechanisms. J. Chem. Educ. 2005, 82 (9), 1402−1407. (3) Ferguson, R.; Bodner, G. M. Making Sense of the Arrow-Pushing Formalism among Chemistry Majors Enrolled in Organic Chemistry. Chem. Educ. Res. Pract. 2008, 9, 102−113. (4) Kraft, A.; Strickland, A. M.; Bhattacharyya, G. Reasonable Reasoning: Multi-Variate Problem-Solving in Organic Chemistry. Chem. Educ. Res. Pract. 2010, 11 (4), 281−292. (5) Grove, N. P.; Cooper, M. M.; Rush, K. M. Decorating with Arrows: Toward the Development of Representational Competence in Organic Chemistry. J. Chem. Educ. 2012, 89 (7), 844−849. (6) Anderson, T. L.; Bodner, G. M. What Can We Do about “Parker”? A Case Study of a Good Student Who Didn’t “Get” Organic Chemistry. Chem. Educ. Res. Pract. 2008, 9 (2), 93−101. (7) Grove, N. P.; Bretz, S. L. A Contiuum of Learning: From Rote Memorization to Meaningful Learning in Organic Chemistry. Chem. Educ. Res. Pract. 2012, 13 (3), 201−208. (8) Bhattacharyya, G. Trials and Tribulations: Student Approaches and Difficulties with Proposing Mechanisms Using the ElectronPushing Formalism. Chem. Educ. Res. Pract. 2014, 15 (4), 594−609. (9) Anzovino, M. E.; Bretz, S. L. Organic Chemistry Students’ Fragmented Ideas about the Structure and Function of Nucleophiles and Electrophiles: A Concept Map Analysis. Chem. Educ. Res. Pract. 2016, 17, 1019−1029. (10) Anzovino, M. E.; Bretz, S. L. Organic Chemistry Students’ Ideas about Nucleophiles and Electrophiles: The Role of Charges and Mechanisms. Chem. Educ. Res. Pract. 2015, 16 (4), 797−810. (11) Cobb, P.; Confrey, J.; Disessa, A.; Lehrer, R.; Schauble, L. Design Experiments in Educational Research. Educ. Res. 2003, 32 (1), 9−13. (12) Flynn, A. B.; Featherstone, R. B. Language of Mechanisms: Exam Analysis Reveals Students’ Strengths, Strategies, and Errors When Using the Electron-Pushing Formalism (Curved Arrows) in New Reactions. Chem. Educ. Res. Pract. 2017, 18 (1), 64−77. (13) Galloway, K. R.; Stoyanovich, C.; Flynn, A. B. Students’ Understanding of Mechanistic Language Prior to Learning Organic Reactions. Chem. Educ. Res. Pract. 2017, 18 (2), 353−374. (14) Galloway, K. R.; Leung, M. W.; Flynn, A. B. Patterns of Reactions: A Card Sort Task to Investigate Students’ Organization of Organic Chemistry Reactions. J. Res. Sci. Teach. 2017, revision requested. (15) Chi, M. T. H. Two Approaches to the Study of Experts’ Characteristics. In The Cambridge Handbook of Expertise and Expert Performance; Ericsson, K. A., Charness, N., Hoffman, R. R., Feltovich, P. J., Eds.; Cambridge University Press: New York, NY, 2006; pp 21− 30. (16) Chi, M. T. H. Theoretical Perspectives, Methodological Approaches, and Trends in the Study of Expertise. In Expertise in Mathematics Instruction; Li, Y., Kaiser, G., Eds.; Springer: Boston, MA, 2011; pp 17−39. (17) Bransford, J. D.; Brown, A. L.; Cocking, R. R. How People Learn: Brain, Mind, Experience, School. In How People Learn: Brain, Mind, Experience, and School: Expanded Edition.; National Academies Press: Washington, DC, 2010. (18) Ericsson, K. A. K.; Krampe, R. R. T.; Tesch-Romer, C. The Role of Deliberate Practice in the Acquisition of Expert Performance. Psychol. Rev. 1993, 100 (3), 363−406.

Although a goal of this study was to see how the patterns of mechanisms curriculum is perceived by the OCII students, the participants were not explicitly asked to sort by mechanism. Instead, the interview prompt asked the participants to create categories however they would like to see how participants would naturally sort the reactions. A future study could directly ask participants to sort based on the mechanism to explore mechanistic patterns that participants with different expertise perceive. Also, this study involved interviewing a small subset of the graduate students training to become organic chemists. Future studies could investigate graduate student development with a purposeful sample of graduate students using the card sort task. Longitudinal studies could also use the card sort to explore how students’ ideas about organic reactions evolve from beginning organic chemistry through completing an undergraduate degree or through graduate education. Additional studies with the card sort task could also be conducted on a larger scale to make more fine-grained comparisons across participants with different experience and knowledge of organic chemistry.

ASSOCIATED CONTENT

S Supporting Information *

The Supporting Information is available on the ACS Publications website at DOI: 10.1021/acs.jchemed.7b00743. Results from inferential statistics and additional information about the categories that the participants created (PDF, DOCX) Cards for the card sort task (PDF, DOCX)

■

ACKNOWLEDGMENTS

This work was supported by the University of Ottawa. The authors thank Dr. Amanda Bongers for her work on the intercoder analysis and the Flynn Research Group for their feedback on the research.

Implications for Research

■

■

Article

AUTHOR INFORMATION

Corresponding Author

*E-mail: [email protected]. ORCID

Kelli R. Galloway: 0000-0001-6080-7322 Min Wah Leung: 0000-0001-7015-8821 Alison B. Flynn: 0000-0002-9240-1287 Notes

The authors declare no competing financial interest. J



Article

(19) Stains, M.; Talanquer, V. Classification of Chemical Reactions: Stages of Expertise. J. Res. Sci. Teach. 2008, 45 (45), 771−793. (20) Camacho, M.; Good, R. Problem Solving and Chemical Equilibrium: Successful versus Unsuccessful Performance. J. Res. Sci. Teach. 1989, 26 (3), 251−272. (21) Kozma, R. B.; Russell, J. Multimedia and Understanding: Expert and Novice Responses to Different Representations of Chemical Phenomena. J. Res. Sci. Teach. 1997, 34 (9), 949−968. (22) Irby, S. M.; Phu, A. L.; Borda, E. J.; Haskell, T. R.; Steed, N.; Meyer, Z. Use of a Card Sort Task to Assess Students’ Ability to Coordinate Three Levels of Representation in Chemistry. Chem. Educ. Res. Pract. 2016, 17 (2), 337−352. (23) Randles, C. A.; Overton, T. L. Expert vs. Novice: Approaches Used by Chemists When Solving Open-Ended Problems. Chem. Educ. Res. Pract. 2015, 16 (4), 811−823. (24) Krieter, F. E.; Julius, R. W.; Tanner, K. D.; Bush, S. D.; Scott, G. E. Thinking Like a Chemist: Development of a Chemistry CardSorting Task To Probe Conceptual Expertise. J. Chem. Educ. 2016, 93 (5), 811−820. (25) Adams, W. K.; Wieman, C. E. Development and Validation of Instruments to Measure Learning of Expert-Like Thinking. Int. J. Sci. Educ. 2011, 33 (9), 1289−1312. (26) Luxford, C. J.; Bretz, S. L. Development of the Bonding Representations Inventory to Identify Student Misconceptions about Covalent and Ionic Bonding Representations. J. Chem. Educ. 2014, 91 (3), 312−320. (27) Brandriet, A.; Bretz, S. L. The Development of the Redox Concept Inventory as a Measure of Students’ Symbolic and Particulate Redox Understandings and Confidence. J. Chem. Educ. 2014, 91 (8), 1132−1144. (28) Chi, M. T. H.; Feltovich, P. J.; Glaser, R. Categorization and Representation of Physics Problems by Experts and Novices. Cogn. Sci. 1981, 5, 121−152. (29) Smith, J. I.; Combs, E. D.; Nagami, P. H.; Alto, V. M.; Goh, H. G.; Gourdet, M. A. A.; Hough, C. M.; Nickell, A. E.; Peer, A. G.; Coley, J. D.; Tanner, K. D. Development of the Biology Card Sorting Task to Measure Conceptual Expertise in Biology. CBE Life Sci. Educ. 2013, 12 (4), 628−644. (30) Rottman, B. M.; Gentner, D.; Goldwater, M. B. Causal Systems Categories: Differences in Novice and Expert Categorization of Causal Phenomena. Cogn. Sci. 2012, 36 (5), 919−932. (31) Lin, S.-Y.; Singh, C. Categorization of Quantum Mechanics Problems by Professors and Students. Eur. J. Phys. 2010, 31 (1), 57− 68. (32) Mason, A.; Singh, C. Assessing Expertise in Introductory Physics Using Categorization Task. Phys. Rev. Spec. Top. - Phys. Educ. Res. 2011, 7 (2), 020110. (33) Snyder, J. L. An Investigation of the Knowledge Structures of Experts, Intermediates and Novices in Physics. Int. J. Sci. Educ. 2000, 22, 979−992. (34) Wolf, S. F.; Dougherty, D. P.; Kortemeyer, G. Rigging the Deck: Selecting Good Problems for Expert-Novice Card-Sorting Experiments. Phys. Rev. Spec. Top. - Phys. Educ. Res. 2012, 8 (2), 020116. (35) Wolf, S. F.; Dougherty, D. P.; Kortemeyer, G. Empirical Approach to Interpreting Card-Sorting Data. Phys. Rev. Spec. Top. Phys. Educ. Res. 2012, 8 (1), 010124. (36) Smith, M. U. Expertise and the Organization of Knowledge: Unexpected Differences among Genetic Counselors, Faculty, and Students on Problem Categorization Tasks. J. Res. Sci. Teach. 1992, 29 (2), 179−205. (37) Domin, D. S.; Al-Masum, M.; Mensah, J. Students’ Categorizations of Organic Compounds. Chem. Educ. Res. Pract. 2008, 9, 114− 121. (38) Stains, M.; Talanquer, V. Classification of Chemical Substances Using Particulate Representations of Matter: An Analysis of Student Thinking. Int. J. Sci. Educ. 2007, 29 (5), 643−661. (39) Ausubel, D. P. Educational Psychology: A Cognitive View; Holt, Rinehart, and Winston, Inc.: New York, NY, 1968.

(40) Bretz, S. L. Novak’s Theory of Education: Human Constructivism and Meaningful Learning. J. Chem. Educ. 2001, 78 (August 20018), 1107. (41) Novak, J. D. Human Constructivism: A Unification of Psychological and Epistemological Phenomena in Meaning Making. Int. J. Pers. Constr. Psychol 1993, 6 (2), 167−193. (42) Novak, J. D. Learning, Creating, and Using Knowledge; Taylor & Francis Group: New York, NY, 2010. (43) Johnstone, A. H. Chemical Education Research in Glasgow in Perspective. Chem. Educ. Res. Pract. 2006, 7, 49−63. (44) Johnstone, A. H.; Selepeng, D. A Language Problem Revisisted. Chem. Educ. Res. Pract. 2001, 2 (2), 19−29. (45) Mayer, R. E. Information Processing. In APA Educational Psychology Handbook, Vol 1: Theories, Constructs, and Critical Issues; American Psychological Association: WA, 2012; pp 85−99. (46) Schunk, D. H. Learning Theories: An Educational Perspective, 6th ed.; SAGE Publications: Los Angeles, CA, 2012. (47) Sweller, J. Cognitive Load Theory, Learning Difficulty, and Instructional Design. Learn. Instr. 1994, 4 (4), 295−312. (48) Sweller, J.; Chandler, P. Why Some Material Is Difficult to Learn. Cogn. Instr. 1994, 12 (3), 185−233. (49) Bastian, M.; Heymann, S.; Jacomy, M. Gephi: An Open Source Software for Exploring and Manipulating Networks. Third Int. AAAI Conf. Weblogs Soc. Media 2009, 361−362. (50) Corbin, J.; Strauss, A. Basics of Qualitative Research; SAGE Publications, Inc.: Thousand Oaks, CA, 2015. (51) Charmaz, K. Qualitative Interviewing and Grounded Theory Analysis. In Handbook of Interview Research; Gubrium, J. F., Holstein, J. A., Eds.; SAGE Publications: Thousand Oaks, CA, 2001; pp 675−692. (52) Lincoln, Y. S.; Guba, E. G. Naturalistic Inquiry; SAGE Publications, Inc.: Newbury Park, CA, 1985.

K


A Comparison of How Undergraduates, Graduate Students, and

Recommend Documents