High Structure Active Learning Pedagogy for the Teaching of Organic

Article pubs.acs.org/jchemeduc

High Structure Active Learning Pedagogy for the Teaching of Organic Chemistry: Assessing the Impact on Academic Outcomes Michael T. Crimmins*,† and Brooke Midkiff‡ †

Department of Chemistry and ‡Department of Psychology, University of North Carolina at Chapel Hill, Chapel Hill, North Carolina 27599-3290, United States S Supporting Information *

ABSTRACT: Organic Chemistry is a required course for programs in chemistry, biology, and many health science careers. It has historically been considered a highly challenging course with significant failure rates. As with many science disciplines, the teaching of Organic Chemistry has traditionally focused on unstructured exposition-centered delivery of course material. This report details efforts to transform the teaching of a large section of Organic Chemistry to a more student-centered approach through the use of a highly structured, active learning format. In this study, the authors examine student performance data on homogeneous examinations and course grades for two groups of students at a large public university for comparison between those that received highly structured active learning pedagogy and those that received traditional, unstructured, lecture pedagogy in Organic Chemistry. Data consist of repeated cross sections over 4 different years, with a total sample size n = 766. Regression and propensity score matching (PSM) are used to analyze student academic outcomes. Results suggest that students exposed to a highly structured, active learning pedagogy in Organic Chemistry scored statistically significantly higher on total points earned and final exam scores, and had a higher probability of an increase in their final course grade. Results further suggest that while increased structure in conjunction with active learning improved outcomes generally for all students who received it, those students at the lowest academic achievement levels experienced the most gains. This study is significant in its causal analysis of the impact of highly structured active learning on academic outcomes. KEYWORDS: Organic Chemistry, Chemical Education Research, Collaborative/Cooperative Learning, Problem Solving/Decision Making, Learning Theories FEATURE: Chemical Education Research

■

INTRODUCTION Organic Chemistry is an important, required gateway twocourse sequence for students who wish to study chemistry, biology, and the health sciences such as medicine, dentistry, nutrition, and pharmacy, among others. Historically, it has been considered a daunting “weed out” course where many students see their hopes of pursuing a career in science either ended or greatly deterred. The large content volume and the complex conceptual nature of the material in the course contribute to the challenge for the typical science student. As such, exposition-centered approaches to teaching the subject have resulted in consistent, low learning gains for many students.1 At large public institutions, Organic Chemistry is typically taught in large theater classrooms where the instructor spends most of the in-class time transferring information to students in the form of standard lectures. There is little or no learning structure provided for the student in the way of formative assessments and feedback. The Department of Chemistry at the University of North Carolina at Chapel Hill (UNC-CH) used this instructional approach for Organic Chemistry for much of © XXXX American Chemical Society and Division of Chemical Education, Inc.

the past 35+ years (and likely much longer). Several years ago, a snapshot of combined D/F rates of a range of first and second year science courses at UNC-CH indicated a very high overall D/F rate (18% for all students and >30% for all underrepresented groups) for Organic Chemistry I. These shocking numbers, together with the anecdotal observations of generally low student performance, led to the evaluation of how organic chemistry, as well as other first and second year STEM courses on campus, might be redesigned to improve student learning outcomes.

■

ACTIVE LEARNING Active learning is generally understood to include any instructional method that actively engages students in the learning process while in the classroom.2 Some common components of active learning pedagogy include collaborative learning, Received: August 30, 2016 Revised: January 23, 2017

A

DOI: 10.1021/acs.jchemed.6b00663 J. Chem. Educ. XXXX, XXX, XXX−XXX

Journal of Chemical Education

Article

cooperative learning, and problem-based learning.2 In contrast, the banking theory of learning which depends on the narrator depositing information to the listener is passive for the learner.3 Active learning can take a variety of forms: flipped classrooms,4 studio format classrooms,5 workshops, group problem solving, tutorials or worksheets completed during class, guided reading questions, use of personal response systems with or without collaborative peer instruction (e.g., think−pair−share),6 just-intime teaching,7 and process-oriented guided-inquiry learning (POGIL).8 A recent report by Freeman that provided a meta-analysis of 225 studies compared the learning outcomes of classes using standard lecture versus active learning methods.9 The study concluded that average exam scores were 6% higher in active learning sections and that students were 1.5 times more likely to fail in a standard lecture format than in an active learning environment. The impressive results obtained using active learning methods in the physics community and more recently the biology community, both at the national level10,11 and locally at UNC,5,12 led to the investigation of the use of a more highly structured format incorporating active learning methods for the instruction of Organic Chemistry. Additionally, Freeman has reported that increased structure in the form of reading quizzes and extensive, Socratic lecturing, and extensive in-class active learning leads to substantial improvements in learning. This report details the methods utilized to incorporate substantially increased course structure in conjunction with active learning methods into Organic Chemistry I at UNC-CH and provides the results of those efforts in the context of evaluation of learning gains. Other reports have appeared that described the use of active learning methods in teaching Organic Chemistry,13−18 but only one has dealt with a large classroom setting, and none have focused on increasing course structure to improve student performance.18

■

an additional conceptual framework for unpacking social integration. Braxton, Milem, and Sullivan28 suggest that “...the college classroom represents one possible source of influence on social integration, subsequent institutional commitment, and college departure” (p 570), and that, as such, “...Faculty use of active learning practices constitute another possible source of influence on the college student departure process” (p 571). One early definition of active learning is any classroom activity that “involves students in doing things and thinking about the things they are doing”.29 Theoretically, this interaction generates a greater sense of community and social integration among students, thus increasing their likelihood of retention. The research presented here operates under the theoretical framework of active learning as a link in the chain of causes of social integration, and its subsequent link to positive student outcomes. We generalize from this broad theoretical base to, more specifically, theorize that active learning may result in positive academic outcomes for students in Organic Chemistry, over and above traditional lecture-format pedagogy. The redesign of the Organic Chemistry course was based on the model of constructivism in which students actively engage in the development of their knowledge, where “Knowledge is constructed in the mind of the learner.”30 Students take their pre-existing knowledge together with the information they are assimilating to construct their own knowledge base. By increasing course structure to include formative preclass assignments, students acquire some information related to the topic of the day prior to class. Thus, a highly structured course allows students to build more complex concepts upon those they already possess, and relate the new information to their existing knowledge. This approach reduces the intrinsic cognitive load during the class period, thus allowing the student to relate difficult concepts to the underlying principles of the topic without the need to assimilate all the new terminology and concepts during class.31 In addition, students accomplish these connections in the presence of the expert (instructor) who can help expose misconceptions as they assimilate the information, rather than students attempting to acquire the knowledge in more vacuous settings such as their dormitory or the library. Consequently, a high structure course, which includes active learning, allows the student to construct their knowledge much more efficiently and completely.

THEORETICAL FRAMEWORK

Course Structure

Haak19,20 recently reported on the effect of increasing course structure on student performance in an introductory biology course. Low structure is defined as a traditional lecture style course with 2−3 hour exams plus a final exam for evaluation. A moderate structure course is defined as utilizing in-class active learning such as clicker questions with a weekly short practice exam of five questions. A highly structured course adds preclass reading assignments with short, low stakes quizzes before class, and expands in-class activities to nearly eliminate traditional lecturing from the course.19,20 Increasing course structure to a highly structured level was observed to substantially increase student performance, reduce failure (D/F) rates, and reduce achievement gaps.19,20 Active learning, as a pedagogical practice, is undergirded by Tinto’s21 interactions list theory of college student departure. Tinto’s theory, widely referenced in the study of undergraduate retention, suggests that social integration, among other factors, is important for retention. Additional scholars have attempted to explain the social integration component of Tinto’s theory, suggesting that integration has to do with institutional type, organizational attributes, individual motivations for attending college, financial aid, fulfillment of college expectations, sense of community in residence halls, and student involvement.22−27 However, the role of the classroom in contributing to several of Tinto’s theoretical determinants of college retention presents

■

REDESIGNED COURSE FORMAT The introductory organic chemistry curriculum at UNC-CH consists of a two-course sequence that serves approximately 1000 students in each course during the regular academic year. The course is taught in a large lecture theater setting of approximately 225 students per section. Classes meet either three times per week for 50 min or twice per week for 75 min. The focus of this research was the first course in a 2-course sequence. The content is typical of a first course in organic chemistry and includes a review of bonding, structural theory, and acid base theory from general chemistry as well as nomenclature, organic structure, stereochemistry, electrophilic addition reactions, nucleophilic substitution reactions, and free radical reactions. Organic synthesis and detailed reaction mechanisms as well as the kinetics and thermodynamics of the reactions are included. The course is populated by chemistry majors, biology majors, and other students majoring in disciplines in preparation for health careers, such as medicine, dentistry, and pharmacy. Prior to implementation of the course B



Article

redesign, the format was one of a traditional low structure lecture course where students attended a lecture by the instructor and optionally worked selected problems from the textbook that were not graded or counted as part of their final grade. Other than their own personal motivation, there was no impetus for students to complete the textbook problems, and there was no requirement for students to assimilate any information before class since class was essentially for information transfer. The grade was determined by performance on three “hour exams” and a final exam. The instructor held standard office hours as well as a weekly hour long question and answer session outside regular class time. The modified course was designed as a high structure19,20 course, with students held accountable for material before class, during class, and after class. The redesigned course incorporated a combination of teaching and learning strategies detailed here.

Box 1. Learning Objectives for Substitution and Elimination • Predict the products of the reaction of various alkyl halides: (methyl, 1°, 2°, 3°, allylic, benzylic) with various nucleophiles/bases and predict which mechanistic type (SN1, SN2, E1, or E2) is governing the reaction • Interpret rates of reactions in the context of changes in concentration of alkyl halide or nucleophile/base for substitution and elimination reactions • Predict the relative strength of various nucleophiles/ bases • Predict the relative leaving group ability of various Lewis bases such as I−, Br−, Cl−, H2O, etc. • Predict the relative reactivity (rates of reaction) of various alkyl halides (methyl, 1°, 2°, 3°, allylic, benzylic) with various nucleophiles/bases • Draw reaction mechanisms (including transition states) for SN1, SN2, E1, and E2 reactions of alkyl halides and nucleophiles/bases using intermediates and arrows to show the movement of electrons • Interpret and draw reaction coordinate diagrams for SN1, SN2, E1, and E2 reactions including • Relative stability (energy) of reactants, products, intermediates, and transition states • Rate-determining step, fast steps, slow steps, reaction kinetics • Energy of activation, DG of reaction: exergonic, endergonic, exothermic, endothermic • Predict and explain the relative stability of carbocations • Predict the relative rates of substitution and elimination in solvents of different polarities • Predict the regiochemistry and stereochemistry of the products of substitution and elimination reactions upon exposure of various alkyl halides to different bases; predict how changing the conditions of the reaction (strength of base/nucleophile, solvent, temperature, concentration) can affect the outcome of the reaction • Intepret Newman projections with respect to which orientation would lead to E or Z alkenes upon E2 or E1 elimination of alkyl halides. • Predict the regiochemistry and stereochemistry of the products of substitution and elimination reactions upon exposure of various cyclic alkyl halides to different bases

Learning Objectives

Students were provided with a detailed list of learning objectives for each section of the course at the beginning of the semester to help guide their study. See the full learning objectives for the class in the Supporting Information. An example of representative learning objectives for nucleophilic substitution and elimination are shown in Box 1. Assessments were directly aligned with these objectives and with course content. Preclass Reading and Video Assignments

Students were instructed to watch one or more short videos and/or read specific sections of the textbook before coming to class. Students were provided with a detailed schedule of which textbook sections and/or videos would be covered during class and which problems in the textbook were relevant to the covered material. The videos were recorded by the instructor using screen-cast methods with Camtasia and a drawing tablet connected to a laptop. Slides were prepared and inserted as .png files one at a time into the drawing program Sketchbook Pro. The “mini-lecture” was delivered verbally while the slides were annotated using Sketchbook Pro as the lecturer spoke over the slides. The Sketchbook Pro artists program allows different pen styles and colors as well as on-screen editing. Slides were changed as needed and the “mini-lecture” proceeded. The videos were similar in nature to Khan Academy videos. Videos were generally from 5 to 15 min in length, and a total of 75 videos were prepared for the Organic Chemistry I course.

more on asking rather than telling in a way that facilitated student engagement.

Preclass Quizzes

Clicker Questions and In-Class Problem Sets

The students were held accountable for the preclass material by a required, short online quiz (generally 3−5 questions) from the textbook problem database. This online quiz was typically due before class, or a short quiz was administered using clickers at the beginning of class. These quizzes, combined with in-class clicker questions and postclass problem assignments in the textbook problem database, constituted about 12% of the total grade.

In class, a series of clicker questions and group problem sessions were presented with explanation of concepts before or after the question was posed. Typically, a clicker question was presented, and students were given time to think about the question and answer using classroom response systems such as Turning Technologies, i-clicker, or Poll Everywhere. If the responses were 70% correct or more, the instructor gave a brief explanation of the answer and the class proceeded. If less than 70% of the responses were correct, the class was given the opportunity to discuss the question with their immediate neighbors and the poll was retaken followed by the instructor’s explanation. This process was repeated throughout the class period with occasional short lectures between the clicker questions to clarify or discuss the connections between concepts. On some days, a more complex problem set was utilized

Socratic Lecturing

The instructor presented material and posed frequent questions to the class and solicited answers from students who raised their hand, cold-called on individuals, or solicited a group response through raising of hands. The purpose of the Socratic approach was to increase student attention and engagement and obtain feedback for the instructor. The format focused C



Article

for the class to engage in the material. The problem set was displayed on the projection screen or distributed prior to class for students to access. The students worked together on the problem set while the instructors and the in-class undergraduate mentors interacted with students by moving around the room to answer questions and pose questions to individual students. Weekly Self-Tests

At the end of each week of class, a longer required set of problems (typically 15−20 problems) was assigned on the textbook problem database that incorporated material from the entire week. Practice Problems and Practice Exams

A selected list of relevant problems was chosen from the textbook that students were encouraged to complete each week as they completed chapters in the text. Prior to the hour exams, students were provided with examples of exam-type questions that they were encouraged to work under exam-like conditions and then self-evaluate the results. This allows students to identify their strengths and weaknesses and focus their preexam study efforts in the proper areas.

Figure 1. High structure increases student engagement and accountability.

Therefore, the strength of this study is in its reliance on observed academic outcomes in response to the redesign of an Organic Chemistry course to a more highly structured format. The research questions for this study are as follows: 1. Did a pedagogical change to a highly structured format including extensive active learning during class impact student performance as measured by total points earned, final exam scores, and final course grades? 2. Were any impacts associated with the pedagogical change to a highly structured format including active learning differential between students performing at differential levels in Organic Chemistry (i.e., students in the bottom quartile compared to students in the top quartile)?

Assessments

Grades were determined by three standard hour exams and a final exam, each weighted 22% plus a daily work grade that was a combination of all the preclass, in-class, and postclass low stakes assignments for a total of 12%. The regular hour exams for the courses were a mix of multiple choice and short answer questions. None of the hour exams were identical from year to year, but the difficulty levels and subject content were comparable. The final exams were multiple choice and were essentially identical from year to year with only limited changes in a few questions. Examples of final exams are available from the author upon request. Thus, a highly structured approach was utilized where students were held accountable for their engagement with the information with low stakes formative assessments before class with a short problem set (quiz), during class with clicker questions and larger problems sets, and af ter class through an end of week review problem set. This arc of student engagement, along with accountability for their learning, provided for a weekly routine wherein students were accountable for being actively engaged with the curriculum to an increasing degree by the end of the unit. Whereas a traditional approach may only hold students accountable for their learning with a single exam af ter a substantial amount of instruction (i.e., the review problem sets or even with just an exam), this high structure format compels students to take an active role in learning at each step of the learning process throughout the unit (Figure 1).

■

DATA The observations used for analysis were drawn from 4 groups of students taking Organic Chemistry with the same professor. The first 2 groups occurred prior to the active pedagogy reform, during 2002 and 2003. The second 2 groups occurred after the reform, during 2013 and 2014. Despite the time gap, the instructor maintained test questions that were homogeneous in content and difficulty level among each group, and kept sufficiently detailed records for analysis. Fixed effects for time were used to mitigate any omitted variable bias due to the time lag. The data allowed for pooled cross-sectional regression analysis and propensity score matching (PSM).

■

METHODS The overall analytic strategy used in this study is grounded in regression, using ordinary least-squares regression (OLS) to determine correlations between the intervention (high structure active learning pedagogy) and student outcomes. The Supporting Information provides additional technical information on the statistical analyses. Quantile regression is used to determine differential effects of active learning across the range of student performance. Logistic regression is used to estimate the probability of a student who received active learning pedagogy having a higher final course grade than a student who received instruction in the traditional lecture format. Finally, propensity score matching (PSM) is used to examine a possible causal link between highly structured active learning and improved academic outcomes.

■

RESEARCH QUESTIONS The primary focus of this research was to determine if a highly structured course utilizing active learning in Organic Chemistry positively impacts student academic outcomes in the context of a large class (>200 students) taught in a lecture theater setting. While self-reported data from surveys can yield important information about the impact of active learning on student engagement and desire to continue in science generally and the field of chemistry specifically, observational data is necessary to examine whether or not a highly structured course using active learning methods is relevant to improving academic outcomes. D



Article

regression calculates the probability of a final grade changing (i.e., from a D to a C− or C, etc.) if the student was in the treatment group. We derived a binary variable (coded either 0 or 1) that indicated if a given student’s fitted value from an OLS (ordinary least-squares) regression for total points earned was higher than the maximum value of total points earned for each assigned letter grade. Logistic regression was used to calculate the impact of treatment on the derived binary variable, controlling for the same covariates used in the linear regression models (academic ability, race/ethnicity, gender, and firstgeneration student status).

The analytical approach represents breadth and depth in examining the relationship between a change in pedagogy and students’ academic outcomes. We use different measures of academic outcomes (total points earned, final exam grade, and probability of final course grade improving) to interrogate the practical significance of findings in each area of the analysis. In other words, the variety of statistical tests are robust to safeguard against reporting statistically significant findings that may not be of practical importance. Specifically, while improvements on final exams and total points earned are a positive academic outcome, they may be less important if students still end up receiving a lower course grade. Because final course grades have a large impact on students’ abilities and preferences to pursue a STEM major or go on to a higher level STEM course for which this gateway course is required, final course grades as an outcome for analysis have pragmatic significance. However practically significant this area of the analysis may be, though, the causal analysis presented here is of great import as it allows us to determine if the correlation between active learning pedagogy causes improved academic outcomes. For these reasons, the analytic strategy presented here covers three academic outcome measures (dependent variables) using quantile regression, logistic regression, and propensity score matching. Box 2 provides a list of variables used within the analyses.

■

PROPENSITY SCORE MATCHING Propensity score matching was used to match and compare students between the treatment and comparison groups based on these observable characteristics to control for differences in these factors. PSM was used to analyze students’ total points earned and final exam grades. While it is useful and important to know that active learning may be associated with improved student achievement, as measured by total points earned or final exam scores, the significant contribution of this analysis is that it provides a causal analysis. In order to draw causal inferences, data must be collected within a randomized framework (randomized control trial) or must meet the necessary assumptions of an appropriate quasiexperimental research design. In the study described herein, students were not randomly assigned to either receive the intervention or not: students in certain cohorts received it while students in other cohorts did not. In quasiexperiments, counterfactual thinking is used to achieve a condition of “like randomness” in order to draw causal inferences.37 We use propensity score matching (PSM) in a quasiexperimental design to target the causal impact of active learning on academic outcomes. PSM balances observable characteristics within each group (treatment and comparison) to estimate the probability of being in the treatment group based on observable characteristics.38 It is preferred to exact matching because it overcomes what is known as “the curse of dimensionality”, that the data needed for reliable estimation increases exponentially when dimensions increase, and because it offers less biased estimates than exact matching.39,40In this analysis, the available covariates for PSM matching included SAT Math and Verbal scores, which serve as a proxy for individual baseline ability upon beginning the course. The other available, observable covariates included demographics such as race/ethnicity, gender, and firstgeneration college student status.

Box 2. Variables Considered in the Statistical Analysis Independent Variables Race/ethnicity Gender identity First-generation college student status SAT math score SAT verbal score Dependent Variables Total points earned Final grade Final exam grade SAT scores were included as covariates to control for overall academic ability, independent of differences in instructional strategy. Total SAT scores correlate with all three dependent variables at R2 = 0.26−0.27. Disaggregated by quantitative and verbal scores, SAT scores correlate with all three dependent variables R2 = 0.28−0.29 for SAT math scores and R2 = 0.17−0.18 for SAT verbal scores.

■

■

QUANTILE REGRESSION Quantile regression was used to delineate the distribution of gains in total points earned as well as final exam grades. Quantile regression is commonly used in educational settings due to the interest researchers and educators have in assisting students who are performing below average or poorly in comparison to their peers.32−36 Simultaneous quantile regression was used to assess the impact of high structure pedagogy across the distribution of total points earned and final exam scores at the 25th, 50th, and 75th percentiles.

DESCRIPTIVE STATISTICS The data consist of four years of cross-sectional data on student demographics and outcomes in Organic Chemistry, taught by the same professor using the same database of test questions. Summary statistics for both groups are given in Table 1. These include the total number of points earned in the class (from exams and any other assignments), final exam scores, final grades students received in the course, and total SAT scores (including quantitative and verbal scores). Final exam scores are percentages (i.e., 64.52%), and final grades are on the 10-point scale (i.e., 5.84 out of 10). Of note in the summary statistics, students in the comparison group earned an average grade of 5.8 (C+), while students in the postintervention group earned an average grade of

■

LOGISTIC REGRESSION Logistic regression was used to inspect the possible impact of the reform pedagogy on students’ final grades. Logistic E



■

TOTAL POINTS EARNED Table 2 provides the results of the regression model.6 The quantile regression analysis suggests that while students in

Table 1. Summary Statistics for Student Outcomes in Organic Chemistry Mean

Standard Deviation

Minimum

Variable Comparison Group, N = 371 Total points earned 203.82 36.84 73 Final gradea 5.84 2.89 1 Final exam scoresb 64.52 13.08 29 SAT quantitative 652.96 64.82 440 SAT verbal 618.32 76.37 390 Treatment Group, N = 395 Total points earned 229.23 35.90 61 Final gradea 6.28 2.69 1 Final exam scoresb 74.66 12.82 33 SAT quantitative 670.48 69.21 350 SAT verbal 637.29 75.90 350 a

Final grades are on the 10-point scale. percentages.

b

Article

Maximum

Table 2. Quantile Regression Analysis for Increase in Students’ Total Points Earned (N = 766)

288 10 92 800 800

Variable Effect on the treatment Group Confidence interval

296 10 98 800 800

a

25th Percentile

50th Percentile

75th Percentile

30.65a (6.12) 18.63−42.67

29.59a (4.85) 20.07−39.09

23.10a (3.81) 15.61−30.57

p < 0.01.

every quantile experienced gains in points earned after the pedagogy reform, students in the bottom quantile experienced higher gains than any other group (30.65 points compared to 29.59 or 23.10). Tests of equality of the coefficients for each quantile showed that they are statistically significantly different between groups as well, meaning that students in the bottom 25th percentile had a gain, on average, of 31 points on their total points earned associated with the active learning intervention and that their 31-point gain is statistically significantly higher than the learning gains of students in the 50th and 75th percentiles.

Final exam scores are

6.28 (B−). While the increase in the final grade is important, the reduction in the standard deviation is also important when understanding the possible impact of the intervention as simple grade inflation over time could reasonably contribute to improved grades. However, between the pre- and postintervention groups, the standard deviation for final grades went down from 2.89 to 2.69, suggesting that the spread of final grades shrank somewhat in the postintervention group, in addition to grades simply improving. Students’ SAT scores served as a proxy for academic ability. That is, some portion of students’ achievement in Organic Chemistry is attributable to academic ability coming into the course, rather than instruction and engagement with the curriculum. With inclusion of SAT scores as covariates in the regression models, the portion of academic achievement attributable to individual academic ability for each student is accounted for. This is true for both the treatment and comparison groups, despite differences in their mean SAT scores. Further, mean differences between the two groups in SAT scores do not impact the propensity score matching analysis; despite differences in SAT scores between the two groups, adequate matching between groups without replacement was still achieved. This means that there were enough students in both groups with similar enough SAT scores on which to match. Finally, there is a time difference between the treatment and comparison group as well. Differences attributable to the passage of time between the two groups are accounted for using a time fixed effect. This allows the model to control for unobservable possible effects on the outcomes of interest associated with the year in which the course was taught. Differences between students in attitudes, use of technology, etc., that are attributable to the passage of time are controlled for through the time fixed effect portion of the regression models.

■

PROPENSITY SCORE MATCHING ON TOTAL POINTS EARNED The PSM analysis shows that the average treatment effect on the treated (ATT) for students was an increase of 22.7 points on their total points earned, statistically significant at the 0.01 level. This suggests that, on average, students who received active learning pedagogy earned approximately 23 additional points due to the highly structured format, controlling for other factors such as academic ability, race/ethnicity, gender, and first-generation student status. Table 3 shows the average treatment effect on the treated (ATT) on total points earned. Table 3. Average Treatment Effect for Total Points Earned (N = 766)

a

Additional Points Earned

95% Confidence Level

22.70a (5.72)b

(14.92−30.47)

Points. bp < 0.01.

■

FINAL EXAM GRADES Table 4 provides the results of the regression analysis. The quantile regression is similar to that of total points earned in that students in the 25th percentile saw the largest increase Table 4. Quantile Regression Analysis for Students’ Final Exam Grades (N = 765)

■

FINDINGS Overall, the findings of the combined analyses suggest that all students benefited from the highly structured format in Organic Chemistry, and that those students in the 50th and 25th percentiles benefitted the most, on total points earned, final exam grade, and the likelihood of earning a higher final grade.

Variable Effect on treatment group Confidence interval a

F

25th Percentile

50th Percentile

75th Percentile

11.78a,b

10.49a

8.59a

(1.57) 8.71−14.85

(1.17) 8.20−12.78

(1.35) 5.94−11.23

Points. bp < 0.01 standard deviation in parentheses. DOI: 10.1021/acs.jchemed.6b00663 J. Chem. Educ. XXXX, XXX, XXX−XXX


Article

(11.78) on their final exams associated with experiencing active learning in Organic Chemistry. However, students in the 50th percentile saw very similar gains associated with the reformed pedagogy. Tests of equality of the coefficients for students in the 25th percentile and those in the 50th show that they are not statistically significantly different from zero. The PSM analysis given in Table 5 shows that the ATT for students was an increase of 10.22 points on their final exam

statistically significant at the 0.01 level. The odds ratio is 1.71, meaning that, for a one-unit increase in treatment (in other words, experiencing active learning pedagogy), we expect a 1.71 increase in the log-odds of the final grade increasing to the next level of assigned letter grade, holding all other independent variables constant. Given that approximately 40% of students received a “C” or below in the pretreatment group, further analysis was conducted to examine the odds of students in the lower end of the grade distribution experiencing a grade change. Table 8 shows

Table 5. Average Treatment Effect for Final Exam Scores (N = 765)

a

Additional Points Earned

95% Confidence Level

10.22a (7.39)b

(7.51−12.93)

Table 8. Odds Ratio of Change in Final Grade if Student Received “C” or Lower (N = 269)

b

p < 0.01. Points standard deviation in parentheses. a

scores, statistically significant at the 0.01 level. This suggests that, on average and all else being equal, students enrolled in the highly structured course using active learning pedagogy earned approximately 10 additional points on their final exams, due to the redesigned course elements.

COURSE GRADES The distribution of final grades was non-normal and was not amenable to standard transformation procedures. To better understand the role of the statistically significant increase in total points earned, though, we examine the relationship of total points earned to students’ earned letter grades. Specifically, the following analysis examines the role of increases in total points earned to changes in assigned final letter grade. Table 6 shows

Comparison Group Final grade “C” or lower, N 151 Total in students in group, N 371 Students receiving final grade of “C” or 41 lower, % Treatment Group Final grade “C” or lower, N 118 Total in students in group, N 395 Students receiving final grade of “C” or 30 lower, %

Standard Deviation

2.80

1.03

2.85

0.98

p < 0.01 standard deviation in parentheses.

EFFECT SIZES The findings suggest that when a highly structured format using extensive active learning is used in Organic Chemistry, all students achieve higher scores based on total points earned, final exam scores, and final grades in the course. Common measures of effect sizes include Cohen’s d and Glass’s Δ, derived from mean differences. These standardized measures of effect size allow for comparison to other studies finding statistically significant effects of active learning on academic outcomes. Table 9 provides the effect sizes from comparison of the means of total points earned and final exam score. Table 9. Mean Comparison Effect Sizes for Total Points and Final Exam Scores Parameters

the percentage of students receiving “C” or below in both the pre- and postperiods. There was an overall decrease in students receiving grades “C” or below by about 10% in the treatment group. Table 7 provides the results of the logistic regression analysis, and suggests that students exposed to the reformed pedagogy had a greater likelihood of experiencing a letter grade change,

Cohen’s d Glass’ Δ Cohen’s d Glass’ Δ a

Table 7. Odds Ratio of Change in Students’ Final Grade (N = 766)

a

6.804a (6.35)

■

Table 6. Comparison of Students Receiving a Grade of “C” or Lower Pre- and Post-Treatment Mean

Odds Ratio

Treatment

that if a student whose grade was “C” or below experienced the active learning pedagogy, her/his log odds of having a higher assigned final letter grade is 6.804, statistically significant at the 0.01 level, holding all other covariates constant. This supports the quantile regression analysis that suggests that students in the lower end of the distribution of total points earned were associated with the highest increase in total points earned when exposed to active learning pedagogy in Organic Chemistry. Similar to the results of the quantile regression analysis, students at the bottom end of the grade scale had a higher probability of their final grade increasing to a higher letter grade in association with active learning pedagogy.

■

Observed

Student Group

Student Group

Odds Ratio

Treatment

1.71a (2.99)

Effect Sizea

95% Confidence Interval

Total Points Earned 0.70 0.71 Final Exam Score 0.78 0.79

0.55−0.84 0.56−0.86 0.64−0.93 0.64−0.94

Small = 0.20, medium = 0.50, large = 0.80.

Both Cohen’s d and Glass’s Δ effect sizes suggest a moderate to high practical significance of active learning pedagogy on student achievement on total points earned and final exam scores. Other measures of effect size are derived from the regression analysis, taking into account the covariates in the regression model. These include eta-squared (η2) and omega-squared (ω2).

p < 0.01 standard deviation in parentheses. G



Article

perhaps a random-effects model may yield less biased estimates. However, this analytic approach would require more than one instructor to statistically compare a model that uses fixed instructor effects to a model using random instructor effects. In either scenario, more than one instructor would be necessary to statistically test for instructor-specific effects. In the end, with the data available, we are unable to disentangle the year/cohort effects from instructor effects attributable to possible withinperson changes across the two time periods. We view this as a limitation of the study that is attributable to the limitations of the data available for analysis. As with many program evaluations, the analysis is inherently limited by the data at hand. Students in the treatment cohort, due to advances in technology and available course-study materials, likely had greater access to study materials via the Internet and tools such as Course Hero. However, beyond time fixed effects, there is no data available to control for individual-level differences in use for external study materials such as Course Hero. The analysis presented here would be strengthened if such data were available; however, the time fixed effects do mathematically absorb the variance associated with the passage of time. As with all fixed effects though, this fixed portion cannot be decomposed to further understand exactly what aspects of the passage of time impacted the outcomes of interest. That is, we cannot know if increasing access to external study materials over time impacted student learning, but we can control for confounding effects overall.

Omega-squared effect sizes are provided for comparison as they present a more conservative estimate. The findings presented in Table 10 suggest that both measures of effect based on the regression analysis are moderate for total points earned and final exam scores. Table 10. Regression Analyses Effect Sizes for Total Points Earned and Final Exam Scores Parameters η2 ω2 η2 ω2

Effect Size

95% Confidence Interval

Total Points Earned Effect Measure 0.10a 0.07−0.15 0.10b 0.07−0.14 Final Exam Score Effect Measure 0.13a 0.09−0.18 0.13b 0.09−0.18

a Small = 0.02, medium = 0.13, large = 0.26. medium = 0.06, large = 0.14.

b

Small = 0.01,

Previous educational research has denoted that eta-squared values around 0.20 are of policy interest when they are based on measures of academic achievement.41 While the effect of receiving active learning pedagogy presented only moderate effect sizes in an overall regression, the middle and bottom quantile regression models had η2 values that were much higher (0.18 for total points earned and 0.21 for final exam scores). From a policy perspective, then, the results suggest that educational reforms to increase the use of high structure active learning pedagogy in Organic Chemistry have the potential for a significant, positive impact on student learning.

■

IMPLICATIONS: CHEMISTRY LEARNING The effect sizes found are in line with previous research on the use of active learning in STEM classes.2,9,45 This research specifically suggests that high structure active learning is a powerful method for improving student academic outcomes in Organic Chemistry, a course that is commonly a STEM major required course across a variety of colleges and universities. Given that the findings show that students in the bottom 25th percent of the class generally experienced greater academic improvement associated with active learning pedagogy, it is likely that this method could substantially decrease the exit of students from STEM fields. Moreover, this has implications for equity and diversity as students who intend to major in a STEM field but who end up leaving STEM fields for other college majors are disproportionately students of color, from low socioeconomic backgrounds, and first-generation students.46 By stemming the tide of students who are steered out of STEM fields by Organic Chemistry classes and other traditional “weed out” classes, active learning presents an opportunity to disassemble the societal barriers that have been prohibitive for increasing diversity in STEM fields.

■

LIMITATIONS The research presented here does not yet fully encapsulate the effects of high structure active learning in Organic Chemistry on students. Future research should include longitudinal data, along with more in-depth observable data on students. For example, if data were available for students from both cohorts on their intended major versus their actual major, along with graduation data, and pursuit of a graduate degree, researchers could begin to better understand possible long-term impacts of active learning in science gateway courses such as Organic Chemistry, including if it may affect generally students’ pursuits of scientific inquiry. Additionally, due to the homogeneity of the instructor and the assessments used, the analysis of student data was able to yield causal results through propensity score matching, which has been shown to be the most effective quasiexperimental design when compared to a randomized control trial, and to date a randomized control trial for the study of active learning in college Organic Chemistry has yet to be published.42,43 On the basis of the research standards of the What Works Clearinghouse for educational research, though, a randomized control trial remains the gold standard, and this would be a highly desirable research design for a future study.44 The continuity of the instructor presents a lower variance in outcome measures due to instructor differences than if the comparison group had been taught by a different instructor. If that had been the case, a fixed effect would likely be necessary to account for the unobservable variance associated with individual instructors. However, another perspective may suggest that the one instructor, given the time between the two groups of students, may have changed significantly in enthusiasm or beliefs about student learning. This perspective suggests that

■

IMPLICATIONS: RESEARCH While this research suggests that a highly structured format using active learning improves academic outcomes among students, future research should investigate the link between active learning, social integration, and the more distal outcome of college retention. Specifically, longitudinal study of cohorts of students who receive active learning instruction versus traditional, lecture-based instruction could shed light on both observed outcomes such as graduating in a STEM field, college GPA over time, and STEM course-taking, but also on other factors closely identified with predicting retention such as H



Article

student performance in science, engineering, and mathematics. Proc. Natl. Acad. Sci. U. S. A. 2014, 111, 8410−8415. (10) Deslauriers, L.; Schelew, E.; Wieman, C. Improved Learning in a Large-Enrollment Physics Class. Science 2011, 332, 862−864. (11) Montelone, B. A.; Rintoul, D. A.; Williams, L. G. Assessment of the Effectiveness of the Studio Format in Introductory Undergraduate Biology. CBE-Life Sci. Educ. 2008, 7, 234−242. (12) Eddy, S. L.; Hogan, K. A. Getting Under the Hood: How and for Whom Does Increasing Course Structure Work? CBE-Life Sci. Educ. 2014, 13, 453−468. (13) Bradley, A. Z.; Ulrich, S. M.; Jones, M.; Jones, S. M. Teaching the Sophomore Organic Course without a Lecture. Are You Crazy? J. Chem. Educ. 2002, 79, 514−519. (14) Paulson, D. R. Active Learning and Cooperative Learning in the Organic Chemistry Lecture Class. J. Chem. Educ. 1999, 76, 1136− 1140. (15) Fautch, J. M. The flipped classroom for teaching Organic Chemistry in small classes: is it effective? Chem. Educ. Res. Pract. 2015, 16, 179−186. (16) Rossi, R. D. ConfChem Conference on Flipped Classroom: Improving Student Engagement in Organic Chemistry Using the Inverted Classroom Model. J. Chem. Educ. 2015, 92, 1577−1579. (17) Dinan, F. J.; Frydrychowski, V. A. A Team Learning Method for Organic Chemistry. J. Chem. Educ. 1995, 72, 429−431. (18) Flynn, A. B. Structure and evaluation of flipped Chemistry courses: Organic and spectroscopy, large and small, first to third year, English and French. Chem. Educ. Res. Pract. 2015, 16, 198−211. (19) Freeman, S.; Haak, D.; Wenderoth, M. Increased course structure improves performance in introductory biology. CBE Life Sci. Educ. 2011, 10, 175−186. (20) Haak, D. C.; HilleRisLambers, J.; Pitre, E.; Freeman, S. Increased Structure and Active Learning Reduce the Achievement Gap in Introductory Biology. Science 2011, 332, 1213−1216. (21) Tinto, V. Leaving College: Rethinking the Causes and Cures of Student Attrition, 1st ed.; University of Chicago Press, Chicago; 1987. (22) Guiffrida, D. A. Toward a cultural advancement of Tinto’s theory. Rev. Higher Ed. 2006, 29, 451. (23) Yi, P.-s. Institutional climate and student departure: A multinomial multilevel modeling approach. Rev. Higher Ed. 2008, 31, 161. (24) Pike, G. R.; Hansen, M. J.; Childress, J. E. The Influence of Students’ Pre-College Characteristics, High School Experiences, College Expectations, and Initial Enrollment Characteristics on Degree Attainment. J. College Student Retention: Res. Theory and Practice. 2014, 16, 1−23. (25) Stephens, N. M.; Brannon, T. N.; Markus, H. R.; Nelson, J. E. Feeling at Home in College: Fortifying School-Relevant Selves to Reduce Social Class Disparities in Higher Education. Social Issues and Policy Review 2015, 9, 1−24. (26) Valentine, J. C.; Hirschy, A. S.; Bremer, C. D.; Novillo, W.; Castellano, M.; Banister, A. Banister, A. Keeping at-risk students in school: A systematic review of college retention programs. Educational Evaluation and Policy Analysis 2011, 33, 214−234. (27) The Pell Institute for the Study of Opportunity in Education. Moving beyond AccessCollege Success for Low-Income, First-Generation Students. http://www.pellinstitute.org/downloads/publicationsMoving_Beyond_Access_2008.pdf (accessed Jan 2017). (28) Braxton, J. M.; Milem, J. F.; Sullivan, A. S. The influence of active learning on the college student departure process: Toward a revision of Tinto’s Theory. J. Higher Educ. 2000, 71, 569. (29) Bonwell, C. C.; Eison, J. A. Active Learning: Creating Excitement in the Classroom; The George Washington University, School of Education and Human Development: Washington, DC, 1991; Report No. 1. (30) Bodner, G. M.; Klobuchar, M.; Geelan, D. The many forms of constructivism. J. Chem. Educ. 2001, 78, 1107. (31) Seery, M. K.; Donnelly, R. The implementation of pre-lecture resources to reduce in-class cognitive load: A case study for higher education Brit. J. Educ. Technol. 2012, 43, 667−677.

student institutional commitment, fulfillment of college expectations, and sense of community.

■

ASSOCIATED CONTENT

S Supporting Information *

The Supporting Information is available on the ACS Publications website at DOI: 10.1021/acs.jchemed.6b00663. Distributions of the data and regression analysis (PDF, DOCX) Learning objectives (PDF, DOCX)

■

AUTHOR INFORMATION

Corresponding Author

*E-mail: [email protected]. ORCID

Michael T. Crimmins: 0000-0003-0313-6010 Notes

The authors declare no competing financial interest. Additional figures and tables demonstrating the distributions of outcomes and the results of analysis by subgroup are available upon request. Subgroup analysis by under-represented minority status, first-generation college student status, and gender were conducted. A representative final exam is available from the corresponding author upon request.

■

ACKNOWLEDGMENTS The authors acknowledge Jennifer L. Krumper, Kelly A. Hogan, and Laurie J. McNeil for helpful discussions. Assistance from Jennifer L. Krumper with preparation of preclass videos is also acknowledged. The authors acknowledge generous financial support from the Association for American Universities sponsored by the Leona M. and Harry B. Helmsley Charitable Trust. Finally, authors acknowledge the editor and anonymous reviewers of the journal for their helpful insights and comments.

■

REFERENCES

(1) Grove, N. P.; Hershberger, J. W.; Bretz, S. L. Impact of a spiral Organic curriculum on student attrition and learning. Chem. Educ. Res. Pract. 2008, 9, 157−162. (2) Prince, M. Does active learning work? A review of the research. J. Eng. Educ. 2004, 93, 223−231. (3) Hooks, B. Teaching To Transgress: Education as the Practice of Freedom; Routledge: New York, 1994. (4) Seery, M. K. Flipped learning in higher education Chemistry: emerging trends and potential directions. Chem. Educ. Res. Pract. 2015, 16, 758−768. (5) Sorensen, C. M.; Churukian, A. D.; Maleki, S.; Zollman, D. A. The New Studio format for instruction of introductory physics. Am. J. Phys. 2006, 74, 1077−1082. (6) Mazur, E. Peer Instruction: A User’s Manual; Prentice Hall: Upper Saddle River, NJ, 1997. (7) Mazur, E. Interactive Teaching: Promoting Better Learning Using Peer Instruction and Just-in-Time Teaching; Prentice Hall: Upper Saddle River, NJ, 2004. (8) Moog, R. S.; Creegan, F. J.; Hanson, D. M.; Spencer, J. N.; Straumanis, A.; Bunce, D. M.; Wolfskill, T. In POGIL: ProcessOriented Guided-Inquiry Learning. Pienta, N. J., Cooper, M. M., Greenbowe, T. J., Eds.; Chemists’ Guide to Effective Teaching; Prentice Hall: Upper Saddle River, NJ, 2009; Vol. II, pp 90−107. (9) Freeman, S.; Eddy, S. L.; McDonough, M.; Smith, M. K.; Okoroafor, N.; Jordt, H.; Wenderoth, M. P. Active learning increases I



Article

(32) Houck, E. A. Intradistrict resource allocation: Key findings and policy implications. Educ. Urban Soc. 2011, 43, 271−295. (33) Alejo, J.; Gabrielli, M. F.; Sosa-Escudero, W. The distributive effects of education: An unconditional quantile regression approach. Economic Analysis Review 2014, 29, 53−76. (34) Castellano, K. E.; Ho, A. D. Contrasting OLS and quantile regression approaches to student “growth” percentiles. J. Educ. Behav. Stat. 2013, 38, 190−215. (35) Martins, P. S.; Pereira, P. T. Does education reduce wage inequality? Quantile regression evidence from 16 countries. Labour Economics 2004, 11, 355−371. (36) Eide, E.; Showalter, M. H. The effect of school quality on student performance: A quantile regression approach. Economics letters 1998, 58, 345−50. (37) Morgan, S. L.; Winship, C. Counterfactuals and Causal Inference: Methods and Principles for Social Research; Cambridge University Press: New York, 2007. (38) Rosenbaum, P. R.; Rubin, D. B. The Central Role of the Propensity Score in Observational Studies for Causal Effects. Biometrika 1983, 70, 41−55. (39) Itang’ata, M. J. A Comparative Study of Exact Versus Propensity Matching Techniques Using Monte Carlo Simulation. Dissertation, Western Michigan University, Kalamazoo, MI, 2013. (40) Stuart, E. A. Matching Methods for Causal Inference: A Review and a Look Forward. Statistical Sci. 2010, 25, 1−21. (41) Hedges, L. V.; Hedberg, E. C. Interclass Correlation Valuses for planning group-randomized trials in education. Educ. Eval. Policy Anal. 2007, 29, 60−87. (42) Dehejia, R. H.; Wahba, S. Propensity score-matching methods for nonexperimental causal studies. Rev. Econ. Stat. 2002, 84, 151−161. (43) Bertrand, M.; Duflo, E.; Mullainathan, S. How much should we trust differences-in-differences estimates? Q. J. Econ. 2004, 119, 249− 275. (44) Seftor, N.; Constantine, J.; Cody, S.; Ponza, M.; Knab, J.; Deke, J.; Monahan, S. What Works Clearinghouse: Procedures and Standards Handbook 2011 (NCEE 2011-XXXX); National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education: Washington, DC, 2011. (45) Cahill, M. J.; Hynes, K. M.; Trousil, R.; Brooks, L. A.; McDaniel, M. A.; Repice, M.; Zhao, J. Q.; Frey, R. F. Multiyear, multi-instructor evaluation of a large-class interactive-engagement curriculum. Physical Review Special Topics-Physics Education Research 2014, 10, 020101. (46) Chen, X.; Soldner, M. STEM Attrition: College Students’ Paths into and out of STEM Fields. Report No.: NCES 2014-001; NCES, IES, U.S. Department of Education: Washington, DC, 2013. http://nces. ed.gov/pubsearch/pubsinfo.asp?pubid=2014001rev (accessed Jan 2107).

J


High Structure Active Learning Pedagogy for the Teaching of Organic

Recommend Documents