Article Cite This: J. Chem. Educ. XXXX, XXX, XXX−XXX
pubs.acs.org/jchemeduc
Improving College Student Success in Organic Chemistry: Impact of an Online Preparatory Course Christian Fischer,*,† Ninger Zhou,† Fernando Rodriguez,† Mark Warschauer,† and Susan King‡ †
School of Education, University of California, Irvine, 3200 Education, Irvine, California 92697-5500, United States Department of Chemistry, University of California, Irvine, 2133 Natural Sciences II, Irvine, California 92697-2025, United States
J. Chem. Educ. Downloaded from pubs.acs.org by UNIV AUTONOMA DE COAHUILA on 04/18/19. For personal use only.
‡
ABSTRACT: This quantitative study examines the impact of a three-week online organic preparatory course for chemistry undergraduates that is designed to improve student performance in the subsequent organic chemistry course series (N = 1,289). Organic chemistry often serves as a gatekeeper for students pursuing careers in science, technology, engineering, or mathematics (STEM). Because many students are underprepared for the rigorous organic chemistry series, and consequently are at greater risk of failing it, an online preparatory course was offered that emphasized topics that students frequently struggle with when they enter organic chemistry. The average treatment effects of participation in the online preparatory course on students’ subsequent organic chemistry course grades were analyzed utilizing inverse-probability weights with regression adjustment. The analyses indicate that participation in the online preparatory course led to an improvement in subsequent organic chemistry course performance of approximately one-third of a letter grade (e.g., C+ to B−). Notably, students typically at-risk in college environments (i.e., low-income students, first-generation college students, underrepresented minorities) showed commensurate gains when compared to their non-at-risk counterparts. Consequently, this study provides an example of a low-cost intervention that can increase student learning and achievement in organic chemistry. In addition, this study contributes to the nascent research base that examines more distal effects of online course participation. KEYWORDS: Chemical Education Research, Second-Year Undergraduate, Internet/Web-Based Learning, Computer-Based Learning, Testing/Assessment, Organic Chemistry, Lewis Structures, Resonance Theory FEATURE: Chemical Education Research
■
INTRODUCTION Organic chemistry has a reputation as a challenging subject.1−3 The content is broad, detailed, and heavily conceptual, and many students are unprepared for the difficulty level of this rigorous gateway course. Successful completion of organic chemistry is required for students majoring in chemistry, biology, pharmaceutical sciences, public health science, or chemical engineering, yet failure rates for this class are high, sidelining many students who aspire to pursue a career in science, technology, engineering, or mathematics (STEM).4−7 Students entering organic chemistry after successful completion of general chemistry have varying levels of preparedness. With an Atoms-First general chemistry series,8,9 the first term of general chemistry covers most of the prerequisite topics for entry into organic chemistry. Depending on Advanced Placement (AP) credit transfer policies of departments and universities, students with an AP Chemistry test score of 4 or 5 might bypass all or some of these general chemistry courses. As the quality of instruction, covered topics, and adopted teaching practices might vary across students’ high school AP Chemistry courses,10 students who rely on AP Chemistry may have different levels of preparedness for © XXXX American Chemical Society and Division of Chemical Education, Inc.
organic chemistry. In addition, the time lag between when students take AP Chemistry in high school and the first term of organic chemistry in college can pose a problem. Retention of the necessary content is often limited when students do not encounter organic chemistry until the beginning of their sophomore year of college. Other students, beyond those with AP credits, can also face challenges in chemistry, such as students who performed poorly in their general chemistry courses, or who faced a time gap between their general chemistry and organic chemistry courses. The transition from general to organic chemistry is not only a problem for many students but also an administrative problem, as underprepared students who take the sophomore organic chemistry series and earn failing grades create bottlenecks in the year-long sequence. The need for students to retake courses in the organic chemistry series generates a large demand for off-sequence course offerings. These additional courses pose substantial logistical challenges to the Received: December 6, 2018 Revised: March 22, 2019
A
DOI: 10.1021/acs.jchemed.8b01008 J. Chem. Educ. XXXX, XXX, XXX−XXX
Journal of Chemical Education
Article
and significantly lower withdrawal and failure rates.25 Such findings suggest the potential benefits of online learning materials to enhance student achievement and retention.32 Another study explored the use of video podcasts in the form of video podcast tutorials that focus on challenging chemistry concepts.15 These video podcasts consisted of the instructor’s voice and handwriting, demonstrated the problem-solving process, and elaborated on the challenging concepts.15 Students’ pre/postexam outcomes suggested that these video podcasts contributed to improvements in student performance.15 Online learning through video podcasts has also been used in preparatory learning activities for chemistry courses. For instance, researchers examined the use of online prelaboratory learning tools in an organic chemistry class.29 The videos were created in a format similar to the video podcasts: instructors used voice-over on PowerPoint slides to elaborate on the laboratory setup and procedures.29 The students reported that the prelaboratory videos helped them with the attitudes toward and the understanding of the experiments.29 Although online learning can provide students with meaningful learning experiences, students often do not perform as well in online settings compared to traditional face-to-face courses.33−36 In particular, students who are traditionally at-risk in college environments face even greater challenges in online learning than their non-at-risk counterparts.37,38 These differences in course performance across course modalities, and student populations, are usually explained by the increased need in fully online courses for students to self-regulate their learning experience and to have appropriate levels of prior knowledge to learn independently.39−41 While prior research has extensively examined these near effects of online course participation (i.e., course grades in current course), analyses of more distal effects of online course participation (i.e., course grades in subsequent courses) are less common. There has been even less research on the connection between the use of preparatory online courses by at-risk students and more distal outcomes. Given the challenging nature of organic chemistry and its essential role in the STEM trajectory in postsecondary education, there is a value in understanding the relationship between participation in online organic preparatory courses and longer-term performance, especially for students traditionally at-risk in college environments. Consequently, this study will examine the impact of a preparatory organic chemistry online course on students’ subsequent course performance.
department as personnel and space capacity for the simultaneously offered laboratory sessions are limited. To address these problems, an online three-week organic chemistry preparatory course was developed, with the goal of making organic chemistry more accessible to underprepared students and helping students begin the year-long organic chemistry series with sufficient preparation and confidence.
■
ONLINE LEARNING IN HIGHER EDUCATION Online instruction has continually expanded in higher education settings. In 2014, more than a quarter of college students enrolled in at least one online course in the United States.11 This growth is mirrored by higher education administrators increasingly recognizing the potential of online courses to not only provide students with cost-efficient instruction for high-volume courses but also increase the depth and breadth of course offerings to students.12−14 Recent developments in online learning tools have enriched the modalities through which students acquire knowledge in higher education settings. Among others, video podcasts that integrate audio and visual modalities have gained popularity in online learning in a variety of disciplines.15−17 Some prominent formats of video podcasts include instructors producing tutorials for challenging concepts by utilizing screen recordings, voice-overs, and annotations.15,16 Previous research has suggested several benefits for using podcasts in higher education, such as promoting student motivation and performance.16,18−20 For example, in one study, 191 undergraduate students from several majors reported positive motivational outcomes from using online podcasts.18 This research has shown that instructors can convey their social presence through podcasts.16,18 Students may find podcasts more personal and relatable compared with seeing only a collection of imagery and textual information. In addition, podcasts allow students to learn about the material regardless of the restrictions on timing and locations.16,18 In particular, video podcasts have been shown to be advantageous as preparatory learning material that students can interact with either prior to formal classes or alongside formal instruction.16 In one study, students in an entry-level college math class reported that video podcasts which focused on math problem solving helped them to learn new content during out-of-class self-study.21 In summary, video podcasts have shown promise for higher education due to their multimodal features, social presence, and ease of access unbounded by time and space.
■
■
ROLE OF ONLINE COURSES IN COLLEGE CHEMISTRY EDUCATION With the rising popularity of online learning, chemistry educators have embraced online courses in higher education settings.22−27 Similar to students of other STEM disciplines, chemistry students have benefited from a range of online learning environments.28−31 For instance, previous research has identified the advantages of using web-based learning materials in engaging low-achieving students and promoting understanding of molecular models in a freshman-level chemistry class.22 Similarly, in another study, organic chemistry subject matter content that was traditionally taught in face-toface lectures was transformed to online lecture videos to free up class time for interactive learning activities.25 Compared to students in the traditional lecture format, students in the flipped organic chemistry course had significantly higher grades
RESEARCH QUESTIONS This study examines the following three research questions. • Research question 1: What student characteristics are associated with participation in the online preparatory course? • Research question 2: What is the effect of participation in the online preparatory course on student performance in the subsequent organic chemistry course? • Research question 3: What is the effect of participation in the online preparatory course on student performance in the subsequent organic chemistry course for students typically at-risk in college environments (i.e., lowincome students, first-generation college students, underrepresented minorities)? B
DOI: 10.1021/acs.jchemed.8b01008 J. Chem. Educ. XXXX, XXX, XXX−XXX
Journal of Chemical Education
Article
Figure 1. Exemplary screenshot of podcast illustrating instructor’s real-time writing out of chemical structures with annotations.
■
METHODOLOGY
podcasts utilized a tablet/stylus combination with voice-over, instead of PowerPoint-type presentations, to mirror the boardwriting or document camera techniques typically used in teaching organic chemistry. This allowed the instructor to write out chemical structures and short notes in real time and encouraged students to become conversant in the language of chemical structure, which is of great importance to acquiring the deep conceptual understanding required in organic chemistry (Figure 1). The Canvas environment distributed 17 written assignments. For each written assignment, the answer sheet was uploaded after the assignment due date together with an additional podcast of the instructor providing detailed explanations and a solution path. In addition, the Sapling Learning Online Homework system was used to distribute three online assignments and a final exam. The Sapling system automatically graded student assignments with immediate feedback to the student about the student’s level of understanding of given topics. The instructor was available to students solely via email. Notably, the course was designed for uncomplicated implementation at other university campuses in order to address the unmet but universal need for a course that gives students the confidence and skills they need to succeed in organic chemistry. In order to provide open access to these voluntary learning opportunities to students from all backgrounds, students are only charged a nominal fee for access to the ebook44 and Sapling Learning (total cost $28). Students are awarded a certificate of completion, but there is no grade and the course does not appear on a student’s transcript.
Study Setting
This quantitative, empirical study was conducted at a large public research university in Southern California, and it examines students who enrolled in the organic chemistry course series (N = 1,289). This study is connected to a large National Science Foundation-funded research project (Award 1535300) that investigates online learning environments in higher education settings. This study was approved by the institution’s Internal Review Board. Institutional data for this study were provided from a number of sources on campus including Admissions, the Office of Financial Aid and Scholarships, Summer Session, the Registrar’s Office, the Office of Institutional Research, and the Office of Information Technology. Identifiers of students in the online preparatory course were provided by the instructor. Students were considered enrolled in the online preparatory course if they completed a single assignment in the Sapling Learning Online Homework system (N = 224). Students who registered for the online preparatory course but did not attempt a single assignment in the Sapling Learning Online Homework system (N = 100) were treated as if they did not enroll in the online preparatory course. Overall, students who were enrolled in the online preparatory course received an average of 65.8% (standard deviation: 37.7%) in the Sapling Learning Online Homework system. Notably, 58.0% of students received more than 70% of Sapling points, and 18.8% of students received more than 99% of Sapling points. Format of Online Preparatory Course
Measures
The three-week online preparatory course was advertised to all students who were enrolled in the first quarter of organic chemistry about 2 weeks prior to the start of the online preparatory course. The preparatory course was administered using the Canvas Learning Management system and the Sapling Learning Online Homework platform.42 With a focus on Lewis structures and resonance structures, topics include converting condensed structures into Lewis structures, drawing resonance structures, using arrow-pushing formalism43 to move between resonance structures, ranking the importance of resonance structures from most important to least important, and drawing resonance hybrids. Alongside a custom ebook,44 lecture-style instruction was provided through 16 video podcasts with accompanying notes. Students were encouraged to fill in notes as they viewed the podcasts. The
The dependent variable in research question 1 indicates whether students enrolled in the online preparatory chemistry course (0, no enrollment; 1, enrollment). The dependent variable in research questions 2 and 3 is an interval-level variable that describes student grades in the subsequent organic chemistry course (A+, A = 4.0; A− = 3.7; B+ = 3.3; B = 3.0;. . .; D = 1.0; D− = 0.7; F = 0.0). In the analysis, student grades are treated a continuous variable. Every grade change (e.g., C+ to B−, B− to B, B to B+, etc.) is defined as a onethird of a letter grade change. Independent variables in research question 1 and covariates in research questions 2 and 3 include student demographic information including student gender (0, male; 1, female), English language learner status (0, English or English and C
DOI: 10.1021/acs.jchemed.8b01008 J. Chem. Educ. XXXX, XXX, XXX−XXX
Journal of Chemical Education
Article
Table 1. Descriptive Student Information
Parameter
Full Sample (N = 1289)
Online Prep Participation (N = 224)
Mean,a %
Mean,a %
Course grade Prep participation
2.569 0.174
Female English language learner Raised by single parent First-generation college student Low-income student Underrepresented minority In-state residency
0.653 0.302 0.171 0.402 0.373 0.247 0.891
UC score High School GPA Units at college admission STEM major
239.4 4.013 26.63 0.704
SD 1.016
20.14 0.212 23.11
SD
2.975 0.885 1.000 Demographics 0.737 0.277 0.147 0.304 0.362 0.161 0.938 Academic Preparation 247.7 19.45 4.077 0.190 35.14 22.51 0.714
No Online Prep Participation (N = 1065) Mean,a % 2.484 0.000
SD 1.022
0.636 0.307 0.177 0.423 0.376 0.266 0.881 237.6 4.000 24.84 0.702
RQ1b
19.85 0.215 22.84
RQ2/3c
DV
DV IV
IV IV IV IV IV IV IV
COV COV COV COV COV COV COV
IV IV IV IV
COV COV COV COV
a Mean scores are reported for continuous variables, percentages for dichotomous variables. bRQ = research question, DV = dependent variable, IV = independent variable. cCOV = covariate.
another language is student’s first language; 1, language other than English is student’s first language), whether students were raised by a single parent (0, no single parent; 1, raised by single mother or single father), first-generation college student status (0, neither parent holds a Bachelor’s degree; 1, at least one parent holds a Bachelor’s degree or higher), low-income status (0, not flagged as low-income based on family household income and household size using 185% of the U.S. poverty line; 1, flagged as low-income), racial/ethnic minority status (0, White or Asian or Asian American; 1, Black or African American, Latino or Hispanic, American Indian, Alaska Native, or Pacific Islander), and in-state residency status (0, out-ofstate student; 1, in-state student). Additional covariates included student academic preparation continuous variables such as student admission score (i.e., UC score that is based on student ACT/SAT performance), student high school grade point average, and the number of college units at the time of admission, as well as a variable that describes whether or not students declared a STEM major (0, no STEM major or undeclared; 1, declared a STEM major). Table 1 lists descriptive information for each variable included in the models for both the full sample and across online preparatory course participation. Notably, 4.6% of all students received an F in the subsequent organic chemistry course (first-generation college students, 5.6%; low-income students, 6.7%; underrepresented minority students, 5.0%). Also, 88.5% of all students received a C− or higher in the subsequent organic chemistry course (first-generation college students, 87.1%; low-income students, 85.9%; underrepresented minority students, 84.3%).
The second and third research questions analyzed the impact of online preparatory course participation on student performance in the subsequent organic chemistry course. Students who participated in the online preparatory course can be viewed as the “treatment group”, and students who did not participate in the online preparatory course as the “control group”. As the first research question might detect underlying differences between treatment and control group students (e.g., selection effects based on student demographics, academic preparation, etc.), ordinary least-squares regression analysis might yield biased treatment effect estimates. To account for these underlying differences, this analysis applied a propensity score matching approach. Propensity scores are probability estimates of students in the treatment condition that account for confounding variables to address self-selection in the treatment group. Propensity scores are frequently used to reduce sample biases in nonexperimental causal settings.47 Instead of exact matching, nearest neighbor matching, or other propensity score approaches that would substantially affect the sample size in the models, this study uses a doubly robust propensity score matching method, inverse-probability weights with regression adjustment.48,49 This approach reduces biases in the treatment effect estimation through the assignment of weights to each student based on their propensity scores and provides more robust estimates compared to ordinary leastsquares regression analysis.50 Notably, the inverse-probability weights with regression adjustment use robust standard errors using linear outcome models and logistic treatment models to estimate the average treatment effect on the treated (i.e., the effect of the treatment averaged across all students in the treatment condition).48,49 Research question 2 utilized the full sample (N = 1,289), whereas research question 3 applied subgroup analyses to estimate heterogeneity effects for at-risk college student populations.51 The subgroups included low-income students, first-generation college students, underrepresented racial/ ethnic minority students, and their non-at-risk counterparts. Assumptions of the propensity score matching approach were tested and fulfilled. For instance, balance statistics of the model-adjusted differences in means and ratio of variances of
Analytical Methods
The first research question applied logistic regression analysis with robust standard errors to predict student participation in the online preparatory course.45,46 The assumptions of the logistic regression models were tested and fulfilled. For instance, the linearity of independent continuous variables and log odds were confirmed using Wald tests. Similarly, the computation of variance inflation factors (VIFs) indicated the absence of multicollinearity (VIF < 3). D
DOI: 10.1021/acs.jchemed.8b01008 J. Chem. Educ. XXXX, XXX, XXX−XXX
Journal of Chemical Education
Article
covariates between treatment and control indicated reasonable overlap. Similarly, χ-squared overidentification tests for covariance balance indicated that covariates were balanced for all models. Notably, students’ UC scores, high school GPA, and units at college admission were z-score transformed to aid the interpretation of the results. Also, a list-wise deletion missing data approach was applied to only include observations with full information on all covariates. The models still had sufficient statistical power. Missing data on all variables was below 3%, except for students’ UC scores, which had 4.6% missing data.
Table 2. Logistic Regression Model with Robust Standard Errors Predicting Online Preparatory Course Participationa Parameter
z
p Value
−6.06
