Online Homework Put to the Test: A Report on the Impact of Two

Aug 14, 2013 - Office of Institutional Research for Undergraduate Education, ... Two different online homework systems were administered to students i...
1 downloads 0 Views 278KB Size
Download PDF
Article pubs.acs.org/jchemeduc

Online Homework Put to the Test: A Report on the Impact of Two Online Learning Systems on Student Performance in General Chemistry Jack F. Eichler* and Junelyn Peeples Department of Chemistry, University of California, Riverside, Riverside, California 92521, United States Office of Institutional Research for Undergraduate Education, University of California Riverside, Riverside, California 92521, United States S Supporting Information *

ABSTRACT: Two different online homework systems were administered to students in a first-quarter general chemistry course. This study used a multiple regression model to control for the students’ academic and socioeconomic background, and it was found that students who completed the online homework activities performed significantly better on a common comprehensive final exam than students who did not participate. More specifically, it was found that students who completed a precourse assignment on an adaptive-responsive homework system (ALEKS; Assessment and Learning in Knowledge Spaces) could expect on average their final exam score to increase by over 13 points when compared to nonparticipating students. Students who completed a precourse assignment on a traditional responsive homework system (MasteringChemistry) also saw an average increase in their final exam score by roughly 8 points versus those who did not participate. Students who worked on the online homework for the entire quarter saw even greater gains in their final exam scores compared to nonparticipants. These findings suggest responsive online homework in general, and a responsive−adaptive learning system driven by knowledge space theory in particular, has a significant positive impact on student performance in the first-quarter general chemistry course. KEYWORDS: First-Year Undergraduate/General, Chemical Education Research, Computer-Based Learning FEATURE: Chemical Education Research

■

INTRODUCTION It has been previously shown that there is a strong correlation between completion of assigned homework problems and student success in general chemistry.1 This likely surprises few educators given the fact completing the assigned homework gives the students a chance to practice the necessary problemsolving skills and to engage in the conceptual material outside of the lecture hall. With the advent of the Internet age, the use of online homework systems has become more commonplace in undergraduate general chemistry courses, and has likely replaced traditional handwritten homework for most instructors. Even though the evidence is divided as to whether online homework has a clear advantage over traditional handwritten exercises in improving student learning,2−4 there are benefits to using online or computer-based homework systems, notably the increased efficiency in distributing, collecting, and grading the student work. The use of online computer based systems also provides the opportunity to create “smart” homework systems that can respond individually to students in a way not possible with handwritten instructor graded homework, particularly for large enrollment courses where providing © 2013 American Chemical Society and Division of Chemical Education, Inc.

individualized feedback might be logistically unfeasible. Despite the fact that online homework may or may not outperform handwritten homework activities, a significant amount of evidence demonstrates that online or computer-based homework exercises do improve student learning and success rates. Richards-Babb et al. have found that online homework activities improved student success rates in a second-term general chemistry course (measured by the percentage of A, B, and C grades),5 and Botch et al. have reported higher success rates and higher grades in general chemistry and prenursing chemistry courses.6 Additionally, Arasasingham et al. have provided evidence of improved student performance on conceptual versus algorithmic tests3 and higher achievement on final exams in the first-year general chemistry course sequence,7 and Freasier et al. reported improved course grades in the general chemistry sequence.8 Within the realm of computer-based homework learning systems, adaptive learning is a relatively new player on the scene. An “adaptive” learning system is generally defined as a technology that gives the learner immediate feedback during Published: August 14, 2013 1137

dx.doi.org/10.1021/ed3006264 | J. Chem. Educ. 2013, 90, 1137−1143

Journal of Chemical Education

■

the course of the learning activity, or in which the flow or content of learning activities is altered.9 However, computerbased learning systems that simply provide feedback or hints to the student are not truly adaptive, as all of the students still navigate the same set of problems at essentially the same pace. We define learning systems that provide hints and feedback in this manner “responsive”. A responsive learning system responds to individual student mistakes, but does not fundamentally change the content or the order in which the learning activities are delivered on a student-by-student basis. One widely used online learning system that incorporates responsive learning is MasteringChemistry (disseminated by Pearson Publishing).10 Instead of simply indicating to students whether they have answered correctly or not, MasteringChemistry provides specific hints and tutorials to a student based on how the problem was missed. Students who miss the problem for a different reason are given different feedback; students who do not miss the problem do not receive feedback and can move on to the next concept. However, even though students receive tailored feedback and guidance by the online learning system, all students still navigate through the same set of learning exercises in the same order, despite the fact some individuals may not be ready to move on to more advanced topics. A more recent entry into the arena of general chemistry online learning is the ALEKS system (Assessment and Learning in Knowledge Spaces).11 This system uses periodic assessments to quickly determine what students know within the general chemistry learning space, and individuals subsequently navigate through a specific set of learning exercises. Once a student has achieved mastery in that set of concepts and skills, he or she can move on to more advanced topics. However if a student fails to demonstrate mastery of a concept in a future assessment, he or she is required to relearn that topic prior to moving on to new course material. The advantage with this system is that each student is given a tailored set of learning activities that line up with that individual’s specific location in the course knowledge space, and perhaps more importantly, students are required to demonstrate continued mastery of requisite material throughout the entire course. We define this type of learning system as “responsive−adaptive”, as students are given both feedback and guidance while completing the learning modules (responsive), and are also placed into individualized learning spaces based on their performance on the assessments (adaptive). Though the MasteringChemistry system has been discussed in a media review in this Journal,12 to our knowledge an analysis of online responsive homework systems and their impact on student learning has not been published. In an effort to determine the impact of these online systems on student learning, MasteringChemistry and ALEKS were implemented in a first-quarter general chemistry course at a large public research institution. The two online learning systems were implemented in two different sections of general chemistry that were taught by the same instructor and participated in the same lecture activities. The performance of the students from the two treatment groups was measured by a common cumulative final exam,13 and the impact of the online learning systems was measured by comparing the final exam scores of the treatment groups with self-selected control groups that did not complete the online learning activities. The results of this analysis are described herein.

Article

STUDY DESIGN

Research Question

Our overall goal was to answer the question: What is the impact of a responsive−adaptive online learning system that uses periodic assessment to place students in a tailored learning space (ALEKS) on student performance compared to the impact of a responsive online homework system (MasteringChemistry)? Class Structure and Data Collection

Table 1 summarizes the class structure and number of students who participated in the MasteringChemistry and ALEKS treatment groups. Both classes were held on a Monday− Wednesday−Friday schedule and met between 11:00 a.m. and 2:00 pm. All lecture and discussion group activities were the same for both sections.14 Students in both sections were given the option of completing a precourse assignment and chapter assignments for the duration of the quarter. Students were given extra credit for completing just the precourse assignment and additional extra credit for completing the remaining chapter assignments. The assignments in both systems were aligned with Chapters 2−7 in the McMurry, Fay Atoms First textbook, and a summary of the topics included in each assignment for both MasteringChemistry and ALEKS can be found in the Supporting Information.15 As seen in Table 1, a majority of the students in each section chose to complete the precourse assignment. This assignment included topics from the mathematics skills review and introductory topics from Chapter 1 (units, measurement, dimensional analysis, etc.). Students were deemed to have successfully completed the MasteringChemistry precourse assignment if 90% of the questions in the assignment were answered correctly, and the threshold for successful completion of the ALEKS precourse assignment was defined by having mastered 90% of the topics in the assignment. A much smaller percentage of students opted to complete the chapter assignments for the remainder of the quarter. Successful completion of the MasteringChemistry assignments required the students to have an average score of 70% or higher on assignments for Chapters 2−7. Successful completion of the ALEKS system required that the students completed 70% of the learning topics in the assignments for Chapters 2−7. Students who completed the ALEKS assignments also completed seven assessment exams. These assessments were used to determine which learning objectives were placed in the students’ learning modules, but were not used to determine the students’ grades. The student grades were automatically calculated and tabulated by the online homework systems, and reported as a percentage of questions or topics completed (i.e., scores were reported as a whole number score out of 100). Students who scored below the success thresholds describe above were not included in the statistical analysis, and students who did not do any of the online homework activities (i.e., did not register for the online homework access) were used as the control group. It is noted that optional handwritten practice problems were posted online for each chapter, and these were available to students who did and did not participate in the online homework. Though we were not able to track how many, or how often students did these problems, anecdotal evidence suggests a large proportion of students from both populations routinely did these problems throughout the quarter. Additional data that was collected for both participants and nonparticipants included their final exam grades, 1138

dx.doi.org/10.1021/ed3006264 | J. Chem. Educ. 2013, 90, 1137−1143

Article

demographic backgrounds, and academic profile information. These were used to assess whether participating in the online homework system truly had a significant impact on the final exam performance.

44 51

There were students who started to complete the Chapter 2−7 assignments, but who quit during the course of the study. These students were not included in the statistical analysis.

Statistical Analysis

To determine whether participating in the ALEKS system or MasteringChemistry assignments resulted in any statistically significant impact, this analysis imposed a linear regression model that held constant the demographic and academic background characteristics such as gender, race/ethnicity, high school GPA, SAT scores, and first-generation college status. This model “provides us with an equation describing the nature of the relationship between two variables”.16 The goal was to determine whether students who participated in the online homework systems would perform better on their final exam, and therefore improve their overall performance in the course. The regression model provides a prediction of the impact of the treatment variable administered to the given population, and the result produced when calculating the regression equation provides a correlation coefficient between the dependent and independent variables. By applying a multiple regression equation, it was possible to evaluate the dependent variable’s outcome given the results to each of the independent variables included in the model. Ultimately, the value of each online homework system’s impact on student final exam grade performance was determined by holding constant the other independent variables that could potentially be contributing factors to the participant’s performance on the exam. The variable definitions for the dependent variable and each of the independent variables used in this analysis are described in Table 1 in the Supporting Information. The SPSS (Statistical Package for the Social Sciences) predictive analytics package was used to run the descriptive statistics and multiple regression analyses. The multiple regression model applies the following equation to account for the dependent and various independent variables in order to calculate the effect on the dependent variable (dependent variable Y = final exam score; the number of independent variables, p, was equal to 13). Y = a + b1X1 + b2X 2 + ... + bpX p

(1)

Table 2 provides the descriptive statistics for the different independent variables used to calculate the impact of the dependent variable for the populations of students who participated in the ALEKS and MasteringChemistry online homework assignments, all participants of the online homework, and all nonparticipants enrolled in the course. This table provides the average and standard deviation for the final exam scores, high school GPA, and SAT scores, as well as the percentage of students who were from the various ethnic and socioeconomic groups. It is noted that the demographic breakdown of the student populations in this study is similar to that seen on the entire University of California, Riverside (UCR) campus: 40% Asian; 29% Latino; 17% Caucasian; and 8% African American.

■

RESULTS

Regression Analysis

Tables 3 and 4 report the regression analysis for ALEKS and MasteringChemistry online homework participants compared to the corresponding nonparticipants for each section, holding constant the student background characteristics and academic preparation independent variables (gender, race/ethnicity, high

a

78 97a

MasteringChemistry ALEKS MWF/11:00−11:50 a.m. MWF/1:00−1:50 p.m. Section 1 (Eichler) Section 2 (Eichler)

McMurry, Fay: Atoms First (1−7) McMurry, Fay: Atoms First (1−7)

349 540a

196 302

a a

Online HW System Day/Time of Class Text (Chapters) Course Section (Instructor)

Table 1. Summary of Class Structure and Student Participants

Total Enrollment, N

Completed Precourse Assignment, N

Completed Chapter 2−7 Assignments, N

Nonparticipants, N

Journal of Chemical Education

1139

dx.doi.org/10.1021/ed3006264 | J. Chem. Educ. 2013, 90, 1137−1143

Journal of Chemical Education

Article

Table 2. Descriptive Statistics Used To Calculate the Impact of the Dependent Variable on Sample Populations Mean Values (Standard Deviations) for Population Samples Independent Variables

ALEKS

MasteringChemistry

Online HW Participants

Nonparticipants

Final exam score Femalea African Americana Native Americana Hispanica Asian/Pacific Islandera Caucasiana First-generation statusa Low-income statusa High school GPA SAT verbal SAT math SAT writing On campusa Hours spent in HW systemb

77.64 (12.24) 0.55 0.04 Insufficient sample 0.26 0.59 0.11 0.47 0.48 3.68 (0.32) 532 (82) 604 (88) 546 (86) 0.60 43.7 (14.9)

78.51 (12.12) 0.50 0.07 0.01 0.26 0.54 0.12 0.50 0.47 3.68 (0.32) 539 (88) 606 (88) 547 (87) 0.51 7.7 (5.4)

78.01 (12.18) 0.53 0.05 0.00 0.26 0.57 0.11 0.48 0.48 3.68 (0.32) 535 (84) 605 (88) 547 (86) 0.57

64.63 (17.22) 0.28 0.06 Insufficient sample 0.29 0.46 0.18 0.52 0.60 3.55 (0.29) 526 (79) 583 (86) 539 (80) 0.37

a

Tabulated values represent the fraction of the specific population that is composed of this ethnic group or socioeconomic trait. bAverage time for all students who participated in the online homework assignments (preassignment and/or all assignments).

Table 3. ALEKS Online Homework Assignment Regression Results Estimated Impact on Final Exam Gradea

Table 4. MasteringChemistry Online Homework Assignment Regression Results Estimated Impact on Final Exam Gradea

b Values (Standard Error) for Population Samples Independent Variables ALEKS impact on final exam score Female African American Native American Hispanic White First-generation status Low-income status High school GPA SAT verbal SAT math SAT writing On campus

One Assignment, No Controls 14.96b (2.07)

b Values (Standard Error) for Population Samples One Assignment

All Assignments

13.57b (2.31)

20.09b (2.90)

−1.71b (1.95) 1.12 (4.24) Insufficient sample −6.40b (2.39) −0.72 (3.01) 2.54 (2.17)

−3.33 (2.89) 5.25 (6.03) Insufficient sample −8.75b (3.21) −1.45 (4.02) 6.21b (2.74)

−0.21 (2.06) 7.99b (2.82) 0.01 (0.02) 0.02 (0.01) 0.01 (0.02) 0.33 (1.95)

−0.68 (2.90) 5.92 (4.47) 0.02 (0.02) 0.02 (0.02) 0.00 (0.03) −4.12 (2.68)

Independent Variables MasteringChemistry impact on final exam score Female African American Native American Hispanic White First-generation status Low-income status High school GPA SAT verbal SAT math SAT writing On campus

a

Column one represents impact of online homework without any controls. Columns two and three indicate impact of online homework while holding all other independent variables constant, as well as the correlation factors for the other independent variables. bStatistically significant at the 0.05 level (two-tailed).

One Assignment, No Controls b

10.37 (2.28)

One Assignment b

All Assignments

7.83 (2.28)

10.09b (2.72)

−4.37b (2.18) 1.79 (4.24) Insufficient sample −1.12 (2.63) 0.53 (2.75) 2.31 (2.39) −2.25 (2.29) 7.07b (3.36) 0.02 (0.02) 0.06b (0.02) −0.02 (0.02) 3.42 (1.98)

−5.08 (2.78) 1.96 (5.09) 0.29 (13.09) −5.93 (3.57) −4.47 (3.92) −0.59 (3.23) −0.93 (3.14) 11.43b (4.27) 0.03 (0.03) 0.03 (0.02) −0.01 (0.02) 4.26 (2.59)

a

Column one represents impact of online homework without any controls. Columns two and three indicate impact of online homework while holding all other independent variables constant, as well as the correlation factors for the other independent variables. bStatistically significant at the 0.05 level (two-tailed).

school GPA, SAT scores, campus residency, and firstgeneration college status). The first column shows the estimated impact (the correlation between the dependent and independent variables) on final exam scores based on having only participated once in the online homework without holding any of the controls constant. Columns two and three also report estimated impact of participation in the online homework for one assignment and all assignments, holding constant the other independent variables for the two subpopulations. This provided an opportunity to test whether the amount of participation in online homework increased or decreased the impact on students’ final exam score. For example, in Table 3, a student who participated in the ALEKS

online homework assignment at least once could expect to increase his or her final exam score by nearly 14 points; whereas, if the participant participated in all the ALEKS assignments, his or her final exam score could potentially go up by 20 points. Finally, the regression results for all students who participated in either of the online homework systems (ALEKS and MasteringChemistry) compared to nonparticipants are reported in Table 2 in the Supporting Information. The same methodology described for Tables 3 and 4 is employed in the results reported in Table 2 in the Supporting Information. Students who participated in either of the online 1140

dx.doi.org/10.1021/ed3006264 | J. Chem. Educ. 2013, 90, 1137−1143

Journal of Chemical Education

Article

homework systems were observed to see an increase in their final exam score by roughly 11 points. The table also shows that the students who participated in all assignments for either homework system earned a final exam score that was nearly 16 points higher than that of nonparticipants. Historically at UCR women and minority groups are not well represented in STEM (science, technology, engineering, and mathematics) majors and have been observed to experience lower retention and graduate rates in these areas of study compared to nonminority students. There is an ongoing effort to identify programs that might provide targeted academic assistance to underrepresented student populations interested in STEM fields; therefore, analyzing whether the online homework activities have a more positive or negative impact on these student populations is warranted. It is also noted that female and Hispanic students in our study had a negative correlation with final exam performance (see Tables 3 and 4); this provided further incentive to determine the impact of online homework on these demographic groups. The impact of participating in either of the online homework systems on the final exam performance of each demographic group, while holding all other independent variables constant, is reported in Table 3 in the Supporting Information. Column one reports grade impact on final exam score without holding any background characteristics and incoming academic profile constant. Columns two and three represent the impact of the online homework participation by how often a student participated.

ment alone seemed to provide a significant positive impact on student performance, as students who did the MasteringChemistry and ALEKS precourse assignments scored approximately 8 and 13 points, respectively, higher than nonparticipants on the final exam. Students who completed all of the assignments scored even higher on the final, as students in MasteringChemistry and ALEKS scored approximately 10 and 20 points higher than nonparticipants, respectively. Based on the data reported here, it is concluded that completing all of the assignments on the ALEKS system had the most significant positive impact on student final exam performance. Analysis of the combined ALEKS and MasteringChemistry student populations also indicated that participating in some form of online homework had a positive impact on the final exam performance. Students who only completed the precourse assignments for either of the online homework systems saw an 11-point increase to their overall final exam scores, whereas students who completed all of the assignments for either system experienced a 16-point increase to their overall scores (see Table 2 in the Supporting Information). When the students were broken down into subpopulations, it was found that all students in every social/ethnic group experienced a positive impact to their final exam score if they participated in online homework activities, even after holding constant their socioeconomic and academic characteristics. Analogous to the larger MasteringChemistry and ALEKS populations, these results generally indicate that students who completed all of the online homework assignments could expect to see an even greater impact in their final exam score compared to those students who completed just one online homework assignment (see Table 3 in the Supporting Information). The only exception was Caucasian students, who could potentially see a 10-point increase to their overall final exam scores after having participated in one online homework activity, but saw no further significant impact if they had completed all of the online homework. Even more noteworthy is the fact that female and Hispanic students who had negative correlations with the final exam score (see Tables 3 and 4) scored approximately 21 and 20 points higher, respectively, than their corresponding nonparticipants (see Table 3 in the Supporting Information). It is noted that even though the ALEKS and MasteringChemistry populations appeared to be similar based on a comparison of the descriptive statistics (Table 2), the populations were found to be different when the independent variables were correlated with final exam score. In particular, it is evident that the ALEKS population had weaker correlation between high school GPA and final exam scores, and more negative correlation between the demographic background and final exam score than the MasteringChemistry population (see Tables 3 and 4). This might explain why the ALEKS homework had a more positive impact on the student exam performance than MasteringChemisty, despite the fact the overall exam averages were similar.

Impact on Final Exam Performance

The populations in the three groups of students (ALEKS participants, MasteringChemistry participants, and nonparticipants) are roughly similar, though there are some subtle differences in the background characteristics for the nonparticipants (e.g., the nonparticipants had a slightly lower percentage of female students, and a slightly higher percentage of low-income status students). As seen in Table 2, the final exam average for the students who did not do any online homework activities is over 10 points lower than that of the students who did the precourse assignment and/or the entire sequence of homework assignments: 64.63 ±17.22 for nonparticipants; 77.64 ±12.24 for ALEKS; and 78.51 ±12.12 for MasteringChemistry). Because the group of nonparticipants was self-selected, one might expect these students to have a weaker academic profile. However, it is noted that the nonparticipants had high school GPAs and SAT scores that were not greatly different from the treatment groups. Additionally, the regression analysis was carried out in order to gain a more clear understanding about the impact of the online homework on the students’ final exam performance, while holding academic background and other independent variables constant. As one might expect, high school GPA was found to have a positive impact on student final exam performance, particularly for the population of students who completed the MasteringChemistry assignments (see Tables 3 and 4). This confirms the notion that the students’ academic preparation does indeed have an impact on their performance in the first quarter of general chemistry. However, when holding this and all other independent variables constant, participating in some form of online homework had a significantly positive impact on the final exam performance (see Tables 3 and 4 and Table 2 in the Supporting Information). Completing the precourse assign-

■

DISCUSSION

Positive Impact of Online Homework

The data reported here clearly indicate that the responsive (MasteringChemistry) and responsive−adaptive (ALEKS) online homework systems have a positive impact on student performance on a cumulative final exam. As one would likely predict, a more sustained intervention that lasts the entire 1141

dx.doi.org/10.1021/ed3006264 | J. Chem. Educ. 2013, 90, 1137−1143

Journal of Chemical Education

Article

academic term had a more significant impact. The students who completed all of the MasteringChemistry homework assignments scored more than 2 points higher on the final than students who did just the precourse assignment, and students who did all of the ALEKS assignments scored more than 6 points higher than students who did only the precourse assignment. The success of the students who did all of the ALEKS assignments is particularly noteworthy; these students scored on average 10 points higher than students who completed all of the MasteringChemistry assignments. The positive impact of the online homework activities with final exam performance reported here generally agrees with previous studies that report the impact of online homework on student learning. In particular, Arasasingham et al. reported a significant correlation between student performance on an online homework system and performance on the final exam in large introductory general chemistry courses.3 Even though the study by Arasasingham et al. did not do a regression analysis that controlled for student academic background, the authors did find weak correlation between precourse placement scores and online homework performance, suggesting that the online homework provided an even starting point for all students regardless of their academic preparation, and that the homework activities therefore had some meaningful impact on the students’ final exam performance. This result is similar to our finding, which indicates that even though student academic preparation (as measured by SAT scores or high school GPA) has a positive impact on final exam performance, the online homework had a significant and positive impact on student final exam scores when academic background variables were held constant. It is also noted that the positive impact of the MasteringChemistry and ALEKS precourse assignments on their own is not unprecedented. Botch et al. also found that an online preparatory activity had a positive impact on student performance in an introductory general chemistry course.6 This previous report indicated that users of the online preparatory homework system, which included topics such as basic mathematics skills, measurement and calculations, and the basic structure of matter and chemical reactions, resulted in significantly higher final course grades for participants compared to nonparticipants. These topics are similar in scope to the topics assigned in our ALEKS and MasteringChemistry precourse assignments, and it appears that using an online homework system to help ensure that students have mastered these fundamental concepts increases student success in a general chemistry course.

Figure 1. Correlation between total time spent on the ALEKS online learning system and performance on the final exam. Only students who spent six hours or more on the system are included in the correlation: r = 0.227; p = 0.0085. Students who spent fewer than six hours in the system were not included as they were likely students who quit using the system or who only did the precourse assignment.

The plot of time spent on the ALEKS system versus final exam score indicates a very weak correlation, which is corroborated by the low Pearson’s correlation coefficient value (r = 0.227; p = 0.0085). It is apparent from this correlation that increased time on task does not explain the increased student performance for students who worked on the ALEKS system. Though it might seem counterintuitive that time spent on the homework system correlates weakly to final exam performance, there are a couple of factors that might explain this finding. Increased time spent in the ALEKS system can be caused by an improper use of the periodic assessments by the students. Students are instructed to do the assessments using only the resources provided by the ALEKS interface (periodic table, calculator, etc.) and not to use other outside sources. This is due to the fact that the assessments are used to place the students in the proper learning space, and if their assessment performance is exaggerated they are asked to do learning activities that are beyond their current capabilities. Students subsequently spend significantly longer periods of time navigating their way through the learning exercises, and given that the student then likely struggles with the more advanced topics, more time on task would not likely correlate to better performance on exams. Another reason that students might end up spending more time in the system can be due to the fact that the student may frequently forget previously learned material, therefore he or she is required to relearn these topics in future learning exercises. Again, even though the student may spend more time on task, it may actually reflect a weaker ability to master the requisite material, which would likely not result in an increased final exam score. Even though there is weak correlation between time on task in the ALEKS system and performance on the final exam, it is worth explaining why the ALEKS system requires significantly more time on task than the MasteringChemistry system. This can be attributed to the fact that ALEKS does not simply have the students complete a set of assigned problems, but it instead gives the students multiple problems for a given concept until he or she has demonstrated repeated mastery. Additionally, the students are required to do multiple assessments in addition to the learning modules, and if specific learning objectives are forgotten, they must be relearned by the student in future

Explaining Online Homework Impact on Final Exam Scores

Table 3 reports the average time spent in the online homework systems, and it is clear that students spent a significantly larger amount of time in the ALEKS system (43.7 ± 14.9 h) than in the MasteringChemistry system (7.7 ± 5.4 h). Although it is beyond the scope of this work to determine what specific aspects of the homework systems actually led to the gains on final exam scores, it is pertinent to comment on potential factors that led to the increased student success in this study. In order to gain some insight about the potential link between time on task and final exam grade, the correlation between the total time in the ALEKS system and final exam score was plotted for students who spent six or more hours in the homework system (see Figure 1). 1142

dx.doi.org/10.1021/ed3006264 | J. Chem. Educ. 2013, 90, 1137−1143

Journal of Chemical Education

■

homework objectives. These factors led to significantly more time on task compared to a traditional responsive homework system like MasteringChemistry. If time on task is not the likely contributor to improved performance, the significantly larger increase in final exam scores for the population of students who completed all of the ALEKS assignments can therefore possibly be attributed to the unique nature of demonstrated mastery in the learning mode and repetitive assessment. As stated above, students do not simply work on a single problem until they are able to get it correct; instead they must complete a set of different problems for each concept until they have demonstrated mastery. Subsequently, periodic assessments are used to ensure that the students are working on the concepts that they are ready to learn. This helps optimize the productivity of the students in the learning activities. Additionally, the assessments ensure that the students have retained their previously learned knowledge. If students forget requisite concepts, they must relearn them in the next cycle of adapted learning activities, and this ultimately leads to longer-term retention of the material. This combination of using the learning system to demonstrate full mastery and using assessments to place the students in learning activities that match their current level of understanding and ensure retention very well could have resulted in the significant positive impact on student final exam performance. To conclude, even though more time on task is required compared to traditional responsive homework systems, time on task alone does not correlate to performance, as students may demonstrate mastery in variable amounts of time. Conversely, all students navigate through the MasteringChemistry knowledge space in the same order, and at essentially the same pace. While students receive responsive feedback and hints in the MasteringChemistry system, their learning activities never vary from those of their peers, regardless of whether the students are ready to move on to more advanced topics or not. More importantly, it is possible for students to complete the learning activities in later assignments, even if they may have forgotten previous knowledge. The fact that the responsive system used in MasteringChemistry does not ensure the retention of previous knowledge likely resulted in it having a less positive impact on student performance on the final exam.

■

Article

ASSOCIATED CONTENT

S Supporting Information *

List of topics included in each chapter assignment for MasteringChemistry and ALEKS; description of the dependent variables used in the linear regression analysis (Table 1); linear regression analysis data for students who participated in either online homework system (Table 2); linear regression analysis data for the impact of the online homework on different ethnic groups (Table 3). This material is available via the Internet at http://pubs.acs.org.

■

AUTHOR INFORMATION

Corresponding Author

*E-mail: [email protected]. Notes

The authors declare no competing financial interest.

■

REFERENCES

(1) Cuadros, J.; Yaron, D.; Leinhardt, G. J. Chem. Educ. 2007, 84 (6), 1047−1052. (2) Fynewever, H. Chem. Educ. 2008, 13, 264−269. (3) Arasasingham, R. D.; Taagepera, M.; Potter, F.; Martorell, I.; Lonjers, S. J. Chem. Educ. 2005, 82 (8), 1251−1262. (4) Fynewever found that there were no significant grade differences between students who did the online homework and those who did traditional hand-graded homework,2 whereas Arasasingham et al. found that an online homework system resulted in significantly higher levels of student learning compared to hand-written homework based on textbook problems.3 (5) Richards-Babb, M.; Drelick, J.; Henry, Z.; Robertson-Honecker, J. J. Coll. Sci. Teach. 2011, 40 (4), 81−93. (6) Botch, B.; Day, R.; Vining, W.; Steward, B. J. Chem. Educ. 2007, 84 (3), 547−553. (7) Arasasingham, R. D.; Martorell, I.; McIntire, T. M. J. Coll. Sci. Teach. 2011, 40 (6), 71−79. (8) Freasier, B.; Collins, G.; Newitt, P. J. Chem. Educ. 2003, 80 (11), 1344−1347. (9) Howard, L.; Remenyi, Z.; Pap, G. 9th Int. Conf. Eng. Educ. 2006, T3K11−16. (10) MasteringChemistry Home Page. http://masteringchemistry. com/site/index.html (accessed Jul 2013). (11) What is ALEKS? Web Page. http://www.aleks.com/about_aleks (accessed Jul 2013). (12) Shepherd, T. D. J. Chem. Educ. 2009, 86 (6), 694. (13) The final exam was composed of 40 multiple-choice questions worth 2.5 points each, plus 5 additional bonus multiple-choice questions worth 1 point each. The 40 questions were equally distributed among the topics in Chapters 1−7, and there was an even split between computational and conceptual questions. The exam was scored out of 100 points, with the highest grade possible being 105/ 100. (14) All students attended discussion group recitations once per week. These meetings involved having graduate teaching assistants provide supplemental instruction, and the students took weekly quizzes in these meetings. (15) MasteringChemistry has both responsive tutorial-based questions and traditional end-of-chapter questions that do not provide feedback to the students. The assignments given in this study were composed entirely of questions from the pool of responsive tutorialbased questions. (16) Kachigan, S. Multivariate Statistical Analysis, 2nd ed.; Radius Press: New York, 1991. (17) One example of an institution that uses responsive−adaptive learning assignment as a course prerequisite is the University of Texas−Austin. http://cns.utexas.edu/academics/placement/alekschemistry-assessment (accessed Jul 2013).

CONCLUSION

The data shown here suggest that simply having students complete a precourse fundamentals assignment using online homework systems helps prepare them for the first-quarter general chemistry course, and subsequently improves student performance on the final exam. In particular, it was found that the responsive−adaptive ALEKS learning system had a larger positive impact on student final exam performance than a more traditional responsive online homework system. Though future studies should aim to determine with more certainty how this type of responsive−adaptive learning system impacts student learning, the results reported herein suggest that chemistry educators should strongly consider incorporating this type of online learning system into their general chemistry programs and, at the very least, chemistry departments should consider using responsive−adaptive learning systems as a way to remediate students in general chemistry.17 1143

dx.doi.org/10.1021/ed3006264 | J. Chem. Educ. 2013, 90, 1137−1143