The Evaluation of a Hybrid, General Chemistry Laboratory Curriculum

Article Cite This: J. Chem. Educ. XXXX, XXX, XXX−XXX

pubs.acs.org/jchemeduc

The Evaluation of a Hybrid, General Chemistry Laboratory Curriculum: Impact on Students’ Cognitive, Affective, and Psychomotor Learning Kory M. Enneking,† Graham R. Breitenstein,‡ Amelia F. Coleman,‡ James H. Reeves,‡ Yishi Wang,§ and Nathaniel P. Grove*,‡

Downloaded via UNIV OF LOUISIANA AT LAFAYETTE on May 6, 2019 at 01:49:03 (UTC). See https://pubs.acs.org/sharingguidelines for options on how to legitimately share published articles.

†

Department of Biology and Marine Biology, University of North Carolina Wilmington, Wilmington, North Carolina 28403, United States ‡ Department of Chemistry and Biochemistry, University of North Carolina Wilmington, Wilmington, North Carolina 28403, United States § Department of Mathematics and Statistics, University of North Carolina Wilmington, Wilmington, North Carolina 28403, United States ABSTRACT: The laboratory has occupied an important place in the general chemistry curriculum for well over a century, despite the fact that many have voiced concern about its value and utility. In an effort to potentially increase capacity in our general chemistry courses, we developed and implemented a hybrid laboratory curriculum that consisted of alternating face-toface and virtual laboratory experiments. This study sought to better understand the impact that this hybrid approach had on students’ cognitive, affective, and psychomotor learning. The results suggest that students taught using the hybrid approach developed similar cognitive and psychomotor skills in comparison to students taught using a traditional laboratory curriculum; however, their affective outlook toward chemistry was significantly lower. KEYWORDS: First-Year Undergraduate / General, Chemical Education Research, Laboratory Instruction, Internet / Web-Based Learning FEATURE: Chemical Education Research

■

INTRODUCTION For over a century, chemists have viewed experiential, laboratory-based learning as an essential component of both secondary and tertiary-level chemistry education.1−3 Faculty envision the laboratory setting as a place where students can begin to make meaningful connections to the material they learn in not only their chemistry lecture but also many of the other science and mathematics courses they are enrolled in.4 At its best, the laboratory provides an opportunity for students to engage in scientific thinking and decision-making and to develop the skills necessary to effectively communicate in the scientific world.4 It is also one of the few places outside of a formal research setting where students can gain experience with the important manipulations and techniques that are valued by the field and to learn how to use equipment and instrumentation.3 In short, when structured appropriately, the chemistry laboratory experience presents students with a unique opportunity to integrate their cognitive, affective, and psychomotor skills, a necessary requirement for meaningful learning.5 The goals outlined above are as numerous as they are diverse, and there is question whether many laboratory experiences achieve them in any measurable capacitymost especially those experiences at the introductory level.6−12 All too often, and frequently in an effort to deal with already large classes and ever-increasing enrollments, faculty have elected to use laboratory materials that seem to circumvent the meaningful learning process. Procedures are provided to © XXXX American Chemical Society and Division of Chemical Education, Inc.

students in great detail, and indeed, the results of the experiment can often be predicted based on the information provided in the introductory materials. While students do have an opportunity in many cases to develop their psychomotor skills, the structure of such experiences strip them of the ability to practice their scientific thinking and decision-making.

■

THE USE OF VIRTUAL CHEMISTRY EXPERIENCES IN THE LABORATORY

Almost as soon as computers began appearing in classrooms in the late 1970s, examples of virtual educational materials for use in the chemistry laboratory began appearing in the research literature.13−18 Wiegers and Smith, for example, reported on the use of “computer-assisted instruction” as a prelaboratory assignment for students in an organic chemistry laboratory course.19 Although the graphical interface is crude by today’s standards, the program required students to perform simulated chemical reactions and manipulate virtual glassware. In using the system, students felt like they better understood what they were doing in the laboratory, and in many cases, their use led to a more efficient experience for them when in the laboratory.19 Received: August 7, 2018 Revised: April 12, 2019

A

DOI: 10.1021/acs.jchemed.8b00637 J. Chem. Educ. XXXX, XXX, XXX−XXX

Journal of Chemical Education

Article

responsibility.” In other words, for learning to be meaningful, it must integrate elements of the cognitive, affective, and psychomotor domains. Faculty goals for the laboratory tend to emphasize the cognitive and psychomotor domains,3,49,50 whereas students believe the primary reason for laboratory work is the acquisition of technical proficiency.51 In many of the discussions faculty and staff had with researchers, the affective domain was largely ignored, although of no less importance than the others.51

Although more recent systems have incorporated more realistic 3D graphics20−22 and have even begun to make use of virtual reality environments,23−25 many chemistry instructors have been hesitant to use such tools as replacements for faceto-face (FTF) laboratory experiments.26 A former editor of this Journal, for example, once wrote27 (p. 475): “A “laboratory” program that is completely virtual cannot provide students with the same knowledge of chemistry that a real laboratory program can.” This belief that students’ learning will be hampered in such environments is not uncommon.28,29 Available evidence, however, would suggest that students who are taught using virtual laboratory materials learn as much as those taught using more traditional modalities.26,30−32 Indeed, studies have documented tangible benefits from their use, although it is important to acknowledge that not all of these are unique to virtual laboratory experiences: • Well-designed virtual laboratory materials are studentcentered and inquiry-based, promoting higher order thinking and the development of critical thinking skills.33,34 • Virtual laboratory experiences can provide designers with the ability to integrate particulate-level animations and simulations that are not possible with static laboratory manuals. Some studies have reported greater conceptual understanding among students from using virtual experiments as a result of these elements35,36 and others.37−44 • Students have the ability to work independently and at their own pace when using virtual materials. When paired with traditional laboratory experiences, they can also help students prepare beforehand to avoid commonly made mistakes.35,36 • Virtual laboratory experiments can serve as low-cost alternatives in situations where resources do not allow for completion of their traditional counterparts or where it would be dangerous to do so.45,46 • Virtual laboratories can provide flexibility for institutions facing disruptions to their normal course offerings and schedules. For example, the Department of Chemistry and Biochemistry at the University of North Carolina Wilmington was recently faced with the loss of its building because of Hurricane Florence.47 The use of online, virtual laboratory experiments permitted students enrolled in general chemistry to finish out the semester, which would not have been possible in the absence of such resources.

■

RESEARCH GOALS At the University of North Carolina Wilmington (UNCW), nearly 2200 students enroll in general chemistry every year. As the university has grown over the last several years, so too has the demand on general chemistry, a prerequisite course for many STEM and health majors. Unfortunately, because of a lack of laboratory space, the Department of Chemistry and Biochemistry cannot offer any additional sections of the course as it currently exists. As such, we began to explore alternative methods for offering our general chemistry 1 laboratory curriculum that would allow us to ease this bottleneck. Ultimately, this led to the consideration of a hybrid laboratory curriculum in which students alternated between completing virtual laboratory (VL) experiments 1 week and FTF experiments the next. This approach is described in more detail below. Before implementation of the hybrid curriculum on a large scale, however, we wanted to better understand the impact that this change would have on our students’ experiences. In considering the requirements for meaningful learning as described by Novak, the following research question was generated and guided this work: In comparison to a traditional laboratory, what impact does a hybrid laboratory approach have on general chemistry students’ cognitive, affective, and psychomotor learning?

■

EXPERIMENTAL METHODS

Research Population

UNCW offers a single introductory chemistry sequence for its students, general chemistry 1 and 2. Because of this, the backgrounds and interests of students enrolled in general chemistry 1 are broad, though weighted toward those students intending to major in chemistry, biology, marine biology, environmental science, engineering, computer science, earth and ocean sciences, exercise science, public health, clinical research, and nursing. Most students, approximately 70%, are in their first year. The sex distribution of students in the course roughly mirrors that of the student body at large, approximately 60% female and 40% male.

■

THEORETICAL FRAMEWORK Ausubel’s Assimilation Theory posits that meaningful learning occurs when the learner consciously elects to integrate new information into the existing cognitive framework, weaving the new with the old, and in the process, creating strong ties between the two. This is typically contrasted with rote learning where the learner merely memorizes new information. Although learning a concept meaningfully takes a greater initial investment in cognitive resources, the information is retained longer and makes the acquisition of related concepts easier.48 Beyond the deliberate decision to adopt a meaningful mindset, Novak suggested that5 (p.18): “Meaningful learning underlies the constructive integration of thinking, feeling, and acting leading to human empowerment for commitment and

Research Environment

At UNCW, all sections of general chemistry 1 are coordinated: all use the same book, follow the same lecture schedule, require students to complete the same online homework assignments, and to take the same hourly examinations and final. In nearly all semesters, students also complete the same series of laboratories and in the same sequence. Most of the experiments conducted are expository in nature, providing students with an extensive background, detailed procedures, and a series of straightforward postlab questions. For the purposes of this experiment, a single section of the course in both Spring 2015 (112 students out of a total course enrollment of 477) and Fall 2015 (83 students out of a total B



Article

Figure 1. LearnSmart Laboratories opening screen for “Calorimetry”.

Figure 2. Screenshot of a typical LearnSmart Laboratories learning resource.

experiments contained in the product. As illustrated in Figure 1, each experiment consists of an introductory “core concepts” section to provide basic familiarity with the concepts covered in the experiment, followed by one or more simulations that allow students to manipulate chemicals, glassware, and equipment to carry out experimental measurements and collect and analyze data. Each section of a lab typically requires between 30 and 60 min to complete. The simulations were designed to address issues that typically arise in a traditional laboratory setting, thus potentially providing a detailed prelab for a corresponding FTF experiment. Once the student completes a simulation, a report detailing his or her performance on specific skills is generated, and the student is given the option to redo the experiment for additional credit. Much like other LearnSmart resources, LearnSmart Laboratories feature adaptive questions

course enrollment of 664) instead participated in a hybrid laboratory curriculum. The hybrid laboratory approach consisted of alternating FTF and VL experiments implemented so that half of the population of students performed the FTF experiments in any given week while the other did the VL experiments. The FTF experiments were a subset of the ones used in the regular laboratory curriculum,52 and the VL experiments were selected from a total of 16 virtual laboratory exercises available in LearnSmart Laboratories, an online laboratory simulation product developed by McGraw Hill Publishing Company.53 It is important to note that in the context of this research, the term “virtual” is used to describe a computer-based laboratory simulation that students completed individually outside of the formal laboratory space. The LearnSmart Laboratories were designed to provide a thorough overview of each of the virtual C



Article

complete each FTF laboratory experiment in 3 h, and at the conclusion, turned in a short summary of their findings along with answers to a few postlab questions before leaving.

and learning resources that are designed to explain concepts that the students have not yet mastered. An example learning resource is shown in Figure 2. One of the authors (Reeves) was part of the development team for these online materials. In order to avoid conflicts of interest, Reeves was not involved in the collection or analysis of data for this study. The laboratory schedule for both the hybrid lab group and the traditional lab group for Fall 2015 is provided in Table 1. The schedule for Spring 2015 was similar to only slight modification to account for the dates of spring-specific, university-wide holidays. Students typically worked in pairs on each experiment, though in a few instances, were asked to work in larger groups of four. Students were expected to

Data Collection

As previously outlined, meaningful learning requires elements of thinking, feeling, and doing, and as such, the sources of data collected as part of this evaluation were selected to provide insights into the impact the hybrid laboratory experience had on these domains of knowledge. Table 2 summarizes these sources. Table 2. Summary of Data Sources Source

Table 1. Comparison of General Chemistry 1 Hybrid and Traditional Lab Schedules, Fall 2015 Week Group 1 2

A B A

3

B A B

4

A

5

B A B

6

A B

7

A B

8

A

9

B A

10

B A

11

B A B

12

A B

Hybrid Lab Group Activity LabSmart Laboratory: Density No Lab In Lab: Lab Policy; Lab Safety; Sig. Figure; Density Experiment LabSmart Laboratory: Density No Lab In Lab: Lab Policy; Lab Safety; Sig. Figure; Density Experiment LabSmart Laboratory: Stoichiometry No Lab In Lab: Stoichiometry Loss of CO2 Experiment LabSmart Laboratory: Stoichiometry LabSmart Laboratory: Calorimetry In Lab: Stoichiometry Loss of CO2 Experiment In Lab: Calorimetry Experiment LabSmart Laboratory: Calorimetry No Lab

Traditional Lab Group Activity No Lab

Fractional Crystallization Experiment

Stoichiometry Loss of CO2 Experiment

Aspirin Synthesis Experiment

Calorimetry Experiment

Atomic Emission Experiment

In Lab: Unknown Solid Identification Experiment LabSmart Laboratory: Reactions in Solution LabSmart Laboratory: Reactions in Solution

Exchange Reactions

14

B A

Mandatory ACS Lab Exam

The American Chemical Society (ACS) General Chemistry Laboratory Assessment is an instrument designed by the ACS Examinations Institute to measure student learning in the general chemistry laboratory.54 The exam consists of six laboratory scenarios: stoichiometry, volumetric analysis, calorimetry, spectrophotometry, identification of acids and bases, and kinetics. As one of the first ACS Exams to utilize an electronic delivery platform, the exam was designed to incorporate video and allows for question types beyond the traditional multiple choice such as drag and drop.54 For the purposes of this evaluation, students were required to complete the calorimetry, stoichiometry, and identification of acids and bases modulestopics that were covered as part of the firstsemester, general chemistry curriculum. All students completed the exam during the last lab meeting of the semester. The Meaningful Learning in the Laboratory Instrument (MLLI) measures students’ expectations for learning in the chemistry laboratory.51 MLLI consists of 31 items divided into three factors: 16 items that measure expectations associated with the cognitive domain, 8 items that measure expectations associated with the affective domain, and 6 items that incorporate elements of both domains. A single indicator question was also included to ensure that students were reading the questions and not just randomly selecting responses. MLLI was administered to students online twice during the semesteronce during the second week of classes and again during the final two. In addition to the typical laboratory write-ups completed during the semester, students were also required to produce a formal laboratory report to accompany the calorimetry experiment they performed during weeks 6 or 7 (see Table 1). In this particular experiment, students used the Dulong− Petit Law to experimentally determine the specific heat of nickel and to then compare it to its known value.52 In comparison to the normal laboratory write-ups where students briefly summarized their findings and answered a few questions, the formal laboratory report required students to produce a statement of purpose, a detailed introduction and background that included at least three scientific references, a

No Lab

In Lab: Atomic Emission Experiment

A

Cognitive, Affective Cognitive Cognitive, Psychomotor

See ref 53. bSee ref 50. cIncluded as part of the data analysis during Fall 2015 only.

Formulas and Equations Experiment

No Lab

13

Cognitive

a

Significant Figures and Density Experiment

In Lab: Calorimetry Experiment LabSmart Laboratory: Spectrophotometry

In Lab: Unknown Solid Identification Experiment Comprehensive Lab Practical

Domain(s)

American Chemical Society (ACS) General Chemistry Laboratory Assessmenta Meaningful Learning in the Laboratory Instrument (MLLI)b Calorimetry Lab Report Comprehensive Laboratory Practicalsc

Unknown Solid Identification Experiment

Comprehensive Lab Practical Mandatory ACS Lab Exam

B D



Article

Table 3. ACS General Chemistry Laboratory Assessment Scores for Hybrid and Traditional Lab Groups Mean Score (SD), by Scenario Semester

Treatment Group

N

Spring 2015

Hybrid Lab Traditional Lab Hybrid Lab Traditional Lab

112 365 83 581

Fall 2015

Acid−Base 11.4 11.2 11.3 11.2

Calorimetry

(3.7) (3.9) (4.5) (3.9)

13.0 12.1 12.8 13.0

Stoichiometry

(3.0) (3.4) (3.6) (3.4)

9.9 9.8 9.2 9.3

(2.4) (2.7) (2.4) (2.8)

Table 4. Summary MLLI Results for Hybrid and Traditional Lab Groups Mean Scorea (SD), by Domain and Administration Cognitive Semester

Treatment Group

N

Spring 2015


112 365 83 581

Fall 2015

Pre 69.2 65.6 75.3 70.5

(10.9) (14.1) (9.8) (12.2)

Affective Post

62.5 64.2 58.2 65.4

Pre

(13.8) (12.9) (10.2) (9.9)

63.8 63.8 67.1 60.6

(18.8) (18.9) (15.6) (19.8)

Cognitive/Affective Post

59.5 64.3 50.9 56.4

(23.6) (17.6) (14.2) (12.1)

Pre 62.9 59.3 63.9 58.5

(15.4) (16.5) (11.7) (16.7)

Post 54.6 57.6 58.3 60.2

(19.5) (15.6) (17.4) (14.6)

a

Numbers represent students’ confidence, in percentage, measured by the Meaningful Learning in the Laboratory Instrument.

using Multivariate Analysis of Variance (MANOVA).55,56 MANOVA is similar to ANOVA except it allows the examination of the impact of multiple, dependent variables on an independent variable. The MANOVA results indicated that none of the differences between the hybrid laboratory groups and the traditional laboratory groups were significant (p = 0.854). This suggests that the replacement of half of the FTF experiments with corresponding VL experiences did not have a negative impact on students’ cognitive understanding of acid/base chemistry, calorimetry, or stoichiometry. These results are similar to those previously reported in the literature.26,30−32

complete summary of their results and the analyses they conducted, and a conclusion. Although the laboratory experiment was performed as a group, each student was required to produce an individual report. At the end of the semester, students were asked to complete a comprehensive lab practical. For this activity, each student was provided with a separate question or goal and a list of available laboratory equipment and chemicals they could use to answer the question or accomplish the goal. Students were expected to use the knowledge they gained throughout the semester to decide upon the types of data they needed to collect, propose a reasonable procedure to collect that data, and determine how they would analyze it. Before initiating the experiment, the graduate teaching assistant was responsible for approving the proposed procedure to ensure that students were not using too much of any given chemical or were planning to do something unsafe. This approval did not consider whether the procedure would lead to a successful outcome. Throughout the lab period, the teaching assistant also monitored the students’ actions as they executed their plan so that a portion of their grade for the practical was ultimately based on their technical proficiency. This provided us with an opportunity to explore the impact of the laboratory modality on students’ psychomotor skills.

Meaningful Learning in the Laboratory Instrument (MLLI)

MLLI was administered to students online both at the beginning and end of the Spring and Fall 2015 semesters. For each of the 30 items contained in the MLLI, students were asked to use a slider bar to indicate their percent agreement with each statement. As such, scores could theoretically range from 0% (completely disagree with the statement) to 100% (completely agree with the statement). The scores for the items were averaged to create a composite value for the cognitive, affective, and cognitive/affective factors. Survey reliability was established through Cronbach α. In line with the results reported by Galloway and Bretz,51 α for the cognitive and affective domains were generally well above the 0.7 reliability threshold (α ranged between 0.710 and 0.881). The α-values for the cognitive/affective scale ranged between 0.517 and 0.570. Galloway and Bretz described similar results and speculated that the divide between the cognitive and affective domains may be quite large within students’ minds and that little integration occurs, thus leading to knowledge fragmentation.51 This fragmentation subsequently manifests itself in lower α-values for this particular scale.51 Previous cross-sectional57,58 and longitudinal59 studies exploring how expectations for learning chemistry evolve over the course of general chemistry 1 document a slight decline in expectations during the semester. The MLLI results collected from the students as part of this research (see Table 4) demonstrated a similar trend, although in general, the declines observed for the students that participated in the hybrid laboratory sections were much larger than those for the traditional sections.

■

RESULTS AND DISCUSSION As described above, the evaluation of the hybrid laboratory approach relied upon four inputs of datathe ACS General Chemistry Laboratory Assessment, MLLI, student-generated calorimetry laboratory reports, and students’ scores on the comprehensive laboratory practicalthat we hoped would provide insights into its impact on students’ cognitive, affective, and psychomotor learning. The results from these various data sources are presented below. ACS General Chemistry Laboratory Assessment

An individual score for each of the three scenarios was calculated for each student. The results are included in Table 3 for both the Spring and Fall 2015 semesters. In comparing the means included in Table 3, there appear to be few differences. Indeed, with the exception of the calorimetry results from Spring 2015, all differences between the groups were within 0.2 points of each other. The data were explored for significance E



Article

changes for the students that participated in the traditional laboratories represented significant improvements in expectations, all 13 unique changes for the hybrid laboratory students were declines. In considering the text of these items, it would seem as if the traditional laboratory students developed more positive views about the role of laboratory instrumentation and felt more comfortable handling chemicals. This is not surprising given the additional time they spent in the lab working with instrumentation and chemicals versus their colleagues that utilized the hybrid approach. They did, however, report a greater expectation of feeling intimidated about the lab which may be related to their belief that the laboratory procedures were not as simple as they initially expected them to be. By comparison, the expectations that the hybrid laboratory students had for focusing on the conceptual underpinnings of chemistry, for making connections within the material, and for understanding what the data they collected because of their experiments meant were not met during their time in the course. Further, students expected to face more confusion and frustration as a result of their laboratory instruction, and perhaps most worrisome, had significantly lower expectations that what they were learning would be valuable outside of the classroom. It may be that the lack of a “real world” experiment in the hybrid curriculum like the aspirin synthesis that the traditional laboratory groups performed contributed to this decline. At the same time, however, it is unlikely that the inclusion or omission of a single experiment had such a dramatic impact on students’ attitudes.

The data were again analyzed for statistical significance using MANOVA. Initially, the MANOVA revealed no significant differences between the hybrid and traditional laboratory groups (p = 0.1654); however, the postresults did (p = 0.0319). Post hoc analyses using the Bonferroni correction showed that the results for the cognitive scale were significantly lower (p = 0.0104) for the hybrid laboratory students versus the traditional laboratory students. Further, with the exception of the Fall 2015 cognitive/affective results, all pre-post mean comparisons for the hybrid laboratory groups were significantly lower. A more detailed, item-by-item analysis was performed on the MLLI results by separately comparing pre/post-averages for all 30 survey items for both groups of students. Statistically significant changes were flagged, and a list of items was assembled for each group. Each list included only those items that changed significantly and were unique to that particular group. The text of each item is included in either Table 5 Table 5. Unique MLLI Statements Showing Significant Change for Traditional Lab Students Because of Laboratory Instruction Change

Item

“When performing experiments in my chemistry laboratory course this semester, I expect...”

Improved Declined Declined Declined Improved Improved Declined Improved

Q2 Q6 Q15 Q17 Q18 Q27 Q28 Q29

To worry about finishing on time. To be confused about how the instruments work. The procedures to be simple to do. To “get stuck” but keep trying. To be nervous when handling chemicals. To be intrigued by instruments. To feel intimidated. To be confused about what my data mean.

Student-Generated Calorimetry Laboratory Report

Approximately halfway through the semester, all students were required to produce a formal laboratory report for the calorimetry experiment. This report contained a statement of purpose, a detailed introduction and background that included at least three scientific references, a summary of the results and the analyses that incorporated any necessary calculations and graphs, and a conclusion. Students also provided the answers to several postlab questions that asked them to think about how the calorimetry concepts they were learning about in their lecture course related to what they observed in the laboratory setting and to make predictions based on the results of their experiment. The formal laboratory reports presented us with an opportunity to evaluate how students critically analyzed their data, the evidence they used to draw their conclusion, and its soundness; how they weaved the references throughout their report and whether they did so appropriately; and their overall written communication skills. As part of its internal general education evaluation program, UNCW has been using the Valid Assessment of Learning in Undergraduate Education (VALUE) rubrics developed by the Association of American Colleges and Universities (AAC&U) for several years.60 The VALUE rubrics were released in 2009 to: “provide needed tools to assess students’ own authentic work, produced across students’ diverse learning pathways, fields of study and institutions, to determine whether and how well students are meeting graduation level achievement in learning outcomes that both employers and faculty consider essential.”60 For the purpose of this study, the calorimetry laboratory reports were evaluated using the Critical Thinking, Information Literacy, and Written Communication rubrics. Copies of these rubrics are available by visiting the AAC&U’s VALUE rubric Web site.61 Given the length of each laboratory report

Table 6. Unique MLLI Statements Showing Significant Change for Hybrid Lab Students Because of Laboratory Instruction Change

Item

“When performing experiments in my chemistry laboratory course this semester, I expect to...”

Declined Declined Declined Declined Declined Declined Declined Declined Declined Declined Declined Declined Declined

Q1 Q4 Q5 Q7 Q9 Q10 Q12 Q16 Q19 Q21 Q22 Q24 Q29

Learn chemistry that will be useful in my life. Feel unsure about the purpose of the procedures. Experience moments of insight. Learn critical thinking skills. Be nervous about making mistakes. Consider if my data makes sense. Feel disorganized. Be confused about the underlying concepts. Think about chemistry I already know. Be frustrated. Interpret my data beyond only doing calculations. Focus on procedures, not concepts. Be confused by what my data mean.

(Traditional Lab students) or Table 6 (Hybrid Lab students); significant improvements in expectations are denoted by a “improved” in the change column while significant declines are indicated with a “declined”. Note that the lists generated for both semesters were similar in content. The differences between the two lists are starknot only in their scope, but also their contentand underscore the broader trends described above. Whereas five of eight unique F



Article

Table 7. Composite Scores for the Critical Thinking, Information Literacy, and Written Communication Learning Outcomes Composite Mean Score (SD) Semester

Treatment Group

N

Spring 2015


23 62 29 139

Fall 2015

Critical Thinking 1.5 1.7 1.4 1.4

(0.78) (0.68) (0.23) (0.23)

Information Literacy 1.9 2.0 1.9 2.0

Written Communication

(0.85) (0.73) (0.21) (0.21)

1.8 1.8 2.1 2.1

(0.62) (0.65) (0.24) (0.26)

and reinforced the results reported previously for the ACS General Chemistry Laboratory Assessment: the integration of the VL experiences did not appear to impact, either positively or negatively, students’ cognitive development in the context of the general chemistry laboratory setting.

(approximately 8−10 pages each) and the number of students in the course (approximately 480 in Spring 2015 and 700 in Fall 2015), it was not possible to analyze each report. As such, 3−5 laboratory reports were randomly selected from each general chemistry section and analyzed using the VALUE rubrics for their level of proficiency regarding critical thinking, information literacy, and written communication. A total of 85 reports (62 traditional, 23 hybrid) were evaluated from students enrolled in the course during Spring 2015 and 168 reports (139 traditional, 29 hybrid) were evaluated from students enrolled in the course during Fall 2015. Each VALUE rubric consists of several dimensions and performance levels that range from 1 (benchmark) to 4 (capstone). Performance descriptors are provided for each level to help guide evaluators, and they are encouraged to assign a zero to a dimension in the event that it is completely missing from the student-generated artifact.60 Before the student researchers began using the rubrics to evaluate the calorimetry lab reports, they worked with the corresponding author to ensure an acceptable level of consistency in their ratings. The specific terms associated with the performance descriptors were discussed together as a group, and three laboratory reports were collaboratively evaluated. Any differences were reviewed until consensus was reached. Subsequently, the students were asked to independently evaluate a group of 10 randomly selected laboratory reports (these reports were separate from the pool of reports described above and not included in the final analyses) and to compare their results. Initial agreement was determined to be 0.76, which rose to 0.90 after discussion. Given this level of consistency, the students were then tasked with rating the randomly selected laboratory reports, the results of which appear in Table 7. The student researchers assigned a score of 0−4 for each separate dimension; a mean was calculated for the dimensional scores to generate a composite score for each report for each of the three learning outcomes. In general, the four positions of each dimension roughly correspond to the skill level expected of that group of students. So, for example, it would be expected that first-year students will perform around the benchmark (1) level or that third-year students will score around milestone (3) proficiency. When viewed through this lens, the composite scores reported in Table 6 align with expectations given the population of students served by our general chemistry coursesapproximately 70% first-year students. When the scores for the critical thinking, information literacy, and written communication rubrics were examined to ensure nonviolation of the underlying assumptions associated with the use of MANOVA, problems were detected. Strong correlations among the dependent variables existed and their variances were not homogeneous. As such, we elected to instead use a generalized Friedman rank sum test to explore the data for statistical significance.62 These analyses revealed that none of the differences were significanti.e., p > 0.05

Comprehensive Laboratory Practical

Toward the end of the Fall 2015 semester, students completed a comprehensive laboratory practical as a final examination for their general chemistry laboratory course. Each student was provided with an individual, open-ended question or goal and a list of chemicals and laboratory instrumentation/equipment that they would have access to to successfully answer the question or achieve the goal. As students completed their experiment, the graduate teaching assistants walked around and observed their laboratory technique and technical proficiency. As such, not only did the comprehensive laboratory practical provide additional insights into the students’ cognitive understanding of the material, but also allowed us to compare the psychomotor skills they developed as a result of a semester working in the general chemistry laboratory. To ensure consistency in their grading of the comprehensive laboratory practical, the graduate teaching assistants were provided with a detailed grading rubric that guided them in assigning points for the procedure the students developed, the technical proficiency they displayed throughout the experience, the quality and analysis of the data they collected, and how well they used that data to answer their question. For the purposes of this study, we compared both the average scores for students’ technical proficiencyinsight into their psychomotor skillsand the overall score assigned for the practical. In both cases, there were no statistically significant differences observed between the hybrid laboratory students and the traditional laboratory students. It was again encouraging to see no impact on students’ cognitive abilitiesin this case, abilities that extended well beyond simply understanding content and encompassed such essential skills as being able to plan an experiment and to be able to appropriately analyze data to answer a question. The results would also suggest that despite the hybrid laboratory students spending about half as much time in the actual lab setting as their traditional counterparts, the time differential did not impede the development of fundamental laboratory skills. It is important to note, however, that although these data provide a glimpse into students’ psychomotor development, it is merely thata glimpse. Additional research is necessary to determine whether there are differences that this single data collection event failed to capture in students’ technical competence. Alternatively, it may be that differences may only begin to manifest themselves in subsequent chemistry laboratory courses. G



■

Article

STUDY LIMITATION It is important to note that as is the case with all research, there are limitations to this study that future work must look to address. Although the experiment was conducted to ascertain the impact that the incorporation of the VL experience had on students versus those that were enrolled in a more traditional FTF laboratory setting, there were additional modifications made to the course. Indeed, there were several other variables introduced into the system because of the way the VL course was structured. The VL students alternated between two different modalities during the semester, one of whichthe virtualwas likely foreign to many of them. The discomfort and frustration students likely experienced when first using the VL materials may have impacted the attitudes they developed throughout the course. As such, it may be important to develop a more structured experience for students at the beginning of the semester as they first learn to use the online materials. By its very design, the groups completed different assignments, and even in instances where the two groups were completing the same laboratories, they often were offset by a week. Finally, it is important to note that different sections of the laboratories were taught by different graduate teaching assistants. The graduate teaching assistants were an essential component of the educational experience, and although they received weekly training with the general chemistry laboratory coordinator, there were undoubtedly differences among sections. It is not inconceivable that some of the differences documented between the two groups of students in this study were caused by these variations between the two educational environments.

much of the scientific world, and to gain some level of independence.3 It is important to note, however, that priority is generally given to goals associated with the cognitive and psychomotor domains in the undergraduate chemistry laboratory, despite the fact that all three are required components of meaningful learning.5 The hybrid laboratory students reported significantly higher expectations of being confused or frustrated by their work in the laboratory and for spending less time thinking about the underlying chemistry concepts, what they mean, and how they fit together. Most troubling, however, was the expectation that the material they were learning would be less useful in their daily lives. In such an environment, it is not unreasonable to expect students to give up more easily or to decide to more readily engage in rote memorization. Indeed, why would students expend the additional cognitive resources necessary to engage in meaningful learning if they believe it will have little utility outside of the chemistry classroom? Future research must focus on exploring these trends in more detail and hopefully finding ways of ameliorating them. In the current study, the hybrid laboratory students alternated weekly between VL experiments and the traditional, didactic FTF experiments. We hypothesize that it may be possible to reverse some of the more egregious declines by modifying the existing FTF experiments to specifically address some of the affective concerns identified. For example, the amended experiments could incorporate more real-world examples, or provide more scaffolded experiences to help decrease feelings of confusion and frustration among the students, or prompt students to explicitly think about the deeper meaning of what they are studying and the connection among concepts, or perhaps some combination of all three. It is also important to explore the difference in scope between the traditional and hybrid environments. In other words, what impact did the traditional students completing experiments on nearly double the number of topics have compared to the more limited curriculum completed by the hybrid students? It would also be valuable for future research to explore other ways of utilizing the VL experiments. In this case, they were used in place of FTF experiments. It may be possible to mitigate some of the affective declines by instead using them as supplements. As stated previously, the VL experiments often highlight areas where students make mistakes or attempt to draw their attention to important details they might otherwise miss. As such, they could potentially serve as effective prelaboratory activities, which previous research indicates could lead to enhanced cognitive development.20,43,63,64 Regardless, for the hybrid laboratory curriculum to have any chance of being a successful alternative to the traditional curriculum already in place, it must first be modified to better support students’ affective development. In so doing, we hope to provide students with a more solid foundation upon which to build their understanding and to encourage more meaningful learning of chemistry.

■

CONCLUSIONS Meaningful learning requires the intentional integration of the cognitive, affective, and psychomotor domains.5 In situations where this integration process does not fully occur, students may resort to rote memorization, which will ultimately lead to a less robust understanding of the discipline. This study sought to better understand how a hybrid curriculum that incorporated both virtual and face-to-face laboratory experiences impacted general chemistry students’ development in these three essential areas. When compared to students completing a more traditional laboratory program, the evidence suggests that there are few differences in the hybrid laboratory students’ cognitive growth. Their understanding of acid/base chemistry, calorimetry, and stoichiometry was comparable, as were their critical thinking, information literacy, and written communication skills. These results are similar to those reported previously in other research studies involving the use of virtual chemistry experiments.26,30−32 The limited information that we have also suggests that the decreased time spent working in the laboratory setting did not retard students’ psychomotor skills, although additional research is necessary to more carefully study the effects on this domain. In particular, future research must track students beyond the first semester of general chemistry to understand if there are any long-term impacts on their technical skills. A much more dramatic difference in students’ affective learning, as measured by the MLLI, was observed during both semesters in which the hybrid laboratory curriculum was utilized. When describing their affective goals for the chemistry laboratory, faculty generally express a desire that it help students make connections to the real world, for students to engage in the types of collaborative behaviors that characterize

■

AUTHOR INFORMATION

Corresponding Author

*E-mail: [email protected]. ORCID

Nathaniel P. Grove: 0000-0003-4354-6655 H



Article

Notes

(22) Ullah, S.; Ali, N.; Rahman, S. U. The Effect of Procedural Guidance on Students’ Skill Enhancement in a Virtual Chemistry Laboratory. J. Chem. Educ. 2016, 93 (12), 2018−2025. (23) Winkelmann, K.; Scott, M.; Wong, D. A Study of High School Students’ Performance of a Chemistry Experiment within the Virtual World of Second Life. J. Chem. Educ. 2014, 91 (9), 1432−1438. (24) Gal, Y. a.; Uzan, O.; Belford, R.; Karabinos, M.; Yaron, D. Making Sense of Students’ Actions in an Open-Ended Virtual Laboratory Environment. J. Chem. Educ. 2015, 92 (4), 610−616. (25) Winkelmann, K.; Keeney-Kennicutt, W.; Fowler, D.; Macik, M. Development, Implementation, and Assessment of General Chemistry Lab Experiments Performed in the Virtual World of Second Life. J. Chem. Educ. 2017, 94 (7), 849−858. (26) Hawkins, I.; Phelps, A. J. Virtual laboratory vs. traditional laboratory: which is more effective for teaching electrochemistry? Chem. Educ. Res. Pract. 2013, 14 (4), 516−523. (27) Moore, J. W. Academic Extensions of Gresham’s Law. J. Chem. Educ. 2008, 85 (4), 475. (28) American Chemical Society. Importance of Hands-on Laboratory Activities. https://www.acs.org/content/acs/en/policy/ publicpolicies/education/computersimulations.html (accessed October 24, 2017). (29) College Board. AP Chemistry: Course and Exam Description. http://media.collegeboard.com/digitalServices/pdf/ap/ap-chemistrycourse-and-exam-description.pdf (accessed October 24, 2017). (30) Bernard, R. M.; Abrami, P. C.; Lou, Y.; Borokhovski, E.; Wade, A.; Wozney, L. How does distance education compare with classroom instruction? A meta-analysis of the empirical literature. Rev. Educ. Res. 2004, 74 (3), 379−439. (31) Cavanaugh, C.; Gillan, K. J.; Kromrey, J.; Hess, M.; Blomeyer, R. The Effects of Distance Education on K-12 Student Outcomes: A MetaAnalysis; Learning Point Associates: Naperville, IL, 2004. (32) Tatli, Z.; Ayas, A. Effect of a Virtual Chemistry Laboratory on Students’ Achievement. Educ. Technol. Soc. 2013, 16 (1), 159−170. (33) Smetana, L.; Bell, R. Computer simulations to support science instruction and learning: a critical review of the literature. International Journal of Science Education 2012, 34, 1337−1370. (34) Stone, D. Teaching chromatography using virtual lab exercises. J. Chem. Educ. 2007, 84, 1488−1496. (35) Diwakar, S.; Kumar, D.; Radhamani, R.; Sasidharakurup, H.; Nizar, N.; Achuthan, K.; Nedungadi, P.; Raman, R.; Nair, B. Complementing education via virtual labs: Implementation and deployment of remote laboratories and usage analysis in South Indian villages. International Journal of Online and Biomedical Engineering 2016, 12, 8−15. (36) Mejias Borrero, A.; Andujar Marquez, J. A pilot study of the effectiveness of augmented reality to enhance the use of remote labs in electrical engineering education. J. Sci. Educ. Technol. 2012, 21, 540−557. (37) Geban, Ö .; Askar, P.; Ö zkan, I.̈ Effects of Computer Simulations and Problem-Solving Approaches on High School Students. Journal of Educational Research 1992, 86 (1), 5−10. (38) Burke, K. A.; Greenbowe, T. J.; Windschitl, M. A. Developing and Using Conceptual Computer Animations for Chemistry Instruction. J. Chem. Educ. 1998, 75 (12), 1658. (39) Akpan, J. P.; Andre, T. Using a Computer Simulation Before Dissection to Help Students Learn Anatomy. J. Comput. Math. Sci. Teach. 2000, 19 (3), 297−313. (40) Dori, Y.; Barak, M. Virtual and physical molecular modeling: Fostering model perception and spatial understanding. Educ. Technol. Soc. 2001, 4 (1), 61−74. (41) Kennepohl, D. Using computer simulations to supplement teaching lab in chemistry for distance delivery. J. Distance Educ. 2001, 16 (2), 58−65. (42) Huppert, J.; Lomask, S. M.; Lazarowitz, R. Computer simulations in the high school: Students’ cognitive stages, science process skills and academic achievement in microbiology. International Journal of Science Education 2002, 24 (8), 803−821.

The authors declare no competing financial interest.

■ ■

ACKNOWLEDGMENTS The authors would like to thank the general chemistry students and faculty for volunteering to participate in this research. REFERENCES

(1) Spear, E. B. Problems in the experimental pedagogy of chemistry. Journal of Educational Psychology 1915, 6 (4), 231−241. (2) Wiley, W. H. An experimental study of methods in teaching high school chemistry. Journal of Educational Psychology 1918, 9 (4), 181− 198. (3) Bretz, S. L. Evidence for the Importance of Laboratory Courses. J. Chem. Educ. 2019, 96, 193−195. (4) Bretz, S. L.; Fay, M.; Bruck, L. B.; Towns, M. H. What Faculty Interviews Reveal about Meaningful Learning in the Undergraduate Chemistry Laboratory. J. Chem. Educ. 2013, 90 (3), 281−288. (5) Novak, J. D. Learning, Creating, and Using Knowledge; Taylor & Francis Group: New York, NY, 2010. (6) Blosser, P. E. A Critical Review of the Role of the Laboratory in Science Teaching; The Ohio State University: Columbus, OH, 1980; p 165. (7) Hofstein, A.; Lunetta, V. N. The Role of the Laboratory in Science Teaching: Neglected Aspects of Research. Review of Educational Research 1982, 52 (2), 201−217. (8) Bryce, T. G. K.; Robertson, I. J. What can they do? A review of practical assessment in Science. Studies in Science Education 1985, 12 (1), 1−24. (9) Hodson, D. Re-thinking old ways: Towards a more critical aproach to practical work in school science. Studies in Science Education 1993, 22, 85−142. (10) Lazarowitz, R.; Tamir, P. Research on using laboratory instruction in science. In Handbook of Research on Science Teaching and Learning; Gabel, D. L., Ed.; Macmillan: New York, NY, 1994; pp 94−130. (11) Hofstein, A.; Lunetta, V. N. The laboratory in science education: Foundations for the twenty-first century. Sci. Educ. 2004, 88 (1), 28−54. (12) Hofstein, A.; Mamlok-Naaman, R. The laboratory in science education: the state of the art. Chem. Educ. Res. Pract. 2004, 8 (2), 105−107. (13) Alpert, D.; Bitzer, D. L. Advances in Computer-based Education. Science 1970, 167 (3925), 1582−1590. (14) Smith, S. G.; Sherwood, B. A. Educational Uses of the PLATO Computer System. Science 1976, 192 (4237), 344−352. (15) Cavin, C. S.; Cavin, E. D.; Lagowski, J. J. Study of the efficacy of computer-simulated laboratory experiments. J. Chem. Educ. 1978, 55 (9), 602. (16) Herman, H. B.; Dilts, J. A. Computer enhancements of general chemistry and quantitative analysis laboratories. J. Chem. Educ. 1978, 55 (10), 654. (17) Moore, C.; Smith, S.; Avner, R. A. Facilitation of laboratory performance through CAI. J. Chem. Educ. 1980, 57 (3), 196. (18) Smith, S. G.; Chabay, R. Computer games in chemistry. J. Chem. Educ. 1977, 54 (11), 688. (19) Wiegers, K. E.; Smith, S. The use of computer-based chemistry lessons in the organic laboratory course. J. Chem. Educ. 1980, 57 (6), 454. (20) Woodfield, B. F.; Catlin, H. R.; Waddoups, G. L.; Moore, M. S.; Swan, R.; Allen, R.; Bodily, G. The Virtual ChemLab Project: A Realistic and Sophisticated Simulation of Inorganic Qualitative Analysis. J. Chem. Educ. 2004, 81 (11), 1672. (21) Woodfield, B. F.; Andrus, M. B.; Andersen, T.; Miller, J.; Simmons, B.; Stanger, R.; Waddoups, G. L.; Moore, M. S.; Swan, R.; Allen, R.; Bodily, G. The Virtual ChemLab Project: A Realistic and Sophisticated Simulation of Organic Synthesis and Organic Qualitative Analysis. J. Chem. Educ. 2005, 82 (11), 1728. I



Article

(43) Limniou, M.; Papadopoulos, N.; Giannakoudakis, A.; Roberts, D.; Otto, O. The integration of a viscosity simulator in a chemistry laboratory. Chem. Educ. Res. Pract. 2007, 8 (2), 220−231. (44) Chen, S. The view of scientific inquiry conveyed by simulationbased virtual laboratories. Comput. Educ. 2010, 55 (3), 1123−1130. (45) Achuthan, K.; Murali, S. A comparative study of educational laboratories from cost and learning effectiveness perspective. Advances in Intelligent Systems and Computing 2015, 349, 143−153. (46) Arnaud, C. H. New approaches to undergraduate lab classes. Chem. Eng. News 2017, 95 (16), 26−27. (47) Widener, A. UNC Wilmington chemists relocate in wake of Hurricane Florence. Chem. Eng. News 2018, 96 (44), https://cen.acs. org/education/UNC-Wilmington-chemists-relocate-in-wake-ofHurricane-Florence/96/i44 (accessed April 2019). (48) Ausubel, D. P. Education Psychology: A Cognitive View; Holt, Rinehart, and Winston: New York, NY, 1968. (49) Bruck, L. B.; Towns, M.; Bretz, S. L. Faculty Perspectives of Undergraduate Chemistry Laboratory: Goals and Obstacles to Success. J. Chem. Educ. 2010, 87 (12), 1416−1424. (50) Bruck, A. D.; Towns, M. Development, Implementation, and Analysis of a National Survey of Faculty Goals for Undergraduate Chemistry Laboratory. J. Chem. Educ. 2013, 90 (6), 685−693. (51) Galloway, K. R.; Bretz, S. L. Development of an Assessment Tool To Measure Students’ Meaningful Learning in the Undergraduate Chemistry Laboratory. J. Chem. Educ. 2015, 92 (7), 1149− 1158. (52) Kieber, C.; Kieber, R. J.; Ward, C. R. General Chemistry 101/ 102 Laboratory Manual, 5 ed.; Kendall Hunt: Dubuque, IA, 2012. (53) McGraw Hill Education. Connect Online Access for LearnSmart Labs General Chemistry. https://www.mheducation. com/highered/product/1259406776.html (accessed April 2019). (54) Examinations Institute Laboratory Assessment. https://uwm.edu/ acs-exams/instructors/assessment-materials/laboratory-assessment/ (accessed April 2019). (55) Krzanowski, W. J. Principles of Multivariate Analysis: A User’s Perspective; Oxford University Press: Oxford, 2000. (56) Hand, D. J.; Taylor, C. C. Multivariate Analysis of Variance and Repeated Measures: A Practical Approach for Behavioural Scientists; Chapman and Hall: Boca Raton, FL, 1987. (57) Grove, N.; Bretz, S. L. CHEMX: An Instrument To Assess Students’ Cognitive Expectations for Learning Chemistry. J. Chem. Educ. 2007, 84 (9), 1524. (58) Galloway, K. R.; Bretz, S. L. Measuring Meaningful Learning in the Undergraduate Chemistry Laboratory: A National. J. Chem. Educ. 2015, 92 (12), 2006−2018. (59) Galloway, K. R.; Bretz, S. L. Measuring Meaningful Learning in the Undergraduate General Chemistry and Organic Chemistry Laboratories: A Longitudinal Study. J. Chem. Educ. 2015, 92 (12), 2019−2030. (60) Rhodes, T. Assessing Outcomes and Improving Achievement: Tips and Tools for Using the Rubrics; Association of American Colleges and Universities: Washington, D.C., 2009. (61) Association of American Colleges and Universities. VALUE Rubrics. http://aacu.org/value-rubrics (accessed April 2019). (62) Wittkowski, K. M. Friedman-type statistics and consistent multiple comparisons for unbalanced designs of missing data. J. Am. Stat. Assoc. 1988, 83, 1163−1170. (63) Burewicz, A.; Miranowicz, N. Effectivenes of multimedia laboratory instruction. Chem. Educ. Res. Pract. 2006, 7, 1−12. (64) Supasorn, S.; Suits, J. P.; Jones, L. L.; Vibuljan, S. Impact of a pre-laboratory organic-extraction simulation on comprehension and attitudes of undergraduate chemistry students. Chem. Educ. Res. Pract. 2008, 9 (2), 169−181.

J


The Evaluation of a Hybrid, General Chemistry Laboratory Curriculum

Recommend Documents