This is an open access article published under an ACS AuthorChoice License, which permits copying and redistribution of the article or any adaptations for non-commercial purposes.
Article Cite This: J. Chem. Educ. 2019, 96, 1300−1307
pubs.acs.org/jchemeduc
Improving Learning Outcomes in Secondary Chemistry with Visualization-Supported Inquiry Activities Mike Stieff*
Downloaded via 109.94.221.189 on July 17, 2019 at 07:54:47 (UTC). See https://pubs.acs.org/sharingguidelines for options on how to legitimately share published articles.
Department of Chemistry, University of Illinois, Chicago, Illinois 60607, United States ABSTRACT: Inquiry activities with component visualization tools are increasingly prevalent in K−12 STEM classrooms; however, evidence of their efficacy has primarily been collected from controlled laboratory studies that lack ecological validity or from small-scale classroom interventions that assess learning outcomes proximal to the intervention. Here, the long-term, differential impact of visualization-supported inquiry activities on learning outcomes was examined in the context of secondary chemistry. Student learning was assessed on summative content assessments completed by a cohort of 1152 students who used visualization-supported inquiry activities from The Connected Chemistry Curriculum during a school year. Results demonstrate that the cohort performed significantly better on ACS Conceptual Exam assessment items aligned with curriculum learning objectives. This study demonstrates that visualization tools embedded in inquiry activities result not only in short-term gains but in long-term improvements relative to business-as-usual practices. KEYWORDS: High School/Introductory Chemistry, Computer-Based Learning, Inquiry-Based/Discovery Learning, Chemical Education Research, Testing/Assessment FEATURE: Chemical Education Research
■
INTRODUCTION Visualization tools, such as animations and simulations, embedded in inquiry activities have been shown to improve student learning outcomes, particularly in chemistry.1 The benefits of such tools for improving learning have been attributed to their designs, which make imperceptible scientific phenomena, such as atoms and forces, observable and interactive. Importantly, the benefit of visualization tools relates to how students use them: their positive impact is more effective and reliable when they are coupled with inquiry activities2−6 or pedagogical supports.7−9 With such scaffolds, student interactions with the visualization tool are supported so that they can investigate the visualization as they would observable, tangible phenomena in the laboratory.10 There is good evidence for the efficacy of visualization-supported inquiry activities for improving learning in several STEM domains.3,11,12 Notwithstanding this evidence, the potential of visualizations for improving long-term learning in a chemistry classroom remains unknown, as extant studies have been limited to laboratory studies and short classroom interventions. As such, we know that visualization tools can improve learning over short periods, but the effect of learning with these tools on learning outcomes cannot be assumed. In this paper, the efficacy of visualization-supported inquiry activities embedded in a yearlong secondary chemistry curriculum, The Connected Chemistry Curriculum, was investigated to identify the potential of such activities to improve long-term learning outcomes. As with other such inquiry activities, previous studies of individual Connected Chemistry Curriculum (CCC) activities have reported positive learning outcomes among secondary students in multiple settings.13−17 However, none of these studies have taken up as an object of inquiry the long-term impact of the curriculum on student learning in a self-contained chemistry © 2019 American Chemical Society and Division of Chemical Education, Inc.
classroom. The present study examines how visualizationsupported inquiry activities benefits student learning in the short term and whether these benefits remain evident on summative achievement assessments at the end of the school year.
■
SUPPORTING CHEMISTRY LEARNING WITH VISUALIZATION TOOLS IN INQUIRY ACTIVITIES Visualization-supported inquiry activities have been shown to address multiple obstacles for chemistry learning. These activities attempt to make explicit the information embedded in two-dimensional representations and to provide a visual display of scientific phenomena for students to observe directly. Such activities provide the student with multiple, linked representations that foster conceptual understanding and emphasize the connections among the particulate level, the macroscopic level, and symbolic representations.18 Here, students learn science by viewing dynamic visualizations of particle interactions side-by-side with graphical outputs, displays of actual phenomena, and chemical formulas. This is in contrast to lecture-based pedagogies that almost exclusively rely on verbal explanations of concepts in which students receive no opportunity to interact (directly or virtually) with the phenomena they are learning. Visualization-supported inquiry activities offer at least three primary advantages to science learning beyond traditional didactic methods that do not employ visualization tools. First, learning environments that employ visualization tools provide students with direct perceptual access to the phenomena of interest in the science classroom. For example, visualization Received: March 12, 2019 Revised: May 31, 2019 Published: June 22, 2019 1300
DOI: 10.1021/acs.jchemed.9b00205 J. Chem. Educ. 2019, 96, 1300−1307
Journal of Chemical Education
Article
implemented a CCC activity in one classroom and covered the same material using business-as-usual practices in another classroom. Immediately before and after instruction in both classrooms, the teachers administered a standardized assessment testing knowledge of concepts related to the activity learning objectives. The results of this analysis revealed that students who completed CCC activities achieved higher scores. Although the effect size was small, further analysis of items that required students to produce chemistry representations in their explanations revealed significant differences in the quality of learning between the two conditions. The results indicate that CCC helps students to improve (1) understanding of the relationship between macroscopic observations and submicroscopic events, (2) reading comprehension by relating visualization to information communicated in text, and (3) adoption of the epistemic practices modeled by teachers. These results suggest that CCC, as one example of an inquiry curriculum with a visualization tool component, has strong potential to improve long-term learning gains in the classroom. However, no studies have been conducted that analyze student learning while the students are engaged in the systematic use of the curriculum over the course of an entire school year. Further, previous studies on the efficacy of the curriculum and its related materials have focused on comparing learning outcomes between groups of students who use CCC and those who do not.
tools can present rich visual displays of molecules and atoms to provide students with additional resources to support their understanding of chemical phenomena and their interactions. This access supports student learning by presenting information via multiple modalities that students can interrogate to recognize recurring patterns, support scientific claims, and engage in argumentation practices inherent to scientific inquiry.3 Second, visualization tools support students’ competence in understanding scientific representations that are used to communicate. In science, multirepresentational visualizations of imperceptible objects and phenomena make explicit the information embedded in external representations with interactive visual displays.18 Such tools often include multiple representations that are dynamically linked to help students both perceive the relationship between the representing and represented worlds and connect various external representations together.18 Using such tools, students can improve their ability both to generate accurate scientific representations and to use them appropriately for describing imperceptible interactions.12,16 Across disciplines, such tools help students to develop a more integrated knowledge of both scientific representations and domain knowledge.19 Finally, visualization tools enhance student opportunities for gaining the related knowledge and skills of scientific inquiry when embedded in inquiry-based curriculum materials. By providing students with direct perceptual access to the world of imperceptible phenomena, such curricula support students learning the investigative practices that characterize the chemical sciences. For example, teachers and students can appeal to molecular simulations to explore submicroscopic explanations to questions such as “Why do liquids and gases flow?” Likewise, highly interactive visualization tools that offer opportunities to modify system parameters in simulations support students’ learning of experimental techniques. With guidance from curriculum support materials and teachers, students can generate their own research questions that require them to plan investigations that occur in both the laboratory and in a visualization. Here, we examine the efficacy of visualization-supported inquiry activities embedded in The Connected Chemistry Curriculum (CCC).20 The Connected Chemistry Curriculum Project has focused on the iterative development of print and software materials for supporting the teaching and learning of secondary general chemistry with supplementary curriculum materials. The curriculum makes central use of a visualization tool to improve conceptual understanding and representational competence in secondary chemistry. CCC employs visualization software that includes visual representations embedded in a modeling environment as the core component of inquiry learning activities throughout an academic year. To that end, the curriculum includes over 100 visualization-supported inquiry activities that focus on active student-driven explorations of molecular-level simulations and animations. As with other such curriculum, CCC was designed to provide students with direct visual access to scientific phenomena with visualization tools, to support the linking of multiple representations, and to provide students with opportunities to engage in modeling and argumentation practices relevant to the discipline of chemistry. Previous studies of CCC have shown that specific curriculum activities directly improve student learning in chemistry.12,16,17 In one such study, individual teachers
■
PRESENT STUDY The present study examined the efficacy of a longitudinal implementation of visualization-supported guided inquiry activities for supporting chemistry learning by using a research design that involved one cohort of students alternatively using both CCC and a district-mandated curriculum. The cohort completed pre- and post-summative chemistry achievement assessments for each CCC content unit and a comprehensive summative achievement assessment at the end of the school year. To determine the impact of CCC on student learning, gains for each content unit assessment were analyzed, and performance on CCC-aligned items on the end-of-year assessment were compared with that on nonaligned items. The main prediction was that achievement would be highest on items that assessed learning objectives targeted by Connected Chemistry Curriculum units. To validate that The Connected Chemistry Curriculum promoted learning relevant to the learning objectives of each implemented unit, it was also predicted that achievement would be greater on each of the unit post-tests compared with on the pretests.
■
STUDY DESIGN
Participants
Participants included 12 chemistry teachers in 10 secondary schools located in one school district in the U.S. mid-Atlantic region with 1152 students enrolled in their general chemistry classrooms. During the year of the study, all participating teachers used the same district-supported curriculum and textbook and replaced three content units with three units from The Connected Chemistry Curriculum that targeted the same learning objectives as the replaced units. Instruments
Chemistry learning was measured with three summative unit assessments from The Connected Chemistry Curriculum and the 2001 ACS General Chemistry (Conceptual) Exam.21 1301
DOI: 10.1021/acs.jchemed.9b00205 J. Chem. Educ. 2019, 96, 1300−1307
Journal of Chemical Education
Article
Figure 1. Curriculum materials of The Connected Chemistry Curriculum, including a standalone software application (left) and supporting workbooks (right).
Connected Chemistry Curriculum Summative Unit Assessments. For this study, Particulate Nature of Matter, Chemical Reactivity, and Chemical Equilibrium Unit Assessments were administered. Connected Chemistry Curriculum summative unit assessments were developed by the curriculum designers in consultation with five secondary chemistry teachers (∼40 cumulative years of experience in secondary chemistry instruction) and a district science assessment specialist (∼26 years of experience in science assessment). Each unit assessment included both fixed-choice and shortanswer items that assessed the target learning objectives for the related unit. Content validity was established via comparison with existing ACS assessments. All items are scored with a binary rubric. Each test displayed excellent reliability for an academic achievement assessment: Particulate Nature of Matter, Cronbach’s α = 0.92 (36 items); Chemical Reactivity, Cronbach’s α = 0.90 (23 items); and Chemical Equilibrium, Cronbach’s α = 0.81 (31 items). American Chemical Society 2001 General Chemistry Exam (Conceptual). To assess chemistry achievement at the end of the school year, a summative achievement assessment composed of 40 items from the 60-item ACS General Chemistry exam was employed. Twenty items assessed content instructed by CCC, and 20 items assessed content instructed by the local district textbook-based curriculum. The 40 items were selected from those items that assessed content taught by the participating school district during the research study; the excluded 20 items assessed content outside the scope and sequence of the district curriculum. The conceptual units in the final list included Particulate Nature of Matter, Chemical Reactivity, Chemical Bonding, Stoichiometry, Gas Laws, Chemical Equilibrium, and Descriptive Chemistry. Excluded items included Nuclear Chemistry, Acid−Base Chemistry, Thermodynamics, and Kinetics. The reliability of the intact assessment is reported as KR-21 = 0.86, and the 40-item scale employed here displayed acceptable reliability for an academic achievement assessment (Cronbach’s α = 0.74).
nuclear processes (PS1C), definitions of energy (PS3A), and energy in chemical processes (PS3D). Additionally, the curriculum attends to all seven crosscutting concepts included in the NGSS. Most notably, the entire curriculum centers around investigations of a model system of atomic and molecular behavior (Concept 4: Systems and System Models) to determine the relationship between submicroscopic processes and phenomena and observable properties of matter (Concept 3: Scale, Proportion, and Quantity and Concept 6: Structure and Function). In each lesson, students search for patterns in gathered data (Concept 1: Patterns) and manipulate system variables to determine the influences of different factors on molecular behavior (Concept 2: Cause and Effect: Mechanism and Explanation). Finally, students work to explain how energy is released and absorbed from different chemical transformations and how certain macroscopic quantities (e.g., concentration, pressure, temperature, etc.) are the results of stochastic processes (Concept 3: Energy and Matter: Flows, Cycles, and Conservation and Concept 7: Stability and Change). Curriculum materials for the present study were taken from v1.0. CCC is a comprehensive set of curriculum materials (Figure 1) that includes multiple instructional activities that provide students with opportunities to problematize their learning, generate demand for knowledge, support refinement of understanding, and illustrate the applications of chemistry. CCC curriculum materials include (1) a stand-alone software application that presents molecular-level simulations designed in Java with supporting Flash animations and (2) a set of curriculum workbooks that scaffold interactions with the software through multiple diverse learning activities. CCC is organized around nine CCC curriculum units, which include multiple guided inquiry Lessons that students can complete within one to three 45 min sessions. Units are organized around nine core disciplinary ideas that are routinely taught in secondary chemistry classrooms across the United States (i.e., Modeling & Matter, Solutions, Chemical Reactions, Pressure & Gas Laws, Kinetics, Thermodynamics, Acids & Bases, Equilibrium, and Nuclear Chemistry), which makes CCC especially easy to integrate into a diverse range of chemistry classrooms. CCC Lessons encourage active exploration in both virtual and physical laboratory environments. Using the developed visualization software, students can manipulate various parameters of a virtual chemical reaction under study in the chemistry laboratory, make predictions on the basis of their
Learning Intervention
The Connected Chemistry Curriculum has been fully described in other publications,12,15,16,19,22 and a summary taken from those descriptions is provided here for reference. Curriculum materials were designed with attention to the principles advocated by the Next Generation Science Standards (NGSS Lead States, 2013).23 In brief, the units cover a core set of disciplinary ideas that include properties of matter (PS1A), chemical reactions (PS1B), types of interactions (PS2B), 1302
DOI: 10.1021/acs.jchemed.9b00205 J. Chem. Educ. 2019, 96, 1300−1307
Journal of Chemical Education
Article
Figure 2. CCC’s molecular laboratory interface.
real world, and elicit prior knowledge. These activities include extended reading activities to support reading comprehension and to help students develop language skills and essential vocabulary related to science content. Demonstration Activities include dynamic demonstrations conducted by the teacher using simulations or wet lab materials to model observation and inquiry skills and encourage students to make predictions. Physical Modeling Activities provide students with alternative models and representations of submicroscopic phenomena, such as magnetic models, related to the virtual simulations. These models enrich student understanding of the relationships between three-dimensional structures and concepts such as bonding and polarity. Finally, Putting-It-All-Together Activities help students reflect on their observations and understanding after a lesson through group presentations, problem solving, and independent research. As a technology-infused curriculum, Simulation Activities are the foundation of CCC and warrant elaboration. Each Simulation Activity provides students with visual representations linked to core disciplinary concepts taught as emergent phenomena. The concept of emergent phenomena recognizes that patterns observed on a macrolevel “emerge” from the interactions among many agents on a microlevel or submicrolevel, according to specific rules that govern individual agent behavior.24 For instance, the specific submicroscopic interactions among the many billions of molecules in a drop of water result in the liquid observed at the macroscopic level. When the water molecules move at relatively high velocities and their interactions are elastic, water vapor is observed on the macroscopic level. Conversely, when the water molecules lack translational motion but vibrate in place, the observer sees solid ice on the macroscopic level. To achieve this, each Simulation Activity makes use of a common molecular laboratory interface (Figure 2). This Java-based interface contains a graphics window, a plotting window, and several variables in the form of input sliders and buttons that students manipulate, allowing them to observe changes in output measurements. The graphics window in the center of the molecular laboratory window is where students observe a visual representation of the interactions among simulated atoms or compounds. The motion and interactions of the molecules in the graphics window result from all of the molecules executing, in parallel, the individual rules that govern their behavior; it is not a static animation linked to predetermined states. For instance, when a student alters the temperature of the system, each molecule individually responds to the temperature change
understanding of relevant chemistry concepts presented in discussion, and then compare their observations of the simulation to their predictions. Consequently, students can monitor their own learning by observing the results of their manipulations of system variables. By facilitating iterative processes of learning through repeated explorations of each simulation, CCC offers chemistry students more than a visualization of the molecular world: it provides students with the opportunity to engage in the same processes of scientific inquiry that define the practice of chemistry. To achieve this, CCC lessons make use of seven different types of CCC Activities to promote learning in chemistry. Three core activities, Laboratory, Simulation, and Discussion, form the foundation of every curriculum unit. In Laboratory Activities, students perform a standard laboratory experiment to gather macroscopic observations about chemical phenomena. Laboratory activities allow students to explore and observe macroscopic properties that are directly visible. CCC is designed to be responsive to school resources and does not mandate a specific experiment but recommends a range of experiments to offer teachers the flexibility to select one on the basis of the resources available to them. In Simulation Activities, pairs of students explore a CCC simulation to understand the nature of the submicroscopic interactions that are responsible for the macrolevel events observed in the laboratory. Each pair completes a guided inquiry activity to explore the simulation, make predictions, and generate explanations about the relations between their submicroscopic observations in the virtual environment and their macroscopic observations in the laboratory environment. Students also have opportunities to draw representations and write narrative explanations of chemical processes before and after viewing a simulation to reflect on their understanding. In Discussion Activities, the teacher leads students through a synthesis of their observations to discuss the conceptual underpinnings that link submicroscopic interactions with observable macroscopic properties. In this activity, the teacher may project a CCC simulation in front of the classroom and work with their students to deduce and affirm abstract concepts and general laws from observations made in both the laboratory and the simulation. Here, students can reflect on their original predictions and resolve unexpected findings on the basis of the classroom discussion. Simulation, Laboratory, and Discussion Activities are supported in CCC with Connecting, Demonstration, Physical Modeling, and Putting-It-All-Together Activities. Connecting Activities capture student interest, make connections to the 1303
DOI: 10.1021/acs.jchemed.9b00205 J. Chem. Educ. 2019, 96, 1300−1307
Journal of Chemical Education
Article
Figure 3. Violin plots of pre- and post-test scores for the three CCC units implemented by participating teachers.
■
by absorbing a quantum of energy from the heated boundary. Rather than merely displaying a new animation of molecular motion at a higher temperature, each molecule changes its behavior as if energy were flowing into the closed system. Thus, students observe directly how changes in a macroscopic system variable, such as temperature, are related to the submicroscopic interactions that in turn perturb other variables. Importantly, the representations in the graphics window are pedagogical models that maintain a high degree of fidelity to scientific models with reduced complexity to help students bootstrap their understanding of domain concepts. So that students can observe the effects of their manipulations of the system on observable variables and learn how to represent these relationships graphically, each interface window includes at least one plotting window and multiple variable outputs.
RESULTS
Direct Improvement from CCC in Content Learning for Implemented Units
To verify that CCC units causally improved learning outcomes of target learning objectives relative to possible pre-existing differences in prior knowledge, chemistry knowledge related to each CCC unit was assessed in participating classrooms before and after each CCC unit was implemented, and researchers observed every day of implementation. On the basis of field reports from observers, participating classes completed ∼85% of CCC curriculum activities across the three content units. Qualitative observations of teacher practices indicated the CCC materials were implemented consistently with the practices teachers learned to use in the professional development workshops. Student learning was assessed before and after each CCC content unit was implemented: teachers administered unit assessments 1 day prior to beginning each unit and 1 day after completing each unit. Only responses from students who completed both the pre- and post-tests were analyzed. All teachers implemented the Particulate Nature of Matter Assessment and the Chemical Reactivity Unit Assessment with 1126 and 968 valid student responses, respectively. Five teachers chose to implement the Chemical Equilibrium Unit Assessment with 267 valid student responses. Seven teachers elected not to administer the Chemical Equilibrium Unit Assessment because of time constraints in their schedules and their personal estimates of the perceived difficulty of the assessment; however, these teachers did implement the Chemical Equilibrium Unit. Large effect improvements from pre- to post-tests were observed for all three content units (Figure 3). On the Particulate Nature of Matter Assessment, achievement increased from 33% (SD = 19%) to 51% (SD = 23%), t(1126) = 763, p < 0.001, Cohen’s d = 0.91. For Chemical Reactivity, student achievement increased from 24% (SD = 27%) to 44% (SD = 22%), t(968)= 668, p < 0.001, Cohen’s d = 0.88. For Chemical Equilibrium, used by five teachers, student achievement increased from 9.6% (SD = 5.6%) to 19% (SD = 5.5%), t(267)= 842, p < 0.001, Cohen’s d = 1.7. Thus, the implementation of each CCC unit appeared to be a primary factor contributing to improving student learning related to the target learning objectives of the respective unit.
Procedure
In a quasi-experimental mixed-design study, 1152 students completed three CCC units (Unit 1: Particulate Nature of Matter, Unit 2: Chemical Reactivity, and Unit 3: Chemical Equilibrium), which were randomly selected from the curriculum to replace the business-as-usual curriculum activities adopted by the partner school district. Curriculum and instructional approaches for business-as-usual units focused on lecture-based pedagogy supplemented by textbook reading and problem solving. All three units were implemented by the 12 participating teachers during the normal course of study in 10 secondary schools in the mid-Atlantic region of the United States. Participating teachers had completed a 3 week professional development course regarding the use of CCC materials. Each teacher integrated the CCC units into the district’s curriculum sequence and implemented them throughout the school year. Units were implemented at each teacher’s discretion, with Unit 1 implemented by all teachers between months 1 and 3 of the school year, Unit 2 implemented between months 4 and 6, and Unit 3 implemented between months 7 and 9. The average duration of each unit was approximately 1 week of instruction (Particulate Nature of Matter: 4 days, Chemical Reactivity: 5 days, and Chemical Equilibrium: 5 days). PDF versions of the simulations and exemplar curriculum materials can be found online.25 Researchers visited each teacher’s classroom every day a unit implementation occurred to observe classroom activities. 1304
DOI: 10.1021/acs.jchemed.9b00205 J. Chem. Educ. 2019, 96, 1300−1307
Journal of Chemical Education
Article
Table 1. Psychometric Properties of the ACS Exam ACS-Reported Psychometricsa Item
Curriculum Alignment
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40
CCC CCC CCC District District District District CCC CCC CCC District District District District CCC District CCC CCC CCC CCC District District District District District District CCC CCC CCC District District CCC CCC CCC CCC CCC CCC District District District
b
Present Study
Difficulty Index
Discrimination Index
Difficulty Index
Discrimination Index
0.96 0.83 0.58 0.55 0.41 0.8 0.46 0.35 0.36 0.77 0.4 0.39 0.36 0.39 0.46 0.34 0.48 0.37 0.51 0.75 0.58 0.29 0.43 0.79 0.32 0.67 0.63 0.85 0.67 0.54 0.49 0.45 0.3 0.44 0.33 0.35 0.43 0.47 0.58 0.47
0.14 0.25 0.64 0.56 0.61 0.37 0.52 0.65 0.31 0.45 0.61 0.11 0.33 0.56 0.64 0.46 0.57 0.58 0.35 0.43 0.55 0.5 0.55 0.54 0.19 0.20 0.60 0.37 0.43 0.38 0.51 0.33 0.21 0.56 0.36 0.36 0.52 0.23 0.46 0.58
0.88 0.75 0.36 0.37 0.23 0.62 0.32 0.19 0.36 0.46 0.17 0.35 0.28 0.21 0.29 0.14 0.32 0.19 0.38 0.61 0.30 0.22 0.13 0.52 0.21 0.47 0.30 0.62 0.39 0.44 0.26 0.31 0.17 0.22 0.21 0.19 0.22 0.33 0.34 0.31
0.22 0.40 0.43 0.32 0.26 0.45 0.51 0.42 0.26 0.64 0.20 0.11 0.27 0.20 0.33 0.25 0.28 0.31 0.37 0.60 0.42 0.17 0.18 0.64 0.05 0.27 0.52 0.57 0.46 0.34 0.29 0.21 0.08 0.10 0.24 0.16 0.34 0.18 0.53 0.30
a
See ref 26. bCCC, The Connected Chemistry Curriculum; District, school-district-supported curriculum.
displayed in Table 1 relative to the psychometric properties of the instrument reported by the American Chemical Society. An independent-samples t test indicated that the average difficulties of CCC-related and district-related items did not differ statistically: t(40) = 1.08, p = 0.31. Given this, differences in performance on items aligned with each curriculum are not likely due to differences between the different types of items. Second, each item on the assessment was scored in a binary fashion, and the total score was analyzed with a repeatedmeasures analysis of variance comparing the subscores for each set of items (district-aligned vs CCC) with students nested in teachers. On average, students performed better than chance (25%) on this fixed-choice response assessment. Furthermore, students performed better on items aligned with the CCC unit learning objectives (M = 36.4%, SD = 16.0%) than on items aligned with learning objectives from the local curriculum (M = 30.8%, SD = 14.4%; F(1, 900) = 371, p < 0.001, ηp2 = 0.29;
Moreover, performance on each pretest demonstrated that students were not generally knowledgeable about the related content assessed on the end-of-year ACS Exam prior to instruction with CCC. Improvement in Long-Term Learning of Domain Content with CCC Relative to Business-as-Usual Practices
To determine the relative impact of CCC and business-asusual practices on long-term student learning outcomes, two analyses were conducted on the end-of-year achievement assessment. First, the difficulty and discrimination indices for this administration of the ACS Exam were calculated. The difficulty index ranks item difficulty according to the proportion of respondents that successfully answer an item. The discrimination index ranks items by the correlation between performance on an item and total score. The itemlevel properties for the 40 items used in this study are 1305
DOI: 10.1021/acs.jchemed.9b00205 J. Chem. Educ. 2019, 96, 1300−1307
Journal of Chemical Education
Article
Figure 4. Violin plots of relative performances of participants on ACS-items aligned with district materials and CCC materials.
the one that did not. Notably, CCC activities were implemented for a period of 1 week on three separate occasions during the school year, whereas the comparison curriculum was used on all other days of the school year. As such, this finding supports the value-added benefits of implementing visualization tools embedded in inquiry activities in STEM classrooms. Moreover, the finding that students performed better under the CCC condition on content assessment items that were comparably difficult to those in the district condition suggests that the benefits observed here from CCC would also be observed broadly in the chemistry classroom. Importantly, the variance among teachers highlights how the benefits of the visualization-supported guided inquiry curriculum for improving learning cannot be assumed without considering the role of the teacher in student learning or other classroom contextual factors. STEM achievement and persistence continues to lag in the United States relative to that in other countries,28 and efforts to improve the number of students pursuing STEM-related degrees in higher education remain an important future target in chemistry education research. Curriculum- and pedagogicalbased interventions continue to show the most promise for improving STEM achievement and persistence in K−12 settings and university settings.29,30 Among these reforms, visualization-supported inquiry curricula have been shown to be particularly effective at facilitating conceptual change and improving classroom engagement,31 which mediate positive learning outcomes. The present findings demonstrate that visualization-supported inquiry curricula can lead to improved learning outcomes relative to more typical textbook-based methods, as reported previously. More importantly, these findings show that the benefits of learning from such tools have a long-term impact on learning outcomes, which can be detected up to several months after the tools were implemented in the chemistry classroom.
Figure 4). Performance between teachers varied significantly: F(1, 900) = 878, p < 0.001, ηp2 = 0.49. A post hoc comparison (Bonferroni correction) of the average performance on each subset of items indicated a large effect from the curriculum (Cohen’s d = 0.91).
■
DISCUSSION AND IMPLICATIONS Prior studies of visualization-supported inquiry activities have shown substantial success in improving student learning outcomes in chemistry. These studies, though positive, have been limited to small-scale, short-term interventions in artificial or real settings. Comparative studies across diverse contexts suggest that innovative learning environments such as these are employed differently across contexts and that not all learners engage with these environments in the same way:27 extended implementations may produce more similar (or more divergent) outcomes from our prior findings. The present study aimed to address this limitation with a longitudinal study of student learning outcomes to determine how the sustained use of visualization-support inquiry activities impacted both short-term and long-term learning outcomes. Student learning from three units of The Connected Chemistry Curriculum was assessed to determine the efficacy of the sustained use of visualization-support inquiry activities. CCC is a secondary science curriculum that aims to improve student learning outcomes in chemistry by improving the student’s representational competence and promoting conceptual change in chemistry through the coordinated use of virtual and physical laboratory environments and a comprehensive professional development program that helps teachers integrate CCC curriculum materials into their classrooms. The efficacy of the curriculum was evaluated with a within-subjects design that compared the performances of individual students against their performances on an end-of-year summative content assessment. In addition, students were assessed before and after completing each CCC unit. The results of the study demonstrate that student content knowledge improved after completing each unit and that students performed better on the end-of-year summative assessment items aligned with the CCC core learning objectives. The significant difference in learning outcomes that was observed demonstrates that the same students exposed to two different curricula here learned more with the one that included visualization-supported inquiry activities than with
■
AUTHOR INFORMATION
Corresponding Author
*E-mail: mstieff@uic.edu. ORCID
Mike Stieff: 0000-0002-1639-891X Notes
The author declares no competing financial interest. 1306
DOI: 10.1021/acs.jchemed.9b00205 J. Chem. Educ. 2019, 96, 1300−1307
Journal of Chemical Education
■
Article
(18) Russell, J.; Kozma, R.; Jones, T.; Wykoff, J.; Marx, N.; Davis, J. Use of Simultaneous-Synchronized Macroscopic, Microscopic, and Symbolic Representations to Enhance the Teaching and Learning of Chemical Concepts. J. Chem. Educ. 1997, 74, 330−334. (19) Stieff, M.; Ryan, S. Explanatory Models for the Research and Development of Chemistry Visualizations. In Pedagogic Roles of Animations and Simulations in Chemistry Courses; Suits, J., Sanger, M., Eds.; ACS Symposium Series; ACS Books, 2013; Vol. 1142, pp 15− 41. (20) Stieff, M.; Nighelli, T.; Yip, J.; Ryan, S.; Berry, A. The Connected Chemistry Curriculum; University of Illinois: Chicago, 2012; Vols. 1− 9. (21) American Chemical Society. General Chemistry (Conceptual) Exam; ACS DivCHED Examinations Institute: Clemson, SC, 2001. (22) Stieff, M. Drawing for Promoting Learning and Engagement with Dynamic Visualizations. In Learning from Dynamic Visualizations: Innovations in Research and Application; Lowe, R., Plötzner, R., Eds.; Springer: Dordrecht, 2017; pp 333−356. (23) NGSS Lead States. Next Generation Science Standards: For States, By States. The National Academies Press: Washington, DC, 2013. (24) Wilensky, U.; Resnick, M. Thinking in Levels: A Dynamic Systems Perspective to Making Sense of the World. J. Sci. Educ. Technol. 1999, 8, 3−18. (25) The Connected Chemistry Curriculum. http://connchem.org (accessed May 30, 2019). (26) ACS Exams. American Chemical Society. http://uwm.edu/acsexams/ (accessed May 30, 2019). (27) Roschelle, J.; Shechtman, N.; Tatar, D.; Hegedus, S.; Hopkins, B.; Empson, S.; Knudsen, J.; Gallagher, L. P. Integration of Technology, Curriculum, And Professional Development for Advancing Middle School Mathematics: Three Large-Scale Studies. Am. Educ. Res. J. 2010, 47, 833−878. (28) Science Performance (PISA). OECD. https://data.oecd.org/ pisa/science-performance-pisa.htm (accessed May 31, 2017). (29) National Research Council. Successful K−12 STEM Education: Identifying Effective Approaches in Science, Technology, Engineering, and Mathematics Workshops; The National Academies Press: Washington, DC, 2011. (30) National Research Council. Promising Practices in Undergraduate Science, Technology, Engineering, and Mathematics Education: Summary of Two Workshops; The National Academies Press, Washington, DC, 2011. (31) Lowe, R., Plö tzner, R.,, Eds.; Learning from Dynamic Visualization: Innovations in Research and Application; Springer: Dordrecht, 2017.
ACKNOWLEDGMENTS Reported data are available upon request from the corresponding author. Data collection was sponsored by a grant from the Maryland Higher Education Commission (ITQ-10-814). Data analysis was sponsored by a grant from the Institute of Education Sciences, U.S. Department of Education (R305A100992). The author acknowledges Jason Yip, Lama Jaber, and Megean Garvin for support with data collection and Kristen Murphy and Jeff Raker of the American Chemical Society Exams Institute for providing the psychometric properties of the published instrument. Special thanks goes to the partner teachers and students in participating schools.
■
REFERENCES
(1) Gilbert, J. K. Visualization: A Metacognitive Skill in Science and Science Education. In Visualization in Science Education; Gilbert, J. K., Ed.; Springer: Dordrecht, 2005; pp 9−27. (2) De Jong, T.; Lazonder, A. W. The Guided Discovery Learning Principle in Multimedia Learning. In The Cambridge Handbook of Multimedia Learning, 2nd ed.; Mayer, R. E., Ed.; Cambridge University Press: Cambridge, 2014; pp 371−390. (3) Linn, M. C.; Lee, H.-S.; Tinker, R.; Husic, F.; Chiu, J. L. Teaching and Assessing Knowledge Integration in Science. Science 2006, 313, 1049−1050. (4) Mayer, R. E. Multimedia Learning; Cambridge University Press: Oxford, 2001. (5) Podolefsky, N. S.; Perkins, K. K.; Adams, W. K. Factors Promoting Engaged Exploration with Computer Simulations. Physical Review Special Topics-Physics Education Research 2010, 6, 0201171− 11. (6) Stieff, M.; Bateman, R.; Uttal, D. Teaching and Learning with Three-Dimensional Representations. In Visualization in science education; Gilbert, J. K., Ed.; Oxford University Press: Oxford, 2005; pp 93−120. (7) Lehtinen, A.; Viiri, J. Guidance Provided by Teacher and Simulation for Inquiry-Based Learning: A Case Study. J. Sci. Educ. Technol. 2017, 26, 193−206. (8) Rutten, N.; van Joolingen, W. R.; van der Veen, J. T. The Learning Effects of Computer Simulations in Science Education. Comput. Educ. 2012, 58, 136−153. (9) Rutten, N.; van der Veen, J. T.; van Joolingen, W. R. InquiryBased Whole-Class Teaching with Computer Simulations in Physics. International Journal of Science Education 2015, 37, 1225−1245. (10) Edelson, D. C.; Gordin, D. N.; Pea, R. D. Addressing the Challenges of Inquiry-Based Learning through Technology and Curriculum Design. Journal of the Learning Sciences 1999, 8, 391−450. (11) Berney, S.; Bétrancourt, M. Does Animation Help Learning: A Meta-Analysis? Computers & Education 2016, 101, 150−167. (12) Stieff, M. Improving Representational Competence Using Molecular Simulations Embedded In Inquiry Activities. J. Res. Sci. Teach. 2011, 48, 1137−1158. (13) Levy, S. T.; Wilensky, U. Crossing Levels and Representations: The Connected Chemistry (CC1) Curriculum. J. Sci. Educ. Technol. 2009, 18, 224−242. (14) Levy, S. T.; Wilensky, U. Students’ Learning with the Connected Chemistry (CC1) Curriculum: Navigating the Complexities of the Particulate World. J. Sci. Educ. Technol. 2009, 18, 243−254. (15) Stieff, M. Connected Chemistry−A Novel Modeling Environment for he Chemistry Classroom. J. Chem. Educ. 2005, 82, 489−493. (16) Stieff, M.; McCombs, M. Increasing Representational Fluency with Visualization Tools. Proceedings of the Seventh International Conference of the Learning Sciences (ICLS), Bloomington, IN, June 27 to July 1, 2006; Barab, S., Hay, K. E., Hickey, D. T., Eds.; Erlbaum, 2006; Vol. 1, p 730. (17) Stieff, M.; Wilensky, U. Connected Chemistry - Incorporating Interactive Simulations into the Chemistry Classroom. J. Sci. Educ. Technol. 2003, 12, 285−302. 1307
DOI: 10.1021/acs.jchemed.9b00205 J. Chem. Educ. 2019, 96, 1300−1307
