Flipped Chemistry Courses: Structure, Aligning Learning Outcomes

Oct 26, 2017 - Combinations of pedagogies have also been described, such as Flipped PLTL (11) and Flipped JiTT (12). A range of emotions can arise whe...
0 downloads 8 Views 888KB Size
Chapter 12

Flipped Chemistry Courses: Structure, Aligning Learning Outcomes, and Evaluation Downloaded by UNIV OF FLORIDA on November 3, 2017 | http://pubs.acs.org Publication Date (Web): October 26, 2017 | doi: 10.1021/bk-2017-1261.ch012

Alison B. Flynn* Department of Chemistry and Biomolecular Sciences, University of Ottawa, 10 Marie Curie, Ottawa, Ontario, Canada, K1N 6N5 *E-mail: [email protected].

In this chapter, I describe the structure, implementation, and evaluation of flipped and blended courses at the University of Ottawa. In a flipped course, lectures (or information transmission) take place out of class (either in video or text form) and in-class time is dedicated to interactive learning activities. The courses discussed were primarily large (>350 students) organic chemistry or spectroscopy courses. One of the flipped and blended courses was a small (~15 students) course in Applications of Spectroscopy, taught in French. In the courses, there was an emphasis on aligning learning outcomes with course components, which is described herein. Overall, the flipped courses had higher grades and student satisfaction compared to courses taught in a lecture or an active lecture format.

Introduction In this chapter, I share the work I have done with flipped courses, as well as evaluating the effects of the changes (1), including some changes since the original publication, such as the structure of my blended course and how others in my department have since flipped their own courses. In a flipped course, lectures (or information transmission) take place out of class (either in video or text form) and in-class time is dedicated to interactive learning activities (Figure 1). Blended course definitions are more varied, with some definitions describing a blended course as a mix of online and face-to-face course components, and others adding the requirement that some face-to-face time © 2017 American Chemical Society Sörensen and Canelas; Online Approaches to Chemical Education ACS Symposium Series; American Chemical Society: Washington, DC, 2017.

Downloaded by UNIV OF FLORIDA on November 3, 2017 | http://pubs.acs.org Publication Date (Web): October 26, 2017 | doi: 10.1021/bk-2017-1261.ch012

be removed. The latter definition is a component of the University of Ottawa’s blended course definition. To call the course “blended” at uOttawa, class time must be reduced by 20–80% (2). I think there is no single correct definition for a blended course, and that a course structure should be chosen based on a combination of the course’s intended learning outcomes, the instructor’s ability and comfort level, and students’ needs. Both flipped and blended course types share similarities with other course structures, such as Peer Instruction (3), Just-InTime-Teaching (4), Team Based Learning (5), Problem-Based Learning (6, 7), and Process Oriented Guided Inquiry Learning (8–10). Combinations of pedagogies have also been described, such as Flipped PLTL (11) and Flipped JiTT (12). A range of emotions can arise when considering a new course structure or being in the process of developing or implementing one, including: excitement, curiosity, anxiety, fear, stress, skepticism, and pleasure.

Figure 1. Summary of course structures.

Before I flipped my courses, I was using an active lecture format—a lecture punctuated by clicker questions. I was using class time to cover definition and simpler ideas. In other words, I was going over the easier material and students were passively listening, for the most part (Figure 2). Students were then responsible for tackling more complex ideas and problems out of class, either on their own or with peers. Over years of teaching the same course, I was going over these same definitions and concepts repeatedly, and felt that there was a better way to spend the little class time that the students and I had during the course. Flipping the course offered a way to more effectively use the three hours a week together, in which the students could do the easier work at home before class, and we could spend class time working through more complex ideas—the harder stuff. By creating in-class opportunities for students to interact with peers, give and receive feedback, and challenge themselves to go beyond what they might do alone, students have opportunities to construct knowledge in a social environment (13, 14). Aligned with this social constructivist perspective, I also try to create an environment for meaningful learning (15, 16). 152 Sörensen and Canelas; Online Approaches to Chemical Education ACS Symposium Series; American Chemical Society: Washington, DC, 2017.

Downloaded by UNIV OF FLORIDA on November 3, 2017 | http://pubs.acs.org Publication Date (Web): October 26, 2017 | doi: 10.1021/bk-2017-1261.ch012

Figure 2. Comparing how class time was used before and after flipping.

Course Contexts and Structure The four courses I first flipped included two large (~400 students) Organic Chemistry courses (I and II), both taught in English, and two third year Applications of Spectroscopy courses. I taught the larger (~150 students) spectroscopy course in English, and the smaller (~15 students) spectroscopy course in French and in uOttawa’s first Active Learning Classroom; the spectroscopy courses had identical learning outcomes. The larger courses were all taught in standard fixed-seating auditoriums. After teaching Applications of Spectroscopy (English) once in a flipped format, I further changed it to a blended format. I used a similar structure for both the flipped and blended courses; Figure 3 gives an overview of the structure, which I also described in an online video (17). Each week begins with communicating the intended learning outcomes (LOs). I have used the Bloom (18) and Structure of Observed Learning Outcomes (19, 20) taxonomies when constructing the LOs; the Next Generation Science Standards also provide a great source for thinking about Los (21). Outside of class, students watch videos and/or read textbook sections related to the LOs. I use existing videos (e.g., from YouTube) whenever I can find ones that align with the LOs. Otherwise, I create them myself. Some videos are simpler explainer videos with handwritten explanations, while others have been created in collaboration with professionals from uOttawa’s Centre for e-Learning (part of our Teaching and Learning Support Service). Examples of videos can be found at OrgChem101.com, an open-access learning tool that scaffolds student learning of core chemistry skills and concepts (22). After watching the videos and before coming to class, students complete an online pre-class test that assesses their baseline success with the LOs, using a chemistry homework program (23). I choose simpler pre-class test questions from 153 Sörensen and Canelas; Online Approaches to Chemical Education ACS Symposium Series; American Chemical Society: Washington, DC, 2017.

Downloaded by UNIV OF FLORIDA on November 3, 2017 | http://pubs.acs.org Publication Date (Web): October 26, 2017 | doi: 10.1021/bk-2017-1261.ch012

among the database of choices and align them as well as I can with the videos. In the future, I plan to create more questions that are even better aligned with the videos.

Figure 3. Flipped course structure. I integrate any areas of difficulty on the pre-class test into the class, where class time is dedicated to interactive learning activities (Figure 4). I choose activities that I think will help students achieve the LOs. These activities can include analyzing animations (24), writing a scientific argument, interpreting data, working with molecular models, etc. We use a document camera to collect written answers or look at molecular models from specific perspectives, and a classroom response system to collect and analyze students’ responses (25). Depending on the activity, the students might work individually or in small groups. Students can download or print the class notes, which consist of questions, data, and other material that does not need to be recopied, similar to Seery’s flipped physical chemistry course (26). The intent is that students spend class time working with information and concepts, rather than copying down written or spoken content.

Figure 4. In class activities.

Out of class, the students can attend tutorial sessions (i.e., discussion group sessions, recitations), office hours, and use the course’s discussion forum (although most prefer to use Facebook or other social media tools). Students complete an 154 Sörensen and Canelas; Online Approaches to Chemical Education ACS Symposium Series; American Chemical Society: Washington, DC, 2017.

Downloaded by UNIV OF FLORIDA on November 3, 2017 | http://pubs.acs.org Publication Date (Web): October 26, 2017 | doi: 10.1021/bk-2017-1261.ch012

online assignment to wrap up each week, which assesses their achievement of the LOs; the assignment is more difficult than the pre-class test. The pre-class test of one week and the previous week’s assignment are due at the same time every week, so that students need only remember one weekly deadline. The blended course structure used in the English Applications of Spectroscopy course is quite similar to the flipped one, except that students have a choice for four of the 22 classes: they can either do an extra module (that would use the fours class periods) or replace that module with a group assignment. Approximately 70% of students in the 2015 course chose to do the assignment, which involved analyzing a series of spectra to determine the structure of a complex unknown compound. When deciding on the format for the assessments, I consider the LOs and students’ needs. For example, I consider how I (or teaching assistants or peers) will assess students’ progress, how students will assess their own progress, and how the feedback process will work. In doing so, I consider whether: formative or summative assessments would be most appropriate, assessment will take place in or out of class, and the assessment should be individual or in group. There are many course components, which gives the potential for students (and the professor!) to become lost or confused. There are three main ways I mitigate these potential issues: (1) In the syllabus, I explain to students the course format, deadlines, and the reasons for the flipped course structure—I can now also include the positive results found in my earlier study (1); (2) I use the Learning Management System to guide students by giving them quick access to the course materials in the left menu bar, and suggesting an order in which to do the activities within each module (Figure 5); and (3) I keep the structure consistent throughout the course, posting notes, assignments, and videos by the same time every week, and keeping the deadlines consistent for students.

Figure 5. Learning management system guides students.

155 Sörensen and Canelas; Online Approaches to Chemical Education ACS Symposium Series; American Chemical Society: Washington, DC, 2017.

Downloaded by UNIV OF FLORIDA on November 3, 2017 | http://pubs.acs.org Publication Date (Web): October 26, 2017 | doi: 10.1021/bk-2017-1261.ch012

Learning Outcomes Anchor the Course Learning outcomes anchor the course, and are considered at all levels of the curriculum: the degree level, program level (27), course level, and module level (18, 20, 28). At the finer-grained module level, I analyze the course structure for alignment between the course components and the module-level LOs (Figure 6). For example, we established a set of organic acid–base LOs that we determined were essential to future studies in organic chemistry and other courses (29). Students were originally not taught how to find or estimate pKa values, nor were they explicitly assessed on that skill (30); rather, students were told to refer to a pKa table, assuming they would know which values to draw from the table. One of our studies revealed that many students lacked this information literacy skill—even high achieving students. In the flipped version of the course, this information literacy skill became explicitly explained, practiced, and assessed in the course, and students’ skills have since improved.

Figure 6. Learning outcomes anchor the course components. In another example of careful alignment of the LOs to the course components, Figure 6 shows opportunities for students to watch videos, practice, and be assessed on identifying the most acidic and basic atoms (within and between compounds). Before flipping the courses, there were not any online activities to practice (and receive feedback) on working with concepts related to basic atoms, which would not be an issue if students were scoring high on assessments. Having seen lower scores on identifying the most basic atoms in structure, we have created numerous questions for students to practice in a variety of contexts (22). As a last example of alignment, we expected that students would have the skills to draw a structure at a given pH, given all the earlier scaffolding tasks (e.g., draw the predominant species of alanine at pH 7.4). However, we found that students struggled with such questions and the associated concepts (e.g., equilibrium, multiple possible reaction pathways, solvent, relationship between pH and pKa). Accordingly, we developed instructional videos and practice activities (22). This type of alignment activity helped me understand what aspects of my course LOs were important, and where I was putting emphasis in the course 156 Sörensen and Canelas; Online Approaches to Chemical Education ACS Symposium Series; American Chemical Society: Washington, DC, 2017.

(through instruction, activities, and assessment). I have been able to rework aspects of the course to put more emphasis on helping students achieve the LOs that I think are most important.

Downloaded by UNIV OF FLORIDA on November 3, 2017 | http://pubs.acs.org Publication Date (Web): October 26, 2017 | doi: 10.1021/bk-2017-1261.ch012

Evaluation I conducted an initial evaluation of the flipped courses, using previous years’ courses as a baseline for comparison (1). Institutional Research Board approval (ethics) was obtained for this study. The study used a Guskey evaluation framework (31, 32), and revealed higher grades and student satisfaction, plus lower withdrawal and failure rates in the flipped courses compared to the previous years lecture (or active lecture) courses (Figure 7 and Figure 8) (1). Factors such as class sizes (over the 5 years of data used in the study, the average enrollment was 374 in Organic I and 386 in Organic II) and time slots in which courses were offered had not changed between years. Full details are provided in a previous article (1).

Figure 7. Final grades compared between lecture and flipped courses. N = 364 – 1226.

Comparing students’ results on identical final exams in two different years (students had never seen any of the questions before) given by the same instructor with the same curriculum revealed higher grades in the flipped course than in the one taught in an active lecture format (a lecture punctuated with clicker questions) (Figure 9) (1). “The average grade on the final exam was higher in 2013 (M=65%, SD=18%) than in 2011 (M=63%, SD=19%). A one-tailed t-test for independent samples revealed a statistically significant difference between the data, t(786) = 1.92, p = 0.03. The effect size was small (Cohen’s d = 0.11) (1).” There could be a few reasons for these higher grades, including natural year-to-year variation or the effect of the flipped course. Course grades were also higher in the flipped course than in four previous years’ courses, including those taught by the same instructor (Figure 7). 157 Sörensen and Canelas; Online Approaches to Chemical Education ACS Symposium Series; American Chemical Society: Washington, DC, 2017.

Downloaded by UNIV OF FLORIDA on November 3, 2017 | http://pubs.acs.org Publication Date (Web): October 26, 2017 | doi: 10.1021/bk-2017-1261.ch012

Figure 8. Lower failure (bottom) and withdrawal (top) rates in the flipped courses. N = 364 – 1226.

Figure 9. Final exam grades compared between active lecture and flipped courses. N = 377 (active lecture) and N = 409 (flipped). Student satisfaction was high in all four courses after they were flipped, as measured by anonymous course evaluation results (1). The question frequently arises of how to reduce student resistance to a new method. In the courses I taught, I did not encounter much student resistance. I believe this is due to having clearly communicated the course structure and reasons for the change (from lecture), and from staying consistent in the format throughout the semester (i.e., there were no 158 Sörensen and Canelas; Online Approaches to Chemical Education ACS Symposium Series; American Chemical Society: Washington, DC, 2017.

Downloaded by UNIV OF FLORIDA on November 3, 2017 | http://pubs.acs.org Publication Date (Web): October 26, 2017 | doi: 10.1021/bk-2017-1261.ch012

surprises thrown at the students). I informally ask for feedback during the course, such as using the classroom response system to conduct an exercise such as StartStop-Continue: what should we start doing in class; what should we stop doing; and what should we continue to do? Usually there are one or two students who ask me to switch back to a lecture format and just tell them what they need to know. This type of request provides a great opportunity to describe what it means to learn—I think about learning as a complementary combination of social constructivism, meaningful learning, and modern information processing theory (33). I frequently use examples from sport, such as asking if we could expect that an Olympic athlete could explain their sport to us, and we could go out this afternoon and perform at their level? What is the role of practice, peers, and a coach? I am not sure that I can convince everyone in the class that active engagement is needed for learning, but I certainly try! To better understand how class time was being used in the flipped class, three research students observed two 80 minute class sessions of a flipped Organic Chemistry II course. To do so, they used the Classroom Observation Protocol for Undergraduate Science, Technology, Engineering, and Math courses (34), or COPUS. Two of the students came to class with the COPUS protocol, which consists of a grid of possible classroom events, one part to record what the instructor is doing, the other to record what the students are doing (Figure 10). A decision was made every two minutes as to what the instructor and students were doing in the class, with each rater working individually. This process generated approximately 1000 data points over the 80 minute class sessions. After the classes, the raters compared their results, with possible outcomes including that both raters agreed that an activity occurred, both raters left an activity blank (i.e., that it did not take place), or the raters disagreed. Raters had a 90% agreement and a Krippendorf’s α = 0.73 for the first class session they observed (35), and 98% agreement and a Krippendorf’s α = 0.95 for the second class session they observed. The findings revealed a high percentage of class time with the students actively engaged, either working individually, discussing in a group, etc. The findings for one of the class sessions is shown in Figure 11; both sessions had a similar distribution of activities.

Figure 10. How COPUS protocol was used to observe flipped classes. 159 Sörensen and Canelas; Online Approaches to Chemical Education ACS Symposium Series; American Chemical Society: Washington, DC, 2017.

Downloaded by UNIV OF FLORIDA on November 3, 2017 | http://pubs.acs.org Publication Date (Web): October 26, 2017 | doi: 10.1021/bk-2017-1261.ch012

Figure 11. COPUS findings for one 80 minute class session. Q = question. Students’ activities are shown on the left; instructor’s activities are shown on the right. The more passive moments for students are pulled away from the pie. A common question arises: do students do the work? While we have not studied this question in detail, we do know that a high percentage of students are clicking on the videos (over 85%) and even more are completing the pre-class test and assignments. Furthermore, classroom attendance is very high at 88%, based on classroom response system participation (25). The workload in the flipped course is not meant to be higher for students than in a regular course. Rather, their time is spent differently. In one example, remaking the videos from the content explained in lecture into digestible chunks made me realize how much more precise and systematic I could be when making a video. My videos started out as short (15 – 20 minutes) and are now shorter (2–5 minutes). There is an increasing number of studies on flipped learning appearing in the literature, even since a recent review (36) and a recent ConfChem on the Flipped Classroom (37). One study in a general chemistry course controlled for the time students spent outside of class and compared students’ final exam scores over three years using the American Chemical Society’s standardized exams (38). The authors found that students’ scores increased by almost one standard deviation in the flipped course compared to the lecture course. Other groups similarly found statistically significantly higher grades and decreases in failure and withdrawal rates in other chemistry courses (39–44). One of those studies (39) also found that students in the flipped course had higher emotional satisfaction and intellectual accessibility compared to the students in traditional lecture courses, as measured by the Attitude toward the Subject of Chemistry Inventory Version 2 (45). Exam and attitudinal gains also have been observed in flipped high school chemistry classrooms (46). Higher grades and/or student satisfaction have been reported in other disciplines, including mathematics and statistics (47), physiology (48), and biochemistry (49). However, not all reports are positive (50–52), showing that future studies are needed on the factors that help students learn in various course formats, in addition to studying the nature of students’ learning outcomes. 160 Sörensen and Canelas; Online Approaches to Chemical Education ACS Symposium Series; American Chemical Society: Washington, DC, 2017.

Downloaded by UNIV OF FLORIDA on November 3, 2017 | http://pubs.acs.org Publication Date (Web): October 26, 2017 | doi: 10.1021/bk-2017-1261.ch012

Conclusions and Next Steps After flipping my own courses, we obtained funding from uOttawa’s Blended Learning Initiative (53) to support a group of professors interested in flipping their own organic chemistry courses. With the funding, the Organic Chemistry I course materials were translated from English to French, and the course was “translated” into a format that another educator could understand. The existing course material was arranged in a way that was understandable to me (e.g., by file types such as videos, course notes, problem sets) but did not help someone else understand how those materials might fit together (the sequence, where each item might be placed, how these might be shared with students, etc.). To “translate” the course materials into a useable form, I hired a student who had recently taken the course. This student created a course calendar with posting dates and other timelines. He also helped set up the course in the learning management system for professors who wanted that help. To date, five professors have used the course materials. Each one used the components they found useful, and adapted them to their own uses and course preferences. Flipping a course takes a lot of work, but I found that it took no more time than teaching a course for the first time. Having flipped my courses, I could not go back to a lecture format—if nothing else, I simply enjoy class time too much now. I get to interact with students and see them interact with each other in a way that was never possible in a lecture. When learning issues arise, we have the time and space to dig into them. I have also started to incorporate advanced questions into the notes I provide to students. Students who finish the assigned class questions quickly and easily can move on to the more challenging ones; I post the answers after class. The most important recommendation I can make to educators making a similar change to their courses is to use a structure and learning outcomes that are simple and consistent for students, and to clearly communicate that structure and intended outcomes to students. Many resources exist for developing courses with active learning components that can be used to lessen the “activation energy” for change.

Acknowledgments I thank the students in my courses, my research group members (FlynnResearchGroup.com), Gilles Lamothe, Jean-Luc D’Aoust, and funding from the Centre for e-Learning (uOttawa), and eCampusOntario.

References 1.

2.

Flynn, A. B. Structure And Evaluation Of Flipped Chemistry Courses: Organic & Spectroscopy, Large And Small, First To Third Year, English And French. Chem. Educ. Res. Pract. 2015, 16, 198–211. Teaching and Learning Support Service. What is a blended course? http:// tlss.uottawa.ca/site/en/what-is-a-blended-course (accessed July 13, 2017). 161 Sörensen and Canelas; Online Approaches to Chemical Education ACS Symposium Series; American Chemical Society: Washington, DC, 2017.

3.

4.

5. 6.

Downloaded by UNIV OF FLORIDA on November 3, 2017 | http://pubs.acs.org Publication Date (Web): October 26, 2017 | doi: 10.1021/bk-2017-1261.ch012

7. 8. 9.

10. 11.

12. 13.

14. 15. 16.

17. 18. 19. 20.

21.

Mazur, E. Interactive Teaching: Promoting Better Learning Using Peer Instruction and Just-in-Time Teaching; Pearson Prentice Hall: Upper Saddle River, NJ, 2004. Novak, G.; Patterson, E. T.; Gavrin, A. D.; Christian, W. Just-In-Time Teaching: Blending Active Learning with Web Technology; Prentice Hall: Upper Saddle River, NJ, 1999. Team-based learning collaborative. Team-Based Learning Collaborative. Team-Based Learning Collaborative; January 1, 2013. Woods, D. R. Preparing for PBL [Problem-Based Learning]; McMaster University: Hamilton, ON, 2006. Woods, D. R. Problem-Based Learning: How to Gain the Most from PBL. chemeng.mcmaster.ca. Donald R. Woods: Hamilton, ON, January 1, 1994. POGIL. POGIL - Process Oriented Guided Inquiry Learning; http:// www.pogil.org/ (accessed September 5, 2017). Moog, R. S.; Creegan, F. J.; Hanson, D. M.; Spencer, J. N.; Straumanis, A. R. POGIL: Process-Oriented Guided-Inquiry Learning; Pearson Prentice Hall: Upper Saddle River, NJ, 2009; Vol. II, pp 90–105. POGIL. POGIL - Process Oriented Guided Inquiry Learning. POGIL Process Oriented Guided Inquiry Learning; January 1, 2011. Robert, J.; Lewis, S. E.; Oueini, R.; Mapugay, Andrea Coordinated Implementation and Evaluation of Flipped Classes and Peer-Led Team Learning in General Chemistry. J. Chem. Educ. 2016, 93 (12), 1993–1998. Lasry, N.; Dugdale, M.; Charles, E. Just in Time to Flip Your Classroom. Phys. Teach. 2014, 52 (1), 34–36. Novak, J. D. Human Constructivism: A Unification of Psychological and Epistemological Phenomena in Meaning Making. Int. J. Pers. Constr. Psychol. 2007, 6 (2), 167–193. Bodner, G. M.; Klobuchar, M.; Geelan, D. The Many Forms of Constructivism. J. Chem. Educ. 2001, 78 (8), 1107. Bretz, S. L. Novak’s Theory of Education: Human Constructivism and Meaningful Learning. J. Chem. Educ. 2001, 78 (8), 1107. Novak, J. D. A Theory of Education: Meaningful Learning Underlies the Constructive Integration of Thinking, Feeling, and Acting Leading to Empowerment for Commitment and Responsibility. Meaningful Learn. Rev. 2011, 1 (2), 1–14. Flynn, A. B. Flipped and Blended Course Structure; https://youtu.be/ xqQmDdTJTr8 (accessed July 13, 2017). Krathwohl, D. R. A Revision of Bloom’s Taxonomy: An Overview. Theory Pract. 2002, 41 (4), 212–218. Collis, K. F.; Biggs, J. B. Using The SOLO Taxonomy. Set Res. Inf. Teach. 1986, 2 (4), 4. Biggs, J. B.; Tang, C. Teaching for Quality Learning; Society for Research into Higher Education and Open University Press: Maidenhead, England, 2007. NRC. Next Generation Science Standards. nextgenscience.org. National Research Council, January 1, 2015. 162 Sörensen and Canelas; Online Approaches to Chemical Education ACS Symposium Series; American Chemical Society: Washington, DC, 2017.

Downloaded by UNIV OF FLORIDA on November 3, 2017 | http://pubs.acs.org Publication Date (Web): October 26, 2017 | doi: 10.1021/bk-2017-1261.ch012

22. Flynn, A. B.; Caron, J.; Laroche, J.; Richard, G.; Bélanger, M.; Featherstone, R. Orgchem101.com: An organic chemistry and metacognitive skill and concept building tool http://orgchem101.com/ (accessed Jul 13, 2017). 23. Sapling Learning. Sapling Learning. January 1, 2016. 24. Deslongchamps, G. Organic Chemistry Flashware. Nelson Education Ltd. January 1, 2007. 25. Top Hat; https://tophat.com/ (accessed July 13, 2017). 26. Seery, M. K. ConfChem Conference on Flipped Classroom: Student Engagement with Flipped Chemistry Lectures. J. Chem. Educ. 2015, 92 (9), 1566–1567. 27. uOttawa: Teaching and Learning Support Service. Program Learning Outcomes; https://tlss.uottawa.ca/site/program-learning-outcomes (accessed September 5, 2017). 28. Goff, L.; Potter, M. K.; Pierre, E.; Carey, T.; Gullage, A.; Kustra, E.; Lee, R.; Lopes, V.; Marshal, L.; Martin, L.; Raffoul, J.; Siddiqui, A.; an Gastel, G. Learning Outcomes Assessment: A Practitioner’s Handbook. Higher Education Quality Council of Ontario HEQCO. January 1, 2015. 29. Stoyanovich, C.; Gandhi, A.; Flynn, A. B. Acid–Base Learning Outcomes for Students in an Introductory Organic Chemistry Course. J. Chem. Educ. 2015, 92 (2), 220–229. 30. Flynn, A. B.; Amellal, D. G. Chemical Information Literacy: pKa Values-Where Do Students Go Wrong? J. Chem. Educ. 2016, 93 (1), 39–45. 31. Guskey, T. R. Does It Make a Difference? Evaluating Professional Development. Educ. Leadersh. 2002, 59 (6), 45–51. 32. Guskey, T. R. Professional Development and Teacher Change. Teach. Teach. Theory Pract. 2010, 8 (3), 381–391. 33. Schunk, D. Learning Theories: An Educational Perspective, 6th ed.; Pearson: New York, NY, 2016. 34. Smith, M. K.; Jones, F. H. M.; Gilbert, S. L.; Wieman, C. E. The Classroom Observation Protocol for Undergraduate STEM (COPUS): A New Instrument to Characterize University STEM Classroom Practices. CBE Life Sci. Educ. 2013, 12 (4), 618–627. 35. Krippendorf, K. Computing Krippendorff ’s Alpha-Reliability; http:// repository.upenn.edu/asc_papers/43 (accessed July 13, 2017). 36. Seery, M. K. Flipped Learning in Higher Education Chemistry: Emerging Trends and Potential Directions. Chem. Educ. Res. Pract. 2015, 16, 758–768. 37. Luker, C.; Muzyka, J.; Belford, R. Introduction to the Spring 2014 ConfChem on the Flipped Classroom. J. Chem. Educ. 2015, 92 (9), 1564–1565. 38. Weaver, G. C.; Sturtevant, H. G. Design, Implementation, and Evaluation of a Flipped Format General Chemistry Course. J. Chem. Educ. 2015, 92 (9), 1437–1448. 39. Mooring, S. R.; Mitchell, C. E.; Burrows, N. L. Evaluation of a Flipped, Large-Enrollment Organic Chemistry Course on Student Attitude and Achievement. J. Chem. Educ. 2016, 93 (12), 1972–1983.

163 Sörensen and Canelas; Online Approaches to Chemical Education ACS Symposium Series; American Chemical Society: Washington, DC, 2017.

Downloaded by UNIV OF FLORIDA on November 3, 2017 | http://pubs.acs.org Publication Date (Web): October 26, 2017 | doi: 10.1021/bk-2017-1261.ch012

40. Shattuck, J. C. A Parallel Controlled Study of the Effectiveness of a Partially Flipped Organic Chemistry Course on Student Performance, Perceptions, and Course Completion. J. Chem. Educ. 2016, 93 (12), 1984–1992. 41. Eichler, J. F.; Peeples, J. Flipped Classroom Modules for Large Enrollment General Chemistry Courses: A Low Barrier Approach to Increase Active Learning and Improve Student Grades. Chem. Educ. Res. Pr. 2016, 17 (1), 197–208. 42. Reid, S. A. A Flipped Classroom Redesign in General Chemistry. Chem. Educ. Res. Pr. 2016, 17 (4), 914–922. 43. Fautch, J. M. The Flipped Classroom for Teaching Organic Chemistry in Small Classes: Is It Effective? Chem. Educ. Res. Pr. 2015, 16 (1), 179–186. 44. Bergmann, J.; Sams, A. Flip Your Classroom: Reach Every Student in Every Class Every Day; International Society for Technology in Education: Washington, DC, 2012. 45. Xu, X.; Lewis, J. E. Refinement of a Chemistry Attitude Measure for College Students. J. Chem. Educ. 2011, 88 (5), 561–568. 46. Olakanmi, E. E. The Effects of a Flipped Classroom Model of Instruction on Students’ Performance and Attitudes Towards Chemistry. J. Sci. Educ. Technol. 2017, 26 (1), 127–137. 47. Peterson, D. J. The Flipped Classroom Improves Student Achievement and Course Satisfaction in a Statistics Course. Teach. Psychol. 2016, 43 (1), 10–15. 48. Akkaraju, S. The Role of Flipped Learning in Managing the Cognitive Load of a Threshold Concept in Physiology. J. Eff. Teach. 2016, 16 (3), 28–43. 49. Ojennus, D. D. Assessment of Learning Gains in a Flipped Biochemistry Classroom. Biochem. Mol. Biol. Educ. 2016, 44 (1), 20–27. 50. Heyborne, W.; Perrett, J. To Flip or Not to Flip? Analysis of a Flipped Classroom Pedagogy in a General Biology Course. J. Coll. Sci. Teach. 2016, 45 (4). 51. Zack, L.; Fuselier, J.; Graham-Squire, A.; Lamb, R.; O’Hara, K. Flipping Freshman Mathematics. PRIMUS 2015, 25 (9–10), 803–813. 52. Yong, D.; Levy, R.; Lape, N. Why No Difference? A Controlled Flipped Classroom Study for an Introductory Differential Equations Course. PRIMUS 2015, 25 (9–10), 907–921. 53. uOttawa. University of Ottawa’s Blended Learning Initiative; http:// tlss.uottawa.ca/site/blended-initiative (accessed Jul 13, 2017).

164 Sörensen and Canelas; Online Approaches to Chemical Education ACS Symposium Series; American Chemical Society: Washington, DC, 2017.