Increasing Enthusiasm and Enhancing Learning for Biochemistry

Aug 15, 2018 - †Department of Computer Science, ‡Department of Chemistry and ... our program is designed to be used in any laboratory around the g...
0 downloads 0 Views 7MB Size
Article Cite This: J. Chem. Educ. XXXX, XXX, XXX−XXX

pubs.acs.org/jchemeduc

Increasing Enthusiasm and Enhancing Learning for BiochemistryLaboratory Safety with an Augmented-Reality Program Bolin Zhu,† Mi Feng,† Hannah Lowe,‡ Jeffrey Kesselman,† Lane Harrison,† and Robert E. Dempski*,‡ †

Department of Computer Science, ‡Department of Chemistry and Biochemistry, Worcester Polytechnic Institute, Worcester, Massachusetts 01609-2247, United States

J. Chem. Educ. Downloaded from pubs.acs.org by UNIV OF SUSSEX on 08/16/18. For personal use only.

S Supporting Information *

ABSTRACT: According to the most recent data from the federal Chemical Safety and Hazard Investigation Board, between 2001 and 2011, more than 120 university laboratory accidents have caused injuries (including one death) and millions of dollars in damages. Laboratory-safety lessons normally comprise lecture slides alongside occasional movies and tours of laboratory facilities. This approach limits the realism of safety instruction within the laboratory. In addition, enthusiasm for laboratory-safety lectures is often low for both instructors and participants. To address these issues, we have developed an augmented-reality (AR) program to increase enthusiasm and enhance the learning experience for laboratory safety using Microsoft HoloLens. AR is an emerging field that uses computer technologies to generate realistic images, sounds, and other sensations that replicate a real environment. When compared with static or one-sided laboratory lectures, our approach creates an interactive learning environment where students must physically move around the laboratory to learn about laboratory safety. As each of the images and holograms for this open-source program can be placed in any location for any laboratory configuration, our program is designed to be used in any laboratory around the globe. KEYWORDS: Upper-Division Undergraduate, Second-Year Undergraduate, Curriculum, Interdisciplinary/Multidisciplinary, Laboratory Instruction, Safety/Hazards, Computer-Based Learning, Inquiry-Based/Discovery Learning, Laboratory Computing/Interfacing, Laboratory Equipment/Apparatus



INTRODUCTION Laboratory safety is an essential component of undergraduate laboratory classes. Effective safety training requires that students be aware of potential hazards, know proper safety measures and how to use safety equipment, and be able to identify their physical locations in the laboratory. Furthermore, safety lessons for laboratory classes must be compliant with appropriate university rules as well as state and federal regulations. Finally, societies, such as the American Chemical Society (ACS), regularly issue guidelines for chemical safety in academic institutions. Laboratory-safety lessons are normally composed of lecture slides alongside occasional movies showing laboratory hazards and tours of laboratory facilities. Although these types of approaches provide all required information to be safe in a laboratory environment, students are often not engaged in learning the material. In order to increase enthusiasm for laboratory-safety lessons and to enhance retention of the material, previous studies have used games, computer simulations, and even comic books to enhance laboratory safety.1,2 In addition, previous initiatives have used online instructional databases to support biochemistry-laboratory classes. Although each of these approaches can improve © XXXX American Chemical Society and Division of Chemical Education, Inc.

laboratory safety, none of these formats provide an interactive environment to learn about the specific laboratory where the student is taking their class that can then be modified for different classroom setups. In order to address these issues, we have developed, implemented, and assessed an augmentedreality (AR) program to teach laboratory safety to undergraduate students. AR is an emerging field that uses computer technologies to generate realistic images, sounds, and other sensations that replicate a real environment. Recently, AR has been used to demonstrate colorimetric titration as well as oxygen-gas generation from hydrogen peroxide.3,4 Under these conditions, laboratory experiments were designed within a specific flowchart to be investigated. To complete our initiative to teach laboratory safety, we initially developed a laboratorysafety curriculum consistent with university, state, and federal policies. Next, a discussion ensued between faculty with expertise in computer science and those with expertise in biochemistry-laboratory safety to plan the most effective way Received: March 7, 2018 Revised: July 16, 2018

A

DOI: 10.1021/acs.jchemed.8b00116 J. Chem. Educ. XXXX, XXX, XXX−XXX

Journal of Chemical Education

Article

Figure 1. Trainee view of the biochemistry-laboratory-safety program. (A) Initial view from the HoloLens, including the list of items to be found in the laboratory. (B) Change of view indicating the student has triggered a new location in the laboratory.

program was assessed as a learning tool. Analysis of this assessment shows that our AR program on biosafety resulted in higher user-experience ratings. In addition, students who used the HoloLens remembered the locations of more safety items within the laboratory when compared with those taught the control lesson.

to deliver this content with virtual objects and a holographic narrator within a real laboratory setting. Following this step, it was decided that Unity3D software would be used to develop this program. Unity3D is a computer program developed by Unity Technologies that can be obtained free of charge and used to develop three-dimensional simulations for computers, consoles, and mobile devices. For ease of use, Unity3D is not required to run this program. Modules were created with this software and reviewed to check for accuracy. The Institutional Review Board (IRB) of Worcester Polytechnic Institute (WPI) granted permission to complete these studies with undergraduate students, using both control (lecture-based) and experimental (HoloLens-based) conditions. The Microsoft HoloLens is a head-mounted, mixed-reality smartglass on which our program is run. Finally, the experience of the AR



METHODS The overall objective of this project was to create an AR program that could be used to teach biochemistry-laboratory safety in a more efficient and enthusiastic manner when compared with traditional lecture-based training methods. Initially, this program was designed to be employed in a single laboratory space. However, to improve the applicability of this program, the design was improved, increasing flexibility so that B

DOI: 10.1021/acs.jchemed.8b00116 J. Chem. Educ. XXXX, XXX, XXX−XXX

Journal of Chemical Education

Article

Figure 2. Finite-state machine representing the actions traversed while the training program is in use.

around the classroom to the locations identified by the Trainer. These locations are marked by colored words (Figure 1A). Once one of these locations is triggered by walking near the item, (1) the colored words turn to white, (2) virtual fireworks appear to indicate a student has reached a location, (3) the number adjacent to the location on the HoloLens screen counts down to zero, and (4) a narrator appears to describe the indicated safety topic as written in Appendix 1 (Figure 1B). Representative videos for the exits and sink can be found in the Supporting Information. The changing of the colors of the words and the fireworks are used to make sure the attention of each and every student is focused on the subsequent video. During initial trials, we noticed that some students did not listen to all of the biosafety material. Adding these items enhanced the learning experience for the participants. Sometimes there are two or more of the same items in the laboratory, such as exits. The program is designed so that at the second and all subsequent locations, the colored words turn white and fireworks appear, but the original spoken text is not repeated. When all locations are at zero, the narrator instructs the student to hand the HoloLens back to the instructor. In order to keep the system progression on track, we introduced a finite-state machine (FSM) into our system. As shown in Figure 2, there are four primary states of color for the system: Initialize in green, Progress in blue, Trainer in orange, and Trainee in yellow; each has its own substates in gray. Every state has its own command associated with it to trigger the event and transition to the next state (blue circles). The arrows indicate the directionality of the signal within the FSM in one or both directions. In parallel to these three general states,

the source code could be used in any laboratory. The software described here, which can be used on AR platforms such as the Microsoft HoloLens and the Magic Leap One, can be freely obtained on GitHub.5 The methodology underpinning the production of this code is described in the following three subsections: Finite-State Machine of Progression, Dynamic Voice Command, and Spatial Localization Based on Origin. Once this laboratory-safety program was created, the next objective was to evaluate the effectiveness of our augmentedreality program when compared with that of a normal lecturebased safety program. Therefore, a controlled experiment with 29 student participants was conducted. We drew on assessment approaches used in prior works on chemistry education,6,7 augmented reality,8,9 and training-effectiveness evaluations to quantify the effectiveness of our approach.10,11 Finite-State Machine of Progression

On the basis of our design, we developed two primary modes in the system: (1) Trainer Mode and (2) Trainee Mode. Trainer Mode allows the instructor to set up the laboratory prior to handing the device to trainees (students), and Trainee Mode is the program used by students for laboratory safety. In other words, in Trainer Mode, the instructor can move around the laboratory and mark locations. Locations that can be marked in the laboratory and the words spoken by the hologram at these locations are indicated in Appendix 1 in the Supporting Information. In Trainee Mode, following program initiation, the student sees a list of items to be found in the laboratory on the left side of the HoloLens (Figure 1A). This list counts down to zero, depending on the number of items, as each location is found in the laboratory. The student can move C

DOI: 10.1021/acs.jchemed.8b00116 J. Chem. Educ. XXXX, XXX, XXX−XXX

Journal of Chemical Education

Article

Figure 3. Voice command in Trainer Mode.

there is also a fifth state (red), which can listen to what is happening in the other four states in the event that the program needs to be reset.

the relatively small amount of storage space of the HoloLens, and the flexibility of use would be limited to one specific laboratory. In order to increase the applicability of this program in many laboratories and to overlap the virtual world and physical world, a starting spot and a focus point that each user must look at when the application starts were created. In other words, the physical and virtual world are connected when each user is standing at the same location and looking at the same spot. This will result in the entire program being oriented the same for all users.

Dynamic Voice Command

Besides flexibility, a priority of our system is that students who have not used AR can readily be trained in biosafety. HoloLens is a powerful AR device that includes capabilities for hand gestures, spatial mapping, and voice recognition. Although the HoloLens does include other features that may be relevant in future training programs, we focus on voice recognition in our system because hand gestures currently require training to achieve proficiency,9 and the voice-command system is easier to use with minimal training, which better reflects the conditions relevant for a classroom of undergraduate students. Voice recognition is a powerful means for users to control the HoloLens. In addition, voice recognition can be migrated to other systems. In this system, a data set of equipment information can be manually entered to initialize training in a particular location. On the basis of the equipment information in the database, a set of corresponding voice commands will be generated upon the start of the application. The grammar of general voice commands for trainers setting up a laboratory with the system is introduced in Figure 3, where “set” + equipment name or “delete” + equipment name generates or removes an equipment node in the virtual world that correlates to the physical world of the laboratory. A full command list for this program is shown in Appendix 2.

Pilot- and Main-Study Participants

To properly assess the effectiveness of this augmented-reality training program, a pilot study was conducted with seven participants to help establish our experimental procedure. We also ran effect-size and power analyses on the pilot study to determine the number of participants needed in the main study. In the main study, 29 students (8 female and 21 male) took part in the experiment. Their ages ranged from 17 to 21 (M = 18.52, SD = 0.83). Before the laboratory-safety training program, each participant filled out a demographic form indicating age, gender, and major as well as experiences related to video games, augmented reality, and laboratory safety. The students were split into two groups: the control lecture course (15 students) and the HoloLens condition (14 students). Each participant took a laboratory-safety pretest with five questions, the score of which was used as the baseline performance. The majority of the participants had no or little augmentedreality experience and were only somewhat familiar with laboratory safety. There was no statistically significant difference between the participants within the HoloLens and control conditions in terms of augmented-reality experiences, laboratory-safety familiarity, or the safety-pretest scores (Table 1). Each participant was randomly assigned to one of two conditions, the control condition or the HoloLens condition. Each of these conditions contained the exact same learning material.

Spatial Localization Based on Origin

The virtual world needs to be correlated to the physical world. Here, we introduce how we implemented this functionality. The virtual space in HoloLens is based on the starting point of the HoloLens camera as its origin, wherein the facing direction is the positive z-axis, the right direction is the positive x-axis, and the up direction is the positive y-axis. Thus, if the application is started in a new location, the mapping of the virtual and physical space will be off by the actual difference between the new starting point and the starting point the last time the program was set up. In other words, two users have to initiate the program with the HoloLens oriented in the same x, y, and z coordinates for them to observe holograms in the same location. Initially, the HoloLens spatial-mapping feature was considered in order to capture the physical space by scanning around the laboratory or room prior to the setup procedure. This solution would be the best scenario and a straightforward solution for our design. However, this method of capture requires a significant amount of physical storage space within

• Control: A traditional lecture-based method was used for the laboratory-safety training program. More specifically, in the biochemistry laboratory, the participant listens to an instructor introduce laboratory safety and point out the locations of every item mentioned in the lecture. D

DOI: 10.1021/acs.jchemed.8b00116 J. Chem. Educ. XXXX, XXX, XXX−XXX

Journal of Chemical Education

Article

Table 1. Single-Choice Questions Given to Participants in Both the Pre- and Post-tests No.

Item

1

Lab coats

2

Fire

3

Shower

4

5

Telephone

Biohazardous waste

Questions and Answer Choices with the Correct Answer Underlined When should lab coats be worn in the lab? A. Never B. Required when you are alone in the lab C. Required when you are doing experiments D. Always required When can you treat a medium-size f ire? A. Never B. When you know the location of a fire extinguisher C. When the fire is not close to you D. Always For chemical spills onto your person, what equipment should you use? A. Sink B. Shower C. Eye-wash station D. Fume hood The telephone located in the lab... A. is to be used to call home, only if you are late. B. is to be used to reach public safety in an emergency. C. should not be used at all. D. None of the above Where should you place the biohazardous waste? A. In a sink B. On a table C. In a red-bag-lined tub D. In a trash can

Figure 4. Framework of the chemistry-laboratory space provided for the laboratory-sketching task. Some example objects with labels are marked in blue.

items mentioned in the training on a schematic of the laboratory (Figure 4); (2) finish a laboratory-safety post-test (identical to the pretest); and (3) complete a postquestionnaire focusing on the training experience. The post-training tasks were completed in the order described above to minimize the influence of the former task on the latter. For example, if the post-test were taken before the sketch task, the items remembered by the participant would be affected by those included in the post-test. Laboratory-Sketching Task

• HoloLens: The AR program was used for the laboratorysafety training program. Here, the students found the locations of items in the laboratory, moved to them, and listened to the holographic narrator. The effectiveness of the training program was evaluated by framing our research questions as follows: • Memory Performance: Does the AR-enhanced program impact students’ abilities to recall the items mentioned in training? • Safety-Test Performance: Does the AR-enhanced program impact students’ performances on a laboratory-safety test? • Training Experience: Does the AR-enhanced program impact students’ laboratory-safety training experiences?

Each participant used the outline of the chemistry laboratory to draw laboratory-safety items that they were taught about either in the control or HoloLens conditions (Figure 4). These laboratory sketches were collected and quantified using the following measures: • Number of sketched items: the number of items mentioned in the training program that were labeled or drawn by the participant in the sketch. Safety Pre- and Post-tests

The answers to the identical safety pre- and post-tests (Table 1) were collected, and the knowledge gained through the training were quantified using the following quantitative measures: • Post-test score: the score of the post-test (0−5 points). • Score increase: the post-test score minus the pretest score.

Biochemistry-Laboratory-Safety-Training Protocol

Each participant took a laboratory-safety training program based on either the traditional method (the control condition) or the AR-enhanced method (the HoloLens condition). If assigned to the HoloLens condition, the participant received a brief introduction on how to wear and use the HoloLens and performed device calibration (as described in the Spatial Localization Based on Origin section) before starting the training in order to see all items in the room at the same locations as all test subjects.

Training-Experience Questionnaire

Subjective data were also collected to quantify the training experience. As shown in Table 2, the ratings for five questions asking about different aspects of the training experience were collected. Written comments on the participants’ preferences were also collected (i.e., things they liked or disliked about the training program).

Biosafety-Assessment Protocol

Experimental Measures

After the laboratory-safety training program was concluded, each participant was moved from the laboratory to a classroom where the laboratory could not be seen. Each participant was then asked to (1) draw a laboratory sketch, labeling all of the

Quantitative and qualitative measures in our experiment were derived from the laboratory sketching task, the safety pre- and post-tests, and the training-experience questionnaire. In addition, the following hypotheses were tested: E

DOI: 10.1021/acs.jchemed.8b00116 J. Chem. Educ. XXXX, XXX, XXX−XXX

Journal of Chemical Education

Article

through the program with only a rudimentary understanding of laboratory safety. In order to address this issue, the program was modified so that only one video could be triggered at a time. In other words, one video would have to be completed prior to a second video being triggered. This approach allows each Trainee to experience each hologram in its entirety prior to the next one starting.

Table 2. Questionnaire Participants Completed To Rate Each of the Conditions on the Basis of the Following Statements No.

Dimension

1

Clearness

2

Knowledge

3 4

Memory Helpfulness

5

Enjoyment

Statements for Student Response:a “Please rate the extent to which you...” Think the training program is clear in presenting the training content. Agree that after the training, you know substantially more about the training contents than before. Think that you will keep the training in good memory. Think the training program helps you learn chemistrylaboratory safety. Enjoyed the training program.

Video Background

The HoloLens has an added advantage in terms of assessing user experience in that the user’s behavior, as seen through the HoloLens, can be recorded as the user goes through the safety protocol. However, after the narrator was filmed and placed within the virtual HoloLens world, a black background was observed behind the narrator instead of the laboratory space. On the basis of research from Color Design1 from Microsoft, the dark color, black in this case, would appear transparent in HoloLens applications. In order to get the exact look on display, a Unity3D Shader program was written to minimize the black-color background within the instruction videos.

a

The scale for each response is 1−6, with 1 indicating Extremely Disagree and 6 indicating Extremely Agree.

Hypothesis 1: The participants in the HoloLens condition on average recall more items mentioned during the biosafety training than those in the control condition. Hypothesis 2: The participants in the HoloLens condition on average perform better in the biosafety tests than those in the control condition. Hypothesis 3: The participants in the HoloLens condition on average have a more enjoyable learning experience of the biosafety training program than those in the control condition.

Memory Performance

Following the biosafety training, each participant was asked to draw a sketch of the laboratory space including the items mentioned in the biosafety training program (14 items in total). The number of items drawn by each participant was tallied. Analysis of this data shows that the average participant in the HoloLens condition recalled more items (M = 12.2 items, 95% CI [11.2, 12.9]) than those in the control condition (M = 10.6 items, 95% CI [9.7, 11.6]; Figure 5A). Given the

Statistical Analysis

Quantitative and qualitative measures were analyzed. In response to concerns about the limitations of null-hypothesis significance testing,11 we modeled our analyses by primarily focusing on confidence intervals and effect sizes. We computed 95% confidence intervals using the bootstrap method and effect sizes using Cohen’s d, the difference in the means of the conditions divided by the pooled standard deviation. We also used the nonparametric Mann−Whitney test to compare the two conditions. Although we included significance tests and related statistics, it was with the intention of supplementing these analyses.



RESULTS

Technical Issues in Program Development

The AR biochemistry-laboratory-safety program was developed in an iterative manner. In other words, as each module was created, the module would be tested, issues with deployment would be identified, and then fixes would be made in the program to address these issues. During this process, challenges included inaccuracy of positioning for items in the laboratory (described above), more than one video playing at a time (see below), and issues related to the background adjacent to the hologram (see below). We have taken efforts throughout the program-development stage to minimize the likelihood that these challenges would appear were other laboratories’ attempts to replicate the current program. Multivideo Stream

In the AR biochemistry-laboratory-safety application, within the Trainer Mode, tags are placed around the laboratory, which, upon triggering by the Trainee, initiate a video of instruction. In an initial iteration, if a user triggered one video prior to the end of an earlier video, more than one video could play concurrently. This could result in a Trainee skimming

Figure 5. Experimental results. Error bars are 95% confidence intervals. The p-values, W-values, and effect sizes showing differences between groups are included. F

DOI: 10.1021/acs.jchemed.8b00116 J. Chem. Educ. XXXX, XXX, XXX−XXX

Journal of Chemical Education



Article

CONCLUSIONS We have developed an open-source augmented-reality biochemistry-laboratory-safety program. This program is designed to be used in any biochemistry laboratory with any configuration. Analysis of testing with 29 undergraduate students demonstrates that students who use this program recall the locations of more items in the laboratory when compared with those taught in the traditional lecture-based format. This is especially notable when considering that students in the lecture-based control condition have the ability to ask questions of the instructor. Even with this ability, students remember the locations of fewer items in the laboratory. In addition, although there were equivalent learning gains about laboratory safety as measured by a laboratorysafety quiz, students enjoyed the HoloLens course more than the traditional lecture-based program.

upper and lower limits of the confidence intervals, the average participant in the HoloLens condition recalled at least the same number of items as those in the control condition and up to 2 more (d = 0.92 [0.07, 1.74]). With this data, our first hypothesis on participants’ memory performances was supported: students participating in AR-enhanced training have better memory performances than those with the traditional lecture-based method. Safety-Test Performance

No significant differences between the HoloLens and control conditions in either the post-test scores or test-score increases were observed (Figure 5B). Thus, the second hypothesis was not supported. The results indicate that the AR-enhanced approach may have a limited effect on students’ test performances. However, this might also be because the test questions were relatively easy. The average pretest scores for both conditions were already equal to or greater than 4.0 points out of 5.0, so there was limited space for improvement.



ASSOCIATED CONTENT

S Supporting Information *

Training Experience

The Supporting Information is available on the ACS Publications website at DOI: 10.1021/acs.jchemed.8b00116.

Each participant answered five questions related to the training experience. As shown in Figure 5C, participants in the HoloLens condition reported better experience in knowledge gained, memory, and enjoyment, whereas the ratings of clearness and helpfulness remained similar. The results are reported in detail below. Question 2 (Knowledge Gain). For the question asking about the extent to which the participant knows substantially more about the training contents than before, the average participant in the HoloLens condition rated higher (M = 4.4, 95% CI [4.1, 4.8]) than that in the control condition (M = 3.7, 95% CI [2.9, 4.2]). The nonparametric Mann−Whitney test shows that W = 71, p = 0.11, and the effect size is d = 0.77 [0.1, 1.43]. Question 3 (Memory). For the question asking about the extent to which the participant will keep the training in good memory, the average participant in the HoloLens condition rated higher (M = 4.4, 95% CI [3.9, 4.9]) than that in the control condition (M = 3.5, 95% CI [2.8, 3.9]). The nonparametric Mann−Whitney test shows that W = 56, p = 0.03, and the effect size is d = 0.93 [0.16, 1.58]. Question 5 (Enjoyment). For the question asking about the extent to which the participant enjoyed the training, the average participant in the HoloLens condition rated higher (M = 5, 95% CI [4.5, 5.4]) than that in the control condition (M = 2.5, 95% CI [2.1, 2.7]). The nonparametric Mann−Whitney test shows that W = 2.5, p = 4.9 × 10−6, and the effect size is d = 0.77 [3.32, 4.29]. In addition to the rating questions, we asked the participants what they liked or disliked about the training. Many participants in the HoloLens condition liked that the program was engaging and fun. Some participants specifically pointed out that the object mapping was intuitive: one participant stated, “It was very intuitive and the controls worked great as in picking up on when you were close enough to trigger a video.” Another stated, “Loved the labeling of real life objects. Made it clear what was what.” The HoloLens group also mentioned some issues related to the technology; for example, the weight of the HoloLens was heavy. In contrast, many participants in the control group mentioned that the program was straightforward but boring.



Appendices: locations and associated text presented by HoloLens narrator and voice-command list (PDF, DOCX) Exit video (AVI) Sink video (AVI)

AUTHOR INFORMATION

Corresponding Author

*E-mail: [email protected]. ORCID

Robert E. Dempski: 0000-0002-7751-1613 Notes

The authors declare no competing financial interest.



ACKNOWLEDGMENTS This work was supported by a Worcester Polytechnic Institute (WPI) Teaching Innovation Grant to R.E.D., L.H., and J.K. Purchase of the Microsoft HoloLens was made possible with funds from WPI’s Messier Memorial Fund.



REFERENCES

(1) Di Raddo, P. Teaching laboratory safety through comics. J. Chem. Educ. 2006, 83, 571−573. (2) Gublo, K. I. A laboratory safety trivia game. J. Chem. Educ. 2003, 80 (4), 425−425. (3) Tee, N. Y. K.; Gan, H. S.; Li, J.; Cheong, B. H. P.; Tan, H. Y.; Liew, O. W.; Ng, T. W. Developing and Demonstrating an Augmented Reality Colorimetric Titration Tool. J. Chem. Educ. 2018, 95 (3), 393−399. (4) Gan, H. S.; Tee, N. Y. K.; Bin Mamtaz, M. R.; Xiao, K.; Cheong, B. H. P.; Liew, O. W.; Ng, T. W. Augmented reality experimentation on oxygen gas generation from hydrogen peroxide and bleach reaction. Biochem. Mol. Biol. Educ. 2018, 46, 245. (5) GitHub Repository for the Dempski Lab. https://github.com/ dempskilab (accessed July 2018). (6) Bowen, R.; Picard, D.; Verberne-Sutton, S.; Brame, C. Incorporating student design in an HPLC lab activity promotes student metacognition and argumentation. J. Chem. Educ. 2018, 95 (1), 108−115.

G

DOI: 10.1021/acs.jchemed.8b00116 J. Chem. Educ. XXXX, XXX, XXX−XXX

Journal of Chemical Education

Article

(7) Blackburn, R. Write my next lecture: Prelecture problem classes and in-lecture discussion to assist case-study teaching of synthesis. J. Chem. Educ. 2018, 95 (1), 104−107. (8) Bai, Z.; Blackwell, A. Analytic review of usability evaluation in ISMAR. Interacting with Computers 2012, 24 (6), 450−460. (9) Henderson, S.; Feiner, S. Evaluating the benefits of augmented reality for task localization in maintenancce of an armored personnel carrier turret. Mixed and Augmented Reality 2009, 2009, 135−1444. (10) Grohmann, A.; Kauffeld, S. Evaluating training programs: Development and correlates of the questionaire for professional training evaluation. International Journal of Training and Development 2013, 17, 135−155. (11) Kirkpatrick, J.; Kirkpatrick, W. Kirkpatrick’s four levels of training evaluation; Association for Talent Development: Alexandria, VA, 2016.

H

DOI: 10.1021/acs.jchemed.8b00116 J. Chem. Educ. XXXX, XXX, XXX−XXX