Chapter 1
Eye Tracking for the Chemistry Education Researcher Downloaded from pubs.acs.org by 188.72.96.137 on 09/28/18. For personal use only.
Eye Tracking as a Research Tool: An Introduction Steven Cullipher,*,1 Sarah J. R. Hansen,*,2 and Jessica R. VandenPlas*,3 1Science
and Mathematics Department, Massachusetts Maritime Academy, Buzzards Bay, Massachusetts 02532, United States 2Department of Chemistry, Columbia University, New York, New York 10027, United States 3Department of Chemistry, Grand Valley State University, Allendale, Michigan 49401, United States *E-mails:
[email protected] (S.C.);
[email protected] (S.J.R.H.);
[email protected] (J.R.VP.).
Eye tracking can be a robust and rich source of data for chemistry education research but is not appropriate for all research questions. There are many variables to consider when deciding to conduct an eye-tracking study, some of which may not be obvious to the novice user. By the end of this chapter, the reader should be able to: (1) decide whether or not the research question can be answered with eye tracking; (2) design an appropriate participant task; and (3) determine which quantitative measures are appropriate to collect and analyze.
© 2018 American Chemical Society
Introduction In recent years, eye tracking has become increasingly popular in the chemistry education research (CER) community. Many researchers are looking to add this technique to their toolbox, but because of its complexity and distinct differences from other CER tools, they have hesitated to make the leap. Outside the field of CER, eye tracking has been employed in many areas, including usability studies, reading research, and visual search tasks (1–12). While these types of studies provide important information on the range and usefulness of eye tracking, the research questions that CER investigates are often unique and distinct. Before venturing deeper into the world of eye tracking, and the remainder of this book, it is important to understand the benefits, capabilities, and limitations of eye-tracking applications. Eye-Tracking Technology: How It Works Most modern eye-tracking systems use a technique called pupil center corneal reflection (PCCR) to track movements of the eyes while viewing a visual stimulus. This technology uses near-infrared illumination to create reflection patterns on the cornea and pupil of the eyes of the user. Image sensors in the eye-tracking unit are then used to capture images of the eyes and reflections patterns of the near-infrared light. Using advanced algorithms for image processing, as well as a physiological 3D model of the eye, the software is able to estimate the position of the eye in space and the point of gaze on the visual stimulus. Sampling rates vary based on the manufacturer and model of eye tracker, and should be considered if purchasing a new eye tracker. A minimum sampling rate of 60 Hz is typically recommended, but a review of the literature will provide best practices for the type of study being conducted.
Eye Tracking as a Useful, Independent Research Tool Eye tracking is a quantitative method for recording a participant’s eye movements as they observe a visual stimulus. Thus, eye tracking can directly answer the questions: • •
•
At what part of the stimulus is the participant looking? How much time does the participant spend looking at a particular part of the stimulus? Does the participant look at a particular part of the stimulus more than the others? In what order does the participant view the various components of the stimulus?
This information alone can be quite useful if the investigator is looking to design, for instance, curriculum materials, dynamic representations, or interactive simulations. Details on designing suitable stimuli for eye-tracking investigations are given in Chapter 3 of this book. 2
Beyond the quantitative data, eye tracking can provide insight into the participants’ underlying cognitive processes as they interact with visual stimuli. Hoffman and Subramaniam have shown that if an individual’s eyes are focused on an object, the attention of the individual is also on that object (13). The relationship between mental processing and eye-movement data has also been studied extensively (14–17). However, to accept the relationship between cognition and eye movements, the investigator must rely on two core working assumptions (14): •
•
The immediacy assumption states that the viewer begins processing the object upon which they fixate immediately and before moving on to the next object. As they begin fixating on the new object, the viewer immediately begins processing this new information. The eye-mind assumption states that a link exists between the eyes and the mind, such that whatever the eye fixates on, the mind processes. Based on this assumption, it can be inferred that commonly occurring eye gaze patterns might represent similar ways of processing a visual stimulus.
Thus, in addition to the questions listed above, eye tracking alone can be useful to answer questions such as: • • •
What part of the stimulus does the participant spend most of their time processing or trying to interpret? How much time does the participant spend processing different parts of the stimulus? In what order does the participant process information presented in the stimulus?
While all of these questions are worthwhile investigations, they only begin to scratch the surface of the types of questions chemistry education researchers are interested in investigating. Eye tracking alone cannot answer questions about what the participant is thinking about the stimulus they are processing, or why they have chosen to process this information in the first place. Such interpretations require the use of auxiliary data collection methods.
Mixed Methods Approaches to Eye Tracking: Methodological Triangulation for Enhanced Confidence As with any research study, the use of additional data channels can serve to boost confidence in the interpretation of results from a single data collection method. Remember that eye tracking alone can only provide information about which parts of a stimulus an individual is processing. It provides no information on the types of processing that occur. Methodological triangulation is the use of more than one data collection method to improve confidence in results when investigating a research question (18). Because of the limitations of eye tracking, 3
most research investigations should rely on methodological triangulation to cement understanding of the quantitative findings. Holmqvist et al. suggest several auxiliary methods to consider, each with their own benefits and shortcomings (19). Many of the alternate data channels that work well with eye tracking are already common tools used by CER investigators, including cognitive interviewing, questionnaires, problem-solving tasks, and thinking aloud. Other auxiliary data collection methods rely on biological responses to gauge neurological functions. Some examples are galvanic skin response, functional magnetic resonance imaging (fMRI), and electroencephalography (EEG). For more information on the use of multiple biometric methods to triangulate eye-tracking results, the reader is referred to Chapter 7. Additional information on other biological data metrics, including those listed, can be found elsewhere (19). Two considerations that the researcher must consider when selecting such auxiliary data collection methods include: •
•
Will the auxiliary data collection method impact the eye-tracking data? Kirk and Ashcraft have shown that verbal interaction with a participant during eye tracking may alter their eye movements (20). Some studies have shown that the increased cognitive load of concurrent verbalizations slows down eye movements and learning processes (21, 22). How will the auxiliary data be linked to eye-tracking data? Most data collection software for eye tracking provides the capability for recording audio and video data while monitoring eye movements. While concurrent verbalization has some drawbacks, the ability to provide an in-the-moment perspective from the participant may serve as valuable data, which can then be directly linked to eye movements during the participant task.
The reader can find more information on identifying appropriate research questions and designing an eye-tracking study to address these questions in Chapter 2.
Eye-Tracking Measures When designing an eye-tracking study, it is important to consider the project as a whole, from formulation of the research question through analysis of findings, from the outset. Because of the nature of eye tracking, decision-making in study design must consider multiple factors. Chief among these decisions will be to determine which measures are best suited for the study. Before discussing these measures, there are some commonly used terms that appear frequently in eye-tracking literature and that you will see throughout this book: •
Fixation: a pause in eye movement in which the retina is stabilized over a stationary object. It is estimated that 90% of viewing time is comprised of fixations, which can last between 150–600 ms (23). 4
•
•
•
Saccade: a rapid movement of the eye that occurs between fixations. During this transition, the viewer is essentially blind. Saccadic movements can be both voluntary and reflexive, so careful consideration is important in analyzing data related to saccades (24). Area of interest (AOI): a region of the visual stimulus in which measurements, primarily fixations, can be aggregated as part of analysis. AOIs can be defined in two ways. First, AOIs can be defined by the researcher before data collection, a technique that is commonly used when the researcher wants to know if a participant looks at a particular region of the visual stimulus. In the second method, AOIs are defined after data collection based on cluster analysis. Cluster analysis aggregates fixation data from all participants to indicate regions of the visual stimulus that have a high concentration of fixations. Scanpath (fixation sequence): a series of eye fixations and saccades, most commonly among AOIs, that occurs when a viewer is exposed to the visual stimulus.
Based on these concepts, there are innumerable measures available to eyetracking users, ranging from the relatively simple to the cumbersome and complex. Below is a discussion of the most frequently utilized eye-tracking measures and their typical applications. The adventurous researcher is referred to Holmqvist et al. and Duchowski for a more extensive compilation of eye-tracking measures (19, 24). Fixation Count and Fixation Duration The most common types of measures used in eye tracking involve fixations. In particular, fixation count and fixation duration are frequently used to interpret participant processing of information. Fixation count refers to the distinct number of fixations within a particular AOI. This is a common measure to indicate how frequently a participant processes information within the AOI. Fixation duration is the length of time that a participant’s gaze remains within a particular AOI. Duration can indicate a participant’s level of understanding, or the complexity, difficulty, or importance of the information, depending on the task and other auxiliary measures. Although research has shown a strong correlation between fixation count and fixation duration, it is important to consider these two measures separately to get a complete picture of how a participant processes the information in a stimulus (25, 26). The researcher should keep in mind that a correlation between fixation count and fixation duration may not exist for their particular study; so it is always important to examine both measures. Some research shows that fixation count is an appropriate indicator of the importance of the information contained in an AOI, with higher fixation counts corresponding to greater importance (27, 28). Fixation duration, on the other hand, is indicative of the complexity of information within an AOI, with longer fixation duration corresponding to greater complexity (7, 29–34). Both fixation count and fixation duration can be analyzed in various ways. For example, fixation duration could be taken as the average duration of individual 5
fixations or the total duration of all fixations within the AOI. An AOI with a high fixation count but a low average duration per fixation can indicate search behavior, showing that the participant has little understanding of the presented information (35). Williamson et al. used fixation duration to examine how students used ball-and-stick representations and electrostatic potential maps to answer questions about electron density, positive charge, proton attack, and hydroxide attack for various molecules (36). Their results showed a correlation between the accuracy of the students’ response and the fixations of students within the provided representation. Hansen used fixation data to examine how students transition between different symbolic reaction representations when solving chemistry problems (33). Results showed that participants shifting their viewing pattern regardless of success and similar viewing patterns were employed by participants who were both successful and unsuccessful in solving visual stoichiometry problems. For a more in-depth discussion of how fixations may be collected and analyzed, see Chapter 4. Fixation Sequence or Scanpath Eye fixation sequences (also known as scanpaths) refer to the order in which a participant’s eye movements shift between AOIs. Whereas fixation count and duration can provide information on the importance and complexity of the various AOIs, fixation sequences can reveal perceptual strategies that people develop for interpreting the sum of a visual stimulus (25, 34, 35, 37–39). These strategies reflect the individual’s cognition processes and could be used to group participants or differentiate between demographically grouped individuals. Cullipher and Sevian used scanpath analysis to compare students at various education levels and problem-solving strategies (35). Their results showed that students with greater understanding showed distinct fixation sequences that correlated to examining the presented information in a direct and reasoned fashion. Additional applications of sequence analysis are discussed in Chapter 5. Pupil Diameter The use of pupil diameter measures, referred to as pupillometry, is another technique that can be used to indicate cognitive functions of viewers. A thorough history and explanation of pupillometry can be found in Chapter 8. In the chapter, Karch also provides evidence for the relationship between pupil dilation and cognitive function. Analyzing Eye-Tracking Data Once the aforementioned measures have been collected by the researcher, some data analysis, likely including statistical comparison, must take place. The type of statistics that can be applied to eye-tracking data share many commonalities with the type of statistics the CER community applies to other quantitative data. However, some special considerations for running statistical 6
analysis of eye tracking data are given in Chapter 6, along with suggestions for using the statistical program R to complete these analyses.
Putting It All Together Building an eye-tracking study takes careful planning and consideration. From selecting appropriate research questions, to designing stimuli and concurrent data collection methods to ensure these questions are properly addressed, to selecting the right metrics to collect, to finding the right statistical analyses to apply to the data, the researcher has many decisions to make. The final two chapters of this book, Chapters 9 and 10, give detailed examples of how CER can be applied to particular topics within CER, and show how other researchers have tackled some of these questions. We hope this book will help the researcher get started and feel more confident making some of these decisions.
References 1.
Van Gog, T.; Paas, F.; Van Merriënboer, J. J. G. Uncovering ExpertiseRelated Differences in Troubleshooting Performance: Combining Eye Movement and Concurrent Verbal Protocol Data. Appl. Cogn. Psychol. 2005, 19, 205–221. 2. Van Gog, T.; Paas, F.; van Merriënboer, J. J. G.; Witte, P. Uncovering the Problem-Solving Process: Cued Retrospective Reporting versus Concurrent and Retrospective Reporting. J. Exp. Psychol. Appl. 2005, 11, 237. 3. Goldberg, J. H.; Wichansky, A. M. Eye Tracking in Usability Evaluation: A Practitioner’s Guide. In The Mind’s Eye; Hyona, J., Radach, R., Deubel, H., Eds.; Elsevier, 2003, pp 493–516. 4. Jacob, R. J. K.; Karn, K. S. Eye Tracking in Human-Computer Interaction and Usability Research: Ready to Deliver the Promises. Mind 2003, 2, 4. 5. Land, M. F. Eye Movements and the Control of Actions in Everyday Life. Prog. Retin. Eye Res. 2006, 25, 296–324. 6. Reder, S. M. On-Line Monitoring of Eye-Position Signals in Contingent and Noncontingent Paradigms. Behav. Res. Methods Instrum. 1973, 5, 218–228. 7. Rayner, K. Eye Movements in Reading and Information Processing: 20 Years of Research. Psychol. Bull. 1998, 124, 372–422. 8. Rayner, K.; Pollatsek, A. The Psychology of Reading; Prentice Hall: Englewood Cliffs, NJ, 1989. 9. Inhoff, A. W.; Radach, R. Definition and Computation of Oculomotor Measures in the Study of Cognitive Processes. In Eye Guidance in Reading and Scene Perception; Underwood, G. M., Ed.; Elsevier Science, Ltd.: Oxford, 1998; pp 29–53. 10. Engbert, R.; Longtin, A.; Kliegl, R. A Dynamical Model of Saccade Generation in Reading Based on Spatially Distributed Lexical Processing. Vision Res. 2002, 42, 621–636. 11. Wolfe, J. M. What Can 1 Million Trials Tell Us about Visual Search? Psychol. Sci. 1998, 9, 33–39. 7
12. Wolfe, J. M. Visual Search: A Review. In Attention; Pashler, H., Ed.; University College London Press: London, 1998. 13. Hoffman, J.; Subramaniam, B. The Role of Visual Attention in Saccadic Eye Movements. Percept. Psychophys. 1995, 57 (6), 787–795. 14. Just, M. A.; Carpenter, P. A. A Theory of Reading: From Eye Fixations to Comprehension. Psychol. Rev. 1980, 87, 329. 15. Rayner, K.; Raney, G. E.; Pollatsek, A. Eye Movements and Discourse Processing. In Sources of Coherence in Reading; Lorch, R. F., O’Brien, J. E. J., Eds.; Lawrence Erlbaum Associates, Inc: Hillsdale, NJ, 1995; pp 9–35. 16. Rayner, K. Eye Movements and Attention in Reading, Scene Perception, and Visual Search. Q. J. Exp. Psychol. 2009, 62, 1457–1506. 17. Anderson, J. R.; Bothell, D.; Douglass, S. Eye Movements Do Not Reflect Retrieval Processes: Limits of the Eye-Mind Hypothesis. Psychol. Sci. 2004, 15, 225–231. 18. Denzin, N. K. The Research Act in Sociology: A Theoretical Introduction to Sociological Methods; Transaction Publishers: Piscataway, NJ, 1973. 19. Holmqvist, K.; Nyström, M.; Andersson, R.; Dewhurst, R.; Jarodzka, H.; Van de Weijer, J. Eye Tracking: A Comprehensive Guide to Methods and Measures; Oxford University Press, 2011. 20. Kirk, E. P.; Ashcraft, M. H. Telling Stories: The Perils and Promise of Using Verbal Reports to Study Math Strategies. J. Exp. Psychol. Learn. Mem. Cogn. 2001, 27, 157. 21. Nielsen, J.; Clemmensen, T.; Yssing, C. Getting Access to What Goes on in People’s Heads?: Reflections on the Think-Aloud Technique. In Proceedings of the Second Nordic Conference on Human-Computer Interaction; ACM: New York, 2002; pp 101–110. 22. Van Someren, M. W.; Barnard, Y. F.; Sandberg, J. A. C. The Think Aloud Method: A Practical Guide to Modelling Cognitive Processes; Academic Press London: London, 1994; Vol. 2. 23. Irwin, D. E. Visual Memory Within and Across Fixations. In Eye Movements and Visual Cognition: Scene Perception and Reading; Rayner, K., Ed.; Springer: New York, 1992; pp 146–165. 24. Duchowski, A. T. Eye Tracking Methodology, 3rd ed.; Springer: London, 2017. 25. Tang, H.; Day, E.; Kendhammer, L.; Moore, J.; Brown, S.; Pienta, N. J. Eye Movement Patterns in Solving Science Ordering Problems. J. Eye Mov. Res. 2016, 9, 1–13. 26. Stieff, M.; Hegarty, M.; Deslongchamps, G. Identifying Representational Competence With Multi-Representational Displays. Cogn. Instr. 2011, 29, 123–145. 27. Hegarty, M.; Mayer, R. E.; Green, C. E. Comprehension of Arithmetic Word Problems: Evidence from Students’ Eye Fixations. J. Educ. Psychol. 1992, 84, 76. 28. Green, H. J.; Lemaire, P.; Dufau, S. Eye Movement Correlates of Younger and Older Adults’ Strategies for Complex Addition. Acta Psychol. (Amst). 2007, 125, 257–278. 8
29. de Corte, E.; Verschaffel, L.; Pauwels, A. Influence of the Semantic Structure of Word Problems on Second Graders’ Eye Movements. J. Educ. Psychol. 1990, 82, 359–365. 30. Tang, H.; Pienta, N. Eye-Tracking Study of Complexity in Gas Law Problems. J. Chem. Educ. 2012, 89, 988–994. 31. Schuttlefield, J. D.; Kirk, J.; Pienta, N. J.; Tang, H. Investigating the Effect of Complexity Factors in Gas Law Problems. J. Chem. Educ. 2012, 89, 586–591. 32. Topczewski, J.; Topczewski, A. M.; Tang, H.; Kendhammer, L.; Pienta, N. J. NMR Spectra through the Eyes of a Student: Eye Tracking Applied to NMR Items. J. Chem. Educ. 2017, 94, 29–37. 33. Hansen, S. J. R. Multimodal Study of Visual Problem Solving in Chemistry with Multiple Representations. Ph.D. Thesis, Columbia University, 2014. 34. Havanki, K. L.; VandenPlas, J. R. Eye Tracking Methodology for Chemistry Education Research. In Tools of Chemistry Education Research; Bunce, D. M.; Cole, R. S., Eds.; ACS Symposium Series 1166; American Chemical Society, 2014; pp 11–191. 35. Cullipher, S.; Sevian, H. Atoms versus Bonds: How Students Look at Spectra. J. Chem. Educ. 2015, 92, 1996–2005. 36. Williamson, V. M.; Hegarty, M.; Deslongchamps, G.; Williamson, K. C.; Shultz, M. J. Identifying Student Use of Ball-and-Stick Images versus Electrostatic Potential Map Images via Eye Tracking. J. Chem. Educ. 2013, 90, 159–164. 37. Havanki, K. L. A Process Model for the Comprehension of Organic Chemistry Notation, The Catholic University of America, 2012. 38. Augustyniak, P.; Tadeusiewicz, R. Assessment of Electrocardiogram Visual Interpretation Strategy Based on Scanpath Analysis. Physiol. Meas. 2006, 27, 597. 39. Slykhuis, D. A.; Wiebe, E. N.; Annetta, L. A. Eye-Tracking Students’ Attention to PowerPoint Photographs in a Science Education Setting. J. Sci. Educ. Technol. 2005, 14, 509–520.
9
