Eye-Tracking Study of Complexity in Gas Law ... - ACS Publications

Article pubs.acs.org/jchemeduc

Eye-Tracking Study of Complexity in Gas Law Problems Hui Tang* and Norbert Pienta Department of Chemistry, University of Iowa, Iowa City, Iowa 52242-1294, United States S Supporting Information *

ABSTRACT: This study, part of a series investigating students’ use of online tools to assess problem solving, uses eye-tracking hardware and software to explore the effect of problem difficulty and cognitive processes when students solve gas law word problems. Eye movements are indices of cognition; eye-tracking data typically include the location, duration, and sequence of subjects’ fixations on the visual representations. Such information is not usually discovered using conventional assessment methods (e.g., written examinations and scores) when measurements of cognitive performance or task difficulty are needed. The results of this study reveal that when compared to successful ones, unsuccessful students spend more time considering their solutions and fixating more on the questions as they attempt to solve the problem. These results demonstrate that eye-tracking is a useful new approach for exploring problem difficulty and student cognitive activities in chemical education research. KEYWORDS: High School/Introductory Chemistry, Chemical Education Research, Problem Solving/Decision Making, Gases FEATURE: Chemical Education Research

■

INTRODUCTION Encouraging students to develop problem-solving skills is one of the important objectives of science education.1 To explore student problem-solving processes in introductory chemistry and the role of cognitive load during these processes, our research group has developed online tools to examine a series of chemistry problems over a range of topics. One of those topics, the gas law problem, requires students to apply Charles’s law to solve for the final volume, given an initial volume and temperature and a final temperature. In a recent study, variables in five complexity factors (gas identity, number format, volume, temperature, and pressure, see columns 1 and 2 in Table 1) were randomly assigned by the software.2 Data were collected and analyzed from a large number of students from several universities. The study found that three problem complexity factors significantly affected the students’ ability to correctly solve the gas law questions: number format, volume, and temperature.2 To investigate why and how these complexity factors influence student ability to solve the gas law problem and to further understand student strategies, an eye-tracking experiment was conducted using a small subset of the questions from the earlier study. People obtain and process visual information during eye fixations.3 Fixations are defined as eye gaze points that remain in one location for a period of time.3−5 Between fixations, rapid eye movements are called saccades.3,5,6 The threshold (i.e., the minimum duration to define a fixation) ranges from 40 to 500 ms in different studies,3−13 although 100−300 ms is normally used. With special software, the locations of eye fixations and saccades can be determined by eye-tracking devices, which typically consist of infrared light sources and sensors receiving reflections from eyes. Compared to the old eye-tracking systems, the new generation does not need a headset and is hence less intrusive. Furthermore, it does not require participants’ heads to be fixed because it compensates for head movements. The calibration takes less than a minute and © 2012 American Chemical Society and Division of Chemical Education, Inc.

usually no recalibration is needed during an experiment. Typically no special accommodation is required for participants who wear glasses or contact lenses.6,14 Eye-tracking hardware uses a special camera that can provide information about the location, duration, and sequence of the visual representations that a subject views.8,15 This so-called gaze information can be obtained while participants are undertaking tasks at their own pace and can be accomplished without any unusual interference. Eye movements are typically the result of cognitive activities; that is, when humans are exposed to text, pictures, or movies, eye movements are direct and objective indicators of attention, which in turn, is linked to cognitive processing.3,16−18 Specifically, the duration of an eye fixation indicates the cognitive complexity of the material. The total number of fixations on a region can be considered an indicator of how important the information in that region is and how efficiently it was transferred to long-term memory.4 The sequence of fixations implies the strategies of processing the information or solving the problem and, thus, the final organization in long-term memory.4,7 Although the hardware can identify a single gaze point, researchers often study areas of interest (AOI) rather than a few points or an entire image. Although eye-tracking research has been conducted for more than a century, modern eye-tracking research using computers and cameras only started about three decades ago.3,6,14 Eyetracking has been applied in fields such as marketing, psychology, Web design, and human−computer interaction.6,14 The field of education research has also used eye-tracking applications in the context of reading,19−21 text and picture comprehension,10 problem solving,7,18,22−26 and instructional design,27 especially when multimedia materials are used.4,8 Published studies in science problem-solving using the eyetracking approach primarily involve arithmetic,9,15,16,28,29 geometry,30 physics,31,32 and a few other areas.5,33 Recently, Published: May 31, 2012 988

dx.doi.org/10.1021/ed200644k | J. Chem. Educ. 2012, 89, 988−994

Journal of Chemical Education

Article

Table 1. Complexity Variables and Cognitive Load Items Cognitive Load Incrementb Complexity Factor Gas identity

Number format

Volume

Temperature

Pressure

Variable

Cognitive Load Item

Ideal gas Mixture gas Unknown gas Decimal Scientific notation General mL to mL mL to L L to mL L to L K to K °C to °C °C to K K to °C Torr atm Blank (no unit)

0.00 0.25 0.25 0.25 0.50 0.00 0.50 1.00 1.00 0.00 0.00 0.50 0.00 0.00 0.25 0.00 0.00

a

Q1

Q1

0

Q3

Q4

0

0

0.25 0.25

Q5

0.25 0.5

0.5

0.5

0 0.5 1 1

1

0.5

0.5

0 0 0

Sumc

0.25

0.25

0.25

1.25

0 2.00

0 0.25

0.25

1.75

2.50

a

Cognitive load item for each complexity variable as assigned in the previous study.2 bCognitive load increment for each complexity factor in each of the five questions used in this study. cTotal cognitive load increment in each question.

problem is to use multiple experimental methods in addition to eye-tracking and performance measures,25 such as interview6,30 and think-aloud.25,39−41 If the problems to be solved are simple enough, it is possible for participants to solve those problems mentally and present solutions aloud.15,23,24,31,42

animations in chemical education have been studied using eyetracking technology.34 The problem-solving studies mentioned above generally used eye-tracking experiments to deeply understand the effect of problem complexity and the difference in solving problems between different groups of subjects. Participants’ eye fixation durations were found to be longer on complex problems than on simple ones. This is because solving complex problems requires temporarily holding more solution steps within working-memory, which results in longer fixation durations.15,16,26,28 When groups with different abilities or backgrounds were compared, fixation durations or response times were found to be shorter for higher-ability subjects than for lower-ability ones.16,33 Furthermore, because eye movements reflect the pattern of dynamic steps of how students process information during problem solving, the sequence of fixations or so-called scanpaths were also investigated.23,24,32,35 Some studies found that novices and experts showed different patterns of eye movements when they solved science problems.5,31 Compared to traditional assessment methods such as examination scores and times to accomplish tasks, eyetracking can provide more subtle and accurate data related to learners’ attention and cognitive processing.5,9,31,36 It can also be used to differentiate novices and experts more distinctly.6 The information obtained from eye-tracking data may provide educators with additional insight into students’ levels of expertise, thereby enabling more effective curricula and assessments.5 Some researchers argue that the extent of the relationship between cognitive processes and measures of eye movement is questionable.37,38 For example, long fixations may imply deep mental processing; however, they may also indicate that a problem solver is merely staring at the problem without any thinking. The latter is usually true when the problem solver considers a problem to be too difficult, a cognitive event accompanied with low frequencies of eye movement. For instance, if a student fails to comprehend a word problem after several attempts, a long fixation could ensue.9 A solution to this

■

METHODOLOGY

Subjects

The participants were volunteer students enrolled in introductory chemistry courses at a large midwestern comprehensive university in 2010. The total number of subjects was 12 and all were subjected to a protocol approved by the local institutional review board. The participants received a gift certificate as compensation for taking part in this research study. Apparatus

A Tobii T120 eye tracker with Tobii Studio 2.0.4 software was used to collect eye movement data. The display of the eye tracker is a 17-in. screen with a resolution of 1280 × 1024 pixels and an accuracy of 0.5°. The eye tracker is a nonintrusive device that tracks and records where and how long a person looks at the screen of the eye tracker; the device appears to be an LCD monitor to the user. A threshold of fixation of 100 ms was selected and the data collection rate was set to 120 Hz.43 The distance from the eyes of a subject to the eye tracker is 60−65 cm. Both the display and the eye-tracking server run on a Windows XP desktop computer. The computer is equipped with 2.81 GHz AMD Athlon 7850 Dual-Core, 4 GB RAM, and an ATI Radeon HD 3200 graphics card. A Wacom Bamboo pen-tablet was used to allow students to write on the tablet in place of scratch paper during the experiment. The tablet was configured so that the writing on the tablet appears concurrently on the screen of the eye tracker, precluding the need for students to move their eyes away from the screen and capturing all the writing by the eye tracker. 989



Article

Figure 1. (A) Screen capture showing the text of question 1, help information, the calculator, and the paint document, which was used as scratch paper. The areas of color (ranging from green through yellow to red) represent the integrated gaze duration, forming a “heat map”. (B) Part of the Web page for question 2 with the question text, table, and defined areas of interest (AOI).

Procedure

eye-tracking data. The questionnaire can be found in the online Supporting Information.

Utilizing the basic layout of the online tool from the previous study,2 five specific questions were selected for use in this study, and the Web pages for each were created using HTML. For the purpose of eye-tracking research, the font size and the spaces between the lines of the questions’ text were enlarged compared to the previous online tool. Each of the five questions featured a unique combination of the complexity variables indicated in Table 1 (last columns, Q1 to Q5, note that larger cognitive load values correspond to more difficulty problems). Although all five questions in this study ask students to solve for the final volume using Charles’s law, there are two formats of question: volumes and temperatures are listed in a table in questions 2 and 5, while all the variables appear in the text of questions 1, 3, and 4. The screen capture of question 1 superimposed with heat map from one participant’ fixation data and part of the Web page of question 2 are shown in Figure 1. All participants answered the five questions in the same order. After reading a brief instruction, the participants sat in front of the eye-tracker monitor. They were asked to use the tablet pen and the calculator in Microsoft Windows XP for a short time so that they could be familiar with these two before answering the questions. Once the eye tracker was calibrated (a required yet simple protocol provided within the Tobii tools), the participants opened the Web pages (i.e., question text and help info areas) of the first question and began to answer. They were allowed to use the calculator and the help info (e.g., a periodic table provided within the question screen). Immediately after completing the eye-tracking experiment, each participant was given a questionnaire to complete. The questionnaire consists of 10 questions, either open-ended or multiple-choice, with the latter requiring students to provide explanations. This instrument asks students to recall their problem-solving strategies and how they considered the order of difficulties of the unit conversions in the experiment. Hence, it was designed to help the researchers to better understand the

■

DATA ANALYSIS

Subjects were divided into two groups based on their performance in the experiment. Five subjects who answered two or fewer questions correctly were categorized into the unsuccessful group; seven subjects who answered more than two questions correctly were in the successful group. As part of the analyses, AOIs for all the pressure, volume, and temperature values that appear in the text (questions 1, 3, and 4) or the table (questions 2 and 5) were defined. For example, the areas of the initial and final volumes are defined as AOIinitial V (or AOI-vi) and AOI-final V (or AOI-vf), respectively. Additionally, an AOI of the table was defined for questions 2 and 5 (AOI-table). An AOI for each question stem (AOIquestion) was also defined. These AOIs appear as rectangles superimposed on the corresponding areas of the gas law problem shown in Figure 1B for demonstration here; however, they do not appear on the screen during the experiment. Furthermore, because complexity factors pressure, volume, and temperature involve both initial and final values, the combinations of their AOI-initial and AOI-final were also calculated and analyzed as a sum. For instance, the number of fixations on volume is the sum of number of fixations on AOI-vi and AOI-vf. In addition to measuring the overall time it takes for a student to accomplish an entire task, we also chose to differentiate each phase of a problem-solving process. Researchers have proposed four phases in mathematical problem solving.15,44 The first phase is problem translation: students build a mental representation for every component of the problem. The second phase is problem integration: problem solvers integrate all mental representations and thus have a whole picture of the problem. The third and fourth phases are solution planning and solution execution, 990



Article

Figure 2. Average time spent on each question (A) and each phase (B: reading, planning, calculating, and the overall sum of the phases). The dashed lines with the question bars indicate the learning effect trend.

Figure 3. Number of fixations on areas of interest (AOI). Part A shows the average number of fixations on each AOI (see Figure 1B). The remaining graphs show the numbers of fixations in each question on these variables: volume (B); temperature (C); and pressure (D).

respectively. A simple two-phase solution process was also reported, which includes an initial reading and the rest of problem solving.16 Because of the characteristics of the design of this study, the problem-solving process is separated into three phases: problem reading, problem planning, and calculation. Problem reading is defined from the time that a problem solver begins reading the problem to the time his or her eye fixations first leave the last word of the problem or the last part of the table. Problem planning is from the end of reading until a problem solver starts to click the button of the calculator. The calculation phase is from the end of planning

until the submission button is clicked. The sum of these phases is the total time a student spends on solving the problem. The variables analyzed in this study included the time spent on each question, the average time spent in each phase, and the number of fixations and fixation length on AOIs. All the analyses were conducted by using 5 × 2 mixed ANOVA tests (five questions by two groups), except for the number of fixation/fixation length on AOI-table, which is a 2 × 2 mixed ANOVA test. Therefore, each test could provide the information of the within-subjects effect (question), the between-subjects effect (group), and the interaction between 991



Article

The statistics of fixation length showed very similar patterns to that of fixation number. Analysis revealed that these two variables were significantly correlated. Thus, fixation lengths are not included in the results.

these two. However, because all the five questions asked students to solve for the final volume using Charles’s law and all subjects answered these questions in the same order, the “learning effect” needs to be taken into account. That is to say, students should become more and more familiar with the questions by continuing to answer the same type of question. They were expected to learn from the previous problem-solving process, and thus, spend less time and perform better on the next one. As a result, only the between-subjects effects of time spent on each question and in each phase are reported in the results. On the other hand, we wanted to investigate how different cognitive load items in the complexity factors pressure, volume, and temperature influence students’ eye movements. Thus, both between- and within-subjects effects as well as the interaction in the mixed ANOVA test of fixation numbers on P, V, and T are included. The criterion for the level of significance is 0.05 in all the cases). This indicates that the numbers of fixations on these AOIs are not different between the successful and unsuccessful groups. 992



Article

(from the easiest to the hardest: “L to L”, “mL to mL”, “mL to L”, and “L to mL”). For the remaining half of the participants, some considered “mL to L” was as difficult as “L to mL”, but none of them thought the latter was easier than the former conversion. Fixation time and numbers are direct and objective measures of cognitive load.45 That is, the fixation number or rate increases when the cognitive load of a task increases.3,14,46 The eye-tracking results show that unsuccessful students fixated more on the entire question when they solved the gas law word problem than their counterparts did. This means that the gas law problem added more cognitive load onto the unsuccessful students; in addition, a new information format in the problem such as the table introduced extraneous cognitive load to this group of students. Although the figures that plot fixation numbers for pressure, volume, and temperature show similar trends to the corresponding cognitive load items, student learning effect and the effect of the table need to be examined independently in future studies. In summary, the present study was undertaken to further understand the complexity factors affecting student ability to solve the gas law problem and the relationship between problem difficulties and eye-tracking data. In the experiment, unsuccessful students fixated more on rereading the questions, including the tables, than did the successful students when they solved the problem. Furthermore, the problem-solving process was divided into three phases: problem reading, problem planning, and calculation. Both successful and unsuccessful groups spent the same amount of time on initial reading, but the unsuccessful students spent more time on the problem planning phase. The results demonstrate that students who were divided into two groups based on their level of success in solving the gas law problem showed different eye-movement patterns, which correspond to their cognitive effort.

■

(4) Slykhuis, D. A.; Wiebe, E. N.; Annetta, L. A. J. Sci. Educ. Technol. 2005, 14 (5/6), 509−520. (5) Tai, R. H.; Loehr, J. F.; Brigham, F. J. Int. J. Res. Method Educ. 2006, 29 (2), 185−208. (6) Bazar, N. S.; Brigham, F. J. Telecommun. Rev. 2007, 46−48. (7) Peebles, D.; Cheng, P. C-H. In Proceedings of the 23rd Annual Conference of the Cognitive Science Society; Moore, J. D., Stenning, K., Eds.; Lawrence Erlbaum: Mahwah, NJ, 2001. (8) Van Gog, T.; Jarodzka, H.; Scheiter, K.; Gerjets, P.; Paas, F. Comput. Hum. Behav. 2009, 25, 785−791. (9) Knoblich, G.; Ohlsson, S.; Raney, G. E. Mem. Cognition 2001, 29 (7), 1000−1009. (10) Chuang, H.-H.; Liu, H.-C. J. Sci. Educ. Technol. 2012, 21 (2), 276−286. (11) Goldberg, J. H.; Stimson, M. J.; Lewenstein, M.; Scott, N.; Wichansky, A. M. In Proceedings of the 2002 Symposium on EyeTracking Research and Applications, New Orleans, Louisiana, March 25−27, 2002; ACM: New York, NY, 2002; pp 51−58. (12) Buscher, G.; Cutrell, E.; Morris, M. R. In Proceedings of the 27th International Conference on Human Factors in Computing Systems, April 4−9, 2009, Boston, MA; ACM: New York, NY, 2009. (13) Blignaut, P. Attention, Perception, & Psychophysics 2009, 71 (4), 881−895. (14) Duchowski, A. T. Eye-Tracking Methodology: Theory and Practice, 2nd ed.; Springer-Verlag: London, U.K., 2007. (15) Hegarty, M.; Mayer, R. E.; Green, C. E. J. Educ. Psychol. 1992, 84 (1), 76−84. (16) De Corte, E.; Verschaffel, L.; Pauwels, A. J. Educ. Psychol. 1990, 82 (2), 359−365. (17) Krugman, D. M.; Fox, R. J.; Fletcher, J. E.; Rojas, T. H. J. Advertising Res. 1994, 34 (6), 39−52. (18) Grant, E. R.; Spivey, M. J. Psychol. Sci. 2003, 14 (5), 462−466. (19) Reichle, E. D.; Rayner, K.; Pollatsek, A. Vision Res. 1999, 39, 4403−4411. (20) Rayner, K.; Liversedge, S. P.; White, S. J.; Vergilino-Perez, D. Psychol. Sci. 2003, 14 (4), 385−388. (21) Magliano, J. P.; Graesser, A. C.; Eymard, L. A.; Haberlandt, K.; Gholson, B. J. Exp. Psychol. Learn. Mem. Cognit. 1993, 19, 704−709. (22) Graesser, A. C.; Lu, S.; Olde, B. A.; Cooper-Pye, E.; Whitten, S. Mem. Cognition 2005, 33, 1235−1247. (23) Yoon, D.; Narayanan, N. H. In Proceedings of the 2004 Symposium on Eye-Tracking Research and Applications, San Antonio, TX, March 22−24, 2004; ACM: New York, NY, 2004. (24) Hodgson, T. L.; Bajwa, A.; Owen, A. M.; Kennard, C. J. Cognit. Neurosci. 2000, 12 (5), 894−907. (25) Waldron, S. M.; Patrick, J.; Duggan, G. B. Q. J. Exp. Psychol. 2011, 64 (3), 485−503. (26) Verschaffel, L.; De Corte, E.; Pauwels, A. J. Educ. Psychol. 1992, 84 (1), 85−94. (27) Eachus, P.; Cassidy, S.; Norgate, S. H.; Marrow, L.; Greene, L. In Proceedings of the Informing Science and IT Education Conference, Varna, Bulgaria, June 22−25, 2008; Informing Science Institute: Santa Rosa, CA, 2008. (28) Green, H. J.; Lemaire, P.; Dufau, S. Acta Psychol. 2007, 125 (3), 257−278. (29) Suppes, P.; Cohen, M.; Laddaga, R.; Anliker, J; Floyd, R. J. Math. Psychol. 1983, 27, 341−369. (30) Bethell-Fox, C. E.; Lohman, D. F.; Snow, R. E. Intelligence 1984, 8, 205−238. (31) Rosengrant, D. In Proceedings of the 2010 Symposium on EyeTracking Research and Applications, Austin, TX, March 22−24, 2010; ACM: New York, NY, 2010; pp 45−48. (32) Epelboim, J.; Suppes, P. Vision Res. 2001, 41, 1561−1574. (33) Liu, C.-J.; Shen, M.-H. J. Sci. Educ. Technol. 2011, 20 (5), 621− 629. (34) VandenPlas, J. R. Animations in Chemistry Learning: Effect of Expertise and Other User Characteristics. Doctoral Dissertation, the Catholic University of America, Washington, DC, 2008.

ASSOCIATED CONTENT

S Supporting Information *

Postexperimental questionnaire. This material is available via the Internet at http://pubs.acs.org.

■

AUTHOR INFORMATION

Corresponding Author

*E-mail: [email protected].

■

ACKNOWLEDGMENTS This work was supported by the National Science Foundation under grants DUE CCLI 06-18600 and DUE CCLI 08-17279. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation. The authors would like to thank Adam Buffington for his suggestions on the development of the questionnaire. We also thank Natalie Ulrich for her comments on an early draft of this manuscript.

■

REFERENCES

(1) National Research Council. National Science Education Standards; National Academy Press: Washington, DC, 1996. (2) Schuttlefield, J.; Kirk, J. S.; Pienta, N. J.; Tang, H. J. Chem. Educ. 2012, 89 (5), 586−591. (3) Rayner, K. Psychol. Bull. 1998, 124 (3), 372−422. 993



Article

(35) Lindström, P.; Andrá, C.; Arzarello, F.; Ferrara, F.; Holmqvist, K.; Robutti, O.; Sabena, C. In Proceedings of the 2nd International Designs for Learning ConferenceTowards a New Conceptualization of Learning, Stockholm, Sweden, March 17−19, 2010. (36) Qu, L.; Wang, N.; Johnson, W. L. In Proceedings of the 10th International Conference of User Modeling, July 24−29, 2005; Ardissono, L., Brna, P., Mitrovic, A., Eds.; Springer: Edinburgh, Scotland, U.K., 2005; pp 70−73. (37) Anderson, J. R.; Bothell, D.; Douglass, S. Psychol. Sci. 2004, 15, 225−231. (38) Horowitz, T. S.; Fencsik, D. E.; Fine, E. M.; Yurgenson, S.; Wolfe, J. M. Psychol. Sci. 2007, 18, 367−368. (39) Guan, Z.; Lee, S.; Cuddihy, E.; Ramey, J. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems, Montréal, Canada, April 22−27, 2006; ACM: New York, NY, 2006; pp 1253−1262. (40) Rhenius, D.; Deffner, G. In Proceedings of the 34th Human Factors Society Annual Meeting, Santa Monica, CA, 1990; pp 1265− 1269. (41) Kaakinen, J. K.; Hyönä, J. Discourse Process. 2005, 40, 239−257. (42) Epelboim, J.; Suppes, P. Vision Res. 2001, 41, 1561−1574. (43) Tobii. Tobii Studio 1.X User Manual; Tobii: Stockholm, Sweden, 2008. (44) Mayer, R. E.; Larkin, J. H.; Kadane, J. B. In A Cognitive Analysis of Mathematical Problem-Solving Ability, Advances in the Psychology of Human Intelligence; Sternberg, R. J., Ed.; Erlbaum: Hillsdale, NJ, 1984; Vol. 2, pp 231−273. (45) Brünken, R.; Plass, J. L.; Leutner, D. Educ. Psychol. 2003, 38 (1), 53−61. (46) Nakayama, M.; Takahashi, K.; Shimizu, Y. In Proceedings of the 2002 Symposium on Eye-tracking Research and Applications, New Orleans, Louisiana, March 25−27, 2002; ACM; New York, NY, 2002; pp 37−42.

994


Eye-Tracking Study of Complexity in Gas Law ... - ACS Publications

Recommend Documents