Toward Preventing Accidents in Process Industries by Inferring the

Subscriber access provided by READING UNIV

Article

Towards Preventing Accidents in Process Industries by Inferring the Cognitive State of Control Room Operators through Eye Tracking Laya Das, Mohd. Umair Iqbal, Punitkumar Bhavsar, Babji Srinivasan, and Rajagopalan Srinivasan ACS Sustainable Chem. Eng., Just Accepted Manuscript • DOI: 10.1021/ acssuschemeng.7b03971 • Publication Date (Web): 27 Dec 2017 Downloaded from http://pubs.acs.org on January 3, 2018

Just Accepted “Just Accepted” manuscripts have been peer-reviewed and accepted for publication. They are posted online prior to technical editing, formatting for publication and author proofing. The American Chemical Society provides “Just Accepted” as a free service to the research community to expedite the dissemination of scientific material as soon as possible after acceptance. “Just Accepted” manuscripts appear in full in PDF format accompanied by an HTML abstract. “Just Accepted” manuscripts have been fully peer reviewed, but should not be considered the official version of record. They are accessible to all readers and citable by the Digital Object Identifier (DOI®). “Just Accepted” is an optional service offered to authors. Therefore, the “Just Accepted” Web site may not include all articles that will be published in the journal. After a manuscript is technically edited and formatted, it will be removed from the “Just Accepted” Web site and published as an ASAP article. Note that technical editing may introduce minor changes to the manuscript text and/or graphics which could affect content, and all legal disclaimers and ethical guidelines that apply to the journal pertain. ACS cannot be held responsible for errors or consequences arising from the use of information contained in these “Just Accepted” manuscripts.

ACS Sustainable Chemistry & Engineering is published by the American Chemical Society. 1155 Sixteenth Street N.W., Washington, DC 20036 Published by American Chemical Society. Copyright © American Chemical Society. However, no copyright claim is made to original U.S. Government works, or works produced by employees of any Commonwealth realm Crown government in the course of their duties.

Page 1 of 34 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

ACS Sustainable Chemistry & Engineering

Towards Preventing Accidents in Process Industries by Inferring the Cognitive State of Control Room Operators through Eye Tracking Laya Das1, Mohd Umair Iqbal2, Punitkumar Bhavsar1, Babji Srinivasan2*and Rajagopalan Srinivasan3* 1

Department of Electrical Engineering, Indian Institute of Technology Gandhinagar, Gandhinagar, Gujarat 382355, India 2

Department of Chemical Engineering, Indian Institute of Technology Gandhinagar, Gandhinagar, Gujarat 382355, India 3

Department of Chemical Engineering, Indian Institute of Technology Madras, Chennai, Tamil Nadu 600036, India

Abstract While modern chemical plants have numerous layers of protection to ensure safety, the human operator is often the final arbiter, especially during abnormal situations. It is therefore not surprising that when operators lose control over the plant, undesirable consequences including property damage, injury, and sometimes loss of lives follow. It is therefore important to continuously monitor the plant operator’s situation awareness based on their cognitive state. In this study, we make the first known attempt to infer the cognitive state of control room operators and its evolution over the course of carrying out tasks in a control room. First, we study the operator’s actions to distinguish consistent actions from inconsistent ones that allows us to identify major events in the evolution of their cognitive state. Next, we conduct experimental studies with human participants and explore the evolution of their cognitive state through patterns in their eye tracking data. Our studies reveal that two eye tracking measures – fixation duration and saccade duration – are sensitive to the cognitive state and can be used to monitor control room operators and thus prevent human error. Keywords: Process Safety, Sustainable Engineering, Human Error, Cognitive Engineering, Human Computer Interaction

*

Corresponding authors: Email: R. Srinivasan – [email protected]; B. Srinivasan – [email protected]

1 ACS Paragon Plus Environment

ACS Sustainable Chemistry & Engineering 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

Introduction Sustainable Chemistry is defined by the Environment Protection Agency of the USA as “the design of chemical products and processes that reduce or eliminate the use or generation of hazardous substances”. The US EPA further goes on to mention, as one of the twelve principles of sustainable chemistry, “to minimize the potential for accidents”. The chemical manufacturing industry however, seems to repeatedly witness accidents that afflict economic and environmental damage as well as cause injury and even cost lives. For example, in September 1998, an explosion occurred at the Esso gas plant at Longford, near Melbourne, Australia that killed two and injured eight. The sequence began with the incorrect operation of a bypass valve, which caused the condensate to spill over on other parts of the system, causing failure of warm oil pumps. Following this, a metal heat exchanger became super cold and therefore, brittle. The operator again made an error of restarting the hot oil flow that led to the fracture of the heat exchanger, releasing and igniting large volume of gas.1 In 2008, the Bayer CropScience facility in West Virginia, USA witnessed an explosion that also claimed lives of two operators. This incident occurred at the time of restarting a methomyl unit after up-gradation of the distributed control system (DCS). A runaway reaction occurred inside a 4500 gallon pressure vessel causing the vessel to explode in the methomyl unit. Highly flammable solvent sprayed from the vessel resulting in immediate ignition with fires for more than four hours. Investigation by the U.S. Chemical Safety Board (CSB) clearly identified inadequate training of operations personnel to operate a new DCS as one of the major contributors to the incident along with violations of the startup procedure and bypassing of safety critical valves by the operators. Significant research effort has therefore been focused on enhancing process safety by developing sophisticated multi-layer automatic control techniques, advanced alarm management systems, support and guidance systems, better equipment design and ergonomic design of human machine interfaces. In process plant, however it is impractical to have complete automation of the system, resulting in a closed loop system operating under human supervision. In such a mode of operation, the human operator is allowed to influence the state, and eventually outcome of an otherwise automated system. As mentioned above, in several of the industrial accidents, “human error” has been identified as one of the factors contributing to the accident – if only the human had better awareness of the situation, or was trained enough to handle abnormalities, corrective actions could have been taken that might have prevented loss of control over the situation. 2 ACS Paragon Plus Environment

Page 2 of 34

Page 3 of 34 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60


It is therefore essential to understand the cognitive processes of the human operator while operating such closed loop systems and how they affect and are in turn affected by the process. In this paper, we demonstrate that tools and techniques from cognitive engineering2 can offer substantial benefits to addressing these problems in the process industry. We first describe the environmental setting characteristic of the process plant control, outlining the sequence of events in a control room in the next section. A brief overview of cognitive engineering with focus on eye tracking is also provided. The problem of understanding, identifying and remedying “human error” has also been addressed in other safety critical domains such as aviation, healthcare and nuclear power plants, which is reviewed in Section 3. A review of the recent works in this direction in the process plant domain is also presented. The experimental setup and methodology adopted in this article are then mentioned in Section 4. We present the proposed methodology for inferring operator’s cognitive state in Section 5 and illustrate it using an example. Results from the experimental studies are reported in Section 6. We conclude with remarks on the application of the results in industrial settings and directions for the future.

Cognitive Engineering Cognitive engineering is multidisciplinary research area that focuses on analyzing the basic cognitive tasks (such as perception, memory, and reasoning) of human operators to understand their mental workload, decision-making process, planning and situation awareness in industrial settings. It involves detailed study of the human worker in either the actual work context or in more controlled environments. Major applications of cognitive engineering include addressing design challenges and human performance evaluation in areas such as aviation, driving, healthcare, and recently chemical process and nuclear power plants. These fields, although largely different face similar challenges in terms of human error. This is because of the human-in-the-loop mode of operation that all these systems operate in. In this section we first describe the environmental setting in the control room and discuss how cognitive engineering, especially eye tracking studies can be employed to address human factors challenges. Role of Control Room Operator in Process Industries The process plant operates as a closed loop system that is supervised by human(s), wherein the human operator is allowed to change the outcome of the system. Such configurations are typically prevalent in contexts where complete automation is impossible or 3 ACS Paragon Plus Environment


impractical; human intervention therefore remains essential at least in some situations (e.g. equipment failure in a process plant). In these situations, the human still has to acquire data/information from the process before triggering actions based on their judgment and experience. The flow of information and the evolution of the cognitive state of the human operator in such a system is shown in Fig 1. It includes the process (the closed-loop chemical plant) from which data regarding critical variables is collected and visualized on a HumanMachine Interface (HMI). The HMI may comprise of various displays as well as alarms (visual and auditory) in order to convey the state of the process to the human operator. The human may also acquire other information about the plant without the HMI such as by visual or aural inspection.3 The human operator uses this information and infers the ‘state’ of the process. Occasionally, for instance when the process drifts towards an undesired state, the human operator takes actions (such as manipulating actuators) based on their judgment in order to ensure that the process remains or returns to a desired state. In order to understand the actions of the human operator, it is necessary to understand the elements that form the basis of such actions. The first step in controlling a process is to acquire information from the HMI. This information is then interpreted to assess and classify the state of the process as being “normal” or “abnormal”. This is followed by generating hypothesis regarding the possible measures that can be taken to maintain the process (such as regular tasks) or to handle the abnormal situation. Based on the information acquired from the HMI and the plausible hypothesis generated, a final set of control actions is executed by the human. Subsequently, information about the process is obtained from the HMI to understand the response of the process to the actions taken. This iterative cognitive process of information acquisition, hypothesis generation and action affects the “cognitive state” of the human operator. While operating in such a setting, the cognitive state of the operator keeps evolving with the events that occur in the control room. For example, when an operator acquires information about a variable, develops a set of hypotheses and finally carries out a set of actions, their cognitive state evolves from “monitoring” to “inferring” to “deciding”. The operator might go through the monitoring and inferring stages multiple number of times before finally acting on the system. However, it is only the information acquisition and action stages of the operator that are observable to an external agent, while the internal cognitive states are hidden. Furthermore, there exists a many-to-one relationship between the internal state of the operator and the actions they decide to take – they might be confident about the action or confused about the situation, while still monitoring the same set of variables and 4 ACS Paragon Plus Environment

Page 4 of 34

Page 5 of 34 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60


carrying out the same set of actions. In this article we explore how cognitive sensors – eye tracking in particular – can be used to identify the unobservable internal state of the operator. Traditionally, monitoring the cognitive behavior of human operators relied on offline, subjective measures based on the operator’s response. Examples include the NASA Task Load Index (NASA-TLX), the Subjective Workload Assessment Technique, and the Workload Profile.4 These cannot take into consideration the cognitive states that lead to operator actions. More recently, the cognitive state can now be inferred through physiological measurements such as eye tracking, electroencephalography (EEG), heart rate variability (HRV), skin conductance. Of these, eye tracking offers the most nonintrusive approach to inferring the cognitive behavior of the human. Eye Tracking Eye tracking refers to the technology of identifying a person’s point of gaze and tracking the movements of eye with respect to the head. It has been a research tool since 1960’s when the technique required subjects to place their heads on a chin rest in dark rooms with controlled illumination and has since then evolved into a highly nonintrusive technology.5 It is widely used in research and in real-life applications such as education and learning, driving, aviation where the subjects are in fact required to move their head and eye to acquire critical information from spatially distributed sources under different ambient lighting conditions. Eye tracking is developed and applied with based on the ‘eye-mind’ hypothesis – the movement of eye serves as a trace of the dynamic attention allocation of the human.6,7 The human eye consists of several structures like pupil, retina etc. that function together to enable visual processing and perception. The retina has varying degrees of sensitivity, with the highest in the fovea. Light from visually interesting areas of the environment (referred to as areas of interest, AOIs) is focused on the fovea by eye movements such as saccades, smooth pursuits and by fixations. For example, during a reading task, the eyes of the readers fixate on words and quickly move between words via saccades. Smooth pursuits are the eye movements that help keep a moving object within the focus of the eye. These movements of the eye can be recorded to identify and monitor areas that are of interest to the operator.



Eye tracking technology identifies the point of gaze (fixation) of the eyes of a person and tracks movements of the eyes with a video camera that record images of eyes.8 Eye tracking provides measures pertaining to eye movement (gaze) and pupil size. Eye movement based measures can be fixation based, saccade based or a combination of the two. Fixation refers to a state where the eye remains still which typically lasts from tens of milliseconds to several seconds.8 Metrics based on fixation include time to first fixation, dwell time, fixation counts, fixation frequency, average fixation duration on each AOI, etc. Saccades on the other hand, are rapid movements of the eye from one AOI to another that typically take 30–80 ms and are the fastest movements made by the human eye. Common saccadic measures are saccade velocity and saccade amplitude. Blink frequency (number of blinks per unit time) is also used as an eye tracking measure. Scan path is a fixation-saccade derived measure that comprises a series of fixations interleaved with saccades. In addition to the gaze derived measures, eye trackers also measure pupil diameter. The pupil diameter is controlled by the iris in order to control the amount of light entering the eye. Pupil diameter fluctuations are involuntary and do not require volitional response thereby the pupil diameter is an important measure for understanding cognitive processes.8 Eye tracking has been applied in a number of safety critical towards understanding human behavior. We next review a few studies in aviation, nuclear power plant, healthcare and process plants.

Eye Tracking Studies in Safety Critical Domains Eye tracking technology has been used to understand the cognitive behavior of operators in several safety critical domains such as aviation, healthcare, nuclear power plants, and driving. In addition to monitoring the operator’s attention allocation and distribution using metrics such as gaze location and pupil diameter the operator’s cognitive state such as fatigue, stress and workload can also be inferred. Eye tracking thus allows systems engineering techniques to be extended to the human component of fairly automated systems that are operated under human supervision. In the following, the applications of eye tracking in three safety critical domains – aviation, nuclear power plant, and healthcare – are discussed. Aviation In the aviation industry, the process being controlled is the aircraft and the human is the pilot, operating in a multi-crew environment (accompanied by copilot). Data acquired 6 ACS Paragon Plus Environment

Page 6 of 34

Page 7 of 34 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60


from the aircraft is presented on cockpit displays such as Terrain Awareness and Warning Systems, Crew Alerting Systems, Flight Control Systems, and Flight Mode Annunciators (FMAs).9,10 These displays are usually distributed throughout the cockpit and also contain alerts relevant to various subsystems of the aircraft.10 The basic tasks of the pilot include: controlling the airplane’s path (aviate), directing the airplane from its origin to its destination (navigate), communicating information and instructions, and managing resources.11-12 However, the tasks of a pilot vary greatly along multiple dimensions during an emergency, i.e., abnormal situation. One of the major dimensions for the crew is to understand the severity of threats and the degree of time criticality associated with a particular abnormal situation.13 In addition, the mitigation of abnormal situation also depends on the amount of task workload, degree of complexity of task, and novelty of a particular situation.13 The pilot has to constantly acquire and process information from the displays and/or alarms and assess the state of the flight. This is referred to as having ‘situation awareness’ defined as “the perception of the elements in the environment within a volume of time and space, the comprehension of their meaning, and the projection of their status in the near future”.14,15 Loss of situation awareness (SA) in perception, comprehension or projection can lead to poor performance in handling an abnormal situation. Most emergency and abnormal situations also lead to a rise in the workload of pilot, which can either fall after a brief period of time, or extend throughout the remainder of the flight.13 Such an increase in workload during abnormal situations can lead to crew errors and less-than-optimal responses which can often be linked directly to inherent limitations in human cognitive processes.13 Stress can also significantly hamper the cognitive performance, by inducing a narrowed attention focus, referred to as attention tunneling.16-17 Attention tunneling can lead to inefficient scanning of the full range of environmental cues, and can direct the individual’s focus on cues that are perceived as most salient or threatening.18 In addition to workload and stress, fatigue and sleep deprivation can also affect the cognitive state of the pilot and lead to impaired performance.19,20 Eye tracking has been used as a tool to understand the cognitive aspects of pilot performance. A study with eight participants examined the detection and response to an unexpected event in a simulator where half the tasks involved information inaccurately (or not) displayed to the pilot through Synthetic Vision System (SVS).18 Their results indicated that pilots who evenly scanned the displays and the outside world and thereby maintained a



regular scan pattern were able to detect the unexpected events while pilots who failed rarely scanned the outside world and focused on the SVS, exhibiting “cognitive tunneling”. A preliminary study was conducted with six participants on the information gathering techniques during normal and abnormal situations in a simulated environment.21 Eye tracking data revealed that pilots spent less time glancing at instruments and focused on fewer instruments in the degraded flight conditions (abnormal situation). In addition, pupil size was observed to vary during different phases of the flight in abnormal situation, indicative of the variations in stress and attention levels of the pilot. An eye tracking study attempted to measure the level of SA of pilots through eye tracking.22 They observed that eye movement patterns and instrument scan behavior can be correlated with the pilot’s SA, i.e., it can be used to predict if and when the pilot was going to lose SA. Another study used attentional focus and scanning entropy to explore the potential of eye tracking measures with twelve participants as indicators of SA of the pilot during simulation tasks.23 The authors introduced abnormalities in the system to hamper the SA and observed shifts in attentional focus and changes in scanning entropy (indicative of information acquisition activities) when the pilot was trying to identify a malfunction (abnormality). Smart Eye Pro eye trackers were used with ten participants to monitor the FMA engagement, a primary component providing information regarding mode changes during flight.24 During autopilot mode transitions, pilots were observed as not monitoring the FMA any more than during periods when there were no transitions.24 This suggested that pilots could easily miss an autopilot mode transition, not notice that it occurred, and therefore, experience a loss of mode awareness. Similar results were found in another study where the pilots were observed to monitor basic flight parameters to a much greater extent than the visual indications of automation configuration, frequently failing to notice mode changes or verify manual modes.25 The study suggested that low system observability and gaps in the pilot’s understanding of complex automation modes contributed to the above problems. Situation Awareness was studied using eye tracking in a fighter pilot simulation scenario.26 Subjects had to simultaneously execute tasks, use navigations systems and perform several other procedures and almost 72% of the pilots failed to identify the activated generator warning light (visual alert) and exhibited poor SA.26 Pilots with better SA also reported lower perceived workload during the task. The pilot’s cognitive workload has also


Page 8 of 34

Page 9 of 34 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60


been related with their operational performance.27 Differences in the pilot’s fixation duration on different displays served as a key factor in distinguishing the response of a pilot with or without SA (measured with the ability to detect a hydraulic malfunction).27 Another study reported on a flight simulator based pilot study instrumented with electromyography and pupillometry.28 Participants who reported higher levels of anxiety in subjective assessment produced large numbers of saccades during a cognitively demanding task. On the other hand participants who did not report elevated levels of anxiety exhibited relatively constant saccade rates. Similarly, increase in pupil size was observed with increase in subjectively reported anxiety. Hence, the influence of cognitive state (i.e. anxiety) during the cognitively demanding task is clearly observable from the saccade rate and pupil diameter. Pupil diameter was also correlated with workload in a flight simulation study under varying cognitive workload conditions.30 The flight simulation task was performed by 12 participants under varying cognitive workload conditions. The behavioral performance of subjects had been studied by a subjective measure (NASA-TLX) as well as a physiological measures. Eye measures, pupil diameter and eyelid opening serves as a sensitive measure to the change in MW. With the increase in workload, the pupil diameter first increased and then decreased. The pupil diameter under the high workload situation is higher as compared to the low and controlled MW cases. On similar lines, flight testing experiment conducted using a flight simulator consisting of a fully functional flight deck with full glass cockpit displays, outside visual projectors, functioning mode control panel (MCP) with autopilot and autothrottle, and standard Boeing 737 controls.31 The experiment was conducted in a manner that the pilot experienced varying flight scenarios within the same task that demand different levels of workload. Workload conditions were varied by changing visibility conditions and automation level. Various eye tracking metrics like number of fixations, mean fixation duration, fixation percentage relative to total no. of fixations, mean and maximum saccadic length etc. were measured and were observed to be affected by task loading. In summary, the above studies bring forth the potential of eye tracking to understand the relationship between the pilot’s mental state (mental workload, stress etc.) and behavior during routine as well as cognitively challenging situations. Several metrics derived using eye tracking such as fixation and saccade derived metrics and pupil diameter were found to offer qualitative as well as quantitative measures which could be used to understand operator behavior and performance.



Nuclear Power Plants Nuclear power plants constitute another safety critical industry with the human operator as a key element of the control loop. Data acquired from a nuclear power plant are presented to the operator in the Main Control Room.32 Decision making in NPPs is knowledge-intensive and accompanied by time pressure.33 The operator is therefore required to be always aware of the current process state and be able to extend their knowledge effectively to predict future state and carry out any necessary control actions, i.e., to maintain SA at all times.34-37 It is therefore imperative to perform cognitive evaluation of operators in the NPP control rooms. Eye tracking has been used to analyze operator performance (both individual and team based) while working with the HMI.38 Fourteen crews with 3 members (a reactor operator, turbine operator and shift supervisor) per crew performed the experiment where each crew had to identify an EOP for the abnormal situation and bring the plant to normal situation. The results revealed that task complexity, presentation complexity, training level and pre-knowledge significantly affect error rate, operation time and personal workload. Each crew was able to successfully diagnose the initiating events of the abnormality but the aforementioned factors influenced the ability of situation recognition and finding out instrumental failure. An experimental study aimed to investigate the relationship between information flow rate of accident diagnosis tasks and operator’s cognitive workload.39 The study compared three methods of measuring the operator’s cognitive workload – information flow rate, subjective measure, and physiological measure. Information flow rates were obtained by imposing time limit restrictions to the time of the accident diagnosis task and NASA-TLX was used as a subjective measure of workload. Eye tracking was used to obtain fixations on various ROIs (region of interests) and operator’s blink frequency which were further used as physiological measure of workload. It was observed that both fixation duration and blink rate decreased when a task was cognitively challenging. Also, longer and more fixations were observed on instruments under high cognitive workload conditions. In another study, participants performed computerized emergency operation procedures (EOPs) of nuclear power plant. Pupil size, blink rate, blink duration, heart rate variability etc.40 were used for measuring workload. The results revealed that eye response measures are useful for understanding temporal changes in workload while cardiac measures are useful for overall 10 ACS Paragon Plus Environment

Page 10 of 34

Page 11 of 34 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60


workload measurement. Pupil dilation was attributed to the need for increased attention and arousal caused due to errors. Blink duration increased over a task period for both EOPs with different levels of complexity. Blink rate was found to be a useful measure for both the peak workload and the procedure workload. A simulation study with eye tracking was performed to evaluate operator’s cognitive workload and performance during reactor shutdown procedure (consisting of primary and secondary tasks) of Nuclear Power Plant (NPP).41 The performance of the secondary tasks (in terms of error rate), subjective workload measure (NASA-TLX) and seven physiological indices were analyzed. The authors used heart rate, heart rate variability, blood pressure, parasympathetic/sympathetic ratio (LF/HF ratio), blink frequency and blink duration as physiological indices. Results showed that blink duration was shorter and blink frequency was lesser during high complexity phase of the task as compared to low complexity phase. The possible reason pointed out for this was that the subjects spent more time watching the interface which therefore decreased their blink frequency and shortened the blink duration. In summary, the benefits of cognitive studies have also been widely reported in the nuclear power plant operations domain, which shares many similarities with process plants. Healthcare The healthcare industry also faces human factors challenges similar to those in aviation and nuclear power plants. Here the process is the patient while the human operator is the care provider (e.g. nursing staff, telemetry technician, and surgeon). The patient’s vital signs are the process data. Monitors often also have alarms related to the patient’s vital signs which indicate an abnormal situation. Alarm management challenges, analogous to those in the process industry, have been reported in the healthcare setting as well. For instance in telemetry, the technician’s task is to monitor vital signs of the patient and communicate abnormal patient vitals to care team.42 In an operation theatre setting (e.g., laparoscopic surgery), the task of a surgeon is to continuously monitor the patient’s vital signs during the surgery.43 Hence, surgeon’s vigilance (attention and alertness) carries utmost importance for improving patient safety.44,45 In addition to expertise, SA has also been considered as significant importance.46 A preliminary study conducted with eight surgeons analyzed visual attention strategies of micro neurosurgeons with varying levels of expertise.47 Eye gaze data was collected while surgeons were observing the four phases of a tumor removal surgery. The distribution of saccadic 11 ACS Paragon Plus Environment


amplitudes revealed that experts’ attention was more compact and locally targeted and that they exhibited longer fixation durations. The results and insights obtained from such studies can be used to improve conduct of micro neurosurgery by building intelligent gazecontingent systems. Similar studies were performed to differentiate the eye gaze patterns of five novice and five experts in a Laparoscopic Surgery simulator.48 It was observed that experts preferred longer target gazing while performing the task while novices’ behavior was varying and switched more between tool and target and had longer fixation duration (information acquisition) on the tool while performing the task. Another eye tracking based pilot study was conducted with four participants for the analysis of ECC (extracorporeal circulation) operation tasks performed during real cardiovascular surgery in the operating room.49 Operating this process demands visual information obtained from various information sources like surgeons, ECC indicators, surgical instruments, displays etc. It was observed that experts dispersed their attention (for information acquisition) more widely than intermediate and novices. Similarly, experts showed transition patterns that were significantly different from novices. An eye tracking study with four experienced and four novice surgeons distinguished the skills and SA of surgeons in a simulated laparoscopy task. The authors found out that novices focus so hard on the surgical display that they do not focus on patient’s condition, while experts balance their glance over the display as well as the patient which is similar to the issue of inattentional blindness in aviation.50 An experiment using simulated surgical task to show the effects of changing the cognitive workload on pupil size.51 An experiment consisted of three subtasks with different levels of cognitive workload requirements. Average normalized pupil diameter and the average rate of pupil diameter change (the slope of normalized pupil diameter over time) were used as workload measures. Average pupil size failed to correlate with the task difficulty, as no significant difference was observed for all tasks. Rate of pupil diameter change was identified as an indicator of changes in workload – during the most difficult task it increases, and drops suddenly when task complexity decreases. Eye tracking was proposed as a solution to understand various aspects of the HMI that aid in information acquisition (using gaze data) and to understand the cognitive load and stress (using pupil diameter). The above studies highlight the potential of eye tracking studies in assessing the expertise, cognitive state, etc., in a surgical environment. 12 ACS Paragon Plus Environment

Page 12 of 34

Page 13 of 34 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60


Process Plants Studies similar to those conducted in the above domains have also been conducted in a process plant setting. Eye tracking measures have the potential to monitor the human operator behavior in the process control room.52,53 Our previous study found that the gaze distribution of participants on different variables during simulated abnormal situations is related to their performance in handling the situation – participants having longer gaze on uncorrelated variables predominantly failed in handling the situation.54 Pupillometry to analyze the workload of participants during abnormal scenarios and found out that the pupil diameter was a good indicator of the cognitive workload during the execution of tasks.55 The workload of participants who failed in handling an abnormal scenario remained high towards the end and so did the corresponding pupillary dilations. On the other hand, participants who successfully completed the tasks exhibited consistently decreasing pupil diameter after the first set of corrective actions. In another study, gaze entropy was used to quantify the level of situation awareness of a participant while handling abnormal situations and found that participants with adequate situation awareness looked at a few variables responsible for abnormal scenario which resulted in a lower gaze entropy.56 Lack of situation awareness resulted in a gaze on relatively large number of variables and a larger value of gaze entropy. The above studies were aimed at identifying patterns in eye tracking measures of participants based on their performance (success/failure). However, the problem of inferring the internal cognitive state of the operator at the time of carrying out routine tasks or handling abnormal situations has not been addressed. In this work, we address the challenge of inferring the unobservable yet critical cognitive state of the operator while carrying out tasks in a control room. We conduct experimental studies to understand the relationship between the operators’ actions and their cognitive state through the responsive changes in the process with carious eye tracking measures. We next describe the experimental methodology adopted in this study.

Experimental Methodology The methodology involves identifying a process, simulating it based on mass and energy balances, making a Human Machine Interface (HMI) and then conducting experiments based on performing certain tasks like controlling the variables to within the acceptable limits in the wake of disturbances or bringing a variable(s) to a new steady state, etc. by human subjects. 13 ACS Paragon Plus Environment


Experimental Setup We have modeled an ethanol production process. The process involves an exothermic reaction between ethene and water leading to the formation of ethanol. The feed goes into the reactor along the line controlled by valve V-102. The reaction takes place in the reactor, and since the reaction is exothermic, to maintain the temperature within acceptable limits and in worst case escalation of temperature to dangerously high levels, there is a coolant line to circulate water in to the jacket of CSTR. A human operator monitors and controls the process using the HMI shown in Fig 2. The coolant (here water) flowrate is controlled by operation of valve V-301. There are different sliders in the HMI which are used to adjust the valve position and accordingly control the flow in the lines. In addition to this there are different tags which provide an operator values of various process variables: F101 (flow rate of the feed), F102 (flow rate of coolant), T101 (inlet temperature of coolant), T102 (exit temperature of the coolant), T103 (temperature inside the reactor) and C101 (concentration of the product in the tank). In addition to this, the trend of different process variables is displayed on the HMI enabling the operators to observe the changes in process variables in response to operator action(s). The operator actions here involve moving of one or more sliders controlling the flow through values to achieve the desired result. Experimental Protocol A team of graduate students of Indian Institute of Technology Gandhinagar with adequate background in control systems engineering and prior experience were trained to be the control room crew for the simulated process. The training began with an induction process with general instructions on using the HMI (Fig 3). Next, the participants were provided instructions on the type of tasks they would be expected to perform as well as detailed instructions for specific tasks (Fig 4). The tasks in the study are of two types: 1. Learning tasks, wherein participants learn the process dynamics – in the form of gain and time constant between variables, and 2. Control tasks, wherein participants are required to bring and maintain a specific process variable within an operating band without crossing the lower or upper limits, within a stipulated amount of time. Each control task is designed in a manner such that only one manipulated variable (slider) is required to complete the task. A typical set of two consecutive tasks instructions is shown in 14 ACS Paragon Plus Environment

Page 14 of 34

Page 15 of 34 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60


Fig 4. In the control tasks, the desired operating region is shown by black dotted lines on the corresponding trend plot while the safe operating limit is shown as a solid red line. The HMI was designed to update the plots after every 5 seconds, while the tag values are updated in the schematic every second. A snapshot of the GUI during a typical control task is shown in Fig 2. A total of 16 tasks were required to be performed by each participant – 8 learning tasks and 8 control tasks. The tasks had varying levels of difficulty based on the values of gain and time constant for the variable to be controlled. These tasks were also repeated by the participants. Out of these, the data collected for the control tasks were used for further analysis. In order to analyze the cognitive behavior of the participants, 12 areas of interest (AOIs) were defined on the HMI. These include the Instruction Panel (IP), action buttons (Start and Finish), the CSTR schematic, and trend panels of the 6 process variables as shown in Fig 2. A Tobii TX 300 remote eye tracker was used for tracking the eye movements of the participants. A standard five point calibration procedure was adopted for ensuring the validity of the gaze data. Eye movements and pupil diameter measurements were recorded with a sampling frequency of 120 Hz. In the next section, we describe the proposed methodology for inferring operator’s cognitive states using eye tracking measures.

Methodology for Inferring Cognitive States of Control Room Operators The control room operator interacts with the process by performing control actions such as opening or closing valves by suitably manipulating the corresponding sliders on the HMI. Further the actions are congruent with the cognitive state of the operator, i.e., they reflect the operator’s mental model at that point in time. This temporal evolution of the cognitive state of the operator as they interact with the process during the course of a task is schematically depicted in Fig 5. The cognitive state of the operator is dynamic and evolves with time as they observe the process, perform control actions and see the effect of the actions on the process’ evolution. Examples of control actions include manipulating a process variable by moving its slider on the HMI. The actions are not all singletons and unique, rather they may be repeated a number of times. A cluster of repeated actions on the same variable and leading to its monotonic change (increase or decrease) within a predefined time window (say 5 seconds) is termed as an event. Each task can therefore be decomposed into a set of events. An illustrative example is presented in Fig 6. The participant is required to maintain C101 between 1440 mmol/lt and 1460 mmol/lt using valve V301 (coolant flow). The 15 ACS Paragon Plus Environment


participant is also instructed to not let the variable cross a limit of 1480 mmol/lt. These bounds are also shown on the trend panel of C101. It can be observed from Fig 6 that the participant carries out the first set of actions (decreasing the coolant flow with V301) at around 46 seconds, i.e., the first event starts at that time. Around 64 seconds, a second event begins when the operator starts actions to decrease the flow rate of the coolant. Event 3 begins at 69 seconds and corresponds to increasing the coolant flow rate. The subsequent events throughout the task are also shown in Fig 6. Our goal in this work is to infer the cognitive state of the operator. For this, we discretize the dynamic cognitive state of the operator. We seek to identify changes in the operator’s cognitive state. The basic premise in this work is that a change in the operator’s cognitive state would reflect in a change in control actions or specifically a change in the events. Some examples of change in events include stopping the manipulation of a variable, manipulating a different variable to control the process and change in the direction of manipulation (such as switching from increasing a variable to decreasing it or vice versa). To detect such changes, we classify a pair of adjacent events as either consistent or inconsistent. If the operator repeats the same actions (same slider moved in the same direction) over a large time window, it indicates that the process is behaving as per the operator’s mental model and there is no change to their cognitive state – such events are called mutually consistent. On the other hand, if the operator changes the direction of a variable or starts manipulating a different variable, it indicates that the operator has moved to a new cognitive state and the adjacent events are different and termed as inconsistent. In the illustrative example above, the cognitive state of the operator at the beginning of the task is ܵ଴ . The beginning of Event 1 at 46 seconds denotes an update to the operator’s cognitive state to ܵଵ. The event beginning at 64 seconds is consistent with the previous event, and hence does not portend an update to the operator’s cognitive state. The operator’s cognitive state changes at 69 seconds when Event 3 with a change in direction of the slider begins. The plots for fixation and saccade duration in Fig. 6 label the different cognitive state of the operator, which update with every inconsistent event during the task. This step wise updating can be interpreted as the temporal evolution of the cognitive state of the operator. The event level eye tracking data can be used to infer the cognitive state of the operator. Fig 6 shows the average fixation duration and saccade duration of the participant


Page 16 of 34

Page 17 of 34 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60


during different events. The following observations can be made from the trend of the fixation duration in relation to the cognitive state and process variable: 1. Inconsistent events lead to a rise in the fixation and saccade duration of the participant: The inconsistent events (marked by vertical black lines on the plots) can be seen to have large values of average fixation duration, as compared to the previous events. Fixation duration of a participant is correlated with careful information acquisition/inspection and deeper information processing of participants.8 The saccade duration is also correlated with the difficulty of the task a participant is carrying out. 8 A rise in fixation duration during inconsistent events can therefore be attributed to the fact that in order to make a change the direction of the slider, the participant has to realize that the process variable is moving in the direction opposite to the desired and decide to counter their previous actions. We hypothesize that these activities of acquiring information from the HMI, perceiving the incorrect direction of the process trend and deciding to change in the direction of the slider manifests in the eye tracking measures as a rises in the fixation duration. Similarly, the increase in saccade duration can be attributed to the increase in difficulty of the task at different events. This can be observed in three out of four inconsistent events in the illustrative example. This observation also justifies why we chose to update the cognitive state of the participant only for inconsistent events. 2. Consistent events do not lead to large increase in the fixation and saccade duration: As opposed to inconsistent events that require a deeper understanding of the process and updating the mental model and state of the participant, the consistent events are a sign of confidence and comfort in carrying out a task, which does not lead to large changes in the fixation duration (in three out of three consistent set of actions). Next, we generalize the above observations by evaluating it on a large number of events carried out by the operator crew members of the ethanol plant.

Results Here we report the results from the application of the above methodology to infer the cognitive state to the crew of four control room operators of the ethanol plant. We define a window around each event that starts 5 seconds before the first action in the event and ends 5 seconds after the last action in the event. The eye tracking data in each window is then extracted, and the AOI with the participant’s gaze is identified. From this point onwards, only 17 ACS Paragon Plus Environment


those events that have at least 70% of eye tracking data available in their corresponding windows are considered for further analysis. The fixation duration, fixation count, saccade duration and saccade count were calculated and analyzed at the event level in all the cases. We first present the results obtained from analysis of consistent and inconsistent events at the participant level. As mentioned earlier, selection of data is based on the availability of at least 70% data. After this, only those tasks were selected that had at least one consistent and one inconsistent event. We present the comparison of the eye tracking measures for these tasks at the participant and task level in the following. The eye tracking measures were analyzed for four participants. The analysis was performed for both consistent and inconsistent events. The observations from the analysis are as follows. Participant one: For this participant there are eight consistencies and four inconsistencies (distributed across all tasks). The average value of fixation duration during consistent events is 461.2 ms and for inconsistent events is 505.4 ms. The average saccade duration changed from 47.7 ms for consistent events to 47.9 ms for inconsistent events while fixation count increased from 23 to 26. At the same time, the saccade count was observed to increase from 37 to 54. This shows that there is an increase in fixation duration as the participant goes from consistent events to inconsistent events. The increase in the fixation duration can be attributed to the fact that the participant would be looking at a specific trend in a particular AOI(s) that might have caused him to change their decision. As already discussed above, a change in state reflects change in decision making of a participant. We looked into the consistent-inconsistent pairs, i.e, a participant moving from a consistent to consecutive inconsistent event in a particular task. Four such pairs were found for this participant out of which two such pairs exhibited an increase in the average fixation duration while the other two showed a decrease. The saccade duration increased in one out of three pairs while remaining almost same for the other pairs. The fixation count did not show any significant change in the four pairs. The saccade count, however, exhibited a marked increase for all the four cases. The increases in fixation durations can be attributed to the deviation of process variable away from the acceptable regions due to operator actions. This observation hints towards the context dependence of eye tracking measures. The same pattern has also been observed and highlighted in the illustrative example. 18 ACS Paragon Plus Environment

Page 18 of 34

Page 19 of 34 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60


Participant two: For this participant, there are eleven consistent event and seven inconsistent events. The mean value of fixation duration for the consistent events was found to be 452.3 ms while for the inconsistent events, it was 601.7 ms again showing an increasing trend as was observed for participant one. The saccade duration however decreased from 42.2 ms for consistent event to 37 ms for inconsistent events, fixation count decreased from 23 to 18. At the same time, the saccade count decreased from 48 to 22. For this participant, three transition from consistent to consecutive inconsistent event was available, of which two exhibited increase in fixation duration, drop in saccade duration. The fixation and saccade counts decreased for two transitions and remained almost constant for one. Participant three: For participant three the results are in agreement with those of the above participants. The number of consistent and inconsistent events were found to be four and two respectively. The average fixation duration increased from 302.8 ms to 378.7 ms for consistent and inconsistent events. The saccade duration increased from 37.5 ms for consistent events to 43 ms for inconsistent events; the fixation count changed from 46 to 47 while the saccade count dropped from 51 to 31. This participant also exhibited only one instance of a transition from consistent to inconsistent within a task that showed an increase in fixation duration and saccade duration, and a drop in fixation and saccade counts. Participant four: For participant four, there were four consistent and two inconsistent events. The average value of the fixation duration for consistent events is 317.6 ms and for inconsistent events is 475.1 msec. These results are in agreement with those for the previous three participants. The saccade duration changed from 32.8 ms for consistent events to 34.2 ms for inconsistent events; the fixation count decreased from 27 to 15 while the saccade count changed from 56 to 52. Out of the two consistent-inconsistent pairs found both of them showed an increase in fixation duration and saccade duration, a decrease in fixation count and no pattern in saccade count as the state changed from consistent to inconsistent. The above results depict that in general for consistent events the fixation duration is on a lower side when compared with the fixation duration for inconsistent events. Similarly the average saccade duration increased for three out of four participants while decreasing for one participant. The fixation and saccade counts, however, did not exhibit any repeated patterns. The average value of fixation duration for all the consistent events for all tasks performed by the four participants turned out to be 422.7 ms when compared with the average value of fixation duration for all the inconsistent events for which it was 508.5 ms. 19 ACS Paragon Plus Environment


Similarly, the saccade duration also showed an increase from 40.5 ms to 44.1 ms when averaged over all tasks and all participants. The summary of the results is also provided in the tabulated form in Table 1. In order to confirm the validity of our results we delved deeper by analyzing the variation of fixation duration at task and adjacent-events level. The results are summarized below. Results at Task and Event Levels Case 1: Task level: This analysis involved measuring the average value of fixation duration by taking into consideration different states within a task, irrespective of whether the transitions from consistent to inconsistent were consecutive. The average value was taken for all the consistent events and compared with the average values for all the inconsistent ones. We observed that out of 12 tasks, fixation duration increased for 9 tasks and decreased for 1 task; saccade duration increased for 6 tasks (decreased for 2 task and did not show a significant change for 1 task), fixation count increased for 4 tasks and saccade count increased for 7 tasks, as the events changed from consistent to inconsistent. Case 2: Transition from a consistent event to adjacent inconsistent event: This analysis is same as the above one, with the tasks chosen such that there exists a transition from a consistent to a consecutive inconsistent event within a single task. Out of the 10 tasks analyzed, it was found that for 6 tasks the fixation duration increased significantly when there was a shift from consistent to inconsistent event; for 3 tasks the fixation duration decreased (by a smaller amount compared to increase observed in the above 6 tasks), and for 1 task the fixation duration did not show a significant change. The saccade duration increased in 5 cases while decreased in two tasks and showed no significant change in the rest tasks. The fixation count decreased for eight tasks while did not have a significant change in two tasks. The saccade count increased for three tasks, decreased for six tasks and did not change significantly for one task. As mentioned earlier, the inconsistent events lead to a significant rise in the average fixation duration of the participant. It is however noteworthy that the change in average fixation duration at around 135 seconds for the illustrative example is large compared to other changes in the measure. This can be attributed to the fact that during this event, the process variable shoots out of the desired band for the second time in the task, imposing additional 20 ACS Paragon Plus Environment

Page 20 of 34

Page 21 of 34 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60


burden on the participant. This observation highlights the context sensitivity of the eye tracking measures. In summary, the fixation and saccade durations generally increase as the participant makes an inconsistent action. Since these are sensitive to deeper process understanding and task difficulty, the observed changes can be attributed to increase in cognitive processing when the participants decide to take a different action.

Discussion The human operator is a critical element in a chemical process plant and form an important component of the “layers of protection” of the plant. The cognitive state of the operator is an unobservable, yet crucial link between the process behavior (information acquired by the operator) and the actions taken on the process. This study explored how eye tracking can be used to infer the cognitive state of the operator in a process plant setting. Operator actions are classified based on their cognitive state into consistent and inconsistent. This classification paved the way for identifying the evolution of the cognitive state of the operator. Experimental studies reveal that eye tracking measures are sensitive to the evolution of the cognitive state of the participants and hence can be used to assess their confidence in handling a situation. In this study, the average fixation duration and average saccade duration were observed to exhibit sensitivity to the cognitive state of participants while others did not exhibit any uniform pattern. Previous eye tracking studies on operator behavior in the process plant setting have aimed at identifying differences in eye tracking measures (typically calculated for the entire duration of the task) of successful and failed participants. The information pertaining to relevance of eye gaze location to a particular task is also often used in such analyses. The proposed methodology however, disaggregates a task into multiple events and examines the operators’ actions and eye tracking measures at the event level, without evaluating their performance (success/failure) in carrying out a task. The proposed methodology therefore allows us to infer the cognitive state of the operator in (near) real time during task execution; it thus offers the possibility to foresee abnormal situations due to human error and prevent them from escalating to accidents. Plant operators usually being experienced may not show frequent changes in cognitive states during the routine operation. However, they may exhibit much more 21 ACS Paragon Plus Environment


frequently changing cognitive states (inconsistencies) during the rarely occurring, yet critical abnormal events. These inconsistencies will be reflected by the fixation duration and which in turn can be used to monitor the operators, especially during safety critical operations. This can help prevent occurrence of any untoward accident such as leakage, spillage, fires, and explosions, or at the least, help minimize the severity of damage caused by an accident. Our future work will focus on gauging the black-box cognitive state of control room operators by studying its dependence on many other eye tracking measures to develop a robust index of ‘unknown to others’ cognitive states during various normal and abnormal situations.


Page 22 of 34

Page 23 of 34 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60


References 1.

2.

3. 4.

5. 6. 7. 8.

9.

10. 11. 12. 13. 14.

15.

16. 17. 18.

Hopkins, A. An AcciMap of the Esso Australia gas plant explosion. Paper presented at the Julkaisussa: Proceedings of the 18th ESReDA seminar on: Risk Management and Human Reliability in Social Context, Karlstad-Sweden, 2000. Chang, Y.; Mosleh, A. Cognitive modeling and dynamic probabilistic simulation of operating crew response to complex system accidents. Part 4: IDAC causal model of operator problem-solving response. Reliability Engineering & System Safety 2007, 92 (8), 1061–1075. Norman, D.A. Cognitive engineering. User centered system design, 31. 1986. Rubio, S.; Diaz, E.; Martin, J.; Puente, J. M. Evaluation of Subjective Mental Workload: A Comparison of SWAT, NASA-TLX, and Workload Profile Methods. Applied Psychology 2004, 53 (1), 61–86. Bradshaw, J. L. Load and Pupillary Changes In Continuous Processing Tasks. British Journal of Psychology 1968, 59 (3), 265–271. Just, M. A.; Carpenter, P. A. A theory of reading: From eye fixations to comprehension. Psychological Review 1980, 87 (4), 329–354. Rayner, K. Eye movements in reading and information processing: 20 years of research. Psychological Bulletin 1998, 124 (3), 372–422. Holmqvist, K.; Nyström, M.; Andersson, R.; Dewhurst, R.; Jarodzka, H.; Van de Weijer, J. Eye tracking: A comprehensive guide to methods and measures, OUP Oxford, 2011. Dill, E. T.; Young, S. D. Analysis of Eye-Tracking Data with Regards to the Complexity of Flight Deck Information Automation and Management - Inattentional Blindness, System State Awareness, and EFB Usage. 15th AIAA Aviation Technology, Integration, and Operations Conference 2015 Cline, T.W.;Alvarez, J.A.; Rockwell Collins, Inc. Bringing critical notifications to a pilot via eye tracking. U.S. Patent 9,457,914, 2016. Billings, C.E. Aviation automation: The search for a human-centered approach. 1997. Ververs, P.M. Understanding a pilot’s tasks. University of Illinois, Aviation Research Laboratory, Savoy, IL. A-1. 1998. Burian, B.K.; Barshi, I.; Dismukes, K. The challenge of aviation emergency and abnormal situations. 2005. Endsley, M. R. Toward a Theory of Situation Awareness in Dynamic Systems. Human Factors: The Journal of the Human Factors and Ergonomics Society 1995, 37 (1), 32– 64. Endsley, M.R.; Farley, T.C.; Jones, W.M.; Midkiff, A.H.; Hansman, R.J. Situation awareness information requirements for commercial airline pilots. International Center for Air Transportation.1998. Jones, D.G.;Endsley, M.R. Sources of situation awareness errors in aviation. Aviation, space, and environmental medicine. 1996. Bundesen, C. A theory of visual attention. Psychological review 1990, 97(4), p.523547. Wickens, C.D. Processing resources in attention. I Parasuraman, R. & Davies, DR (Eds): Varieties of Attention. 1984.



19. Wilson, A.M.; March. Alarm management and its importance in ensuring safety. In Best Practices in Alarm Management (Digest No. 1998/279), IEE Colloquium on (pp. 6-1). IET. 1998. 20. Wilson, G. F.; Russell, C. A.; Caldwell, J. A. Performance and Psychophysiological Measures of Fatigue Effects on Aviation Related Tasks of Varying Difficulty. 2006. 21. Dehais, F.; Causse, M.; Pastor, J, March. Embedded eye tracker in a real aircraft: new perspectives on pilot/aircraft interaction monitoring. In Proceedings from The 3rd International Conference on Research in Air Transportation. Fairfax, USA: Federal Aviation Administration. 2008. 22. Kilingaru, K.; Tweedale, J.W.; Thatcher, S.; Jain, L.C. Monitoring pilot “situation awareness”. Journal of Intelligent & Fuzzy Systems 2013, 24(3), 457-466. 23. van de Merwe, K.; van Dijk, H.; Zon, R., Eye movements as an indicator of situation awareness in a flight simulator experiment. The International Journal of Aviation Psychology 2012, 22(1), pp.78-95. 24. Dill, E. T.; Young, S. D. Analysis of Eye-Tracking Data with Regards to the Complexity of Flight Deck Information Automation and Management - Inattentional Blindness, System State Awareness, and EFB Usage. 15th AIAA Aviation Technology, Integration, and Operations Conference 2015. 25. Sarter, N. B.; Mumaw, R. J.; Wickens, C. D. Pilots Monitoring Strategies and Performance on Automated Flight Decks: An Empirical Study Combining Behavioral and Eye-Tracking Data. Human Factors: The Journal of the Human Factors and Ergonomics Society 2007, 49 (3), 347–357. 26. Yu, C.-S.; Wang, E. M.-Y.; Li, W.-C.; Braithwaite, G. Pilots’ Visual Scan Patterns and Situation Awareness in Flight Operations. Aviation, Space, and Environmental Medicine 2014, 85 (7), 708–714. 27. Li, W.C.; Chiu, F.C.; Wu, K.J. The Evaluation of Pilots Performance and Mental Workload by Eye Movement. 2012. 28. Tichon, J. G.; Mavin, T.; Wallis, G.; Visser, T. A. W.; Riek, S. Using Pupillometry and Electromyography to Track Positive and Negative Affect During Flight Simulation. Aviation Psychology and Applied Human Factors2014, 4 (1), 23–32. 29. Li, W.C., Chiu, F.C. and Wu, K.J., 2012. The Evaluation of Pilots Performance and Mental Workload by Eye Movement. 30. Wanyan, X.; Zhuang, D.; Zhang, H. Improving pilot mental workload evaluation with combined measures. Bio-medical materials and engineering 2014, 24(6), pp.2283-2290. 31. Ellis, K.K.E. Eye tracking metrics for workload estimation in flight deck operations. 2009. 32. O'Hara, J.; Brown, W.; Lewis, P; Persensky, J. The effects of interface management tasks on crew performance and safety in complex, computer-based systems. Nureg/CR6690, 2. 2002. 33. Miao, A.; Zacharias, G.; Kao, S.-P. A computational situation assessment model for nuclear power plant operations. IEEE Transactions on Systems, Man, and Cybernetics Part A: Systems and Humans 1997, 27 (6), 728–742. 34. Bainbridge, L. The Process Controller. The analysis of practical skills 1978, 236–263. 35. Bainbridge, L. Ironies of automation. Automatica 1983, 19 (6), 775–779. 36. Wirstad, J. On knowledge structures for process operators (No. ERGONOMRAAD-R-18). Ergonomiraad AB. 1981. 24 ACS Paragon Plus Environment

Page 24 of 34

Page 25 of 34 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60


37. Hogg, D.N., FOLLES⊘, K.N.U.T., Strand-Volden, F. and Torralba, B. Development of a situation awareness measure to evaluate advanced alarm systems in nuclear power plant control rooms. Ergonomics 1995, 38(11), pp.2394-2413. 38. Jiang, X.; Zheng, B.; Tien, G.;Atkins, M.S.,. Pupil response to precision in surgical task execution. In MMVR (pp. 210-214). 2013. 39. Ha, C. H.; Kim, J. H.; Lee, S. J.; Seong, P. H. Investigation on relationship between information flow rate and mental workload of accident diagnosis tasks in NPPs. IEEE Transactions on Nuclear Science 2006, 53 (3), 1450–1459. 40. Gao, Q.; Wang, Y.; Song, F.; Li, Z.; Dong, X. Mental workload measurement for emergency operating procedures in digital nuclear power plants. Ergonomics 2013, 56 (7), 1070–1085. 41. Hwang, S.-L.; Yau, Y.-J.; Lin, Y.-T.; Chen, J.-H.; Huang, T.-H.; Yenn, T.-C.; Hsu, C.C. Predicting work performance in nuclear power plants. Safety Science 2008, 46 (7), 1115–1124. 42. Kohani, M.; Berman, J.; Catacora, D.; Kim, B.; Vaughn-Cooke, M. Evaluating Operator Performance for Patient Telemetry Monitoring Stations Using Virtual Reality. Proceedings of the Human Factors and Ergonomics Society Annual Meeting 2014, 58 (1), 2388–2392. 43. Zheng, B.; Tien, G.; Atkins, S.M.; Swindells, C.; Tanin, H.; Meneghetti, A.; Qayumi, K.A.; Panton, O.N.M. Surgeon's vigilance in the operating room. The American Journal of Surgery 2011, 201(5),673-677. 44. Weinger, M.B.; Slagle, J. x. Human factors research in anesthesia patient safety. In Proceedings of the AMIA Symposium (p. 756). American Medical Informatics Association. 2001. 45. Schaefer, H.; Helmreich, R.; Scheidegger, D. Human factors and safety in emergency medicine. Resuscitation 1994, 28 (3), 221–225. 46. Schulz, C.M.; Endsley, M.R.; Kochs, E.F.; Gelb, A.W.; Wagner, K.J. Situation Awareness in AnesthesiaConcept and Research. The Journal of the American Society of Anesthesiologists 2013, 118(3), pp.729-742. 47. Eivazi, S.; Bednarik, R.; Tukiainen, M.; Fraunberg, M. V. U. Z.; Leinonen, V.; Jääskeläinen, J. E. Gaze behaviour of expert and novice microneurosurgeons differs during observations of tumor removal recordings. Proceedings of the Symposium on Eye Tracking Research and Applications - ETRA 12 2012. 48. Law, B.; Atkins, M. S.; Kirkpatrick, A. E.; Lomax, A. J. Eye gaze patterns differentiate novice and experts in a virtual laparoscopic surgery training environment. Proceedings of the Eye tracking research & applications symposium on Eye tracking research & applications - ETRA2004 2004. 49. Tomizawa, Y.; Aoki, H.; Suzuki, S.; Matayoshi, T.; Yozu, R. Eye-tracking analysis of skilled performance in clinical extracorporeal circulation. Journal of Artificial Organs 2012, 15 (2), 146–157. 50. Tien, G.; Atkins, M. S.; Zheng, B.; Swindells, C. Measuring situation awareness of surgeons in laparoscopic training. Proceedings of the 2010 Symposium on Eye-Tracking Research & Applications - ETRA 10 2010. 51. Zheng, B.; Tien, G.; Atkins, S. M.; Swindells, C.; Tanin, H.; Meneghetti, A.; Qayumi, K. A.; Panton, O. N. M. Surgeons vigilance in the operating room. The American Journal of Surgery 2011, 201 (5), 673–677. 25 ACS Paragon Plus Environment


52. Ikuma, L. H.; Harvey, C.; Taylor, C. F.; Handal, C. A guide for assessing control room operator performance using speed and accuracy, perceived workload, situation awareness, and eye tracking. Journal of Loss Prevention in the Process Industries 2014, 32, 454–465. 53. Noah, B.; Rothrock, L. Using eye tracking for live measures of workload in a refinery control room process monitoring task. ASM Consortium 2015. 54. Sharma, C.; Bhavsar, P.; Srinivasan, B.; Srinivasan, R. Eye gaze movement studies of control room operators: A novel approach to improve process safety. Computers & Chemical Engineering 2016, 85, 43–57. 55. Bhavsar, P.; Srinivasan, B.; Srinivasan, R. Pupillometry Based Real-Time Monitoring of Operator’s Cognitive Workload To Prevent Human Error during Abnormal Situations. Industrial & Engineering Chemistry Research 2016, 55 (12), 3372–3382. 56. Bhavsar, P.; Srinivasan, B.; Srinivasan, R. Quantifying situation awareness of control room operators using eye-gaze behavior. Computers & Chemical Engineering 2017, 106, 191–201.


Page 26 of 34

Page 27 of 34 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47


Table 1: Fixation and saccade durations of participants Participant No. of No. of No. of No. consistent inconsistent (consistentevents events inconsistent) pairs

1 2 3 4

8 11 1 4

4 7 1 2

4 3 1 2

No. of Mean of Mean of Mean of Mean of times fixation fixation saccade saccade fixation duration duration duration duration duration for for for for increased consistent inconsistent consistent inconsistent events events events events 2 461.9 505.4 47.7 47.9 2 452.3 601.7 42.2 37.0 1 302.8 378.7 37.5 43.0 1 317.6 475.1 32.8 34.2 383.6 490.2 38.9 40.4 Average

27

ACS Paragon Plus Environment


Fig 1: Cognitive States and Behavior of a Control Room Operator


Page 28 of 34

Page 29 of 34 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60


Fig 2: Human Machine Interface of the Experimental Setup



Fig 3: Typical instructions at beginning of experiment


Page 30 of 34

Page 31 of 34 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60


Fig 4: Typical instruction of tasks



Fig 5: Evolution of cognitive state of operator


Page 32 of 34

Page 33 of 34 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47


Fig 6: Illustrative Example

33

ACS Paragon Plus Environment


For Table of Contents Use Only

This paper demonstrates that human cognitive errors that are often the root cause of plant accidents can be detected proactively by analyzing operators eye gaze in real-time


Page 34 of 34

Toward Preventing Accidents in Process Industries by Inferring the

Recommend Documents