Article pubs.acs.org/jchemeduc
Investigating the Effect of Complexity Factors in Stoichiometry Problems Using Logistic Regression and Eye Tracking Hui Tang,*,† John Kirk,‡ and Norbert J. Pienta† Department of Chemistry, University of Iowa, Iowa City, Iowa 52242-1219, United States ABSTRACT: This paper includes two experiments, one investigating complexity factors in stoichiometry word problems, and the other identifying students’ problem-solving protocols by using eye-tracking technology. The word problems used in this study had five different complexity factors, which were randomly assigned by a Web-based tool that we developed. The logistic regression analysis in the first experiment of this study showed that the ability of a student to achieve a correct answer was dependent on three complexity factors: number format, unit, and chemical equation. This was followed by an eye-tracking experiment, which reaffirmed that eye fixation durations were different between students at different levels of success when they solved chemistry word problems. The online tool in this study provides a general instrument to design chemistry word problems with various complexities, as well as a method to quantitatively assess problem difficulty and students’ cognitive load. This can be applied in constructing curricula and examinations with desired difficulties in chemistry courses. KEYWORDS: High School/Introductory Chemistry, First-Year Undergraduate/General, Chemical Education Research, Problem Solving/Decision Making, Stoichiometry FEATURE: Chemical Education Research
■
upon reading comprehension.8,15,16 Understanding the complexity factors and students’ cognitive activities when they solve stoichiometry word problems is the subject of this study. Our research group developed an online tool to simultaneously examine complexity variables in gas law word problems.17 With the use of the online tool, we previously reported that three complexity factors significantly affected students’ ability to correctly solve gas law problems.17 These factors are number format, temperature, and volume. Although the questions investigated might be categorized as being of lower novelty and complexity according to how Bodner18 and others19,20 have classified “problems”, the role of cognitive load was identified in the study.17 Based on the parameter-estimate results from logistic regression, we proposed a quantitative value in cognitive load for each complexity variable and, thus, were able to obtain a cognitive load increment for each problem with different combinations of the complexity variables.17 The cognitive load increments were based on a Bayesian interpretation of the results. That is, a cognitive load item of each complexity variable was subjectively assigned according to the odd ratio eβ of that variable obtained from the logistic regression analysis. The magnitude of the cognitive load item (0 = no additional load, 0.25 = small effect, 0.50 = medium effect, 1 = large effect) increases as the corresponding eβ value decreases. A cognitive load increment is defined as the sum of the cognitive load item values in a problem.17 Thus, a cognitive load increment did not represent the absolute problem complexity; rather, it reflected the relative difficulty of the complexity variables in the gas law problems. Tang and Pienta21 later utilized eye-tracking technology to explore the problem complexities in depth. The eye-tracking results demonstrated that students at different levels of success
INTRODUCTION Problem solving skills are important in academic success in science education.1,2 Students’ problem solving abilities in chemistry are affected by multiple cognitive variables, such as cognitive developmental level3−5 and working memory capacity.6,7 For cognitive developmental level, studies have shown that proportional and probabilistic reasoning abilities are essential for students to understand abstract concepts and, hence, to solve problems in general chemistry.3,8 According to Piagetian cognitive development theory, students possess high levels of aptitude in these two reasoning abilities when they reach the formal operational level.9 However, many first-year college students have not reached this stage and thus lack proficiency in these reasoning skills.3 Another cognitive variable that affects students’ problem solving abilities is their working memory capacity.6,7 According to cognitive load theory (CLT), the working memory can only store and operate on a maximum of seven variables for a short period of time.10,11 There is a close correlation between the cognitive load a problem imposes and its complexity. Researchers have reported that increasing problem complexity to a certain level can lead to overload of the students’ working memory, which results in a dramatic decrease in performance in chemistry problem-solving.6,7,12 Recently, Knaus et al.13 developed an instrument to assign cognitive complexity ratings to items on chemistry exams. The instrument provided a method to quantify the cognitive complexities in chemistry assessment tasks. Studies have shown that solving chemistry problems is a difficult task for students4,9,14 and solving word problems is increasingly difficult because of the extra task of extrapolating the necessary information.8 For example, to solve stoichiometry word problems, students not only have to balance chemical equations and understand and convert between moles and grams but also need to set up mathematical procedures based © 2014 American Chemical Society and Division of Chemical Education, Inc.
Published: May 13, 2014 969
dx.doi.org/10.1021/ed4004113 | J. Chem. Educ. 2014, 91, 969−975
Journal of Chemical Education
Article
used: mol to mol, mol to g, g to mol, and g to g. The last complexity factor, substance, had three different options which asked for the mass of one of the three substances in the chemical equation (i.e., aluminum oxide, aluminum hydroxide, or water). Table 1 lists the possible variables for each
in solving the gas law problems showed different eye movement patterns, which disclosed students’ cognitive efforts. The authors have also discovered the trend that illustrates the correlation between eye-tracking variables and quantified cognitive increments.21 Eye-tracking technology has been used to study problemsolving in math and science education for decades.22−29 The application of eye-tracking in chemical education has started recently, mainly focusing on students’ problem-solving and cognition of chemistry representations.21,30−33 Researchers have found that problem complexity is positively correlated to participants’ eye fixation durations, which reflect their cognitive activities.23−25 Studies also showed that eye movement patterns are different between novices and experts when they solved science or chemistry problems.28,31 However, eyetracking data may yield misleading interpretations. For example, long fixations on a specific region can be caused by various factors: (1) participants find that the information in that region is important or relevant to the problem; (2) the material in the area is interesting; (3) the material is difficult; or (4) participants just stared at the location or object without any related mental activities.22 As a result, eye-tracking experiments are often carried out with complementary methods such as verbal protocols.34−36 In a recent study of how students solved chemistry problem on multiple representational formats in an interview, Stieff30 found that eye-tracking and verbal protocol results were highly correlated and both reflected students’ cognitive processes. We report herein another topical series of problems using the online tool in stoichiometry with the chemical equation 2Al(OH)3 → Al2O3 + 3H2O. The purpose of the present research is to investigate problem complexities in a topic other than gas laws as well as to study students’ cognitive activities using eye-tracking while they solve the stoichiometry problems. Stoichiometry is a difficult topic in chemistry for students because they struggle to (i) understand the concept of mole; (ii) use the units of mass; (iii) balance chemical equations; (iv) interpret word problems; and (v) perform arithmetic operations.8,14−16,37 In order to examine these obstacles, five complexity factors were included when the stoichiometry problems were designed for this study: they were defined as alumina identity, number format, chemical equation, unit, and substance. In a manner similar to the gas law study, these five factors were randomly assigned by the Adobe/Macromedia Flash software each time when students accessed the problem tool. For each of the five complexity factors, different variations were supplied. For alumina identity, there were three different variables. A student’s problem could begin with “Aluminum oxide is the main component of the gemstone known as sapphire and is the working ingredient in sandpaper.”, “Aluminum oxide occurs naturally as the mineral corundum.”, or no descriptive fact is given. For number format, it was one of the three different possibilities: a general number format (e.g., 1.53, a number larger than 1), decimal format (e.g., 0.0153, a number smaller than 1), or scientific notation (e.g., 1.53E2). For chemical equation, students were randomly given either a balanced chemical equation, an unbalanced chemical equation, or an equation given in words (“Synthetic aluminum oxide is formed by heating aluminum hydroxide, also forming water as a byproduct.”). For the unit complexity factor, four different combinations of the units of the reactant (aluminum hydroxide) and the products (aluminum oxide or water) were
Table 1. Variables in Each Complexity Factor Alumina Identity Mineral corundum Gemstone Blank or not specified
Number Format
Chemical Equation
Unit
General
Unbalanced
mol to g
Scientific notation Decimal
Balanced
g to g
None
g to mol
Substance Aluminum hydroxide Water Aluminum oxide
mol to mol
complexity factor. Two example questions illustrate the randomized assignment of the variables (Box 1 and Box 2). The text in bold in the two boxes is to identify the complexity factors; the bold type did not appear in the problems to the students. Box 1. Example Question with a Chemical Equation; Text in Bold Identifies Complexity Factors Aluminum oxide is the main component of the gemstone known as sapphire and is the working ingredient in sandpaper. An unbalanced equation for the formation of aluminum oxide is given below. Determine how many mol of aluminum oxide can be formed from 5.44E3 g of aluminum hydroxide. Al(OH)3 → Al 2O3 + H 2O
Box 2. Example Question without a Chemical Equation; Text in Bold Identifies Complexity Factors Aluminum oxide occurs naturally as the mineral corundum. Synthetic aluminum oxide is formed by heating aluminum hydroxide, also forming water as a byproduct. Determine how many g of aluminum hydroxide can form 3.71 mol of aluminum oxide. Through the use of the online tool, the effects of these complexity factors on students’ ability to solve the stoichiometry problem were examined. The relative difficulties of the components (i.e., variables) within each complexity factor could be identified and thus students’ cognitive load increment for each complexity factor in the word problem could be computed based on the experimental results. Eye-tracking technology was also used to explore the problem complexities and students’ cognitive efforts in depth. Therefore, two experiments are included in this paper. In the first experiment, a large number of participants solved the stoichiometry problems online; in the second experiment, a small number of students from a different population participated in the eye-tracking study and were interviewed afterward.
■
EXPERIMENT 1
Experiment 1 Methodology
Experiment 1 Participants. The participants were students who volunteered for the study in an introductory 970
dx.doi.org/10.1021/ed4004113 | J. Chem. Educ. 2014, 91, 969−975
Journal of Chemical Education
Article
Table 2. Parameter Estimates Model of Logistic Regression Preparative Chemistry, N = 646a Variables
β
p Values
β
p Values
Mineral corundum Gemstone Blank Scientific notation Decimal General No equation Unbalanced equation Balanced equation mol to g g to g g to mol mol to mol Aluminum hydroxide Water Aluminum oxide
1.096 −0.215 0.024 0.000 −0.486 −0.035 0.000 −0.030 −0.064 0.000 −1.340 −1.083 −1.130 0.000 −0.130 −0.082 0.000
0.000 0.285 0.908
1.397 −0.081 −0.038 0.000 −0.255 −0.163 0.000 −0.395 −0.366 0.000 −0.857 −0.659 −0.490 0.000 −0.114 −0.108 0.000
0.000 0.505 0.759
Complexity Factors Intercept Alumina identity
Number format
Equation
Unit
Substance
a
General Chemistry, N = 1752a
0.019b 0.859 0.884 0.746 0.000b 0.000b 0.000b 0.140 0.163
0.038b 0.170 0.001b 0.002b 0.000b 0.000b 0.001b 0.350 0.375
Cognitive Load Items 0.25 0.25 0.00 0.50 0.25 0.00 0.50 0.50 0.00 1.00 0.50 0.50 0.00 0.25 0.25 0.00
Numbers are attempts rather than students. bEvaluated at p < 0.05.
within an individual complexity factor. One variable in each complexity factor group was set as a reference and its β value was set to zero so that all other values within the complexity factor were easily shown relative to that β value.17 More negative β values correspond to more difficult components of the word problem. The intercept βo is defined as the total difficulty when all of the reference complexity variables are present in the word problem. βo values are 1.096 for the preparative chemistry group and 1.397 for the general chemistry group. This indicates that students in the general chemistry group performed better than preparative chemistry students did. In fact, 46.9% of the attempts in preparative chemistry and 59.3% in general chemistry answered the question correctly. The latter solved the problem correctly significantly more often than did the former (χ2 = 29.17, p < 0.001). In both preparative and general chemistry groups, the alumina identity and substance complexity factors did not significantly affect whether students solved the problems correctly or not. On the other hand, the number format and the unit complexity factors had statistically significant relationships with students’ success. For the number format variables, the analysis determined that scientific notation was significantly harder than the general and decimal formats. For the unit complexity factor, all other variables (mol to g, g to g, and g to mol) were significantly more difficult than the reference, mol to mol. Lastly, when the chemical equation complexity factor was considered, the problem giving the balanced chemical equation was significantly easier than the other two situations for the general chemistry group, but this complexity factor is not statistically significant for the preparatory chemistry group. We also assigned a semiquantitative cognitive-load value for each of the complex variables based on a Bayesian interpretation of the outcomes as we described in the previous study17 and the Introduction. They are listed in the last column of Table 2 (0 = no additional load, 0.25 = small effect, 0.50 = medium effect, 1 = large effect). As a result, for each of the 324 possible combinations of the stoichiometry problems, an overall cognitive load increment for each question can be calculated by adding the values of the individual cognitive load item in the
chemistry course at one of two large, Midwestern research universities. There were a total of 2398 attempts in fall 2007 and spring 2009 (each student could try 1−5 attempts). Of these attempts, 646 were from students enrolled in preparative chemistry courses and another 1752 attempts were from general chemistry courses. Because students at one university were assigned a same user ID to log onto the problems if they were in the same class, the total number of participants in this study is unknown. Experiment 1 Instruments. The software program used in this research is the same as the one previously described in the gas law problem study,17 except that different algorithms generated a series of problems for the stoichiometry topics in the present case. The Web-based tool was created with Flash software and could be accessed using the URL and user IDs given to the students. Complexity variables were randomly assigned by the software each time when a student logged onto the tool. The software also recorded the answers that students submitted as well as the steps they performed on the built-in calculator. During the experiment, students were assigned the problems in a manner similar to homework (i.e., with no supervision or conditions) and were allowed to use external sources such as textbooks. Experiment 1 Data Analysis
According to the number of possibilities for each complexity factor described in Table 1, there are a total of 324 (3 × 3 × 3 × 4 × 3) different combinations of the stoichiometry question. The difficulty of each of these different variables (e.g., “g to g” as the initial and final mass units) was determined by logistic regression. Results from the preparative chemistry and general chemistry courses were analyzed separately. The criterion for the level of significance was
