Article pubs.acs.org/jchemeduc
Applying the Multilevel Framework of Discourse Comprehension To Evaluate the Text Characteristics of General Chemistry Textbooks Daniel T. Pyburn and Samuel Pazicni* Department of Chemistry, University of New Hampshire, Durham, New Hampshire 03824, United States S Supporting Information *
ABSTRACT: Prior chemistry education research has demonstrated a relationship between student reading skill and general chemistry course performance. In addition to student characteristics, however, the qualities of the learning materials with which students interact also impact student learning. For example, lowknowledge students benefit from texts that explicitly state relationships between ideas, namely, those that possess high cohesion. Such texts limit the number of inferences a student must draw: a practice crucial for students lacking the requisite prior knowledge to fill in conceptual gaps. In this study, five best-selling general chemistry texts were analyzed using Coh-Metrix, a tool created to measure the linguistic characteristics of discourse and text. Our results constitute a survey of five measures of text difficulty: narrativity, syntactic simplicity, word concreteness, referential cohesion, and deep cohesion. Statistically significant differences were found when comparing the cohesion of the chemistry texts. Therefore, some texts may be more optimal for low-knowledge students than the others. In addition, four of the five texts had an optimal distribution of cohesion. This work also demonstrates the utility of Coh-Metrix as a tool that instructors can use to assess the appropriateness of learning materials for students based on their reading skill and prior chemistry knowledge. KEYWORDS: First-Year Undergraduate/General, Chemical Education Research, Textbooks/Reference Books FEATURE: Chemical Education Research
C
hension of text depends not only on student characteristics such as reading skill and the amount of knowledge within the domain but also on certain characteristics of the text as well. To our knowledge, no studies have used measures of text difficulty to assess and compare introductory chemistry texts. To fill this gap, we report here an analysis of general chemistry texts, focusing on those textual characteristics described by a prominent psycholinguistic framework, the Multilevel Framework of Discourse Comprehension. These characteristics include narrativity, syntactic simplicity, word concreteness, and cohesion. Together, these characteristics provide a multidimensional assessment of text difficulty.
hemistry textbooks serve an important role in student learning within introductory chemistry courses. Texts are a main component of a chemistry course; they are a resource available to all students who can use them to study and reference.1 Many studies have been completed to assess various aspects of introductory chemistry books. The use of analogies, explanations, sources of misconceptions, the effects of various representations, and the types of questions present have been explored in introductory chemistry texts.2−8 For example, Dávila and Talanquer studied the nature of practice problems within general chemistry books.8 Their results show that a majority of problems fall within the application and analysis categories of Bloom’s taxonomy, while few required students to perform at higher levels. In another study, Gkitzia and Tzougraki developed a set of criteria to evaluate chemical representations in school textbooks.1 These criteria included representation type, interpretation of surface features, relationship to the text, the existence and properties of the caption, and the degree of correlation between the components of a multiple representation. The results from studies such as these can be used to assess existing chemistry learning materials or in the design of new materials.1 Although many aspects of chemistry texts have been examined, one feature that has not received much attention is the text itself. Most students in introductory science courses face difficulty comprehending informational text.9,10 Compre© XXXX American Chemical Society and Division of Chemical Education, Inc.
■
BACKGOUND
The Multilevel Framework and Text Cohesion
The Multilevel Framework of Discourse Comprehension describes the comprehension of discourse and text as occurring on different levels.11 These levels include the surface code, textbase, situation model, genre and rhetorical structure, and pragmatic communication. The content of these levels consists of words/word composition; referents and connectives; dimensions of spatiality, temporality, and inferences; discourse category; and goals of the speaker, respectively.12 A computer
A
dx.doi.org/10.1021/ed500006u | J. Chem. Educ. XXXX, XXX, XXX−XXX
Journal of Chemical Education
Article
Table 1. A Summary of the Five Easability Components, Including How They May Each Impact a Student Reading a General Chemistry Text Easability Component Narrativity
Syntactic simplicity Word concreteness Referential cohesion
Deep cohesion
Description
Potential Impact on General Chemistry Students
The degree to which the text follows a storyline, thus tending to contain elements that are fairly familiar to a reader The degree to which shorter and more familiar sentence structures are used The degree to which discourse contains concrete ideas
Chemistry texts are generally expository (informational) in nature and thus would be associated with low narrativity. Given this constraint, there is little an author can do to substantially lessen the text difficulty associated with this component.
The extent to which the text makes connections for the reader via overlapping words and ideas The degree to which the text contains causal, intentional, and temporal connectives
Greater syntactic simplicity implies the use of sentences that are easier for the reader to process. Authors could use simpler sentence structures if they wish, which would lessen text difficulty associated with this component. Greater word concreteness implies that the content contains more concrete ideas and fewer abstract ideas. As chemistry makes use of many abstract ideas, the word concreteness of chemistry texts should be relatively low. Given the broad use of abstract ideas in chemistry, there is little an author can do to lessen the text difficulty associated with this component. Greater cohesion implies that a reader must use domain knowledge less often to fill in conceptual gaps. This is the only text characteristic that has been shown to interact with student characteristics. Students low in domain knowledge benefit from greater cohesion, where students with greater knowledge benefit from less cohesive texts, unless they possess good reading skills. Authors can have great control over the cohesion of a text.
low-knowledge students because these students lacked the knowledge to infer information that was not explicitly stated in the text.20 These studies also concluded that high-knowledge students benefited from low-cohesion texts, a phenomenon referred to as the reverse cohesion ef fect. O’Reilly and McNamara suggested that high-cohesion texts did not permit highknowledge students to use their prior knowledge to fully integrate new information, resulting in a passive processing of the text. However, when reading skill was considered, highknowledge students with good reading skill benefited from high-cohesion texts because these students were more likely to employ reading strategies and their knowledge in processing new information.20 Taken together, these results suggest that reading skill and domain knowledge have varying effects on comprehension based on text cohesion. Therefore, it is important that authors of learning materials for introductory chemistry courses (e.g., textbooks, workbooks, worksheets, etc.) consider characteristics of the text they prepare. Cohesion of texts, for example, will likely have different effects on different subpopulations of students. The work by O’Reilly and McNamara also demonstrates the limitations of traditional readability formulas. For example, longer sentences in a passage may result in a passage being deemed difficult based on the Flesch Reading Ease Score, but a longer sentence may be more cohesive, which will make the passage easier for students with low knowledge within the domain.13
program called Coh-Metrix was developed to evaluate discourse with respect to the first four levels.13−15 Coh-Metrix provides dozens of measures of text difficulty. Using thousands of texts representative of what students would encounter from kindergarten through 12th grade, McNamara, Graesser, and Cai performed principal component analyses on the text difficulty measures provided by Coh-Metrix.16 Their results included five components that closely aligned with the levels of the Multilevel Framework. These components, referred to as the text easability components, are narrativity, syntactic simplicity, word concreteness, referential cohesion, and deep cohesion. Obtaining measures of the five easability components provides a more accurate assessment of text when compared to traditional methods of determining text difficulty. Readability formulas, such as the Flesch Reading Ease Score, do not provide a complete evaluation of text as they only rely on word and sentence lengths.13 Table 1 provides brief descriptions of each of the five easability components. Narratives tell a story and tend to contain elements that are fairly familiar to readers. As a result, texts of high narrativity are easier to read than informational texts.17−19 Syntactic simplicity is the degree to which shorter and more familiar sentence structures are used. Word concreteness is the degree to which discourse contains concrete ideas, as opposed to abstract ideas. As abstract ideas tend to be more difficult to understand, texts containing many abstract ideas tend to be more difficult to read. Thus, the more abstract ideas present in a text, the lower its word concreteness. Finally, the last two components measure cohesion of the text: the degree to which concepts, ideas, and relations within a text are made explicit.13,20−24 Referential cohesion is the extent to which the text makes connections for the reader via overlapping words and ideas, while deep cohesion is the degree to which the text contains causal, intentional, and temporal connectives.16 Of the five easability components, the measures of cohesion are of particular interest because these textual features are known to interact with student characteristics.20,22−25 Research in this area has assessed the effect of a passage’s cohesion on comprehension. For example, O’Reilly and McNamara assessed comprehension of high- and low-cohesion passages from a biology text based on participants’ prior knowledge and reading skill.20 Their results indicated that students with low knowledge in the domain benefited from more cohesive texts. These authors suggested that low-cohesion texts were not suitable for
Research Questions
Recent work has demonstrated correlations between language comprehension, domain knowledge, and performance in general chemistry.26 The ability of students to connect new information to relevant domain knowledge varies, as does the amount of prior chemical knowledge possessed by students entering an introductory chemistry course. However, the dependence of these student characteristics on success is only one aspect of a greater issue. The linguistic attributes of learning materials also play a role in comprehension, and some of these attributes interact with student characteristics. The goal of this study was to assess the textual aspects of chemistry texts commonly used in a traditional two-semester general chemistry sequence. Specifically, the following research questions were explored: 1. Are any general chemistry texts more or less suitable for high- or low-knowledge students? B
dx.doi.org/10.1021/ed500006u | J. Chem. Educ. XXXX, XXX, XXX−XXX
Journal of Chemical Education
Article
Table 2. Results from the Mann−Whitney U Tests Comparing the Five Coh-Metrix Easability Components between General Chemistry Texts and Novels Easability Components Narrativity Syntactic simplicity Word concreteness Referential cohesion Deep cohesion
■
Book Type
Median
U
Z
p
Effect Size (r)
Chemistry Novel Chemistry Novel Chemistry Novel Chemistry Novel Chemistry Novel
18.41 79.24 55.57 64.43 27.09 56.95 73.24 10.48 58.13 29.64
885.00
−11.62
