In the Classroom
Evaluation of Student Learning in Organic Chemistry Using the SOLO Taxonomy Linda C. Hodges* McGraw Center for Teaching and Learning, Princeton University, Princeton, NJ 08544; *
[email protected] Lilia C. Harvey Department of Chemistry, Agnes Scott College, Decatur, GA 30030
Chemistry faculty concerned with helping students develop conceptual understanding may find that assessing students’ learning is not a trivial undertaking. Standard instruments are useful for quantifying student learning on the basis of specific facts or concepts learned, such as the American Chemical Society’s series of course-specific standardized exams, but these may not thoroughly probe the depth of students’ understanding of complex ideas. Recent articles in this Journal have explored different ways to assess the quality of students’ learning in organic chemistry classes (1, 2). Given the complexity of the undertaking, we may often rely on our intuitive ability to “recognize it when we see it”. Establishing clear criteria for distinguishing the level of student work and sharing these criteria with students, however, can save a great deal of time and trouble. Students are clearer about our expectations for them and better able to understand the basis for grading decisions when we clearly articulate our evaluation guidelines. Faculty may also find that developing and using evaluation criteria can help them clarify their own teaching goals (3). In this article we describe our experience in adapting a standardized instrument, the Structure of Observed Learning Outcomes (SOLO) taxonomy (4), for use in an organic chemistry two-semester course sequence at a small liberal arts college for women. Although there are examples in the literature of this instrument being used to evaluate levels of learning in college-level biology classes (5, 6), we have not seen its use discussed in college-level chemistry classes. This instrument may be used both in a formative manner to assess students’ understanding of key concepts throughout a course, and in a summative manner as part of a plan to assess the effectiveness of certain pedagogical strategies or curricular approaches in promoting student learning. This method may be adapted to use as a template or rubric for assigning grades on essay questions, especially when grading is done by more than one person, such as the case when using teaching assistants. The SOLO taxonomy describes student learning in five hierarchical levels related to a student’s ability to apply appropriate concepts in answering questions, connect concepts together coherently, and relate concepts to new ideas. • • • • •
Prestructural: No recognition of appropriate concepts or relevant processing of information Unistructural: Preliminary processing but question not approached appropriately Multistructural: Some aspects of question addressed but no relationship of facts or concepts Relational: Several concepts are integrated so coherent whole has meaning Extended Abstract: Coherent whole is generalized to a higher level of abstraction
Faculty can use the SOLO taxonomy to rank student responses to open-ended questions in terms of increasing structural complexity that reflect depth of understanding (7). We found this method to be a powerful tool for analyzing points of difficulty or confusion in student learning and following students’ progress in their understanding of particular ideas. Faculty can use this tool to identify and document students’ lack of understanding or alternate conceptions more clearly, and thus design more effective intervention strategies. This instrument is adaptable across institutional and discipline contexts, making it a valuable assessment method for anyone interested in scholarly and reflective teaching. Analyzing Points of Difficulty Each author taught a two-semester organic chemistry course sequence for science majors using collaborative learning methods. The class sizes ranged from 19 to 28 students over the course of the two semesters. In one application we were interested in probing the depth of student understanding of how structure affects the physical properties of organic molecules. We asked students on an examination early in the first term to rank the boiling points of three compounds of similar molecular mass but different polarities: methanol, methyl chloride, and ethane. The two instructors defined criteria that corresponded to each level of learning in the SOLO taxonomy and then evaluated students’ responses. Since the evaluation process is subjective, each student’s work was initially evaluated by both instructors using these criteria to validate the instructors’ individual judgments. An individual instructor may meaningfully use this taxonomy, however, as long as the instructor maintains consistent criteria for classifying student responses. There were recurring themes in the students’ answers representative of different levels of their understanding of these concepts. Recognizably correct answers were ones at the Relational or Extended Abstract level, but we were also interested in viewing what might be described as the gradient of student conceptual understanding. We therefore assigned students’ answers to the other levels based on whether they tried to apply inappropriate concepts (Prestructural or Unistructural) or whether they used appropriate concepts in wrong ways (Multistructural) in answering the question. Assigning students’ responses as Prestructural or Unistructural was fairly straightforward since these answers demonstrate that students do not know what concepts to draw from to approach the question. In this question, for example, students’ answers that did not include the concepts of molecular size, polarity, or hydrogen bonding were ranked as Prestructural. If students included one of these concepts
JChemEd.chem.wisc.edu • Vol. 80 No. 7 July 2003 • Journal of Chemical Education
785
In the Classroom
but not another, their answer was classified as Unistructural. We rated responses as Multistructural if they indicated that students knew most of the general concepts needed to answer the question, but the students did not have the depth of understanding necessary to apply the concepts correctly and coherently to answer the question, or the responses showed only partial understanding of all the concepts needed. For example, typical student responses at this level showed that students realized that polarity was a key factor affecting boiling point, but they ascribed this property of the molecules to the lone pairs of electrons rather than looking at polarity of bonds and overall geometry of the molecule. Others neglected to take into account that the alcohol could participate in hydrogen bonding. We saw no examples that illustrated extended abstract thinking on this question. Examples of typical student responses that we assigned to the different levels and the percentage of student answers ranked at that level are shown below. Prestructural (5%): “The CH3CH3 will have the lowest because it will be the least likely to latch onto other molecules because it is saturated and has no lone pairs. The CH3OH will be somewhat interested in bonding because of the single lone pair on the oxygen, but the CH3Cl will be the most interested in bonding with other molecules due to its many lone pairs and therefore, it will have a relatively high boiling point.” Unistructural (21%): “CH3CH3 is the most ‘branched’ molecule. So the intermolecular forces are weak. CH3Cl is not as branched as CH3CH3 so the boiling point must be higher due to stronger intermolecular forces. CH3OH is not as branched as much as CH3CH3 and also, the OH at the end is used when hydrogen ‘bonding’ occurs in between two or more molecules. And hydrogen bonding is strong and since the force is strong the boiling point is high.” Multistructural (32%): “CH3CH3, CH3OH, CH3Cl. The Cl group allows strong polar interactions and is highest. CH3OH is more polar than CH3CH3 so it must take more energy to boil it.” Relational (42%): “Methanol is capable of hydrogen bonding due to its OH group. This means that the bonds between molecules of methanol are stronger, therefore methanol has the highest boiling point. Because of the geometry CH3Cl are polar molecules. The polarity of a molecule has an affect on the boiling point. The more polar a molecule is the higher the boiling point.” When we classified student responses on this question we found that over half were Prestructural to Multistructural in nature. This close examination of student work alerted the instructors to a lack of understanding of an important concept in organic chemistry: polarity as it relates structure to function. The question then became, are students able to progress in their understanding of this key concept as the course proceeds? The instructors followed students’ progress throughout the year by asking and assessing additional questions on that topic using the SOLO taxonomy.
students to discuss polarity of molecules or how a molecule’s polarity affected some aspect of its structure or function. In addition to class “mini-lectures” on the topic, students were asked questions on this topic on worksheets throughout the semester. The focus of these exercises was descriptive, not quantitative. They answered these questions individually and discussed their answers in groups. We designed questions on the midterm and final exams related to the concept of polarity and assessed the responses using the SOLO taxonomy. One question on each exam dealt with effects of polarity on reactivity of the carbonyl functional group, specifically the effect of the presence or absence of electron-withdrawing groups on the susceptibility of the carbonyl compounds toward nucleophilic addition (Figure 1). We did not ask the identical question on both the semester and final exams since students had their earlier exam and could simply memorize their prior exam answer. This difference in the questions on a particular topic posed between the semester exam and the final exam made interpretation of results somewhat more complicated. Using the SOLO taxonomy, however, one is able to pick out the key ideas that demonstrate understanding. We were, of course, most interested in responses at the Relational or Extended Abstract level. Common features that we looked for to assign a student’s response as Relational were: recognition of the greater reactivity of the aldehyde compared to the ketone because of the greater polarity of the carbonyl in the absence of the alkyl substituent; recognition that the ketone was less reactive than the aldehyde because of the presence of the alkyl; recognition that this effect was offset somewhat by the electron-withdrawing effects of the halide substituent; and recognition that the effect of the halide was stronger the closer to the carbonyl that it occurred—the
Midterm: List the compounds shown below in increasing order of reactivity in a nucleophilic addition reaction to the carbonyl. That is, list the least reactive compound first and the most active compound last. Explain your reasoning. O
O
O H
H Cl
Final: Which of the following molecules contains the carbonyl group more susceptible to nucleophilic attack and why? O
O
O OH
H
O
Following Student Progress Recognizing from responses on this early examination question that many students had only a partial understanding of how a molecule’s structure relates to its physical properties, we deliberately designed assignments that asked 786
Cl
O Cl
Figure 1. Midterm and final exam questions that ask students to rank compounds according to their susceptibility to nucleophilic attack.
Journal of Chemical Education • Vol. 80 No. 7 July 2003 • JChemEd.chem.wisc.edu
In the Classroom
extreme example, of course, being the acid halide. Although steric effects on reactivity are important, students were not required to include this discussion for their response to be ranked as Relational. We have found in general that students can recognize these spatial constraints on molecular reactivity much more easily than they can understand electronic effects. Students showed overall improvement on the final exam in understanding the concept of polarity: 28% of student responses were at the Relational level on the midterm, 53% on the final (N = 28); 6 out of 8 of the students who answered at the Relational level on the midterm maintained this level on the final; 9 students improved on the final in their understanding of the concept of polarity to the level of Relational thinking as determined from our assessment of their responses. The small numbers of students involved means that these results are suggestive but not statistically definitive in terms of a positive upward trend in responses on the final. A key observation from these results is that even with additional exercises and class emphasis placed on the idea of polarity, more than a third of the students still had difficulty developing an in-depth understanding of this concept. This result may reflect both the complexity of this seemingly intuitive concept and the challenge that students have in transferring knowledge learned in a specific context to another similar, but not identical, application. Concepts in organic chemistry build continuously throughout the course and instructors rely on students’ ability to transfer information learned in one kind of situation to related contexts. By charting students’ progress in understanding specific key concepts, instructors may sometimes uncover the underlying basis for other points of difficulty. Relating Learning and Assessment with Effective Use of Questions
Develop Questions Focused on a Key Learning Goal Instructors will gain the most useful information from evaluation of the responses to questions that examine students’ understanding of key concepts on which later content builds. Assessing esoteric questions that deal with some of the idiosyncratic behavior in organic reactions using the SOLO taxonomy will likely reveal little about the range of student learning. Only those students who are extremely facile with the material are likely to understand these nuances. Develop Questions Specific to a Concept Evaluation of student learning is clearer when questions distinguish between related but different concepts since students may have some understanding of one issue and not the other. For example, a question that asks students to compare both boiling points and solubilities of a series of compounds may in all likelihood yield confusing information about students’ depth of understanding of either concept as determined by the SOLO taxonomy. Develop Questions That Are Multifaceted The SOLO taxonomy is designed to help instructors examine students’ abilities to make coherent connections and formulate relationships between ideas. For example, items that ask students to rank and explain characteristics or activities
of a certain series of compounds or reactions usually yield more information than asking for an explanation of a single feature of a single molecule or reaction. If an instructor wishes to examine students’ understanding of a certain feature, she may want to ask students to discuss the experimental basis of this idea or to place this concept in the context of an earlier principle, for example. The SOLO taxonomy provides a way to take both a single-point snapshot of student conceptual understanding and to view the panorama of progress in learning as the course proceeds. The nature of the questions used is central to the effectiveness of adapting the SOLO taxonomy for a class. Conclusions The SOLO taxonomy is most useful in identifying points of difficulty and in following students’ progress in understanding specific content issues. Students may demonstrate an Extended Abstract level of understanding in some areas and yet remain Unistructural in their thinking on other topics. Comparing students’ level of understanding on isolated, unrelated concepts throughout a course may not yield particularly helpful information about their progress, unless the instructor wishes specifically to study these kinds of differences. Reflective teachers are often aware of the concepts that students find particularly challenging. Using the SOLO taxonomy, however, can reveal student learning difficulties in a much clearer way than simply noting overall examination scores or anecdotally following student written responses. We could see patterns developing in the types of struggles that students were having with the course material. We were then better able to design particular pedagogical interventions to address these specific issues. After we shared this taxonomy with our students, they were better able to understand our goals for their learning and became more motivated toward learning for understanding and a little less focused on simply getting a desired grade. Acknowledgments We gratefully acknowledge the Pew National Fellowship Program for Carnegie Scholars and Agnes Scott College for their support of this work. Literature Cited 1. Nash, J. G.; Liotta, L. J.; Bravaco, R. J. J. Chem. Educ. 2000, 77, 333–337. 2. Maroto, B.; Camusso, C.; Cividini, M. J. Chem. Educ. 1997, 74, 1233–1234. 3. Walvoord, B.; Anderson, V. A. Effective Grading: A Tool for Learning and Assessment; Jossey-Bass: San Francisco, 1998. 4. Biggs, J. B.; Collis, K. F. Evaluating the Quality of Learning: the SOLO Taxonomy; Academic Press: New York, 1982. 5. Hazel, E.; Prosser, M.; Trigwell, K. Research and Development in Higher Education 1996, 19, 323–326. 6. Lake, D. J. Biol. Educ. 1999, 33, 191–198. 7. Hattie, J.; Purdie, N. In Teaching and Learning in Higher Education; Dart, B. Boulton-Lewis, G., Eds.; The Australian Council for Educational Research Ltd.: Melbourne, Australia, 1998; pp 146–176.
JChemEd.chem.wisc.edu • Vol. 80 No. 7 July 2003 • Journal of Chemical Education
787
