Assessing Student Understanding of General ... - ACS Publications

Research: Science and Education edited by

Chemical Education Research

Diane M. Bunce The Catholic University of America Washington, D.C. 20064

Assessing Student Understanding of General Chemistry with Concept Mapping

W

Joseph S. Francisco, Mary B. Nakhleh,* Susan C. Nurrenbern, and Matthew L. Miller Department of Chemistry, Purdue University, West Lafayette, IN 47907-1393; *[email protected]

“I don’t think anything is supposed to connect in chemistry.” This remark was overheard by the second author as she prepared to begin her first semester of college teaching. How often does this thought reverberate within the mind of a chemistry student? From a constructivist perspective, students and instructors must consider this question as they become engaged in the subject of chemistry. What connections between concepts are important? What can be done to enhance the construction of these connections? Observations similar to those above and our subsequent reflections were the driving force for this study. This paper describes how concept maps were used by students as an alternative study technique and by chemistry professors as an alternative assessment tool to improve students’ conceptual understanding in an introductory chemistry course. Background

Rationale and Research Questions The goal of this study was to investigate how students’ conceptual understanding of chemistry concepts changed through the use of an alternative study and assessment technique: concept mapping. By alternative study technique, we mean that concept mapping provides students with a unique method to construct meaning between important chemistry concepts instead of, or in addition to, the use of rote learning or memorization. By assessment, we mean that concept mapping was used by professors as a means of evaluating a student’s ability to make connections between concepts. Additionally, the concept mapping was used by students and teaching assistants (TAs) as an opportunity to reflect on student connections between concepts. The underlying assumption is that students encounter difficulty as they strive to understand concepts in chemistry (1). A premise upon which we based this study is that for learning to occur, students must construct and reflect upon concepts and experiences to generate personal understanding (2). Therefore, by providing the concept map as an alternative learning tool, we hypothesized that students would reflect on the chemical concepts and that their reflections might result in improved understanding. Chemical educators often have the problem of knowing what difficulties students encounter in chemistry and how to address such difficulties. Therefore, we hypothesized that concept mapping may also offer professors and TAs a method to assess a student’s current understanding of chemical concepts by providing a source of information about student understanding of chemical concepts. Through the use of concept maps in undergraduate chemistry classrooms, enhanced learning opportunities may be provided when students utilize 248

these concept maps for studying, and teaching may be improved when professors and TAs make use of concept mapping as a method of assessing student understanding. These possible improvements in undergraduate education are examples of the findings from this study, which itself is part of a larger research project. The overall project uses action-based research teams to implement change, investigate the results of the change, then use the findings to enhance the educational opportunities provided by the institution. More details regarding this research may be found in Krockover et al. (3). In an attempt to encourage the use of concept maps by students, concept mapping was introduced in a variety of situations during a university chemistry course, including homework, laboratory, and assessment opportunities. The following research questions guided this study: 1. Do students’ understandings of concepts affect their performance on assessments? 2. Do concept maps help students connect chemistry concepts? 3. Do concept maps impact the strategies that students use for learning?

Constructivist Perspective Our theoretical perspective is based upon the constructivist learning theory. Constructivism is a cognitive model of learning that postulates that knowledge is generated by individuals, and that this knowledge is influenced by background, attitudes, abilities, and experience (1). The construction of knowledge, therefore, depends on the interaction of personal experiences with private understandings (4, 5). As educators we must be aware of student understandings and provide opportunities for students to more easily create and retrieve appropriate conceptions (6 ). Design of the Study

Action Research Model Action research was selected as an appropriate model for this study. Action research is a four-stage model of planning, action, observation, and reflection. Within each cycle, specific innovations are introduced within a course, data are collected and analyzed, and suitable revisions planned. Following each cycle of stages, the research continues by evaluating the outcome of the previous cycle, reorganizing, and proceeding through another cycle (7). Further details about this research are planned for a subsequent publication.

Journal of Chemical Education • Vol. 79 No. 2 February 2002 • JChemEd.chem.wisc.edu

Research: Science and Education

Concept Maps We chose concept maps as a tool in the action research cycle to help students construct and reflect upon their chemical knowledge and make connections between concepts. Briefly stated, concept maps are visual representations of understanding (8). Students show this understanding by drawing links connecting key concepts (placed into boxes called nodes) as shown in Figure 1. Single words or short phrases are written on the links providing the essence of the relationship between the concepts. Additionally, connections between key concepts are shown to have directionality by drawing an arrow at one end of the link. Figure 1 provides examples of concept maps drawn by introductory-level chemistry students to represent their knowledge structure of the chemical principles involved in basic atomic structure. Concept map A in Figure 1 shows a nicely developed concept map with links between all concepts. The numerous connections between the links show the student seems to have developed a well-connected knowledge base. Concept map B shows that this student’s concepts were apparently divided into two separate sets of knowledge. The concepts isotope, element, and mass number were linked with each other but had no apparent connections to the other set of concepts (neutrons, protons, and atomic number). This apparent lack of connection between many concepts provides professors and students with feedback regarding specific student’s concepts that need improvement.

atomic number

depends on # of

protons have a set of

different for all have e the same

element have same # of are the same

can have different #'s of have different # of

neutrons

have a different

isotopes

f

mass number

Concept Map A

atomic number

amount (#) of these is the

element amount of these is the

protons as many as this one

atomic weight of an radioactive atom of an

mass number

neutrons isotopes Concept Map B

Figure 1. Examples of concept maps drawn by chemistry students. Concept map A shows a connected knowledge structure; concept map B shows a lack of connections between closely related concepts.

Another interesting aspect of concept map B is that the two separate sets of knowledge can be categorized as macroscopic or microscopic. The terms isotope, element, and mass number hold definitions that characterize concepts within an individual’s real world, that is, within the macroscopic world. The terms neutrons, protons, and atomic number have definitions that require an individual to visualize concepts that remain unseen, that is, stay within the microscopic world. Concept map B indicates that the individual has not established connections between these categories of knowledge, an important skill for chemistry students. Concept maps provide the opportunity for students to incorporate several important learning strategies into their study habits. First, concept maps provide a visual outline of an individual’s knowledge structure regarding specific topics. By drawing concept maps, students can visually interact with the knowledge structure they have constructed. Another important strategy is the ability to recognize possible deficiencies in knowledge structures. By examining their concept maps, students can identify unconnected concepts. The ability to reflect on these visual maps to attain information about a student’s conceptual understanding is important for both professors and students (8). For additional information about the use of concept maps in college and university classrooms, a useful Web site is http://www.wcer.wisc.edu/NISE/cl1/flag/ (9). As briefly reviewed in this Journal (10), concept mapping has been used in chemistry classrooms specifically as a method of instructional or student review (11, 12) or in prelaboratory and postlaboratory activities (13, 14 ), and as an assessment method (15, 16 ). Concept maps have also been used in physics and life sciences to assess student understanding and to provide opportunities for active learning (17–21). In this study, we investigated two uses of concept maps, as active learning opportunities for students and as assessment tools.

Focus Group Protocol Focus group discussions (2 or 3 students per group) were used in the final cycle of the study as a technique to obtain students’ perspectives on the impact of using concept maps. Discussions of this type involved selecting specific topics generated during the research, then organizing opportunities for participants to respond to these topics and listen to the responses of others. This format gave participants an opportunity to comment in response to the opinions of other participants. In this way, participants could more clearly describe their position regarding a topic, provide additional data, and crosscheck previously collected data (22). A script was used to guide the focus group discussion (Appendix AW). These questions ensured that a common core was asked of all students, but we also were free to explore other issues as they arose. Cycles of Implementation Table 1 provides information regarding the subjects, use of concept maps, and data collected during all three cycles of the research. This section discusses the procedures used to implement concept mapping in a large university chemistry course. Cycle One. Students enrolled in the first semester of an introductory-level chemistry course for science and engineering majors were subjects of the first cycle. These participants possessed similar academic abilities owing to stringent en-

JChemEd.chem.wisc.edu • Vol. 79 No. 2 February 2002 • Journal of Chemical Education

249

Research: Science and Education Table 1. Study Design over Three Research Cycles Cycle

Participantsa

Use of Concept Maps in Course

Data Collected

One

fall semester 1997 n = 446

Homework Prelaboratory Postlaboratory

Concept maps from postlaboratory question TA questionnaire Student surveys Student grades

Two

fall semester 1998 n = 437

Homework and recitation activities Quiz and exam questions

Concept maps from five quizzes Grade comparison of concept map quiz question versus paired exam/quiz questions Student surveys

Three

spring semester 1999 Recitation activities n = 345 Optional questions on quizzes Exam questions

aAll

participants were first-year chemistry course engineering and science majors.

rollment standards for the course and common career goals, providing a homogeneous group of participants for the study. The participants received training for constructing concept maps during lectures and recitations prior to receiving any concept map assignments. The course professor, during several lectures, demonstrated the construction of concept maps and emphasized their importance as a learning tool. Additionally, TAs, typically chemistry graduate students, guided students in constructing concept maps by providing a written concept map construction guide and opportunities to practice concept map construction during recitation. Concept map assignments were first implemented during a unit on thermodynamics. Assignments involving concept maps included weekly homework as well as prelaboratory and postlaboratory exercises. The weekly homework for students, beginning during the thermodynamics unit, included traditional textbook problems as well as the construction of one concept map. The concept map consisted of five to ten terms associated with thermodynamics. Students were expected to individually construct a map and then compare it to concept maps of other students during weekly recitation meetings with their TA. These weekly assignments were continued during subsequent topics, then terminated two weeks before the end of the semester. Prelaboratory and postlaboratory concept maps were assigned only during two thermodynamics-related laboratories. Before each thermodynamics laboratory, each student was expected to construct a concept map as preparation for the laboratory. After the second thermodynamics laboratory, the postlaboratory concept map was constructed through a group collaboration. Groups consisting of eight students were assigned four different sets of thermodynamic concepts. Subgroups of two students constructed a concept map for each set. After these were completed, two subgroups combined concept maps, creating a larger concept map consisting of two sets of thermodynamics terms. Finally, the two larger concept maps were merged into one overall postlaboratory concept map containing all the thermodynamics terms initially assigned to the group of eight students. Data collected during the first cycle were from both student and TA sources. The student sources of data included the concept maps drawn for the postlaboratory exercise, overall course scores recorded at various times during the course, and surveys (Appendix BW). TAs also provided a source of data by completing a free-response questionnaire (Appendix CW). Cycle Two. Students enrolled in the same introductorylevel chemistry course one year later were participants in cycle 250

Concept maps from quizzes Focus group interviews

two. Instruction regarding the construction of concept maps was similar to that in cycle one with one important difference. Continuous support was provided to TAs during this second cycle. Feedback from student surveys and TA questionnaires from cycle one suggested that attitudes regarding the importance of concept map construction played a major role in the usefulness of the task. Several TA questionnaires included such comments as “they [students] don’t feel it helps very much with their studying” (TA questionnaire #8, 1997)1 and “they have limited time…all they [students] care is just to solve problems and pass the tests” (TA questionnaire #4, 1997). Student surveys often contained negative responses such as “concept maps…are a complete waste of time” (student survey #272, 1997) and “I don’t find concept maps very beneficial” (student survey #260, 1997). Therefore, time was designated during each staff meeting for TAs to discuss problems regarding concept mapping to improve the attitudes of TAs so as to influence the attitudes of students.

chemical symbol can represent

can represent basic unit of

atom

element react to form

bond to form

compounds basic unit of

molecule

represent represent

with lowest ratio between components is

are formed by reactions between nonmetals

compounds of all nonmetals are

can be formed between metallic & nonmetallic

chemical formulas

with metal & nonmetal

represent represent

empirical formula

molecular compounds

ionic compounds

Figure 2. An example expert concept map. Students used the expert map to make comparisons with the homework concept map they had prepared for recitation.



During cycle two, students were assigned weekly concept maps in addition to traditional textbook problems. The concept maps included a list of five to ten terms associated with the weekly topics. Students were expected to construct the concept map individually, then compare their map to maps of other students from the same recitation section during a class discussion. After the class discussion, an expert map was provided for further feedback to help students evaluate their concept maps. Figure 2 displays an example of an expert concept map provided as a model for students to illustrate the degree of crosslinking an expert might construct in a concept map. Students were told that the expert map did not show all possible correct links, but it was provided for use as a comparison map because of suggestions made on student surveys collected during cycle one. For example, one student described a feeling of frustration “because we had nothing to check our ideas against to see if we had all of the concepts correct” (student survey #138, 1997). Concept maps assigned as homework were not evaluated for points. Early in the semester, the TAs discovered that many students were not constructing their individual concept maps prior to recitation. These findings were reported to the professor during a staff meeting and resulted in the maps being assigned for construction during recitation as a group project. After the construction and group discussion, an expert map continued to be provided for additional feedback. This change in the process by which the concept maps were implemented into the classroom is an excellent example of the flexible nature of action research. Concept maps were also used as questions on student quizzes and exams. Five of nine quizzes during the semester required students to construct a concept map from terms related to these topics: significant figures, isotopes and nucleons, solutions and concentration, oxidation–reduction, and acidic or basic oxides. Students were given concepts for a particular topic and asked to provide linking statements between them. During this cycle, data were again collected from students and TAs. All quizzes containing concept maps were photocopied to generate an archive of student concept maps on quizzes. Student surveys (Appendix DW) were revised from cycle one, then used to assess the attitude of students with respect to concept maps. Additionally, some students were informally interviewed when the fourth author collected the student surveys. TAs provided data through several informal interviews and observations of staff meetings made by the fourth author during the semester. Cycle Three. Participants in the third cycle were students enrolled in the second semester of an introductory chemistry course for science and engineering majors, a continuation of the first semester course discussed in cycles one and two. Students and TAs received training for concept map construction and grading similar to that of the first two cycles. Concept maps were constructed by students for this course on recitation assignments, optional quiz questions, and exams. The recitation assignments, completed each week as group activities, required students to construct a concept map with 6 to 10 important terms related to the topics covered in lecture. No grades were assigned to these concept maps constructed in recitation and expert maps were not provided for feedback. However, discussion of concept maps generated

during recitation provided feedback for students. Concept maps were found as optional questions on three of the eight quizzes given during the semester. Topics covered by these optional maps included vapor pressure, acid–base chemistry, and electrochemistry. Students had the choice of constructing a concept map similar to the recitation maps or answering a conceptual question. Finally, a concept map question was placed on two of the four exams during the semester. During the third cycle, again, data were collected from both students and TAs. The data include concept maps constructed on quizzes, student focus group interviews, and observations of TAs presented during staff meetings. Analysis and Discussion This paper focuses on the analysis of the following data: postlaboratory concept maps collected during cycle one, concept maps collected during cycle two, and focus group interviews conducted during cycle three. Results from the analysis are used to address research questions one and two. Research question three will be addressed in a subsequent article.

Grading Process for Concept Maps Five postlaboratory concept maps from cycle one were randomly selected for analysis. The linking phrases used by the groups of students to connect concepts were coded as correct, correct but noninformative, incorrect, or duplicate. Using these codes, the concept maps were evaluated and the following scoring algorithm was developed: # correct (linking phrases) – # wrong or noninformative (linking phrases) ×5 total # of connections made

The algorithm was utilized during cycles two and three to grade concept maps on quizzes. In the algorithm, the ratio is multiplied by 5 because concept map questions on quizzes were assigned a total of five points. For students to receive any credit on concept map questions, all terms were required to have at least one connecting link in the map. To illustrate the use of this algorithm, scores received for the concept maps in Figure 1, as assigned by the fourth author, would be 5 for concept map A and 0 for concept map B. Interrater reliabilities, which compare scores assigned by two individuals, were calculated to determine the consistency of these codes for scoring concept maps. Two volunteers and the fourth author evaluated 10 randomly selected concept maps from the first quiz given in cycle two, using the grading rules generated from the analysis of the 5 postlaboratory maps from cycle one. The reliability values comparing the volunteers to the fourth author were .84 and .80. These values demonstrate that good internal consistency was associated with the scoring procedure. To check the reliabilities, the three scorers discussed differences in their evaluations of the concept maps and revised the guidelines for differentiating between codes. These guidelines were used by the same evaluators to evaluate a random sample of concept maps from quizzes 4 and 5. The reliabilities for quiz 4 were .77 and .80, and for quiz 5 were .70 and .82. The continued consistency of the interrater reliability scores suggests that these coding rules provided an acceptable


251


isotopes, mass number) with the microscopic world (neutrons, protons, atomic number). Several other examples of pertinent information from concept maps came from comments made by TAs and students. One TA commented that “students with many connections can do multiple-step problems” (field notes, p 44). Student concept maps with many links may be an indication of that student’s ability to solve multistep algorithmic problems. Another TA mentioned that students had a “difficulty with being concise on linkages” (field notes, p 44), showing that the concept maps provided that TA with information about difficulties students were having with concepts. Although some TAs were experiencing teaching for the first time and were possibly less able to recognize difficulties using the concept maps, their use provided the opportunity for TAs, through staff meeting discussions, to become aware of students’ conceptual difficulties. Students could also make use of their concept maps by noting deficiencies within the map structure and confronting this possible lack of understanding. For example, one student recognized the existence of deficiencies in the concept maps, “I have a hard time relating everything together…I couldn’t ever complete all the links” (student survey #115, 1998). Through recognition of these deficiencies, students can identify concepts that need improved understanding. The process of reflecting about connections on concept maps provided an excellent opportunity for students to construct connections with chemistry concepts.

method for scoring student concept maps, although, as with most grading, some subjectivity remains. Overall, the TAs used the same guidelines for grading the concept maps, and they were able to efficiently evaluate student concept maps. However, student and TA attitudes toward concept map evaluation were a cause for concern. Many students questioned the fairness and consistency of grading concept maps. One student, though positive about concept maps as shown in this statement, “Making concept maps is a valuable skill, and learning how to do them is a fine idea”, also commented that “under no circumstances should they be graded for credit, especially using an arbitrary set of criteria” (student survey #182, 1998). This concern was echoed by the majority of students, with the concept of informative versus noninformative being the center of that concern. Likewise, TAs were skeptical about the process of grading concept maps. One TA commented that “[my] time was impinged upon because other TAs didn’t do their job. Students were coming to [me] because their TAs would not help. … Also, TAs were not prepared to deal with concept maps … better instruction was needed for TAs” (research notebook; p 67). Therefore, students and TAs must be better informed about what designates informative connecting phrases and the importance of making informative connections. Although guidelines were developed and discussed, this information may not have been distributed to all the necessary participants with enough vigor. Additionally, positive feedback to students through the grading process is needed to motivate them to be more thoughtful as they construct concept maps. Therefore, TAs must be better trained, not only to grade concept maps more consistently, but to help students construct more thoughtful concept maps. Likewise, students must be better informed as to what designates an informative link and be trained to reflect critically upon their connections. A second outcome from the analysis is that professors, TAs, and students were able to obtain pertinent information about conceptions students hold regarding specific topics. The concept map in Figure 1 provides an example of information pertinent to professors and TAs. Figure 1 contains two concept maps drawn by different individuals for the same quiz question. The first shows numerous links, indicating that this student may have a strong basic knowledge of this topic. The second map, however, shows a student who, possibly lacking basic connections, actually created two isolated maps within the same concept map. Furthermore, upon analyzing the terms in the two isolated maps, it would seem that the student had difficulty connecting the macroscopic world (elements,

Concept Maps Versus Paired Algorithm Questions The concept maps constructed by students on five quizzes taken during cycle two were collected from all sections of the chemistry course. Five sections were randomly selected for analysis, providing a total sample varying between 64 and 111 students. The variation resulted from students in these sections not providing all requested information or being absent during quizzes. Concept map questions from each quiz were paired with algorithmic questions, from the same quiz or from the exams, that covered the same concept. The pairing process involved three steps. First, questions were selected from quizzes and exams that contained concepts similar to those found on the quiz concept map question. Then, the scores from the concept map question–algorithmic question pairs were tabulated. Finally, the number of paired question sets was reduced by combining or discarding paired sets. Steps two and three are discussed further in the following paragraphs.

Table 2. Paired Question Data, Quiz Number 3 (Solutions and Concentration) Concept Map Acceptable

Unacceptable

Algorithmic Question Totals

Algorithm question correct

A 41% (34)

B 36% (30)

C 77% (64)

Algorithm question incorrect

D 7% (6)

E 16% (13)

F 23% (19)

Concept map totals

G 48% (40)

H 52% (43)

sample size 83

Score Type

NOTE: No significant difference was found at ∆ using a chi-square test. A capital letter (A–H) was assigned to each cell as a method of identifying the cells.

252



After the initial selection and pairing of concept map and algorithmic questions, scores from each type of question were compiled by assigning a value of 1 or 0. On concept map questions, students scoring 70% or greater out of the 5 points possible received a 1. Algorithmic questions on quizzes and exams received no partial credit when graded during the semester. Therefore, students getting the algorithmic question correct were assigned a value of 1. An example of these assigned scores for a paired question set is shown in the matrix in Table 2. The box shows the questions for the paired question data set found in Table 2. Raw scores and percentages are shown in each cell of the matrix. For example, cell A represents students having both an acceptable concept map and a correct algorithmic question. Out of 83 students, 34, or 41%, were correct on both questions. Conversely, cell E shows 13, or 16%, of the students incorrectly answering the algorithm and having an unacceptable concept map. Cell C (64, 77%) represents the total number and percent of students correctly answering the algorithmic question, and cell G (40, 48%) represents the same statistics for students with an acceptable concept map. Similar tables were constructed for all paired question sets. The second step of developing the paired question data set was reducing the data by combining or eliminating paired questions. Paired question data sets were combined if the algorithmic questions in two paired question sets were similar and from two different forms of the same quiz. For example, two different forms of quiz 3, which involved isotopes and nucleons, were given to the students. The algorithmic question asked students to correctly identify, in a multiple-choice format, which group of symbols represented a set of isotopes. Both forms of the quiz asked the same question, differing only in the possible multiple-choice answers. Therefore, these two paired data sets were combined. Some paired-question data sets

were discarded owing to indirectly-related paired questions. An example of an indirectly related paired question involved an exam question on the concept of calculating the average atomic mass. Although a concept map question was asked regarding isotopes and nucleons, the map did not specifically encompass the concept of average atomic mass; therefore this paired question set was discarded. The result of data reduction was the generation of 12 paired question data sets. Table 3 shows the percent of students with acceptable concept maps compared to the percent of students correctly answering the algorithmic question for each paired question set. The majority of pairedquestion data sets show that students were more often able to correctly answer algorithmic questions than to construct acceptable concept maps. This trend may be specific to the use of concept maps as conceptual questions in the study and might not be the same if the conceptual question were different. However, the issue of differences in chemistry student performance on algorithmic questions compared to conceptual questions has been previously addressed. It has been shown that the ability to solve algorithmic problems does not imply conceptual understanding (23, 24 ). Data collected in this study show that this same trend exists; that is, students consistently perform better on algorithmic problems than on conceptual questions, in this case concept maps. Examining this trend does not provide evidence that a link exists between conceptual understanding and algorithmic problem solving. However, careful analysis of reversals of the trend, found in data sets #4 and #9, may provide evidence that conceptual knowledge may have a more direct effect on algorithmic problem solving than previously considered. The analysis of paired data set #4 (111 students) confirmed that students have difficulty with a complex algorithmic

Paired Concept Map and Algorithmic Questions Involving Solution and Concentration Concepts

Table 3. Comparison of Students’ Scores on Concept Maps and Algorithmic Questions Students Correctly Answering Algorithm (%)

Paired Question Data Set 1

Isotopes and Nucleons

44

95

# correct – # wrong or noninformative ×5 total # of connections made

2


44

80

3


44

95

0.1 M Na2SO4(aq)

solution

molarity

4

48

24

0.2 mol/L

ionic solute

Na2SO4(s)

Solutions and Concentration

5


48

61

6


48

77

7


48

52

8


48

83

0.1 mol/L

Algorithmic Question As a new technician in a quality control lab you have just been assigned to prepare 0.100 M NH3 solutions in each of five separate 1000-mL volumetric flasks. You will do this by adding a measured volume of 14.8 M NH3 solution to each and then filling the flasks with H2O up to the mark on the neck of the flask. The volume of 14.8 M NH3 solution that you will add to each flask is

Concept on Concept Map Question

Students with Acceptable Concept Map (%)

Concept Map Question Complete a concept map with the terms in the following layout. Be as specific as possible in your connecting phrases. You must include all the terms to get credit. Grading will be calculated as

9

Oxidation–Reduction

37

22

(a) 0.676 mL

(b) 6.76 mL

10


37

79

(c) 67.6 mL

(d) 1.48 mL

11


37

83

12


37

78

(e) 14.8 mL


253


question. The problem was multistep, requiring students to combine stoichiometry, dilution, and unit conversion concepts. Additionally, the stoichiometry within the problem was not a one-to-one ratio. The low percentage of students who answered the algorithmic question of this paired-question set correctly is not surprising, considering the quality of connections students made on concept maps. The concept map paired with this algorithm provided the opportunity for students to make a connection involving non-one-to-one stoichiometric ratios. The terms students used to generate this concept map were 0.1 M Na2SO4(aq), 0.2 mol/L, 0.1 mol/L, solution, ionic solute, molarity, and Na2SO4(s). Using these terms, students had the opportunity to link 0.2 mol/L to 0.1 M Na2SO4(aq) by stating that the Na+ concentration in the 0.1 M Na2SO4(aq) solution would be 0.2 mol/L. Only two students from the sample made this connection. We interpret the absence of this connection as indicating a lack of conceptual understanding of stoichiometric ratios. Additionally, considering that most students missed the stoichiometric ratio connection and only a low percentage correctly answered the algorithm, a relationship may exist between conceptual understanding and the ability to answer complex, multistep algorithmic questions. Students, as shown in previous studies (23, 24 ), can correctly answer algorithmic questions without complete conceptual understanding, but we speculate that as the complexity of the algorithmic question increases, the need for a more complete conceptual understanding becomes more important. The analysis of paired data set #9 (81 students) also reveals a possible relationship between conceptual understanding and the ability to solve algorithmic problems. Students were asked to identify the substance oxidized and the reducing agent in an oxidation–reduction reaction on form A of a chemistry quiz (or the substance reduced and the oxidizing agent, form B). Only 22% were able to correctly identify the corresponding substances in the reactions. A careful analysis of student concept maps revealed a possible reason for this result. The concept map constructed in this paired question set included the following terms: reducing agent (RA), oxidizing agent (OA), oxidation, electrons, oxidation number, and reduction. The connections made for RA and OA on each map were analyzed. Fifty-two percent of the students in the sample incorrectly made links for RA and OA. Only four students in this sample were able to correctly assign RA or OA to the appropriate chemicals in a chemical equation. Interestingly, three of these four tried to link OA to reduction (and RA to oxidation). Although the connections made were incorrect, the attempt to make connections between OA and reduction (RA and oxidation) was appropriate. The analysis of the 48% making correct links for RA and OA showed that 11 students in this sample were able to correctly assign RA and OA in the oxidation–reduction reaction. Of these, eight made correct connections between OA and reduction (and RA and oxidation). However, no student who made correct connections between OA and oxidation (and RA and reduction) was able to correctly assign OA and RA to the appropriate chemicals in the chemical reaction. This analysis indicates that certain connections between concepts are more appropriate than others—that is, are critical links. Students were more likely to be successful at assigning OA and RA in a chemical reaction if they tried to link OA 254

to reduction (and RA to oxidation), even if they could not correctly state the connection. Therefore, a more appropriate link to make is between OA and reduction (and RA and oxidation). The inverse situation also supports this analysis. Students trying to link OA to oxidation (and RA to reduction), although they may have made correct linking statements within the concept map, were not able to answer algorithmic problems about oxidation–reduction reactions. Therefore, we argue that critical links exist, and students must be guided toward making these critical links. If these appropriate connections are not made, problem solving, whether algorithmic or not, will be affected.

Focus Group Discussions Focus group discussions were conducted during the third cycle using an interview guide to promote discussion. A note taker documented participant responses and then reviewed the transcripts to ensure accuracy. Two members of the research team coded the transcripts, using participant statements to generate the categories, which included links, perplexity and ambiguity, concept map scoring, learning style, role of the TA, calculations, suggested recitation techniques, student attitudes, study methods, and test-taking skills. Our analysis suggests that the majority of the categories emphasized student concerns for external factors as affecting concept mapping during the chemistry course. External factors are defined for these purposes as conditions students believe are outside of their control. For example, categories such as links, scoring, learning style, role of the TA, calculations, recitation techniques, study methods, and test-taking skills provide examples of student statements concerned with these external factors. In one example, from the learning styles category, a student said concept mapping is “very stifling” (i-3; s-3; q-5; 1999)2 and another said “everyone has different learning styles” (i-3; s-2; q-5; 1999). The students believed that concept maps forced a particular learning style to be used within the classroom. Similarly, these students believed that constructing concept maps “makes you conform to a way of thinking” (i-3; s-2; q-12; 1999). The perception of these students that concept mapping assigned on homework sets and used for assessment on quizzes and exams resulted in the restriction of learning styles is of concern. Concept maps are meant to provide an opportunity for students to create a visual representation of personal knowledge structures for the purposes of reflection. Unfortunately, students’ perceptions that the concept maps were restrictive may have resulted in establishing a negative attitude toward the use of concept maps. For example, one student commented that this chemistry class seemed more like a “sociology 101 class rather than a chemistry [class]” (i-3; s-3; q18; 1999). During cycle 2, students in several laboratory sections were informally interviewed by the fourth author. These students were also negative toward the use of concept maps, making comments such as Students felt that it was very much imposed upon them. TAs took the expert maps too literally while grading and were not able to evaluate fairly. …unfair to give zeroes for improper concept map construction. paraphrased comments, research notebook, p 65



Through these comments, we can see that some students perceived concept mapping as restricting their learning process. One possible reason for these student perceptions may have been a misunderstanding by students about the proper use of concept maps. Some students seemed to use an algorithmic approach to concept mapping. Again during the informal interviews, some student statements point to this possible misuse of the concept map. Some students felt the concept maps did not force them to study the right material. For example, one girl said some concepts could have multiple links. If she settled on one particular one, the test might cover a different one, which she didn’t place on her concept map to study. paraphrased comments, research notebook, p 65

This student appeared to use the concept map as a way of selecting a path to answer a question, much as a student might select an equation to solve a problem. Other students made statements that implied similar use, or attempted use, of the concept map. Concept maps were not appropriate because they did not help them to solve problems. They did not measure our chemistry knowledge. paraphrased comments, research notebook, p 65

Other external factors of concern to students involved basic physical aspects of the process of concept mapping. For instance, a student worried that phrases on arrows should only go in one place (i-2; s-3; q-4; 1999) and that while preparing concept maps one didn’t know the number of connections (i-2; s-3; q-10; 1999) that would complete the concept map. Without knowing the number of connections, this student stated that “[I] don’t know if I’m done or not” (i-2; s-3; q-10 1999). Finally, one student felt the construction of concept maps could be improved if a correct concept map were provided (i-1; s-2; q-4; 1999). Unfortunately, the student was not able to understand that concept maps do not have a specific format; that is, they are meant to reflect a student’s personal knowledge structure. Again, although concept maps are designed to allow students to creatively illustrate their knowledge structures, students often felt restricted by the use of the concept maps. This attitude may have impeded some students’ opportunities to critically evaluate their personal knowledge structures. Internal factors were also described by students in the focus group interviews, but not to the extent of external factors. Internal factors would include those conditions controlled by the student. For example, while explaining why a student would choose to answer an optional concept map on a quiz, a student indicated a need to think a bit longer before doing the concept map (i-1; s-1; q-9; 1999). Thinking about the concept is a student-controlled factor. Students seem to acknowledge the need for additional time for engagement with the chemistry principles, their goal being the improvement of conceptual understanding. Those students taking the opportunity to engage in this manner possibly did improve conceptually. However, from the coded statements of the focus group discussions, the external factors (48% of statements) weighed more heavily upon the students than the internal factors (25% of statements). Therefore, although the concept

maps gave students excellent opportunities for engagement in chemistry, the perceived negative influences of external factors may have impeded some students, diminishing the possible benefits from the concept mapping tasks.

Response to the Research Questions Research Question 1. Do students’ understandings of concepts affect their performance on assessments? Conceptual understanding appears to be a factor in the performance of students on assessments when complex, multistep algorithmic questions are asked in the assessment. More cross-linked concept maps appear to enhance students’ ability to correctly solve complex problems. Additionally, appropriate conceptual connections appear to have an effect on problem solving. As shown in paired question data sets #4 and #9, students making the non-one-to-one stoichiometric link and students making appropriate connections regarding redox concepts seem likely to solve problems in these areas of chemistry. Research Question 2. Do concept maps help students connect chemistry concepts? Concept maps provide the opportunity for students to make connections between ideas through careful reflection on their constructed concept maps. These reflections provide opportunities for students to reexamine current or missing connections, thereby possibly helping students create a more highly cross-linked and appropriate knowledge structure. However, it still remains the student’s responsibility to take advantage of this opportunity. Focus group discussions indicated that students may have been impeded by their focus on the external factors associated with concept mapping, such as student perceptions that concept mapping restricted their learning process. These external factors and student perceptions possibly led to a negative attitude toward the use of concept maps. Furthermore, these perceptions may have resulted from a misunderstanding of how concept maps might be useful to students. Some students seemed to make use of algorithmic approaches while using concept mapping techniques, which may have led to frustration. Therefore, it may be very important for instructors to carefully guide students regarding how concept maps can be useful as a study technique. A few examples, however, showed that through the process of constructing concept maps, positive impacts may have resulted as students were able to make connections. One student commented that “concept maps helped with math” (i-1; s-1; q-3; 1999) while another said “[my] brain [was] stimulated as long as the pace kept up” (i-3; s-1; q-3; 1999). Through proper attention to how concept maps are used, negative attitudes may give way to more positive results. Summary of Results Concept mapping provides an excellent alternative education tool for students, professors, and TAs. For students, concept maps create opportunities for personal reflection. Through the act of visualizing their knowledge structure, students can recognize their understanding as being incorrect,


255


incomplete, or completely deficient. Professors and TAs are provided this same opportunity, and additionally, may be able to recognize difficulties that students have with specific concepts. As they work with student concept maps, in the classroom as well as while grading, professors and TAs need to realize that important information can be extracted from these visual representations of students’ knowledge. As shown, concept maps drawn for redox and solution concepts provided evidence that students need to organize their knowledge structure in appropriate ways in order to successfully construct meaningful understandings. Also, professors and TAs can analyze concept maps to observe the ability of students to relate the microscopic and macroscopic worlds. Implications for Teaching The versatility of concept maps as an instructional method for professors and as learning tools for students has been shown in this study. In terms of an instructional method, the concept map was utilized for student assessment (both formal and informal), group work questions, and as a focus for discussion during classes led by professors and TAs. As a learning tool, the concept map proved versatile and may have challenged students to think on a deeper level. Concept maps were used for both formal and informal student assessment during several cycles of the study. The formal assessments on quizzes and exams were successful in that methods for grading were developed and the instructional staff was able to efficiently assess the knowledge students indicated visually. The concept maps were best used as formal assessments when students were allowed to freely construct a visual diagram of the concept. The informal assessment occurred as students, professors, and TAs were able to observe and address incorrect, incomplete, or deficient student knowledge. This method of assessment was best utilized when students were allowed time to make comparisons with other student or expert maps and when professors and TAs took time to analyze student concept maps. However, students were resistant to the use of concept maps as graded questions. Although students initially requested that concept maps be graded, they later resisted this use because of the differences between concept maps and traditional exams. Students in science and math have been consistently trained algorithmically, rather than conceptually. Therefore, when exposed to a different type of instructional and assessment method, students were resistant. Furthermore, TAs were also resistant to the use of concept maps. TAs, also traditionally trained algorithmically, had a difficult time adapting to the method. We argue that extensive training of TAs and students is necessary to overcome the entrenched traditional ideas of how individuals learn science knowledge. Concept maps were also used as focus questions during student small-group work sessions and as classroom discussions led by the professor or TA. They provided an excellent method of focusing on key concepts discussed during the week. The combination of concept maps and small groups in recitations generated excellent discussion sessions where students could confront chemistry concepts. Likewise, the use

256

of concept maps by professors and TAs during instruction generated increased discourse during class sessions. Concepts maps were also versatile as learning tools for students. When used appropriately, they challenged students to think on a deeper level. One student commented “[I] have to know more material for concept maps. … I like the ones with more molecular level” (i-1; s-2; q-9; 1999), indicating that the concept maps were a challenge to complete in terms of necessary knowledge, but were motivating when combined with molecular level information. However, regulation of the use of concept maps is essential according to this student. “These [concept maps] would help if [they] were regulated more. As is, people don’t take it seriously and don’t really take the time to learn what is written” (student survey #156, 1997). The use of concept maps must therefore be carefully regulated to facilitate the usefulness of the task. The concept maps required students to use, not abandon, the basic skills considered essential for learning. One student identified tasks needed to be accomplished in order to answer concept map questions. “[L]ooking through notes. [This] simplified relationships in [the] chapter summary” (i2; s-1; q-15; 1999). The concept map assignments required students to use the basic skills of reviewing to construct a conceptual framework capable of answering the concept map question. Finally, as in the case of any learning tool, the attitude of participants was critical. A quote from the focus group discussion stressed this point by stating the “individual must make [the] effort to actively take advantage of [the] format” (i-3; s-3; q-5; 1999). The development of a positive interest in the use of concept maps is therefore needed to attain the benefits mentioned previously. Appropriate training of TAs and students is one very important step toward the improvement of student attitudes as they use concept maps during a course. As educators, it is necessary to respond as students encounter difficulty with chemistry. One response to the difficulty of “I don’t think anything is supposed to connect in chemistry” is to provide students with an opportunity to critically reflect on their prior knowledge and interweave it with new knowledge with which they are confronted. Concept maps provide excellent opportunities for students to reflect. Additionally, they give professors and TAs evidence of the types of difficulties students encounter and why these students believe that connections just don’t exist in chemistry. Acknowledgment We gratefully acknowledge G. H. Krockover for providing the funding opportunity through NSF Grant 9653980-DUE. W

Supplemental Material

The focus group protocol, the student questionnaire and feedback survey, the concept mapping feedback from graduate instructors, and the fall 1998 CHM 155-A concept map feedback survey are available in this issue of JCE Online.



Notes 1. TA questionnaire # or student survey # refers to a number randomly assigned to a particular TA questionnaire or student survey response. The year designates which year the questionnaire or survey was completed. 2. The notation refers to focus group interview transcripts: i-3; s-3; q-5; 1999 explains that focus group interview #3 (i-3) was cited and that student #3 in the interview made the comment.

14.

Literature Cited

15.

1. Nakhleh, M. B. J. Chem. Educ. 1992, 69, 191-196. 2. Wheatley, G. H. Sci. Educ. 1991, 75, 9–21. 3. Krockover, G.; Adams, P.; Eichinger, D.; Nakhleh, M.; Shepardson, D. J. Coll. Sci. Teach. 2001, 30, 313–317. 4. West, L. H. T.; Fensham, P. J.; Garrard, J. E. Cognitive Structure and Conceptual Change; Academic: Orlando, FL, 1985. 5. Tasker, R. Aust. Sci. Teach. J. 1981, 27 (3), 33–37. 6. Osborne, R. J.; Wittrock, M. C. Sci. Educ. 1983, 67, 489– 508. 7. Kemmis, S.; McTaggart, R. The Action Research Planner, 3rd ed.; Deakin University Press: Victoria, BC, 1988. 8. Novak, J. D.; Gowin, D. B. Learning How to Learn; Cambridge University Press: Cambridge, 1984. 9. National Institute of Science Education. Field-tested Learning

10. 11. 12. 13.

16. 17. 18. 19. 20. 21. 22. 23. 24.

Assessment Guide; http://www.wcer.wisc.edu/NISE/cl1/flag/ (accessed Sep 2001). Robinson, W. R. J. Chem. Educ. 1999, 76, 1179–1180. Francisco, J. S.; Nicoll, G.; Trautmann, M. J. Chem. Educ. 1998, 75, 210–213. Elhelou, M.-W. A. Educ. Res. 1997, 39, 311–317. Markow, P. G.; Lonning, R. A. J. Res. Sci. Teach. 1998, 35, 1015–1029. Stensvold, M.; Wilson, J. T. J. Chem. Educ. 1992, 69, 230– 234. Regis, A.; Albertazzi, P. G.; Roletto, E. J. Chem. Educ. 1994, 73, 1084–1088. Pendley, B. D.; Bretz, R. L.; Novak, J. D. J. Chem. Educ. 1994, 71, 9–15. Rice, D. C.; Ryan, J. M.; Samson, S. M. J. Res. Sci. Teach. 1998, 35, 1103–1127. Dorough, D. K.; Rye, J. A. Sci. Teach. 1997, 64 (1), 36–41. Weisenberg, R. C. J. Coll. Sci. Teach. 1997, 26, 339–344. Austin, L. B.; Shore, B. M. Phys. Educ. 1995, 30, 41–45. Adamczyk, P.; Willson, M.; Williams, D. Sch. Sci. Rev. 1994, 76 (275), 116–124. Patton, M. Q. Qualitative Evaluation and Research Methods, 2nd ed.; SAGE Publications: Newbury Park, CA, 1990. Nurrenbern, S. C.; Pickering, M. J. Chem. Educ. 1987, 64, 508–510. Sawrey, B. A. J. Chem. Educ. 1990, 67, 253–255.


257

Assessing Student Understanding of General ... - ACS Publications

Recommend Documents