Research: Science and Education edited by
Chemical Education Research
Diane M. Bunce The Catholic University of America Washington, DC 20064
Modifying and Validating the Colorado Learning Attitudes about Science Survey for Use in Chemistry Jack Barbera*† Department of Chemistry and Biochemistry, University of Colorado, Boulder, CO 80309; *
[email protected] Wendy K. Adams, Carl E. Wieman, and Katherine K. Perkins Department of Physics, University of Colorado, Boulder, CO 80309
⇐
Chemistry content and structure
Coherent framework of concepts
⇐
Source of chemistry knowledge
Describes nature; established by experiment
⇐
Problem solving in chemistry
⇐
address: Department of Chemistry and Biochemistry, Northern Arizona University, Flagstaff, AZ 86011
Experts’ Characteristic Beliefs
⇐
†Current
Fundamental Aspects of Chemistry
⇐
With the extensive development of curriculum innovations and alternative teaching methods in chemistry education there is a need to evaluate the impacts of these various changes on students’ beliefs about chemistry and learning chemistry. Distinct differences exist between novice and expert learners concerning their beliefs about science and learning science; Textbox 1 summarizes these differences in three main areas. Many different terms have been used to refer to the types of views represented in Textbox 1: beliefs, attitudes, epistemologies, and so forth. Unfortunately, none of these terms is consistently interpreted. Following Bauer (2), here we use the term “beliefs” as these generally differ from “attitudes” in that beliefs represent a person’s “personal knowledge or understandings that are antecedents of attitudes and subjective norms”. Fishbein and Ajzen (3, pp 11–16) outline the formation of attitudes, intentions, and behaviors based on one’s beliefs. Good education clearly should result in changes to student beliefs toward those of experts. Studies by House have shown that students’ expectations can be a better predictor of college science performance than previous mathematics or science experience (4, 5). These studies showed that a student’s achievement expectations and self-concept correlated more closely with achievement in chemistry than measures of prior achievements or instruction in mathematics or science. Student beliefs can affect how they learn new information; in turn, students’ experiences can shape their beliefs (6–9). Hume et al. (10, p 667) found “a significant correlation between students’ expectations at the beginning of the semester and learning outcomes.” The measurement of student expectations and beliefs has been an area of investigation in the physics community for some years (11–14), however studies in chemistry are more limited (2, 10, 15). Monitoring student beliefs provides instructors with information about how their teaching methods influence students’ views about chemistry and what it means to learn chemistry. This type of attitudinal information differs from the previous work of Bauer (2) in that his work addresses the students’ self-efficacy about chemistry and learning, not their beliefs about the discipline of chemistry. In order to investigate students’ beliefs about chemistry and the learning of chemistry we have modified the Colorado Learning Attitudes about Science Survey (CLASS), originally designed for use in physics (CLASS-Phys) (11). The survey is
Systematic, conceptbased strategies; widely applicable
Novices’ Characteristic Beliefs Isolated pieces of information Handed down by authority; no connection to the real world Pattern matching to memorized, arcane recipes
Textbox 1. Contrasting novice and expert beliefs juxtaposed relative to three fundamental aspects of chemistry. (Adapted from ref 1.)
designed to be used in a wide range of undergraduate chemistry courses (from survey courses for nonscience majors to graduatelevel courses).1 Survey statements are grouped into categories that reflect various aspects of student thinking. These categories emerge from analysis of student response data using the modified principal component analysis method developed during creation of the original CLASS survey (11). Instrument Design Many surveys probe various aspects of student attitudes about science (2, 13–15). Two designed specifically for chemistry are the Chemistry Expectations Survey (CHEMX) (15) and the Chemistry Self-Concept Inventory (CSCI) (2). Statements in the CLASS-Chem survey are written to be meaningful for a range of students and designed to address a wide variety of beliefs about:
1. Learning chemistry
2. The content of chemistry knowledge
3. The structure of chemistry knowledge
4. The connection of chemistry to the real world
The differences between CLASS-Chem, CHEMX, and CSCI are few yet important. CHEMX differs in three main areas. First, CLASS statements probe students’ beliefs, not
© Division of Chemical Education • www.JCE.DivCHED.org • Vol. 85 No. 10 October 2008 • Journal of Chemical Education
1435
Research: Science and Education
about a specific course, but about chemistry in general; many of the CHEMX statements measure students’ views about a particular course, including laboratories. Second, CLASS wording is carefully selected and tested to provide clear, concise statements with a single interpretation. Finally, the grouping of CLASS statements are determined based on statistical analysis of student responses; CHEMX survey categories are based on predetermined, author-defined groupings. The CSCI assesses students’ self-efficacy in learning chemistry, not students’ beliefs about chemistry itself. Many CLASS-Chem statements are identical to those of the CLASS-Phys survey with the word “physics” changed to “chemistry”; other statements required further rewording. In addition, 11 new chemistry-specific statements involving visualization, reactivity, and molecular structure were added; all statements are listed in the online supplement. Scoring and Administration The scoring and administration is identical to that previously described for the CLASS-Phys survey (11). Brief details will be given here. Students respond to each statement using a five-point Likert scale (strongly agree to strongly disagree). An individual student’s “Overall percent favorable” score is the percentage of responses for which the student agrees with the expert response (including only those statements where experts have consistent views—45 of 50). Similarly, the “Overall percent unfavorable” score is the percentage of responses for which the student disagrees with the expert response. A choice of neutral is neither grouped as favorable nor unfavorable. These individual scores are averaged to determine the “Overall percent favorable” and “Overall percent unfavorable” score for all participating students. Scores and averages are also determined for groupings of statements within categories. Each category contains a number of statements that portray an aspect of student thinking. The CLASS-Chem survey has been administered in over 30 courses at a variety of universities. We administer the survey online and obtain a 75–85% response rate. We reject 10–15% of these responses for one or more of the following controls:
• The same answer was chosen for most statements
• Fewer than three minutes were spent on the survey
• Statement 31 was answered incorrectly (see the online supplement for the survey text)
Statement 31 is used as a filter to discard surveys of those who are not reading the questions. “Please select agree (not strongly agree) for this question”, provides an easy method of determining whether students read the statements. When preparing “matched” data sets to evaluate shifts in beliefs from pre to post, we further reduce our response rate to ~60% of the course population. Validation Our criteria for validity of the statements and survey as a whole are:
1. Interviews that show that the wording and meaning of the statements within the survey are clear to the target population and that their responses (either expert or novice) are consistent with their explanations
2. Consistent responses by experts, providing face validity to the survey and defining the expert response
1436
3. The ability to distinguish between groups that we plausibly expect should have different beliefs about chemistry, namely, nonscience majors and chemistry majors
The statements in CLASS-Chem are validated through interviews with students in chemistry courses as well as with chemistry faculty. Interviews Over 40 students were interviewed during the development of the original CLASS-Phys survey (11). These interviews provide partial validation for the many statements that are identical between the physics and chemistry versions. For the CLASS-Chem survey, 10 additional students were interviewed. They were selected from a range of courses—from chemistry for nonmajors to junior-level organic chemistry—and included 5 males and 5 females. During these interviews, the students first completed the survey on paper and then were asked about their major, course load, educational interests, good and bad class experiences, and future goals. Then, the interviewer read each of the statements with the students asked to give their response and an explanation for their response. Most students freely provided thoughts and comments on all statements; those that did not were prompted to explain their answer choice. Interview Results Student interviews resulted in minor wording changes in two of the statements (statements 27 and 38) originally reworded from CLASS-Phys. Students expressed some confusion because of the fact that the words “equation” and “formula” are readily interchangeable in physics and have the same mathematical meaning; in chemistry, however, these words by themselves are ambiguous. If a statement did not obviously indicate which form of the words was being used, then the prefix “chemical” or “mathematical” was attached. These changes provided unambiguous interpretations on all statements for all the interviewees; in addition, this provided clear correlations between students’ written survey responses and their verbal reasoning for their choices. Three of the original 42 CLASS-Phys statements were removed based on student comments reflecting ambiguities, and because these specific statements were not grouped into any of the eight categories in the physics version. Dropping these statements also allowed us to add more chemistry-specific statements (as we did not want to exceed a total of 50 statements). After testing 20 additional “chemistry-specific” statements, we kept the 11 that we thought were most valid and informative. In doing this, it was discovered that words such as “intuitive”, “theories”, and “structure” do not have unambiguous definitions for introductory students and thus required revisions in the vocabulary of several statements. Faculty Surveys The CLASS-Chem survey has been given to over 50 chemistry faculty from several universities; these responses are used to define the “expert response” and also to provide additional feedback about the statements themselves. Consistent expert responses were given for 45 of the 50 statements (89.2% average agreement among respondents, 4.0% average disagreement, 6.8% average neutral response). The statements with consistent responses are indicated in the online supplement; these define the expert view.
Journal of Chemical Education • Vol. 85 No. 10 October 2008 • www.JCE.DivCHED.org • © Division of Chemical Education
Research: Science and Education
Categorization of Statements A significant difference between the CLASS surveys and other surveys is the method by which the statement groupings are determined. Categories in the CHEMX and its physics partner, the MPEX, are defined a priori by the developer, yet often there is little correlation between student responses to statements within a category (11). To identify self-consistent facets of student thinking, CLASS-Chem groupings are based on correlations in student responses (11). The iterative process of using reduced-basis factor analysis depicted in Adams et al. (11) was used to optimally incorporate the 11 additional statements into categories in the chemistry survey. The final CLASS-Chem categories and robustness values are listed in Table 1. These categories are quite similar to the original CLASS-Phys categories. We see that the statement groupings produce similar robustness (11) for most categories regardless of the statement context (physics versus chemistry). The new statements are dispersed through five of the original categories identified in CLASS-Phys, and one new category. Statements 1 and 9 were removed from the Conceptual Connections and Conceptual Learning categories, respectively, because of poor correlation with the other category statements. All additions and deletions (relative to the corresponding CLASS-Phys categories) strengthened each category in terms of robustness. These nine categories encompass 36 of the 45 statements that showed “expert” consistency. The additional statements, which accurately distinguish important novice–expert beliefs, are included in the “Overall” score but are not included in any category because they do not correlate well with other statements in that category. Thus they measure different uncategorized aspects of student thinking. Concurrent Validity The CLASS-Chem is designed to measure students’ beliefs about chemistry and the learning of chemistry. Therefore, an appropriate concurrent validity test is to show that the instrument can measure a difference in beliefs between populations that would plausibly be expected to have different beliefs: for example, students in an introductory chemistry course and undergraduate chemistry majors. As shown in Figure 1, students in both general and organic chemistry who are chemistry majors have more expert-like beliefs as measured by the CLASS-Chem than the nonchemistry majors surveyed (who were biology majors, nonscience majors, or other science majors) in these same courses. The values for the Overall category are p = 0.016 and for Personal Interest they are p = 0.001 for majors versus nonmajors in general chemistry I. Similarly, the p values are 0.208 (majors) and 0.010 (nonmajors) for organic chemistry I. Reliability Studies Using fall 2006 data of students from nine chemistry courses at two different universities, the average Cronbach’s α value for CLASS-Chem is 0.89. This value falls into the “good” range (16),
and it is not so high as to suggest excessive redundancy in the statements (17). Because the population that enrolls in first-term general chemistry each year is large (N > 800) and consistent (admissions standards and student population characteristics are nearly identical year-to-year), we believe a better indicator of the
Overall Personal Interest
80
Survey Response Similarity to Experts (%)
Four statements do not have consistent faculty responses. Three of these are about how students think they learn (statements 5, 10, and 39) and are included to provide additional information for instructors; these statements are not included in the expert–novice score. Statement 8 has been kept as a probe of students’ beliefs about the nature of science. The last unscored statement is number 31, described earlier.
60
40
20
0
Introductory General I General I (nonmajors) (majors)
General I Organic I Organic I (honors) (nonmajors) (majors)
Student Population Group Figure 1. Percentage comparisons of expert-like scores for “Overall” and “Personal Interest” categories for a range of courses and majors. Error bars represent the standard error for each data set. Table 1. Comparative Robustness of CLASS-Phys and CLASS-Chem Surveys, by Categories1 Survey Response Subcategories
Survey Statement Numbers2
CLASS-Phys Survey Robustness3
CLASS-Chem Survey Robustness3
Personal Interest
4, 13, 16, 28, 34, 36
8.20
7.74
Real World Connection
34, 36, 41, 43
7.32
9.68
Problem Solving: General
15, 18, 19, 21, 28, 30, 40, 47, 49, 50
6.50
7.16
Problem Solving: Confidence
18, 19, 40, 47
7.39
6.60
Problem Solving: Sophistication
6, 24, 25, 28, 40, 47, 50
8.25
8.48
Sense Making/ Effort
13, 21, 26, 27, 38, 42, 46, 48, 49
5.91
6.17
Conceptual Connections
6, 7, 15, 24, 37, 38, 50
5.57
6.01
Conceptual Learning
1, 6, 7, 12, 24, 25, 47
5.71
6.71
Atomic–Molecular Perspective of Chemistry
2, 11, 17, 29, 33, 44
NA
7.13
1See
ref 11. 2Bold type indicates statements added for the chemistry survey version. 3Robustness values range from 0 to 10, with 10 being the most robust.
© Division of Chemical Education • www.JCE.DivCHED.org • Vol. 85 No. 10 October 2008 • Journal of Chemical Education
1437
Research: Science and Education
reliability of this survey is the results of the test–retest method for this course (18, 19). As Table 2 shows, the high correlation between students’ responses in fall 2005 and fall 2006 for all statements on the CLASS-Chem confirm the reliability of the survey. Similar results have been seen for the CLASS-Phys survey (11), supporting the assumption that the student populations have negligible year-to-year variation. Applications A full discussion of the use of the now-validated CLASSChem survey is beyond the scope of this paper, and will be addressed in future publications. In addition to characterizing students’ beliefs, shifts in beliefs over the course can be correlated to various teaching methods. Here we present one example, the shift over one semester of general chemistry I. Table 3 shows disturbing shifts in students towards more novice-like beliefs,
similar to what have been observed for introductory physics courses (11). Studying interventions that will avoid this regression in beliefs and successfully shift beliefs to be more expert-like is an ongoing area of research. Work is underway to show that approaches similar to those that have proven successful in at least avoiding this regression in beliefs in physics (11) will also work in chemistry. Another example of research using CLASS is the comparison of how biology majors view physics and chemistry (20). This study revealed some surprising differences in students’ beliefs about the two disciplines. For example, students have more novice-like beliefs about chemistry compared to physics, specifically seeing chemistry as more about memorizing and less about the real world. Biology majors also agreed less often (by 29%) to statement 38, “It is possible to explain chemistry ideas without mathematical formulas”, than to the same statement about physics. Conclusions
Table 2. Correlations of Fall 2005 and Fall 2006 Survey Responses Favorable (%)
Neutral (%)
Unfavorable (%)
0.99
0.95
0.99
Note: N = 1218 (for Fall 2005, N = 521; for Fall 2006, N = 697).
Table 3. Comparison of General Chemistry I Students’ Favorable Responses at Beginning (Pre), and End (Post) of One Semester of Chemistry Survey Response Categories
Favorable Favorable Post–Pre Standard Precourse Postcourse Response Error Response (%) Response (%) Shift (%) in Shift
This paper describes the development and validation of the CLASS-Chem instrument, which can be used to study students’ beliefs about chemistry and the learning of chemistry.1 This instrument is a useful tool for characterizing students’ beliefs in a broad range of undergraduate chemistry courses, evaluating the impacts of teaching practices, and investigating correlations between beliefs and other important educational measures (e.g., content learning, interest, or choice of major). Pursuing research on better understanding students’ beliefs and how instruction affects these beliefs is likely to provide important insights into several issues. Salient among these is recruitment and retention of students majoring in chemistry and improving the perception of chemistry among students in the larger service-oriented population taking chemistry courses.
Overall
53
48
−5
1
Personal Interest
53
44
−9
2
Note
Real World Connection
58
46
−12
2
Problem Solving: General
59
53
−6
2
1. The CLASS-Chem survey can be found online and in PDF format at the CLASS Web site: http://CLASS.colorado.edu (accessed Jun 2008). Excel scoring templates are also available; they provide users with an easy format for processing the survey data.
Problem Solving: Confidence
64
56
−8
2
Problem Solving: Sophistication
44
40
−4
2
Sense Making/ Effort
66
56
−10
1
Conceptual Connections
55
51
−4
2
Conceptual Learning
42
40
−2
2
Atomic– Molecular Perspective of Chemistry
52
51
−2
2
This work has been partially supported by the National Science Foundation. We thank: Elias Quinn and Mindy Gratny for their assistance; the many chemistry faculty who worked with us; and the members of the CU PER group for useful discussions. Jack Barbera acknowledges partial support from the National Science Foundation IGERT program. Literature Cited
Note: N = 697.
1438
Acknowledgments
1. Hammer, D. M. Cognition and Instruction 1994, 12, 151–183. 2. Bauer, C. F. J. Chem. Educ. 2005, 82, 1864–1870. 3. Fishbein, M.; Ajzen, I. Beliefs, Attitude, Intention, and Behavior: An Introduction to Theory and Research; Addison-Wesley: Reading, MA, 1975. 4. House, J. D. Int. J. Inst. Med. 1994, 21, 1–11. 5. House, J. D. Int. J. Inst. Med. 1995, 22, 157–167. 6. Bransford, J. D.; Brown, A. L.; Cocking, R. R. How People Learn; National Academy Press: Washington, DC, 2002.
Journal of Chemical Education • Vol. 85 No. 10 October 2008 • www.JCE.DivCHED.org • © Division of Chemical Education
Research: Science and Education 7. Hammer, D. M. Am. J. Phys. 2000, 68, S52–S59. 8. Redish, E. F. Teaching Physics with Physics Suite; John Wiley and Sons: New York, 2003. 9. Seymour, E.; Hewitt, N. Talking about Leaving; Westview Press: Boulder, CO, 1997. 10. Hume, D. L.; Carson, K. M.; Hodgen, B.; Glaser, R. E. J. Chem. Educ. 2006, 83, 662–667. 11. Adams, W. K.; Perkins, K. K.; Podolefsky, N. S.; Dubson, M.; Finkelstein, N. D.; Wieman, C. E. Phys. Rev. ST Phys. Educ. Res. 2006, 2, 2–15. 12. Lederman, N. G.; Abd-El-Khalick, F.; Bell, R. L.; Schwartz, R. S. J. Res. Sci. Teach. 2002, 39, 497–521. 13. Redish, E. F.; Saul, J. M.; Steinberg, R. N. Am. J. Phys. 1998, 66, 212–224. 14. Elby, A. Am. J. Phys. 2001, 69, S54–S64. 15. Grove, N.; Bretz, S. L. J. Chem. Educ. 2007, 84, 1524–1529. 16. George, D.; Mallery, P. SPSS for Windows Step by Step: A Simple Guide and Reference, 11.0 Update, 4th ed.; Allyn and Bacon: Boston, 2003.
17. Streiner, D. L.; Norman, G. R. Health Measurement Scales: A Practical Guide to Their Development and Use; Oxford University Press: New York, 1989. 18. Fraenkel, J. R.; Wallen, N. E. How To Design and Evaluate Research in Education, 3rd ed.; McGraw-Hill: New York, 1996. 19. Wilson, M. Constructing Measures: An Item Response Modeling Approach; Lawrence Erlbaum Associates, Inc.: Mahwah, NJ, 2005. 20. Perkins, K. K.; Barbera, J.; Adams, W. K.; Wieman, C. E. PERC Proceedings 2006, 883, 53–56.
Supporting JCE Online Material
http://www.jce.divched.org/Journal/Issues/2008/Oct/abs1435.html Abstract and keywords Full text (PDF) Links to cited URLs and JCE articles Supplement The CLASS-Chem survey instrument
Chemistry Education Research Connections: Survey Development Diane M. Bunce, CER Feature Editor These two articles (1, 2) help move chemical education research on survey development into a new era. This new era is characterized by two things: (i) a theoretical base outside the chemical education literature, and (ii) validity and reliability measures for instruments that have a strong statistical foundation. Others in the chemical education field (3, 4) have helped contribute to this new sophistication in survey research design. No longer is it acceptable for a researcher to sit alone in an office to develop a survey to administer to students and report the results as publishable research. Surveys must be developed with a theoretical construct in mind and often this construct about learning has been developed in fields other than chemical education. Bauer and Barbera et al. (1, 2) both point to research on beliefs versus attitudes that have been established in other disciplines. Once the construct such as beliefs or attitudes is defined through the literature, arguments are presented in these articles about why this construct has some meaning within the field of chemical education. Both researchers make the distinction between attitudes or beliefs and achievement. This is a point that is often overlooked by researchers when reporting survey results. Once the theoretical construct such as attitudes or beliefs has been established, these authors take the reader through a careful discussion of decisions that were made in the design of the instrument from Bauer’s discussion of the semantic differential model to Barbera et al.’s description of how the CLASS-Phys instrument was modified in the creation of the CLASS-Chem instrument. Especially important in Barbera et al’s discussion is how the initial modifications were tested and revised. Both authors describe this validity process in terms of student interviews and initial use and analysis of the instrument with a separate student sample. Validity testing using factor analysis is an especially strong approach because it relies on data that can be analyzed for the presence of grouping or factors within the survey questions. Grouping, factors, or subscales
editorial
provides the researcher with a more useful way to explain the results of a survey rather than presenting student responses as percentages on individual questions. Once a group, factor, or subscale has been identified, the survey results can be presented as measures of identifiable variables. Validity, however, is only one part of the process. The other part is reliability. Both authors calculated and correctly interpreted the classic statistical measure of reliability (Cronbach’s α). They also both took the reliability test a step further by measuring a test–retest statistic. This second statistic provided a more informed answer to the question of whether the surveys described were reliable. Without validity and reliability results that the survey developer can use to modify the instrument, the data collected from an implementation of the survey may be either in error or uninterpretable. Research on how students learn must be based upon the data provided by instruments that have been proven to measure what they say they are measuring and are shown to provide stable results. Anything less could lead to erroneous results or interpretation of data. Both of these authors have done an admirable job of describing the arduous task of conceiving, developing (or modifying) survey instruments, proving that the instruments are valid and reliable and helping the chemical education community to understand the process. Others in the field can build on this process and use it as a model to develop or modify new instruments that will help us measure the variables involved in the complex operation of learning. Literature Cited 1. Bauer, C. F. J. Chem. Educ. 2008, 85, 1440–1445. 2. Barbera, J.; Wieman, C. E.; Perkins, K. K. J. Chem. Educ. 2008, 85, 1435–1439. 3. Bauer, C. F. J. Chem. Educ. 2005, 82, 1864–1870. 4. Grove, N.; Bretz, S. L. J. Chem. Educ. 2007, 84, 1524–1529.
© Division of Chemical Education • www.JCE.DivCHED.org • Vol. 85 No. 10 October 2008 • Journal of Chemical Education
1439
