Chemistry in the Community-ChemCom A Five-Year Evaluation Frank X. Sutman, Program Director, EHR National Science Foundation,1800 G Street, NW, Washington, DC 20550 Matthew H. B N C ~ Department of Science Education, Temple University, Philadelphia, PA 19122 Nature of ChemCom Chemistry in the Communiw, or ChemCom, is a yearlong chemistry course developed through the efforts of the Education Division of the American Chemical Society and supported through the National Science Foundation. Development and in-service teacher education activities revolving around ChemCom result from expenditures of about 1.5 million dollars over a period of eight years. ChemCom is designed for high school students who may or mav not be colleee bound ( I ) : for that croup of students whh enter high sihool chemistry with m k i a l interest in the ouantitative aspects of the subject and who should benefit from varied experiences with-some level of higher order cognitive processes in approaching both science and the significant issues facing them personally and facing the nation. ChemCom was developed against a backdrop of general concern that most of the present high school level chemistry texts and courses are encyclopedic in nature and utilize the 'laboratory" out of context of the materials considered during formal lecture-discussion sessions, or utilize it simply as a means of verification of what is already known. These more traditional programs give little consideration to the relationshin amone science. technolow. and societal issues whose effktive r&olution'depends &part upon knowledge of chemistry and interest in chemistry by the "nonscience major" component of our citizenry. The need for orecolleee level courses with this kind of emphasis has been expressed in many recent national reports such as those of the National Assessment of Educational Progress (NAEP) (2)and the American Assoc. for the Advancement of Science (3).ChemCom, also, was designed to motivate students to learn more chemistry through addressing "burning" personal and societal issues. The example below will enable the reader to distinguish between the "thrust" of ChemCom and that of most other high school level chemistry programs. The chapter of the ChemCom text titled: "Petroleum: To Build or to Burn?" is introduced with a detailed presentation of the significant role of petroleum in our society. Students are challenged to assess items in their homes and to find those that are made from by products of petroleum. Consideration of the role of petroleum in national and global economics, through simple bar graph and two coordinate graphic representations is followed by a series of investigations emphasizing the "mixture" nature of petroleum. how the components of this mixture are separated, why we wish to separate these components, and hob chem: ists have discovered that certain ~ m ~ e r t iof e sthe components are related to their chemicai s&ctures. Study of the structures of the varied particles that make up petroleum give reason for an understanding of chemical bonding and of isomers. and how these chemical representations assist scientists /n their explanations for the cornposition and the properties of petroleum and of other substances. An intro564
Journal of Chemical Education
duction to reaction mechanisms (an elementary description of how certain molecules react with other molecules) is followed both by a qualitative and quantitative consideration of bow heat from the oxidation of petmleum is converted to more useful electrical energy The value of a comparative standard for this conversion leads to a n investigation of heat of combustion. This investigation calls upon the need to compare the quantity of heat released during the oxidation of comparable numbers of particles. This leading to a rationale for a clearer understanding of the mole concept and some of its applications. The chapter offers ample opportunities for students to read tables of data and graphs of these data. It also offers opportunities for students to prepare graphs and data tables and to interpolate and extrapolate or predict from these. The chapter concludes with the section titled "Putting It All together." This section is designed to involve students as a team of "experts," who are to use a set of assumptions in preparing for a public debate on appropriate future uses for petroleum in our society as a source of energy andlor as raw material for product development. A parallel "Putting It All Together" section appears as part of each of the other eight chapters. Clearly, this example indicates an approach to the study of chemistry and its applications significantly different from the usual. In "petroleum" the "organic" content is utilized as the vehicle for understandine traditionallv. taueht chemical principles at a reasonable conceptual level. Hands-on investigations are integrated into this and all other chapter presentations, and these investigations play a more significant instructional role than is the case in more traditional chemistry courses. The "laboratory," rather than divorced from the other components of instruction, is imbedded in the instruction in the form of investigations whose completion and understanding leads naturally to the development of and understanding of the next instructional step. Basic mathematics skills are called upon; but not presented simply algorithmically. Where algorithms do appear a rationale is given for their use.
-
-
Evaluation Process The ChemCom approach to school level chemistry materials called for some form uf independent evaluation during the developmental and trial stages. The independent evaluation process was conducted by a team of five researchers, over a period of the five years. It will be useful for the reader to understand that the independent evaluation was looked upon by the development team for ChemCom as a " mildly necessary evil". The components of this evaluation process included: Development and analysis of an opinionnaire used to validate the ChemCom obiectives: Development and analysis of students' responses to the cogniriveend ofchapter tee and the culminatingcxamination;
Determination of the nature of the student and teacher field test nonulatiom: Assessment of teachers' reactions to ChemCom as a mntribution to the advancement of instructional practice in high school chemistry.
..
The impact of the evaluation proeess as it relates to the future role of ChemCom (the so-called ChemCom approach) in high school chemistry instruction pervades the presentation herein. Amore detailed and analytical report of the evaluation is on file both at the Education Ofice of the American Chemical Society and the Education and Human Resources Directorate Office of the National Science Foundation. ChemCom Objectives The table lists the 11 ChemCom objectives. Review of these quickly indicates that certain objectives are typical of more traditional high school chemistrv courses (for example, ". . . to learn b&ic chemical facts i n d knowledge"i. Others likedtobetter understand the importance ofacquiring appropriate scientific information before making decisions about related societal issues," reflect a departure from tradition. These 11objectives appeared in slightly different wording as part of the initial proposal to the National Science Foundation seeking support to develop ChemCom. The validity of these objectives was determined utilizing two approaches; both of which are components of a statistical procedure referred to as The P r o w s Discrepancy Model (4). This model is widely accepted by social scientists. I t is a statistical procedure for analyzing and interpreting the opinions of experts and other experienced personnel. The first approach asked members of the ChemCom Steering Committee, and the initial writers to indicate the extent to which each objective should be associated with ChemCom. On a scale of 1, (should not be) to 5 (definitely should be a n objective), the mean response for each of the 11 objectives was between 4.0 and 5.0 The Prows Model indicates that a mean of 4 or above on the five-point scale, indicates objectives that are "very acceptable" or valid. The second approach directly utilized the opinionnaire in the table. Each member of the five groups constituting the producers and users of ChemCom were asked, during the pilot testing, to indicate to what extent each objective should be a standard for ChemCom? They then were asked to what extent ChemCom met each soecific obiective? The five groups surveyed for this anal& wereihe members of the Steering Committee, the teachers who comprised the writing teams for the pilot version of ChemCom, the ChemCom editorial staffresponsible for revising the piloted version for use in the field testing, the teachers who taught the pilot version of ChemCom, and the students enrolled in-the pilot classes. The Provus model allows for the use of factors to account for differences in sizes of the responding groups. Following the Prows Discrepancy Model the numerical mean responses for each group was determined; and discrepancy values among the opinions held by the various participating groups regarding the appropriateness of the standards and the level of performance were calculated. The Provus Model indicates that an average discrepancy value of 1.0 or less, on a four-point difference scale represents a very high level of agreement regarding the validity of such objectives. The range was from 0.38 - 1.08 with onlv one mean of the 11discreoancv values sliehtlv above 1.0:~heseresults indicate adition.& support f& thk validitv of the obiectives. P values of 0.01 - D 0.001 calculated ffom the discrepancies indicate further ihat the process for developing ChemCom was extremely on target.
Cognitive Tests Development and Results Based upon the state of the art and supported by the prominent research literature on assessment of student learning, a t the time that the ChemCom evaluation was initiated, a decision was made to utilize a multiple choice format for the student assessment (5-7). It will be helpful to note. however. that a t the time of this writing ChemCom staff are continuing to explore alternative modk of assessine students'learnine throueh the "ChemCom exoerience". For the evaluation process. test items were initially constructed by the teachers comprising the initial writing team for each chapter. The ratio of test items emphasizing understanding ofchemical knowledge versus those that emphasized applications of this knowledge to societal issues was set a t approximately 2:l. The initially teacherwritten items were rewritten by the evaluators to assure claritv of oresentation. content accuracv. ". a reasonable sampling of content, consistent format, minimal redundancv and cueine. -. and a readabilitv level of made 9 f 1. The validation of items was accomplished by. a panel of . five chemistrytchemistry education pn~f~ssionals to assure that each Item responded adequately to the ChemCom objectives and that each was correctly &tegorized a s an item to assess either chemical knowledge of that - or applications .. knowledge. Two student groups from an inner-city science-oriented high school were selected to review each item prior to the pilot testing. One group had completed a "traditional" course in high school chemistry The other had no formal chemistry instruction. Both groups commented that they wished they had had chemistry experiences that were applied to the resolution of environmental issues. This (student) opinion was reflected repeatedly during each step of the evaluation. Questions not utilized in the pilot testing were held for later use in the final examination. The tests for each ofthe nine chapters as well as the final examination are included in the i'kacher's Guide that accompanies the ChemCom textbook published by Kendall-Hunt Publishers (8). The text and teacher's guide materials were ready for distribution to schools in a "revised printing" form in 1988. A shortened form of the Loneeot Test for Coenitive Development was administered to the approximkely 1000 students involved in the pilot testing (9).As expected, a modest significantly positive correlation (0.4) between students' levels of cognitive development and their ability to answer the application of knowledge questions was determined. This indicated that the tests consisted ofreasonable numbers of items designed to measure students' abilities t o apply content and tasks, a n d t h a t these "application items" were valid. The analyzed data also indicated that teachers should not expect students with very low levels of cognitive development to perform well on the application questions. Teachers who decide to use the ChemCom tests will be encouraeed bv the fact that d o t i n g and field testing involved over 3f00 students and 84 teachers in seven diverse sites across the wuntrv eauallv distributed between urban schools and suburbanand rural schools. Also encouraging is that:
-
-
-
. .
-
the overall test relisblllty factor is 0.80. the index ofdifficulty . range . is from 0.3-0.5;and 'although the items included one-third requiring higher order thinking, the mean lndex of drflirulty is 0.45-extremely close to the projected index of 0.50. the readability level of the tests is grade 9 f 1;similar to the readabilitv level calculated for the text. the results of rhc testing program indicate that students complcting the entm year-long ChemCom course significantly uutperformrd students complcting more tradmunal Volume 69 Number 7 July 1992
565
Opinionnaire Used To Assess the ChernCorn Objectives
Objectives
Importance 1
No 1. Gives students the opportunity to learn basic
chemical facts and knowledge. 2. Gives students opportunity to understand how to deal with societal issues using chemical knowledge.
3. Gives students opportunity to interpret scientific
information. 4. Gives students opportunity to see how certain
personal problems can be solved utilizing chemical knowledge. 5. Contains materials that are understandable to students (i.e.,reading level, graphs, diagrams, and science terminology. 6. Helvs students to understand better the importance of ac6iring appropriate scientific information before making adecision about related societal issues. 7. Helps students recognize that each solution to a complex societal problem may produce new problems.
2
Low
3
Med
4
High
5
Very High
Should be an objective Has been met by ChemCom Should be an objective Has been met bv ChemCom Should be an objective Has been met bv ChemCom Should be an objective Has been met by ChemCom Should be an objective Has been met by ChemCom Should be an obiective Has been met by ChemCom Should be an objective Has been met bv ChemCom ~~~
~~~
~~
~
,
Should be an objective Has been met by ChemCom 9. Gives students opportunity to learn how to interpret Should be an objective scientific information. Has been met by ChemCom 10. Helps students better appreciate the scope and the Should be an objective limitation of technology. Has been met by ChemCom I t . Gives students opportunity to become familiar with Should be an objective some important issues involving interactions among Has been met by ChemCom science,technology, and society. 8. Helps students to identify alternative courses of action in dealing with societal issues.
college preparatory chemistry on test items designed to ass e s ~h l h chemistry learned and application^ of chemistry.
Teacher Population
Those who are considering adopting ChemCom can be confident in that the program was field tested utilizing a realistic sample of science teachers. Of the 84 teachers who were involved in the seven locations across the country (California, Washington, DGMaryland, Colorado, Louisiana, New York City-New Jersey, Texas, and Virginia) two thirds had 30 or more credits of academic background in chemistry, four-fifths had taught for six years or more; yet two-fifths had taught chemistry five years or less. Student Population
Three-fifths of the chemistry students in the test schools were enrolled in college preparatory programs; with twofiiths in noncollege preparatory programs. Ninety percent of these students were enrolled in grades 10-12.As indicated earlier, half of the student population attended urban schools while the other half attended surburban or rural schools. Investigations
The teachers supplied valuable editorial input to the ChemCom staff especially regarding the "workability" of the "laboratory investigations". Significant revisions of these investigations were incorporated into the final commercial publication of ChemCom.
566
Journal of Chemical Education
Teachers' Reactions
The field test teachers indicated conflicting ethnographic data regarding their perceptions of the nature of high school chemistry. They indicated that high school chemistry should meet t h e "societal objective" of ChemCom for all students; yet, that the components of more traditional chemistry emphasizing more analytical skill develo~ment(such as mathematical Dmblem solvinp, -. equation writing, etc.) is important for students to experience in order to prepare effectively for college, especially for those who will major in science in college. Conclusion and Issues
I t is clear, from this study of ChemCom, conducted during its development and field testing, that the program is a valid, highly functional instructional program when used with those sewndary-level students for whom it was designed (those enrolled in grades 10-11 and who read slightly below grade level). Since experience with ChemCom does not deter students from learning significant chemistry wntent, there is potential for ChemCom to substitute for more traditional high school level chemistry courses. Although not studied formally here there is some assurance that ChemCom motivates more able students, beyond those who are motivated through more traditional courses, to pursue the next level of advanced study in chemistry andlor other sciences.
Beyond the Study: Future of High School Chemistry
This presentation should not end without briefly addressing the impact of ChemCom on school-level instruction, a t least, in the areas of chemistry and mathematics. The apparent contradiction between the need to prepare all high school level chemistry students to be successful in beginning college level science, especially courses for majors, and the need for all students to experience and to understand the role of chemistry in resolving societal issues continues to occur against a backdrop of studies in science education, over the years. These studies continue to indicate that college chemistry faculty place relatively low value on the amount of chemistry content "presented" at the high school level, with more value placed on effective preparation in mathematics, as well as motivation of students to pursue a study in chemistry and other sciences. This formal study of ChemCom, may indicate that a "ChemCom" type appmach to high school chemistry can equally serve the noncollege bound, as well as college bound and nonmajors alike. The more critical issue for college bound science majors may well be the nature of and the extent of mathematics experience necessary to develop basic and higher order mathematics skills required to be successful in analytically oriented science subjects. A national attempt to address this issue is inherent in the instructional standards reflected in the Curriculum and Eualuation Standards for School Mathematics (10).
A "ChemCom a~proach"to instruction presents some ODportunity for students to apply math skills; yet tl;e ChemCom objectives do not give claim to meeting the mathematical needs of future science students. What ChemCom appears to accomplish is affording - opportunitv -for students-& begin to cons-hct their own world view if chemistry and its applications, and t h r o u ~ hthis become motivated to furthe; study in the sciences: Whatever unfolds, i t is clear from the independent evaluation of ChemCom, that ChemCom has the potential to play a significant role in shaping both the content of and the approach to high school level chemistry instruction in the years ahead. Literature Cited 1. ware, S. "Testimony regarding the ACS,po"sod Chemcom Cvrrivrriulum, before the US. House of Representatives Science and Technology Committp.,"June 7, 1989. 2. Nationoi Asmaarmart o f S c i e 196S1973:A ~ Copauk ofDeruiotian o f c h o w in SerOnce Aehiewmnr Science R w r t No. 64s-W, US. Gmernment Rioting Office: Waahinpton, DC,lW5. 3. Science F o r A 1 l A m e d n e : S u m ~ofPmjecl2061, Americao Assodstion forthe Advancement of Science, 1988. 4. Pmws, M . Diaemmney Evduntion far EdumrionolPmgmm I m p m w m f and As. 8085mnt;McCutchoon: Berkeley, CA, 1911. 5. Dale, E.; Chall, J. Educ Res Bull 1948.27,11.20. 6. Wan, J.;Summerlin,L.Sci. Tpoch. 1972,11,3W6. 7. Doran,R., BasieMeorummnf fndEff11tululI1 S s i e i e I m t t t t u l n , NatimalScienee Teachers haodation, 1980. pp P I E . 8. Tuekman, 6.Measudw Edumflond 0 u t m m : F u n d n m n t a l s ofl'eatiw: H a m u t , Brace and Jwano%eh:Sarasota, FL,1975, pp 3-13. 9. Chemistry in the Communrty: KendaU-HuntDubuque, 1.4.1388. 10. Cur~culumondEuolIlionSton&rdsfol.Schaol mothemotlea: National Counnlof Teachusof Mathematics: Reston. VA. 1989.
Volume 69 Number 7 July 1992
567
