EVALUATING THE RESULTS
l a b o r a-t o r
"For the teacher who worries some about his standards, examinations and judgment there is at least some consolation in recalling that very likely no student ever returned to thank him for making i t easy, but a good number probably returned to thank him for proving to them that they could do a good deal more with their minds and their energies than they ever believed possible." SO, 449 (1973)
ViSlling Scientist Program. 1977
WTL SYMPOSIUM PAPER
Robert C. Brasted Department of Chemistry 207 Pleasant St. S.E. University of Minnesota Minneapolis, MN 55455
Profiles in Learning I. How well do we teach?
The Instrument The author has been associated with the Division of Chemical Eduation's Examination Committee for a decade and a half in a number of capacities. These include the Inorganic Examination Subcommittee, The General Chemistry Subcommittee, and the Executive Committee. In each of these efforts and in each subgroup, he felt strongly that the program could serve the communitv of chemical educators in wavs not being pursued at the time. The o p p o r t u n ~ t i eiordata adlrcriun were inrredihlv larae. As Chairman o i the 1975 Examination for ~ e n e r a i ~ h e m i s t and r y with the help of a large number of comvetent and dedicated teachers. as well as with the blessing o i t h e Executive Committee, &e reasonably radical changes were mnrle in the iormnt of the 1975 rxamination. In addition to changes in the examination that year (and in subsequent years), an effort was made to collect additional descriptive statistics that had not heen collected in previous questionnaires attached to each examination. Tables 1and 2 summarize the descriptive statistics as well as the ten different subsets (described later) of instructional topics that the committee agreed would be a minimum, providing both information on the areas of instruction as well as enough auestions to make the inter~retationof the data meaningful. In the graphics (bar format) that follow, we have omitted some statistics when the N values are too small to be significant. ..-.. Probably the most divergent section (from the usual format) is that ind&ntr.dby S u h m X,n set of ten Irlkmratory questiuns. T h r w not only attcm~)tedtu Find out whether s t ~ ~ d e ngrasp ts some of the basics or laboratory procedures, but alsowere assembled in a matrix format which reasonably well eliminated the "guess" contribution inherent in a one out of four answer format. In manufacturing this section, the subcom'Presented in part before the W. T. Lippincott Symposium, Joint Congress, Japan-USA, April 1979. 82 1 Journal of Chemical Education
mittee recognized the impossibility of probing the variety of laboratory efforts carried out in all institutions. Rather, a series of "mini" problems were presented that might be thought-provoking, as well as relating descriptive chemistry to laboratory operations. The twelve questions (Discipline Statistics) which were asked of each student taking the examination (Table 2) could have been enlarged easily, however, the permutations and combinations derived from these are more than considerable. The author is indebted to the University of Minnesota Council on Liberal Education, Center for Educational Development for Financial Support, and especially to the Measurement Service Center for their help in data storage, retrieval, and statistical analysis. This source of funding also in part explains why (unsid&rsbleattrntion is devutrd to the perfimnance of Univcrsit). of Minnt:sr,ta studenrs and our particular approach to teaching General Chemistry. An attempt is made to report the major results and conclusions in three reasonably separate Profiles. The first Profile is entitled "Profiles in Learning. How Well DwWe Teach?" The second is a study based upon performance and general background, "Performance in the First Year Course in Chemistry Based Upon Secondary School Preparation." The third Profile is one that relates to, "Multi-Section Instruction for Large Numbers of Students in the Introductory Course." The description of the instrument and the rationale will not be repeated in the successive Profile articles. In abstracting and retrieving the data it is obvious that the individual choices for the total number of students who took the 1975 examination over a two-year period, amounted to something in the order of 6.8 X los. It is possible that this is the most comprehensive study of its kind ever attempted. Over 3,500 students participated. Presenting these data alone represents a most formidable task. Interpretation in our time
Table 1. Sub-sets or Areas of Learnlng (No. of Questions) framework touches on the impossible. The choice in presentation was to consfruct har-graphirs, 01 visuals, that show I. Stoichiometry of Reactions (6) trtmdsund rrlativr percentepez. as well as percentilw rather 11. SVucture. Bonding, Periodicity and Geometry of Molecules (13) than to produce innumeral)le tahles ras visuals~that n ~ u l dhe Ill. States of Maner (8) evaluated only with intensive study. IV. Aeid-Base Chemistry and Solution Stoichiomehy (10) No. instrument is better than the directions urovided for its V. Equilibria: Homogeneous and Heterogeneaus (5) VI. E!ectrochemistry. Oxidation end Reduction (6) udministration. A diffirulty in interpretation immrdiarely Vll. Thermochemistry. Thermodynamicsand Kinetics of Reactions (6) arose as the data frwn our own Universitv uf Minnesota stuVlll. Descriptive Chemistry (12) dents were transferred to computer cards. Several problems Total I-IX = 70 IX. Carbon Chemistry (Organic)(4) are mentioned. One of the largest colleges in our university Overall = 80 X. Laboratory (Matrix)(10) is called "The College of Liberal Arts!' Obviously, these students are in fact registered in a four-year state university, however, many of them (with justification) checked the item on the descriptive statistic sheet as being in a "liberal arts Table 2. Descriptive Profile of Student PopulationB college." Fortunately we were able to segregate these students from those who are legitimately part of the Liberal Arts ColQuestion1. Type of Institution in which you are registered lege contributing data. a. Liberal Arts College Still another inconsistency, which fortunately could be b. State University (or college) corrected, was tht. checking by many o i the students in our c. Private University d. Two year College OWI l'niversity of the "One-Quartrr C ~ ~ u r s t : " ( Q t ~ e e t i ~ ~ n ~ ~ ( a r ) . Question2. Class (Year in College) 'I'hese students dcr~~illly were finishing t h e s w m d quarter of F Freshman a 7'1m-Quurrrr C ' o ~ ~ r yet i t . obviously the? were inderd in in S sophomore a me-uuarrer (our>c16erFiei. l ; L , From the statistics of over J Junior 1500 &dents from other colleges and universities, it is evident Sr Senior that our own University is unique in providing a two-quarter Question3. Sex srquenre, rather than a one year (2 semestrri, in whirh an M Male attempt is made to cmnplett~the material found in the "usual" F Female general chemistry tt8xl. Independent st~xliesvcr:fy this sitQuestion 4. Length of Cwrse a. One quarter 11a1iun. b. Two quaters It is hoped that the reader will understand, if not appreciate, c. One semester the reasons for making the distinrtion in so many instances d. Two semesters o i "A11 Students" velna "llniversity uf Minneaotn Studena." Question5. Number of lecture periaddwk. (equivalent of 50 min. periods) W r do not feel that our student hody \,ariessignificantly from a. One those of other state universitirs, thus conclusions rearhed irom b. TWO data on this group should be representative and useful. C. Three For the most Dart. the data are presented and are expected d. Four to be substantial. w h e n exceptibnally unique features are e. Fire observed. thev will be highlighted. However, it is fair game to Quenion6. Number of Hours Lablwk a. One indulge in raiionalizatioi and interpretation. ~~~~
Descriptive Statistics of Students Figure 1 summarizes the general orofile of students (exeluding examination performance). Data for both the "nonUniversity of Minnesota" and the University of Minnesota students are included. Only some of the more pertinent statistics will he highlighted. We were pleased with the general equitability of the student contribution from the various kinds of institutions. We do note differences in the number of students by class or the year in which the student has taken the course and thence the examination. For the University of Minnesutu s n ~ d e n t swe find almoit as many sophimores as freshmen. It is ior thir, among other reasons, that the author takes exception to the general title of the first-year course as being "freshman chemistry." As was noted earlier, we felt obliged to senarate the Minnesota students from others since as noted above, we are unique in our offering a two-quarter course. We also felt that we did not wish to "swamp out" the data in our particular offering of four lectures per week. A somewhat sur~risinestatistic is that of an a.~.~ r e c i a bnumber le of studtmts being offered rhemistry courses withwlt any discussion or rvciratitm r~eriwk.Firure !includes background file" study will report folly Descriptive ~ t a t i s t i c s later :~ on this phase of the total studs. A ~escriptiveStatistic allowing useful interpretation is that of the curriculum of study (or major) indicated by thestudent (Fig. 3). Our homogeneous group a t Minnesota is again separated from the non-Minnesota student for certain interoretations and "lumoed" toeether for others. When such has been done, the legend will so indicate. Twogroupings in this cateeorv s some iustification. A recent indeoen.. . ~. e r h a o need drnt survey within our university inrii
3
4
W~~M-,"."
Figure 20. Performance in arithmetic (A) versus descriptive (Dl subsets related to descriptive statistics (all students). For Institutiongraphs, adcorrespond to options a-dunder Duestion 1 in Table 2. For Length of Course graphs, a-d correspond to options a-dunder Question 4 in Table 2.
are nearly as many different chemistry courses as there are colleges, universities, and indeed, teachers of chemistry. The subdivisions that were provided in the Descriptive Statistics s e-e m- to a reasonable. if not a universallv acce~ted. ~ nrovide ~ ~ breakdown. Figure 9 involvesall students althoigh there is a heavv "inout . from the Minnesota class as is reflected in the large number of students reporting on the two-quarter sequence. The two alternatives, two-quarter and two semesters, account for over 90% of the total. The data on a one-quarter course are too insignificant to he here included. The onesemester contribution a t 7% was borderline hut is included. The conclusion that can be drawn from these data is inescapable, the two quarters with a four-lecture format provides a good foundation of general principles. There is little justification for a one-semester sequence course in chemistry if we hope to accomplish the ohjectives that are usually set forth in a general chemistry course and are accepted by the General Chemistry Examination Subcommittee. I doubt that we could ever convince an administrative "calendar" committee to suggest that a two-semester course be abbreviated. The data, however, suggests that such might be possible. Regardless of the length of the course, we find "Structure," "Electrochemistry," "Thermo," and the very brief "Organic" subsets standing out. Those of us who have battled administrative edicts and legislative funding committees recognize their search for proof that the laboratory is of little or no use in the
Table 3. Subsets as Predictors of Excellence or A Perlormance First Quarter %(Raw)
Subset I. 11. Ill. IV. V. VI. Vll. Vill. IX. X.
(Stoich.) (Struct.) (States) (Acid-base) (Equil.) (Eiectro.)
I-X.
Second Quarter %(Raw)
%lle
%
%iie
(Descrip.) (Carbon) (Lab.)
81.6 79.3 80.4 78.6 80.8 71.8 79.3 72.8 75.8 67.6
91 90 94 90 92 83 90 79 94 87
80.8 79.8 82.6 72.8 83.6 75.5 77.8 76.8 75.3 69.6
90 90 96 79 94 88 90 82 94 90
3 5 2 10 1 8 6 7 9 11
5 5 1 11 3 9 5 10 3 5
(2)
79.4
95
80.2
96
4
1
(Thenno.)
Table 4. Dlscrimlnation Based on A-D Spread In Subsets
I. (Stoich.) 11. (Struct.) Ill. (Slates) W . (Acid-Base) V. (Equil.) VI. (Eleclra.) Vll. (Therm.) Vill (Descrip.) IX. (Carbon) X. (Lab.) I-IX (2)
.
Predictor %ile
Dill. in%
Diff in%ile
R'
A%
A%iie
36.1 35.6 27.4 43.2 55.6 38.5 40.8 35.1 48.8 35.4 40.7
38 28 49 62 56 50 39 59 39 49 72
4 5 I 10 2 9 6 9 7 8 3
7 8 11 3 1 6 5 10 2 9 4
10 11 6 2 4 5 8 3 8 6 1
husrage rankbg basad on A wads w1ormancs only.
instruction of chemistry. This study (Fig. 10) seems to suggest that the money invested in the laboratory is well spent. In virtually every subset the performance is a function of the laboratory experience. The three-hour session seems to be not only the most commonly adopted but also the most productive. If we were to base all of our conclusions on the single experimental section, "The Laboratory Matrix" (X),we might be tempted to move to administration's "side!' We doubt the validity of such a deduction on several grounds. Although the differences in performance in the 2-, 3-, and 4-hour periods are not large, the 2-hour period still is a t a disadvantage. It is also difficult to compare on a percentile basis a ten by ten matrix format with our usual 4 choice-set of questions. The experiment had a number of objectives, not all of which were related directly to the laboratory concept. As stated earlier, we hoped to interrelate common sense deductions with Descriptive Chemistry. Perhaps we were not as effective in this effort as we hoped; however, later data will suggest that the experiment provides useful data. Figure 11 allows some meaningful deductions with regard to the number of "Discussions" and the role tbat this part of a course plays in instruction. A problem in collecting statistics is evident in differentiation between a "discussion" period and a "recitation." A consistent thread is found in the fact tbat when there are no discussions, there appears to be a comparable increase in lecture time. Later data support the fact that there is benefit from small group interaction (recitation or discussion) and a reasonable assumption that both junior and senior staff are productive in instruction. There is certainly no evidence to suggest that larger numbers (two or three) of discussion sessions are especially productive. The percentile ratings for "descriptive" subsets on first glance seem strange, hut it seems to tell us that the exposure to the lecturer has been better than multiple periods of "recitation." Only a very small sample size was available for "three recitations." Certain proponents of teaching innovative techniques and some educational psychologists suggest that the lecture approach and the lecturer have outlived their usefulness. Figure 12 suggests otherwise when a sample size (N) of 3500 students is considered. Even with this base we felt that 1% for 1 and 5
lecture courses did not warrant inclusion in our eraohics. . It would be difficult to argue against the student performing well if not a t a superior level in the 3 and 4 lecture period courses. Additional data will be provided in the late; Profile I11 on "Large Sections." A Review and General Survey with "All Students" Included In the Data Base In the comments that follow. "All Students" (Universitv of Minnesota with those from other state universities) are olaced in a sinele data bank and the comoarisons with subsets are reviewed Giefly. It is not evident (Fie. 13) that the students from the liberal arts and the two-year colleges perform a t a superior level, even thouah it might have been a reasonable assumption considering the smaller classes and more persrlle '.ke\.punchrr."'l'h~: author s,ill resist the temptation to philosophiie on the changes that have occurred in our chemistry program because of hand calculator use. Figures 10, 20, and 21 depict the remaining descriptive statistics in relation to the student's ability to attack arithmetic and descriptive questions. Conclusions are not easily drawn other than the factors that have been alluded to earlier. Strong and/or weak performances in one are also noted as strone and/or weak nerformances in the other. From the . generally poor performance in "Descriptive Chemistry" (the Subset VIII). . . we mieht have expected an inversion in these percentile graphics ithat is, D lower than A). It would seem that we have chosen in our subset questions in the arithmetic area ones that are somewhat moredifficult than the descriptive auestions. Perhaps we are also attempting to compare "appies" and "orang&" different modes of instruction and learning. We shall leave these data to the reader for interpretation. ~~~
e~
Grade Prediction From Subsets
Data were obtained from enough students from the University of Minnesota to allow a rather unique and, we think, significant interpretation of subset performance. We were able, by searching course records of hundreds of students as well as from data provided by the students themselves, to relate the course letter grade not only in the quarter in which the standardized examination was given hut also from the first quarter since a standardized examination was not adapted to this first 10-week quarter course. Separate runs were made through the data to analyze suhsets as a function of the quarter. Both percentiles (against national norms) and raw percent data are recorded in Table 3. The same deductions as were made early reappear. The best 88 /
Journal of Chemical Education
performance found for these A students on a percentile basis is in "States of Matter." We were pleased that the summation or average of all questions was the best predictor of our (A) student. Perhaps this predictor is not surprising, however, it must be born in mind that with the Minnesota student, this final examination is only 30% of the total grade. There is a 40% contribution from four objective hour examinations and a 30% contribution from laboratory work which includes unknowns, technique, quiz work, lab final, and a personal estimate. We must be examining something that is meaningful in this 1975 examination. All data point to the fact that this is a very valid instrument. There is little to choose from insofar as predicting a performance using the second quarter data in the several sections: "Carbon," "Equilibrium," "Structure," "Stoichiometry," and "Thermo." These students, as would be hoped, performed admirably in all. Though we need not he ashamed of the performance in Descriptive Chemistry and Equilibrium, these two areas fall reasonably far down the list as predictors. We have attempted still another analysis based upon the student's spread from A to D in performance. This is a differentiation of the A student performance from the D student in all subsets. We are assuming that the D grade is a passing, however, a minimum performance to satisfy the department, though it may not necessarily satisfy a particular college or curriculum of study. Previous data substantiate the fact that the number of students in the D category though small is statistically significant. In each subset the average performance of the A students was computed, and again, the average performance in each subset for the D students. Both raw percent and percentiles are found in Table 4. We were searching for those suhsets in which there seemed to be the greatest extremes indicating or inferring differences attributable to our instructional capabilities. This approach in a sense establishes a validity or correlation coefficient for our suhsets. Included also is the ranking of the A students that has been derived from Table 3. It is used simply for comparison purposes. For all of the criticism leveled a t our laboratory Subset X (experimental) section, it would appear that we have a reasonable discriminator. The suhsets in which we find the greatest disparities in raw percent match reasonably well those for percentiles. The former should provide the more precise data since no comparisons are made with other institutions. Where deviations in ranking occur, they are as expected. These are, for instance, subsets where students all do quite well (States of Matter) or those in which there is reasonably uniform poor performance (where else hut Descriptive Chemistry). The Carbon Subset is mixed simply hecause the field is so vast, the amount of time we have to devote to the subject is so small, and the number of questions is also small. By whatever measure, by whatever subset, by whatever descriptive statistic we use in our study, we must "get o?" with better and more meaningful ways of teaching (perhaps even defining) Descriptive Chemistry. In summary, after examining the graphics-obviously a tedious operation if the viewer is not to he completely converted to a sitting or walking diffraction grating-it is evident that only the surface has been scratched in the potential information derivable. The succeeding Profiles (I1 and 111) will provide new interpretations hut hased upon the same instrument. We must do something to impress our state Boards of Education, advisors in secondary schools, among the others, that entering students must have the best possible preparation in mathematics and science. We as teachers must search out some different ways of leavening our instruction to lead to a better understanding of chemical change and the behavior of matter; that is, the meaningful and interesting applications of the principles and not just the principles for their own sake.
