G. F. Atkinson university 01 Waterloo Waterloo, Ontario Canada N2L 3G1
II Profile Reporting of Assessment in Chemistry
Teaching chemists are already familiar with a form of profile assessment: the transcript of course marks. This tells that a student is "strong" in organic mechanisms hut "weak" in thermodynamics. Instead of stopping with the average -arade for all courses taken in the .vear...oerformance is considered course-by-course for two main reasons. First, to improve both the advice the student is aiven about future courses and the methods by which he is taight. Second, to improve the usefulness of reports and references about the student which may he sent out. I t is argued that this familiar example is not the most meaningfd fnrm of profile. Present attention is focussed on carrying the hreaking-down onestage further, sostudent accomplishment in a lecture murse is represented not by a single mark, hut hy a profiling of levels oiattainment on a numher of a s ~ e c t of s the course. which will nor usuall\. coincide either with'the topic headings of the syllabus or with the separate auestions of the examination. In the spring of 1976, the Assessment Group of the Education Division of the Chemical Society (London) held a conference on profile assessment, which is attracting some attention as a fairer and more informative wav of describing student performance. This paper sets out some information on strengths and weaknesses of the technique.
~.~~
~~~~~
~
The Nature of a Proflle A profile is a means of presenting the results of a n assessment. While i t does not necessarily require change in the means of assessment or in teaching methods, considering how best to devise a profile may lead to reviewing course ohjectives and adopting new methods of assessment. For each aspect of student performance to he shown. the instructor setsup stages of achievement. A numher of firms are commonly used. A numerical scale of five with labelling words at the extremes The student quickly and correctly performs the mathematical operations required in the course. never 1 2 3 4 5 always It is hetter to give more than two verbal clues, either general to a numher of items, or particular to one In marking these scales, use the meanings: 1 = never, 2 = seldom, 3 = about as often as not, 4 = usually, 5 = always. or In writing an, experiment report, this student: is incoherent, writes unlinked fragments, usually writes connectedly, seldom lacks cohesion, always writes coherently. By reviewing test papers, submitted work, and other assessment information about a student, the instructor selects the appropriate stage on each scale. The resulting assessment may then he presented in various ways. Extreme forms might he a list of the numbers chosen for each asnect. or a runnine narrative incorporating the chosen phrges. intermediate forms of nresentation include bar eranhs. and the classic "profile" bf line segments joining the ends bf such bars. Items for Profiles These are best selected after defining the scope of the whole desired description. Though aspects of "the whole person" may influence course performance, it is desirable to report
only a few of these most closely linked to academic performance, such as originality in reporting a problem-solving course or work style (on a scale from "best alone" to "best in a large group") in reporting a laboratory course. Within such " eeneral limits. items to he profiled mav be suggested by course ohjectives. Kempa offers these general ohiectives of chemistrv courses as useful starting ~ o i n t s 1) knowledge of essential facts, principles, conventions, and
terms
2 ) application of these to problem solving 31 rr:Iical interpretnrion of iniormstion 41 skill in handl~ngchemicals and manipulating apparatus mgain
practical knowledge skill in observation and measurement competence in reporting and commenting in writing on experiments 7) knowledee of the role of chemistrv in societv 8) ability tosolve chemical problemsby devising and carrying out suitable experiments 5) 6)
I t is desirable to review alternative labellings or scalings which will often suggest ways of avoiding ambiguous interpretations by graders or by profile users. Once a list of objectives and of possible profile items is considered, a strong argument for profiling becomes apparent. Clearly, the same traditional single mark could he earned by students whose learning under these various heading* had hwn substantially ditterent, and t h mark ~ w d d conceal rather than reveal sixnificimt differences in attainment and in readiness for furlher work. This may not worry an instructor accustomed to offering a choice of any 5 of 8 questions on a n examination, and thus to obtaining supposedly comparable marks earned on up to 56 different groups of questions. Should it? The profile is intended to make the description of the student's achievement concise, clear, and complete. There must he some optimum numher of items for this purpose. Some redundancv is inevitable and desirahle to imnrove reliability. How is selection of items to be done? The formal techniques of factor analysis are time-consuming and demand experienced assistance. In practice, these will often prove to he neither possible with respect to time nor justifiable with respect to cost. I t will often suffice to select items agreed in previo~~s factnr analysis studies to he relatively independmt, and hence in need of separate presentation. Thus, items relaced ro remembering and items related to reasoning should appear separately. Key ancillary skills such as mathematics will also merit separate presrntation. Whether to single out commnnication skills is a moot point. It is easy to see that lack of these will influence the results obtained under almost anv evaluation system. However, some argue that if improved communication of chemistry is not a stated course objective, communication skills should not appear in a profile. Some profile items will he single-ended, with scales running from the non-occurrence to unfailing occurrence of the desirahle skill. Other items mav he double-ended. neither end being necessarily good compared to the other, A d the range not representing the presence or absence of a single skill. The broader the range of intended users of the profile, the greater the task of constructing it. If profiles come into widespread use, knowledge of how to interpret them will
spread, but it cannot yet be assumed. This problem is hidden with traditional grading as we assume without inquiry that everyone knows what 75% means. With a profile system, the attention of the instructor in the next course and of the prospective employer of agraduate may focus on quite different items. Thus, items presented must serve a variety of users' needs without confusion. Problems With Profile Reporting Althoueh nrofile reportine overcomes some deficiencies of traditional grade reporting, it is not without its own problems. To introduce consideration of some of these, let us look briefly a t another familiar example of profile reporting: the student assessment of university teaching. Student assessments are often reported as a profile. The teacher is rated on a five-point scale on various aspects of overall pertinninnce. 'l'he five numherx attached to the scale mav or mav not be uniformly divpersed along a number line re$esentiig the quality or achievement heing reported. These numbers are labels, not measures, ordinals, not cardinals. Even if verbal meanings are attached, users still tend to treat the numbers as measures, and apply to them operations like averaging or taking standard deviations which are suitable with cardinal numbers. The literature of teaching evaluation is full of examples of this practice. This problem also exists in profiling student performance. Moreover, these non-linear number scales may be of different lengths. That is, a high achievement on one may contribute no more to overall nerformance than an averaee achievement on another. ~urth'ermore,the main distinct"aspects of the profile such as reasoning and remembering should be thought of as lying in different dimensions of the measurement space, forestalline s i m ~ l eaddition even anart from the foreaoinz difficulties. when double-ended items, repesenting the ha< ance of two factors rather than the absolute amount of either, are considered, it is quite clear that simple averaging to reduce the nrofile to a familiar mark is not an acceptable opera-
-.
tion
Consider also the problem of combining ratings by several instructors into one profde. Are two "hests" plus two "worsts" the same as four "averapes"? In general, profiles m i s t be preserved and used as profiles, and not casually reduced to other forms of presentation. Even if overtreduction to a number is avoided. care must betaken to avoid seeing a student as "about a 4" and rating him in terms of his more obvious departures from this norm toward better or worse. Rating scales manged so in some cases the large number is good, and in other cases i t is the undesirable end, help t o counteract this tendency. There is a general question of the standard against which the profile should measure the student. Some items have obvious ohiective criteria. such as how often calculations are right. 0ihers use normkive ratings. What is the appropriate norm? This class? A few past years' classes? Some wider group? The wider the group chosen to set the norm, the sbarner the question of whether the rater has adequate kno&ledge of this group. I t may prove useful to construct a profile of the hoped-for achievement of a successful student, br even a differential profile of before-and-after standards showing typical changes sought. Some readers may wish to pursue this notion in the direction of grade contracting or of student participation in grading. Like a traditional mark. a profile becomes a lahel attached to a student. I t may well be easier for John Smith to live down 53% in nhvsical chemistrv than to live down "Is correct in necessaiy mathematical operations: seldom." Yet these may be the two ways of reporting the same occurrence. I t may be necessary to lahel profile items where improvement is hoped for with an exnirv date. Not all items df a profile are equally objective. Some, like quality of reasoning in discussions, may he quite significant
but subiect to varied opinions about the same performance. Items which purport td rate a state of affairs in the student's mind based on the rater's preceptions of the student's expressions of his thoughts &e rather indirect and must he viewed with caution; yet such items are not unknown. (Think of teaching evaluation, with students rating the instructor's attitude to the subject.) Pracilce and Proposal Professor Hinton of Aston University School of Architecture has been using profile reporting successfully. His profiles deal with academic work, subject bias of the student within the overall demee course, problem solving abilities, and work style of the sGdent. The Scottish Council for Research in Education has been studying the use of profiles for describing science courses on the Scottish School Leaving Certificate. Their current proposals suggest a three factor profile: basic skills, subject-related knowledge and reasoning, work-related nonintellectual qualities (e.g., perseverence, enterprise, reliability). Field trials of profile reporting are in progress. Comparison of Profiles If a prospective employer is to use together profiles prepared by different persons concerning different courses, he must have a basis for comparison. Becher has suggested a "profile of profiles," with each form rated on a chart with scales between the following points measurement attainment academic factors general abilities technically defined terms dosed farm (points on scale)
judgment aptitude personal factors subject-relatedabilities commonly understood terms open form (comments)
Such a form could also be useful in judging profiles against other forms of assessment reporting. Consideration of such a form makes clear the differences which must he expected between profiles for internal use and those for external distribution. Summary Profile reporting, which is gaining some acceptance, is a systematic yet flexible form of report, lying between mark transcripts and letters of reference. A clear view of course objectives greatly facilitates its use. Use of profiles will appeal to those who prefer to describe student achievement apart from rank orderine on a sinele scale of merit. Not all the difficulties of ;profilinghave obvious solutions, and the habit of reducine assessment results to summarv numbers by simple arithmetic is one of the most insidious nroblems. Nevertheless. because a diversitv of attainments is needed for mastery bf chemistry, this ~pproachmerits consideration. Not the least of the advantages of such consideration is the new light it throws on other more traditional methods of reporting assessment. Notes and Acknowledgment The proposals of R. F. Kempa and R. A. Becher were presented at the Sloueh conference. "Comolete Assessment?" referred to in the text, held on 12 April 1976. A short account is available. on receint of a stamned addressed envelooe. from of tChemistry, . Kingston ~olGebhnic, Dr. A. C. %cent, ~ ' e ~ Penrbyn Road, Kingston-upon-Thames, Surrey, KT1 2EE, England. Receint of a sabbatical travel erant from the Ontario Universities. Program for lnstructi~nalDevelopment, and the hos~italitvof the Department of Chemistry. University of soitham&on, are aclkowledged with thanks. Volume 54, Number 8, August 1977 1 505
