Why Are Chemists and Other Scientists Afraid of the Peer Review of

Research: Science and Education

Why Are Chemists and Other Scientists Afraid of the Peer Review of Teaching? Charles H. Atwood* Department of Chemistry, University of Georgia, Athens, GA 30602; *[email protected] James W. Taylor Department of Chemistry, University of Wisconsin–Madison, Madison, WI 53706-1396 Pat A. Hutchings The Carnegie Foundation for the Advancement of Teaching, 555 Middlefield Rd., Menlo Park, CA 94025

Background In 1994, the American Association for Higher Education (AAHE) began the project “From Idea to Prototype: The Peer Review of Teaching”. The project organizers noted the importance of chemistry in the teaching of fundamental science by asking chemistry departments from 12 institutions to participate. Representatives from Indiana University Purdue University Indianapolis (IUPUI), Northwestern, Georgia, Michigan, Wisconsin–Madison, and California–Santa Cruz accepted the invitation and attended the initial meeting. The broad-based goal noted in project materials was “to create new roles for faculty in improving and ensuring the quality of teaching and learning.” To accomplish this, the project wanted “to develop and enact strategies for the peer collaboration and review of teaching that are intellectually rigorous, appropriate to the discipline, and practically useful in improving the quality of teaching and learning.” In the ensuing four years, the project developed several techniques that participants report have changed their teaching and raised their awareness of student learning styles. They have also been used effectively in decisions about promotion, tenure, and merit. However, the project confirmed that it is best to experiment with these approaches in low-stakes contexts before employing them in summative ways. The uses of these tools in both kinds of settings have been extensively documented in various publications (see, e.g., refs 1 and 2). However, widespread adoption of these tools and techniques has not been forthcoming. Why has there been such reluctance from chemists and scientists in general to expose their teaching to peer review? Certainly, ignorance of the project and the concept of peer review of teaching plays a role. Lack of time to institute peer review properly can also be a concern. Although there may be other excuses, it appears that fear is one of the most compelling reasons to forestall the implementation of peer review. How ironic that disciplines that pride themselves on the peer review of their research, grant proposals, and even time on expensive instruments can let peer review of teaching be so immobilizing! Peer review of research has miraculously opened the doors to all sorts of inquiry since the peer review process began. Science has basked in the limelight of peer review to claim that only the best science is supported. What makes us believe that peer review cannot do the same for teaching? One explanation is articulated by Lee Shulman, President of the Carnegie Foundation for the Advancement of Teaching: “I now believe that the reason teaching is not more valued in the academy is because the way we treat teaching removes it from the community of scholars” (3).

Imagine the form that chemical research might have taken if it had developed in the same fashion as teaching. Rather than going to meetings to present our results or publishing papers to convince the scientific community of the correctness of our experiments, we would just discuss our results over a cup of coffee with a colleague. We would say to a colleague, “Your experiment is great but it would never work in my lab because the molecules are different and besides, I have more molecules than you do.” We would not treat research experiments this way, but we treat teaching experiments this casually and often do not attempt to duplicate the successful teaching experiments that others have undertaken. We believe it is time to treat teaching with the same intellectual rigor that we afford chemical research, and providing that intellectual rigor is one of the purposes of the peer-review project. This mandate has taken root at 16 institutions across the nation1 and, hopefully, will continue to grow beyond them. This paper will address how the AAHE project began, its present status, the issue of fear, the arguments for peer collaboration and review, a brief review of the results and impact of peer review of teaching, and future plans for the project. The AAHE Peer Review of Teaching Project The conception and design of the project was a joint effort. Lee Shulman brought to the work many years of research with outstanding K–12 teachers and a sense of the complexity of the teaching–learning process. AAHE had been working on the concept of “A Culture of Teaching and Learning” and was interested in helping campuses develop such a culture. Together, Shulman and AAHE sought and received funding from The Pew Charitable Trusts and The William and Flora Hewlett Foundation. The project began formally in 1994 when faculty from a dozen universities met at Stanford University for a week-long intensive session to establish how to begin the peer review of teaching process. This session developed a set of three exercises that guided faculty through a process of reflection, analysis, and documentation of their course design, their teaching of the course, and the results of their teaching in terms of student learning. Subsequently, a “menu” of strategies for peer collaboration and review of teaching was devised and shown to support the improvement of teaching and learning. The following eight basic tools for peer review and collaboration of teaching have been used in a variety of institutional settings, courses, and disciplines. 1. Teaching and course portfolios are selections of teaching materials and student work assembled by faculty to represent their teaching and its connection to student learning. Portfolios are an excellent way for teachers

JChemEd.chem.wisc.edu • Vol. 77 No. 2 February 2000 • Journal of Chemical Education

239


2.

3.

4.

5.

6.

7.

8.

to reflect upon their teaching and take responsibility for its quality. During reciprocal classroom visits, pairs of faculty observe each other’s classes and interview each other’s students. It provides feedback for improvement and engenders collegial collaboration in teaching. Visits by faculty across disciplines have been quite effective in initiating new ideas and approaches to the teaching–learning discussion. The reflective memo is a document prepared by the instructor that outlines the learning goals of the class, how they will be assessed, and what problems are present in teaching the specific class. Departmental teaching seminars are regular, formal occasions for scholarly exchange about teaching and learning. These are similar to the research seminars that are in place in our departments. Teaching circles are small groups of faculty who meet regularly to discuss their own teaching and student learning patterns. Attention to teaching in the hiring process can be as simple as asking candidates to present a teaching seminar as well as a research seminar. Another method is to have departmental members talk with the candidate about their teaching expectations. Coaching and mentoring involves arrangements between groups or individuals that encourage professional teaching development. External peer review mimics what we already expect in chemical research. Materials related to teaching are sent to scholars in the field for appraisal. Some materials that can be included in the peer review are the teaching and course portfolios mentioned above.

As the tools were developed and became more widely used, faculty at participating institutions began to share their experiences with them. AAHE published a compilation of these efforts in 1996 (2), and a Web site under development by the Teaching Academy at the University of Wisconsin– Madison lists a Menu of Options (hence the MOO in the address) to guide faculty and staff through the many powerful choices that have been developed. The URL is http://www. wisc.edu/MOO/. Under a separate but related initiative, T. A. Angelo and K. P. Cross have developed a series of techniques for assessing student learning (4 ). These tools, and more that will be developed as the project grows, give us powerful techniques for improving and assessing our teaching and its connection to student learning. Addressing the Perceived Barriers to Peer Review We have suggested that fear may play a role in the reluctance of chemists to employ peer review to improve teaching and learning in chemistry classes. Evidence of this fear comes from surveys of faculty and administrators who were asked to articulate the reasons they did not implement peer review at their institution. Fear and time were the top two barriers listed. Because of these concerns, we want to directly address ways that fear can be minimized and time can be used effectively.

Fear Fear is natural and is experienced on a regular basis when research is subjected to peer review. Many of our colleagues, 240

however, perceive the difference between the review of research and the review of teaching and learning as arising from concerns about fairness. We have established procedures for the peer review of research whereby specific elements are evaluated. These have been developed over years of scrutiny. Also, the peer reviewer is not normally at the same institution and is presumed not to benefit directly from the outcome of the review. This is not to suggest that the peer review of research is always fair, but it has the comfort of familiarity. What Should Be Reviewed? This leads to the first barrier to the peer review of teaching, uncertainty about what should be reviewed. Because teaching styles are viewed by many as personal, many members of the Peer Review Project have suggested that learning, rather than teaching style, should be the measure of teaching effectiveness. For this reason, the following definition of good teaching is a useful starting point (5): Good teaching is purposeful activity intended to bring about learning in others, and it is good or successful in the degree to which it helps accomplish that intended learning.

With this definition, the focus of the review becomes how to demonstrate that learning has occurred as a result of the teaching process, and does not suggest review of the teaching style. This focus is related to the research review question of what is the long-term impact of the research and suggests that measures of outcomes of teaching may be more objective. How Should the Review Be Conducted? The second barrier to the peer review of teaching is in how the process is reviewed. The above definition suggests that teaching is a “purposeful activity”. In research this would be the experimental design or the approach to the problem. In this context, a successful researcher would demonstrate mastery of the proposed problem by showing how and why the critical steps were chosen and the expected outcomes. The Peer Review Project suggests a similar approach with a document called the “Reflective Memo”. The following are examples of the items that might be included in such a memo: 1. Overview of purpose of course. 2. Specific goals to be achieved and when. 3. Clientele and the mixture of their abilities. 4. Approaches to learning. 5. Special problems for which help from the peer would be welcome.

In this document, the teacher outlines the nature of the class, the intended learning goals, and his or her intellectual and scholarly understanding of the topic, and then, to focus the reviewer’s response, selects one or two areas in which feedback would be most helpful. Who Should Conduct the Review? A third barrier is the choice of the reviewer. In research, we might suggest that our ideas not be submitted to certain reviewers because of past experiences of bias, but funding agencies are not bound by that request. If part of our fear in the review of teaching comes from the possibility that a reviewer could be from our own department or discipline, we have the option to select someone from an allied discipline or to send our teaching material—reflective memo, portfolios, etc.—to a colleague at another institution. What is suggested

Journal of Chemical Education • Vol. 77 No. 2 February 2000 • JChemEd.chem.wisc.edu


here is that if the review is to be a constructive one for the purpose of improving teaching, there is no requirement that it be done by one’s departmental colleagues. In fact, there is a great deal of positive benefit from having a colleague in an allied discipline review the effort because some of the difficulties with teaching are not discipline-specific (styles of student learning, attention span, testing, problem solving, assessing outcomes, etc.). On the other hand, the content of the course may be an area where another chemist could provide particular insight. If the course portfolio has been constructed appropriately, however, it should be sufficiently portable that a chemist at another institution could provide review. When Should the Review Be Conducted? If the processes suggested by the Peer Review Project are practiced for constructive reviews before they are employed for summative purposes (merit and promotion), less fear might remain because the parallels to the peer review of research will be understood. In addition, the question of fairness could be examined using the actual experiences of the faculty to modify behavior perceived as unfair. It would also appear that fairness would require that tenured faculty be the first to experiment with peer review of teaching before subjecting junior faculty to the process. This suggestion would, we believe, advance the status of teaching in our discipline.

Time There is no way to create a path for the improvement of teaching and learning without the expenditure of time. The issue that each of us has to decide is how much time will be given to this aspect of our work. It is curious that only seldom is the review of research considered a time burden when it is primarily for summative purposes (e.g., who gets funded). If we postulate that learning in our classes could increase by some percentage, how much additional time would we spend in a given semester? Would one hour a semester be too great a burden? Our University of Wisconsin–Madison colleague in Electrical and Computer Engineering, John Webster, addressed the problem from an engineering perspective and concluded that most of the practices recommended by the Peer Review Project could be carried out in one hour, as suggested below. Steps for the Reviewer to Conduct Peer Review in One Hour

the course, provided material for the review, and prepared for the reviewer. The questions the reviewer asks the class focus on learning, and there is a mechanism for arriving at consensus rather than having the most vocal student dominate the review. Finally, the reviewer summarizes student comments and his or her own observations on the issues outlined by the instructor in the reflective memo. This process required one hour by the reviewer, but consumed only 15 minutes of actual class time. The time expended by the instructor was considerably longer because of the preparation of the reflective memo and the material needed for the peer to understand the purpose of the course and its development. The one-hour approach is a “one-shot” process, but it is a mechanism for getting colleagues to consider the power of peer review without a large investment of time. Clearly, it would be interesting to do this more than once during a semester to see if the instructor were able to deal creatively with whatever learning problems were identified. It would also be of advantage to do the review early enough that the whole semester is not lost for students who are having difficulty. The current metric for learning, the student questionnaire, has the failing that it is normally given at the end of the semester, so that learning problems are articulated at a point when it is too late to address them. Instructors who decry the use of student questionnaires as unfair do have an alternative, and that is peer review. The student questionnaire takes about the same amount of class time, but the essential reflection and communication by the instructor is missing. The question is whether research is so different from teaching that a focused and intentional approach is effective in the former, whereas it would not be effective in the latter.

Barriers: Conclusion Fear is real and in some cases justified. If peer review can enhance learning, however, we must find ways to prevent fear from becoming a barrier to the implementation of scholarly discussions of learning within and across disciplines. Time will continue to be a barrier, but if effort is focused with good justification for the suggested activities, it may return the dividends that justify it. Those of us who have tried the strategies of peer review are convinced that they are effective and worthy of continued improvement. A national discussion within the discipline of chemistry could be a way to hasten further improvements.

1. Peer reviews course objectives, outline, notes, etc.

10 min

Rationale for Peer Review and Collaboration

2. Peer attends class and makes notes

35 min

What should chemists and other academics gain from submitting themselves to peer review of their teaching? There are four broad arguments for participating in peer review: motivation for self-improvement, generating a more scholarly metric of teaching besides student evaluations, academic recognition of teaching, and alternatives to bureaucratically decreed forms of accountability.

3. Instructor leaves the classroom. 4. Peer asks three questions of whole class

5 min

a. What aspects of the course have helped you learn best, so far? b. Should the course be changed to help you learn better, and if so, how? c. Should the instructor change to help you learn better, and if so, how? 5. Discussion groups of students on answers to questions in #4 5 min 6. Peer summarizes group comments and collects responses

5 min

7. Peer collects student responses and the peer observations; gives these to instructor.

John G. Webster at [email protected]

This short process contains all the elements necessary for a good review provided that the instructor has thought about

Self-Improvement Teaching is a learned experience that grows and improves with collaboration. At present, graduate school training in teaching chemistry is almost nonexistent. Most of us are released in the world of academe with a mandate to teach but with little idea of how. Consequently, our students suffer through our first few terms of teaching while we learn the many tricks and tools of the trade on the job. There has to be a better way to prepare graduate students and faculty for


241


their teaching careers. Furthermore, there has to be a better way to improve and learn from our teaching and to grow in this portion of our careers. Collaboration and reflective practice offer methods for self-improvement as well as for improved training of graduate students. Tools such as teaching and course portfolios, teaching circles, reciprocal classroom visits, and coaching and mentoring are designed to aid in this process. Industrial and research settings have long-established traditions of collaborative investigation and peer participation. Even professional football does a better job of peer review and using teamwork than academia does. Clearly, this is one area in our profession that we can vastly improve.

Student Evaluations Student evaluations as an objective, quantifiable measure of teaching effectiveness may fail because students cannot adequately assess many aspects of teaching simply because they are new learners of the discipline. For instance, the subject level and content of a course are much better evaluated by fellow faculty than by students. Peers are in a unique situation to observe and judge what good teaching is all about. According to Lee Shulman’s research, superior teachers have a good grasp of three tenets of teaching (6, 7 ). They have an excellent grasp of their subject area, basic content; they understand general methods of teaching; and they can transform a concept appropriate to their discipline into terms that students can understand. Clearly, peers have more perceptive insights into these tenets than many students and can tell us how these areas can be improved. Academic Recognition of Teaching In our professional lives, we value few things more than the regard of our peers. Whether locally, regionally, or nationally, we try to impress our colleagues with our abilities as researchers, chemists, and gatherers of grants, subjecting these areas of our work to peer review. Until we elevate teaching by subjecting it to peer review, it will remain the neglected stepchild. Alternatives to Bureaucratic Accountability The least tasteful reason for adopting peer review and collaboration may also be one of the most compelling ones. Across America, outside agencies are attempting to make higher education more accountable. Post-tenure review, abolition of tenure, mandated teaching hours, and other agendas confront us regularly. We are left with the following choice: either some outside agency will set up an accountability procedure for us, or we can devise our own. Peer review can be an important factor in those mechanisms; and, it has the potential to produce actual improvements in the learning accomplishments of students. To look at some of these strategies in more detail, visit the Wisconsin Task Force on Peer Review’s Web site, http://www.wisc.edu/MOO/. (Be sure to capitalize the MOO.) The site contains material on how to conduct a review of a colleague’s teaching, how to design a peer review program, how to develop or improve one’s own teaching, how to show evidence of one’s teaching, how to evaluate evidence from a peer review, and how to raise the profile of teaching in one’s department. Results and Impact of the Peer Review of Teaching John Wright, a chemist at the University of Wisconsin– Madison, used peer review to assess the effectiveness of 242

innovations in his teaching of analytical and equilibrium chemistry. To his dismay, he had found that even A students could not transfer the knowledge imparted in his class from one setting to another. John became convinced that the students had not struggled with the material sufficiently to grasp the deeper context of what they were learning. Accordingly, he transformed his class by incorporating student groups, using computers to apply simple concepts to more difficult problems, having the students read research papers and apply what they read to new problems, and working with students on more open-ended projects that involved designing an approach to a problem. The class changed from teacher-centered delivery to student-centered learning. To evaluate the effectiveness of his new approach to teaching chemistry, John ran a comparison test with a similar course taught in the traditional format by one of his most esteemed colleagues, who was also an outstanding teacher. At the end of the semester, 25 faculty from departments outside chemistry were invited to serve as peer reviewers without knowing which class the students had taken. They were asked to assess the competence of students in their knowledge of the chemistry and to rank the students that they tested. At the end of the study, the rankings of the students in the two classes were compared, and there were positive results to support the student-centered or collaborative learning approach. Furthermore, the faculty who served as peer reviewers learned about new approaches in the classroom and had an opportunity to think and talk together about their own teaching (8). When a teacher embarks on some of these collaborative, more open-ended teaching styles, it is not unusual for student evaluations of the teacher to go down while the learning assessment performance rises. Two authors of this paper, CHA and JWT, have experienced this. However, one of the peer review tools, reciprocal classroom visits, is an effective method to demonstrate to students, colleagues, administrators, and the public that these new methods facilitate student learning. A committee of peers can visit the classroom or lab during the course. There they can assess the effects of the new teaching methods and, if necessary, write a report to the appropriate administrators describing positive effects. On the departmental level, teaching circles can be an effective way to redirect funds, institute change, and develop curricula. In the University of Georgia Chemistry Department, the teaching circle developed into the tool that guided the department through the pains of switching from the quarter to the semester system and establishing the basic course content. Not only were two brand new courses built from the drawing board (a junior–senior-year integrated physical–analytical– inorganic lab and an introduction to modern technology and literature seminar), but also appropriate funding from the department was directed to these courses to assure their success. Moreover, the teaching circle accepted the challenge of looking at how the curriculum could be integrated from top to bottom and whether, for example, spectroscopy is taught at appropriate points and the information is refreshed in the students’ minds in subsequent courses. This is a large enough undertaking that one or two faculty would wither under the load. Thus, the teaching circle helped to distribute the task. More importantly, because a large contingent of the faculty participated in the effort, the new curriculum had a stronger internal endorsement than could have been generated

Journal of Chemical Education • Vol. 77 No. 2 February 2000 • JChemEd.chem.wisc.edu


by any other method. Consequently, the transition year was relatively painless. In this case, genuine peer collaboration was an effective method of departmental transformation. At the Biennial Conference on Chemical Education in Waterloo, Ontario, Erwin Boschmann, Associate Dean of the Faculties at IUPUI, presented a paper entitled “Peer Review as a Catalyst for Institutional Change”. It outlined the procedures IUPUI has in place to make peer review an important piece of the tenure and promotion guidelines. The project began with Chemistry, English, and Nursing, each of which was given $5000 to initiate peer review. After the first phase was finished, another 16 departments applied to join the project and 6 were chosen. Over a three-year period, the promotion and tenure files of the faculty displayed a noticeable trend to include more peer review and emphasis on teaching. Language in the promotion and tenure guidelines has been changed to reflect this emphasis. For instance, the guidelines now ask, “Has the faculty member communicated the results of teaching to peers on campus and across the discipline nationally and internationally? Have peers assessed teaching?” The Provost has spelled out two clear directives to the campus: the peer review of teaching will be an annual expectation; and all hiring interviews should include a pedagogical colloquium. These activities were encouraged with teaching development grants, and the top teaching award on campus carries a $3000 increment to the base salary. At IUPUI, peer review is now a normal part of the business of teaching and learning. In addition, post-tenure review has been adopted as a campus policy, and peer review is a strong component of that review. Future Directions for Peer Review As a result of the AAHE peer review project and related efforts, the idea that teaching, like research, should be peerreviewed is much more in evidence today than it was five years ago. New practices and policies are also increasingly evident. But more work is clearly needed. To continue this work is the purpose of a new national project launched in winter l998, entitled the Carnegie Academy for the Scholarship of Teaching and Learning (CASTL). It is a five-year effort, funded by The Pew Charitable Trusts ($4.76 million) and The Carnegie Foundation ($1.24 million). Like the previous AAHE effort, CASTL is predicated on the idea that teaching should, like other scholarly work, be shared community property. The problem driving the project is not that teaching is bad or somehow deficient, but that it is so often invisible, “disappearing like dry ice”, as Carnegie Foundation president Lee Shulman puts it. What’s lost in this invisibility is what is taken for granted in other forms of scholarly work: the chance, indeed the responsibility, to document what we do, share our work with our scholarly peers, seek out critique and feedback, and contribute to the advancement of thinking and practice in our field. The goal of CASTL is to develop a scholarship of teaching and learning that will make this kind of exchange and progress possible, to (i) foster significant, long-lasting learning for all students, (ii) enhance the practice and profession of teaching, and (iii) bring to teaching the recognition and stature afforded other forms of scholarly work in higher education. Toward these ends, CASTL integrates work on three fronts. The first component, the Pew National Fellowship

Program for Carnegie Scholars, is for faculty from all sectors of higher education—122 of them over the five years of the project, committed to investigating and documenting significant issues and challenges in the teaching of their fields. The Pew National Fellowship Program for Carnegie Scholars is not an award for teaching excellence, nor it is a teachingimprovement workshop; its purpose is to create a community of scholars whose work will advance the profession of teaching and deepen the learning of students. Six chemists are participants in the first two cohorts of the Pew National Fellowship Program for Carnegie Scholars: Brian Coppola from the University of Michigan, James Hovick from the University of North Carolina at Charlotte, Deborah Weigand from the University of Washington, Linda Hodges from Agnes Scott College, Dennis Jacobs from the University of Notre Dame, and Mark Walter from Oaken Community College. The second component is the Teaching Academy Campus Program, intended for campuses prepared to enact a view of teaching as scholarly work and ready to put into wider practice the strategies featured in this article. The third component entails collaboration with the scholarly societies (including the American Chemical Society) and is intended to get the word and work of the Carnegie Academy out into larger faculty communities. Further information about CASTL is available on The Carnegie Foundation Web site, www.carnegiefoundation.org, or by contacting Jacki Calvert, The Carnegie Foundation for the Advancement of Teaching, 555 Middlefield Road, Menlo Park, CA 94025; phone: 650/566-5139; email: [email protected]. More information about AAHE’s work on peer review can be found at www.aahe.org after clicking on Teaching Initiatives. Note 1. Alverno College, Colorado College, Indiana University Purdue University Indianapolis, Kent State University, Northwestern University, Stanford University, Syracuse University, Temple University, University of California–Santa Cruz, University of Georgia, University of Michigan, University of Nebraska–Lincoln, University of North Carolina at Charlotte, University of Wisconsin– Madison, University of Wyoming, Xavier University of Louisiana.

Literature Cited 1. Innovative Higher Education, Vol. 20 No. 4; Simpson, R. A.; Gillespie, K. H., Eds.; Human Sciences Press: New York, 1996. 2. Making Teaching Community Property: A Menu for Peer Collaboration and Peer Review; Hutchings, P. A., Ed.; American Association of Higher Education: Washington, DC, 1996. 3. Ibid.; Introduction, p 1. 4. Angelo, T. A; Cross, K. P. Classroom Assessment Techniques: A Handbook for College Teachers, 2nd ed.; Jossey-Bass: San Francisco, 1993. 5. Teaching and Its Evaluation: A Handbook of Resources; Ory, J. C., Ed.; University of Illinois at Urbana: Urbana, IL, 1990. 6. Grossman, P. L.; Wilson, S. M.; Shulman, L. S. In Knowledge Base for the Beginning Teacher; Reynolds, M. C., Ed.; Pergamon: New York, 1989; pp 23–26. 7. Shulman, L. S. Am. Assoc. Higher Educ. Bull. 1989, June, 8–13. 8. Wright, J. In Making Teaching Community Property: A Menu for Peer Collaboration and Peer Review; Hutchings, P. A., Ed.; American Association of Higher Education: Washington, DC, 1996; pp 79–82.


243

Why Are Chemists and Other Scientists Afraid of the Peer Review of

Recommend Documents