Development and Evaluation of a Prep Course for Chemistry Graduate

A. Flaherty, A. O'Dwyer, P. Mannix-McNamara, and J. J. Leahy . Evaluating the Impact of the “Teaching as a Chemistry Laboratory Graduate Teaching As...
0 downloads 0 Views 351KB Size
Download PDF
Article pubs.acs.org/jchemeduc

Development and Evaluation of a Prep Course for Chemistry Graduate Teaching Assistants at a Research University Gili Marbach-Ad,*,† Kathryn L. Schaefer,‡ Bryna C. Kumi,§ Lee A. Friedman,§ Katerina V. Thompson,† and Michael P. Doyle§ †

College of Computer, Mathematical, and Natural Sciences, ‡College of Education, §Department of Chemistry and Biochemistry, University of Maryland, College Park, Maryland 20742, United States S Supporting Information *

ABSTRACT: This study describes the development and evaluation of a prep course for chemistry graduate teaching assistants (GTAs). The course was developed around three major goals: (i) building a community for new GTAs and socializing them into the department; (ii) modeling teaching with well-documented, innovative teaching and learning techniques; and (iii) helping GTAs to understand their roles within the department and their specific course. The program consisted of a team-taught, sixweek course, which was mandatory for first-year GTAs. It was first offered in Fall 2009 (46 students), and then taught again in Fall 2010 (33 students). The course covered multiple topics including student−GTA communication, student assessment, and teaching strategies. End-of-semester surveys, student evaluations of teaching, and interviews with GTAs were used to evaluate the course. Overall, feedback from the course was very positive. GTAs reported that the course benefited them immediately and even more so after they had taught for several semesters. Student evaluations of teaching showed that, on average, first-year GTAs that had completed the prep course in Fall 2009 and Fall 2010 received significantly higher scores than the previous cohort of new GTAs (who had not completed a prep course) on measures such as effective teaching, respecting students, and being prepared. KEYWORDS: Graduate Education/Research, Professional Development, TA Training/Orientation

■

INTRODUCTION Science programs at large research-oriented universities rely heavily on graduate teaching assistants (GTAs) to assist with undergraduate education, primarily in leading laboratory and discussion sessions of introductory courses.1,2 Faculty and administrators often assume that the graduate students already know how to teach, are enthusiastic about their role as GTAs, understand the structure of the class, are comfortable with being in a position of authority, and thoroughly understand science at the level they are expected to teach.3 However, surveys of 4000 graduate students at 27 universities show that only one-third of them report participating in GTA training sessions in preparation for teaching.4 Moreover, few felt prepared to engage in basic teaching activities such as lecturing, facilitating discussions, and supervising laboratories.4,5 GTA teaching development is often trial-and-error, and most graduate students are left to hone teaching skills on their own, which can be overwhelming for novice graduate students.3,5−7 As a result of concerns regarding the lack of systematic professional development opportunities for GTAs, there have been recommendations to provide GTAs with more training in teaching.8−11 Some institutions have begun to institute orientation programs to help GTAs adjust to their instructional role, which indicates the importance the university places on undergraduate education.12 However, the nature of these programs varies greatly, ranging from university-wide half-day orientations to department-specific full semester courses.1,13 While systematic training programs for GTAs can result in substantial improvements in their teaching behaviors and efficacy,5,11 many of these training programs provide limited © 2012 American Chemical Society and Division of Chemical Education, Inc.

opportunities for in-depth development of pedagogical skills. Moreover, little attention is given to the unique demands of teaching within specific scientific disciplines such as physics, chemistry, geology, or biology.14 As with other scientific fields, the nature of chemistry GTA training varies greatly. A survey of undergraduate chemistry programs approved by the American Chemical Society found that 37% did not provide formal GTA training. Among the institutions that provided formal GTA training, only 17% spent more than one day on training activities.15 Several studies have examined specific chemistry GTA training programs at various institutions. Roehrig et al. found that a semester-long training seminar run by chemistry laboratory coordinators was effective at preparing GTAs for prelaboratory discussions in traditional laboratories, but not for inquiry-based laboratories.13 Nurrenbern et al. compared Purdue University chemistry GTAs that were in a trial GTA training program with those who were not. The trial program taught GTAs effective teaching strategies and the importance of active learning. Results indicate that the trial program could be used effectively to facilitate improvement of graduate instructors’ teaching skills.16 A second common recommendation for improving doctoral education is to build a strong and supportive community for graduate students.8 Few studies have examined the role of peers in the development of GTA teaching skills; however, preliminary research indicates that graduate students value the frequent, informal interactions with peers that are possible within structured programs.17 In our previous experience with Published: May 4, 2012 865

dx.doi.org/10.1021/ed200563b | J. Chem. Educ. 2012, 89, 865−872

Journal of Chemical Education

Article

The six weekly sessions followed a similar template, comprising two parts: 1. Stories from the classroom 2. Topics of the week The first two sessions began with experienced GTAs and faculty members sharing stories of real incidents from their teaching experience. In later sessions, the new GTAs, most of whom supervise labs and a few lead discussion sessions, had the opportunity to share their own stories. This usually fostered discussions, such as how to engage inattentive students, how to deal with students who are caught cheating during tests, preferred ways for instructors to interact with students (e.g., through e-mail, office hours), and appropriate places to schedule meetings with students. Following the stories from the classroom, instructors focused on one of several topics (see Table 1). During these

developing a GTA prep course for biology departments, we discovered from GTAs’ feedback that the course helped them build a community, especially because of the opportunities to meet regularly with experienced GTAs in the departments and teaching personnel outside of their advisors and graduate coordinators.18 Research has found that community involvement influences GTA teaching practices. In a qualitative study, GTA and postdoctoral fellows who were involved in an inquiry community began practicing new teaching techniques and adopted identities as both researchers and teachers. The participants attributed this change to the process of social participation, practice, and apprenticeship fostered by the community.19 Moreover, GTAs who regularly discussed their teaching with mentors and peers tended to show greater improvement in their inquiry-based teaching skills. Gilmore and Hurst found that graduate students who made gains in inquirybased teaching skills over the course of an academic year were more likely to regularly discuss their teaching with their mentors, graduate student peers, or practicing K−12 teachers.6 In response to these recommendations, the Department of Chemistry and Biochemistry at our institution developed a program to enhance GTA teaching preparation and build a community among incoming GTAs. The program consisted of a team-taught, six-week GTA development course, first offered in Fall 2009, which is mandatory for all first-year GTAs. This is in contrast to most of the GTA training programs described in the literature, which are not mandatory and may result in lower participation rates.4 This study aimed to evaluate the effectiveness of the course and provide recommendations for improvement, with the ultimate goal of developing a program that could serve as a national model for graduate student professional development.

■

Table 1. General Topics for the GTA Teaching Preparatory Course Week

General Topic

1

Introductions

2

Student−GTA communication Student assessment Teaching strategies GTA evaluations

3 4 5 6

Presentation skills; Careers

Specific Discussion Elements GTA responsibilities; The student−GTA relationship Difficult situations; Tips for communicating with students Grading exams and lab reports; Maintaining consistency in assessment Making laboratories memorable; Responding to student questions University course evaluations; Self-evaluations; Personal strengths and weaknesses Modeling diverse methods for presenting material; Guest lecture by National Institute of Standards and Technology laboratory director

METHODS discussions, they demonstrated different teaching techniques (e.g., small-group discussions, think−pair−share, and role-play) and explained how to implement them in undergraduate classrooms. The online Supporting Information gives a detailed program for each session. The focus topics for each session were based on discussions among the Chair, instructors in the department, experienced GTAs, and the Director of the Teaching and Learning Center. Because almost all of the GTAs are required to serve as laboratory leaders, the course focused on the pedagogy of laboratory instruction. We followed the literature recommendations to focus on encouraging student−GTA communication, which includes modeling for the GTAs how to encourage their students to ask questions and work in small groups.13,16 We exposed the GTAs to best practices through role-plays, group discussions, and simulations.

Course Description

A six-week mandatory course for all new chemistry and biochemistry GTAs at our institution was offered in Fall 2009 and 2010. The course met one evening a week for 2 h and was held in a large seminar room with moveable seating to encourage small-group work and discussion among participants. The course was developed and led by two people: one who was serving as the Department Chair, and one who was the Director of the Chemical and Life Sciences Teaching and Learning Center. Additionally, three different faculty members and two to three different experienced GTAs from the Department of Chemistry and Biochemistry participated in each session to share their perspectives and experiences and to demonstrate and discuss innovative teaching methods. The participation of faculty members and experienced GTAs also allowed the new GTAs to become acquainted with members of their new department. The deep involvement of the Department Chair emphasized the importance of the course and highlighted departmental priorities. Prior to each session, all participating faculty and experienced GTAs met for a preparation meeting. The course’s goals included: • Building a community for the new GTAs and socializing them into the department. • Modeling teaching with well-documented, innovative teaching and learning techniques. • Helping GTAs to understand their roles within the department and their specific course.

Sample

The course enrollment in 2009 consisted of 23 females and 23 males, and in 2010 consisted of 12 females and 21 males. The number of international GTAs in 2009 and 2010 was 29 and 8, respectively. The GTAs came from diverse backgrounds in term of undergraduate institutions (e.g., liberal arts, research intensive, public, private, small, large). Primary Data Sources

GTAs’ End-of-Semester Feedback Survey. At the end of this six-week course, during the course’s final meeting, GTAs were asked to evaluate the format and topic selection. This was administered anonymously using printed evaluation forms. In Fall 2009, 39 GTAs responded to the survey, while in Fall 2010, 866

dx.doi.org/10.1021/ed200563b | J. Chem. Educ. 2012, 89, 865−872

Journal of Chemical Education

Article

Table 2. Demographics of GTAs Who Were Interviewed GTAs

Year Prep Course Taken

Gender

Ethnicity/Nationality

Class Taught

Class Type

1 2 3 4 5 6 7 8

2009 2009 2009 2009 2010 2010 N/A 2009

Female Female Female Male Male Male Female Female

White/American Asian/American Black/American White/American White/American White/American White/American Asian/Chinese

Introductory Introductory Introductory and upper level Introductory Upper level Introductory Introductory Introductory

Discussion Lab Lab Discussion Lab Lab Lab and discussion Lab

22 students responded. GTAs’ end of-semester feedback survey is provided in the online Supporting Information. Responses to the open-ended questions were analyzed qualitatively using an inductive approach, in which related responses were grouped into subcategories that could be quantified.20 A research team, including a graduate student from the College of Education, a graduate student from the Chemistry and Biochemistry Department, and a science education faculty member, categorized the responses separately and then discussed their categories until they reached agreement. The inter-rater agreement was 90%. University End-of-Semester Course Evaluations. Undergraduate student course evaluations were used as one measure of the teaching efficacy of the GTAs. These evaluations are administered online campus-wide during the last two weeks of the semester and consist of 10 items, 5 of which were reported to administrators and were available for analysis (see Findings section). The university-wide response rate for the evaluations is on average about 65%. Response rates for the chemistry GTAs in this study were 71% for Spring 2009, 68% for Spring 2010, and 62% for Spring 2011. Students are asked to rate their instructors and courses on a scale of 1 to 4, with 4 being highest. The Spring 2010 (n = 29) and Spring 2011 (n = 25) teaching evaluations for the GTAs, who had completed the training course in Fall 2009 and Fall 2010, respectively, were compared to the Spring 2009 teaching evaluations for the previous year’s GTAs (n = 44), who had not completed the training course. Both groups of GTAs had taught for two semesters in the same undergraduate curriculum. Of the 44 GTAs from Spring 2009, 24 were female and 20 were male; 28 were international students and 16 were domestic students. Of the 29 GTAs from Spring 2010, 15 were female and 14 were male; 21 were international students and 8 were domestic students. Of the 25 GTAs from Spring 2011, 11 were female and 14 were male; 5 were international students and 20 were domestic students. Follow-Up Survey. In Spring 2011, GTAs from both prep courses (2009 and 2010) were asked to complete a follow-up email survey regarding how beneficial they found each prep course topic. Our goal was to examine their perceptions of the course after they had accumulated more teaching experience. For the graduate students completing the prep courses, 25 GTAs responded, which is 31.6%. Of the 25 GTAs, 16 participated in the 2009 course and 9 participated in the 2010 course. In terms of demographics, 15 were male and 10 were female; 14 were international students and 11 were domestic students. The GTAs who responded appear to be representative of the sample of GTAs in the training program. Interviews with GTAs and Course Instructors. At the end of the Fall 2010 semester, eight GTAs (who had taken the course and/or participated in the course as an experienced

Served as Experienced TA? Yes, Yes, Yes, No No No Yes, No

2010 2010 2010

2009

GTA) were interviewed regarding their experience in the course. GTA demographics are listed in Table 2. In addition, three instructors from the course were also interviewed regarding their impressions of the course. All the interviews were conducted, recorded on audiotape, and transcribed by one of the authors who is external to the Chemistry and Biochemistry Department and was not involved in teaching the course (see the online Supporting Information for the protocol of the semiconstructed interviews). Once again, the research team, including the graduate student from the College of Education, the graduate student from the Chemistry and Biochemistry Department, and the science education faculty member, categorized the responses separately and then discussed their categories until they reached agreement. Their inter-rater agreement was 90%. The research team coded for major themes, which are presented below.

■

FINDINGS

GTAs’ End-of-Semester Feedback Survey

To learn about GTAs’ reported benefit from the course, the general topics covered in the course were listed and students were asked to rate the benefit of the topics on a Likert-type scale of “most beneficial”, “beneficial”, “not beneficial”, and “not at all beneficial”. If GTAs rated a topic as “not beneficial” or “not at all beneficial,” they were encouraged to explain further. The categories of “most beneficial” and “beneficial” were combined and the percentages of GTAs with this response are reported in Table 3. The majority of GTAs from both 2009 and Table 3. Percentage of GTAs Giving Specific Prep Course Topics a Rating of “Beneficial” or “Most Beneficial” Course Topics

2009 Course (N = 39), %

2010 Course (N = 22), %

Classroom management Student communication Student assessment GTA evaluations Teaching strategies Experience stories Presentation skills Strengths and weaknesses

100 97 97 92 97 87 84 87

95 95 95 95 86 95 89 68

2010 found each of the course topics beneficial. Moreover, they reported that they used what they learned from the course in their subsequent teaching. In 2009, GTAs commonly reported implementing teaching strategies (e.g., guided inquiry, microteaching) that they learned from the course. In 2010, GTAs commonly reported applying what they learned about grading, student communication, and classroom management. In terms of grading, GTAs commented that the course helped them 867

dx.doi.org/10.1021/ed200563b | J. Chem. Educ. 2012, 89, 865−872

Journal of Chemical Education

Article

understand that they need to “ask questions when you are unsure of grading” and “review a bunch [of tests] before marking”. Regarding student communication, one GTA reported, “I have been using the communication techniques by smiling and trying to be pleasant with every student regardless of their attitude.” Finally, in terms of teaching strategies, some students reflected these specific comments: I have learned about how to answer their [students] questions. I don’t tell them the answers directly. Instead I direct them to discuss with each other or refer to the lab manual to find answers by themselves. GTAs also reported that some course topics helped them become more aware of their own teaching performance. These included meeting student and department expectations (n = 9, 18%), grading (n = 6, 12%), fostering communication (n = 6, 12%), and discussing and comparing teaching techniques (n = 6, 12%). Discussions with other GTAs seemed to be particularly important in normalizing various teaching experiences and providing ideas for classroom techniques. For instance, multiple GTAs commented that the most helpful aspects of the course were “experience stories, they let me know that difficulties that I had weren’t just happening to me” and “the sharing of experiences [which were] helpful for understanding and evaluating students and their reactions.” One GTA indicated that the course was helpful in learning “how to interpret students’ questions and how [these questions] reflect your communication skills.” Another commented that, “The system in this institution is rather different from my previous one. Therefore it really helped me figure out what is going on here.” Most GTAs from both years reported that they had positive feelings about the course, liked the course format, and thought the course covered all of topics important for novice teachers. Comments included: “I learned to interact with students and what I am expected to do”; “It was perfectThank you so much for devoting so much time for us”; and “Some demonstrations from experienced lecturers and senior students gave much information.” Parts of the course that GTAs indicated they particularly enjoyed included the presence of speakers and experienced GTAs, scenarios and skits, and discussions among the GTAs. Negative comments about the course generally stemmed from GTAs’ perception of the length and repetitive nature of the course. Comments included: “The first three weeks were informative, but after a while it was redundant”; “Already knew most of what was discussed because of a prior teaching assistant position”; and “The material could have been presented to us in a shorter workshop before the start of class.” Some GTAs felt that other topics should be included, such as “how to manage the time for TA and class” and “more ways to organize class when you are teaching (i.e., how to get class started).” Another GTA indicated, “the workshop on subjectivity in grading student work was especially enlighteningmore of this would have been useful.” As this course is a rare opportunity to guide graduate students in teaching practices before they assume a faculty position, GTAs were asked if they were considering a university career. A large number of GTAs from both years (more than 70% out of the 61 students that responded to the survey) indicated that they are considering an academic career. This is another good rationale for GTA training, because most chemistry faculty members have no formal training and

sometimes their only teaching experience before assuming a faculty position is as a GTA. University End-of-Semester Course Evaluations

Course evaluations were obtained for students who were taught by Spring 2010 GTAs and Spring 2011 GTAs (who had completed the training courses the previous semester), and from Spring 2009 GTAs (who had entered the program before the prep course was available). The course evaluations included five items (the instructor treated students with respect; the instructor was well prepared for class; the instructor was an effective teacher; the course was intellectually challenging; I learned a lot in the course). Figure 1 shows the average score

Figure 1. Average course evaluation scores for second-semester graduate teaching assistants from the 2009, 2010, and 2011 spring semesters.

for each item for each cohort of GTAs who were described in the Methods section above. The data show that in general, evaluations from Spring 2010 and 2011 of the GTAs who had completed the training course had higher average scores compared to the evaluations from Spring 2009 of the GTAs who had not completed the training course. It was expected that the five course evaluation items might be correlated, so a principle components analysis was first conducted to reduce the dimensionality of the data. The data analysis indicated that there were two main principle components, which together accounted for 83% of the variance in the data set. The first component accounted for 58% of the variance and was interpreted as an instructor variable, as it was composed largely of survey items indicating the instructor’s respect for students, preparedness for class, and effectiveness as a teacher. The second component accounted for 25% of the variance and was interpreted as a course variable as it was composed largely of whether the students viewed the course as intellectually challenging. One item (learned a lot) loaded on both components (Table 4). The two principle components were used as dependent variables in separate two-way analyses of variance with year (Spring 2009, Spring 2010, and Spring 868

dx.doi.org/10.1021/ed200563b | J. Chem. Educ. 2012, 89, 865−872

Journal of Chemical Education

Article

the course on a scale from “not at all beneficial” to “most beneficial” (Table 6). The majority of GTAs (more than 88% out of the 25 GTAs that responded to the follow up survey) rated all of the topics covered in the course either “beneficial” or “most beneficial”. None of the topics were rated as “not at all beneficial”. Coverage of strategies for good teaching received the most ratings in the category of “most beneficial” (52%). Communication with students (32%) and representation skills (29%) were the next most common topics rated as “most beneficial”. In addition, all GTAs reported that the course overall was either “beneficial” or “most beneficial” for them.

Table 4. Loading Matrix for Principle Components Analysis Survey Item Statements The instructor treated students with respect The instructor was well prepared for class The instructor was an effective teacher The course was intellectually challenging I learned a lot in this course

Instructor Variable

Course Variable

0.869

−0.273

0.930 0.938 0.254 0.571

−0.190 −0.167 0.861 0.616

2011) and nationality of GTAs (from the United States or not from the United States) as independent variables. Because of the modest sample size and the strong correlations among the original survey items, we performed an analysis of variance on the principle components rather than on each survey item. This approach resulted in greater statistical power and ease of interpretation. For the posthoc comparisons, we chose a single degree of freedom linear contrast because it was the most powerful approach and allowed a direct test of our hypothesis that the students who experienced the prep course showed higher scoring teaching evaluations than those who had not completed the prep course. There was no overall effect of nationality on either principle 1 (instructor variable) or principle 2 (course variable) (Table 5). On average, GTAs from the United States and those from

Interviews with GTAs and Course Instructors

In Spring 2011, to probe more deeply into GTA reflections on the course, after they had accumulated more teaching experience, seven GTAs who took the prep course in 2009 or 2010 were interviewed and asked to describe their overall experience in the course. The GTAs’ feedback was categorized under three main themes: 1. Expressing attitudes toward the course 2. Describing the support from peers and teaching staff 3. Describing how the course reinforced innovative teaching methods Below we will elaborate on each theme, followed by the results of interviews with the course staff. Interviews with the GTAs: Expressing Attitudes toward the Course. GTAs used the terms “interesting”, “enjoyable”, and “positive” to describe the course, and all of them said that the course was helpful in some way. [I]t helped me. I just got my evaluations back from being a TA and I had a 3.7 average out of 4. So I’m sure that part of that is from the course. So overall I think it was a positive experience and it helped me. [GTA 6] Six of the GTAs described how the course changed their approach to teaching, changing their attitudes about teaching, and making them more comfortable in the teaching role. It definitely changed my approach to teaching. I came in and it was a scary type of experience and I didn’t know what to think. It was 24 students versus one, but again hearing those other stories and seeing all these people around me who have done it before and who are doing it at the same time, it made me feel like I wasn’t alone and that helped a lot. [GTA 3] The GTAs felt that the course prepared them for teaching, although more so for supervising a lab section than leading a discussion section. Five of the GTAs described how the course covered all of the important content: “They gave me enough material in the beginning to comfort my fears so I didn’t freak out from all of the intense pressure of teaching a course” (GTA 5). All of the GTAs interviewed said that they would recommend the course to other new GTAs.

Table 5. ANOVA Results for the Principle Components by Year and Nationality Instructor Variable Effect

p-Value

Effect Size

Main Year Main Nationality Interaction Year × Nationality