Comparing Faculty and Student Perspectives of Graduate Teaching

Research: Science and Education edited by

Chemical Education Research

Diane M. Bunce The Catholic University of America Washington, DC 20064

Comparing Faculty and Student Perspectives of Graduate Teaching Assistants’ Teaching

Christopher F. Bauer University of New Hampshire Durham, NH 03824-3598

Romola A. Bernard Rodriques and Janet Bond-Robinson*† Department of Chemistry, University of Kansas, Lawrence, KS 66045; *[email protected]

Attempting to assess and coach individuals in effective teaching is a complex and cyclic endeavor. Teaching involves strategic interactions and problem solving based on understanding of the situation, the discipline, and the population of students that one is teaching. This is true of graduate teaching assistants (GTAs) as well. Feedback from undergraduate students (UGs) and from faculty and other instructors coaching GTAs in teaching provides outside perspectives. At least two perspectives beyond that of the individual GTA are involved in assessing a GTA’s teaching: the undergraduate students (UGs) of the GTA, and the faculty instructor–coaches of a new GTA, for which we use the abbreviation Fac. Assessment of graduate teaching assistants in chemistry is not a new issue (1, 2). In their discussion on teaching assistant assessments, Brooks et al. (2) pointed out the possible unreliability of direct observation and unannounced faculty visits to classrooms and laboratories for the purpose of evaluation. They argued that unannounced evaluations may lead to sampling error when “good” GTAs are evaluated on bad days and “bad” GTAs are evaluated on good days. To minimize this problem our GTAs played a key role in determining the date and laboratory section in which they were to be assessed, usually selecting the second lab of the week. Lab observations, using a remote audio-visual observational system, allowed the instructor to observe and record each GTA and UG interaction within their naturalistic setting and without interference of a human observer in the tight confines of the lab. The audio reception and recording were far superior to that heard when physically observing in the laboratory, and the observer panned, tilted, and zoomed the camera to see pages in lab notebooks, computer screens, and even small writing on the chalkboard. Remote observation was also less tiring so the Fac were able to pay close attention for several hours while following the GTA.

nars facilitated discussion of specially selected readings and teaching experiences with GTA peers. The GTA training seminars and assessments worked in tandem with a weekly TA meeting by the instructor of the laboratory course, which often emphasized procedural teaching using procedural knowledge of general skills, measurement, techniques, and instrument operation. GTAs were first formatively assessed and given feedback two weeks after they began teaching in the laboratory. A second and third formative assessment, separated by two weeks, focused on a GTA’s ability to create an effective learning environment—one in which they integrated chemical concepts underlying the lab investigation and attempted to get their UGs to reason in chemistry. In midNovember GTAs were given a final summative assessment for a grade. Thus, GTAs were observed and given substantial feedback on their teaching a total of four times—with the strongest emphases on conceptual teaching. Since formative assessment of new GTAs was part of the training, the Fac coached and provided opportunities for GTAs to build their understanding and therefore, to improve their performance. While observations of teaching behavior are often viewed as behavioral variables, our program does not simply promote practicing of skills. The hardest part of the strategic use of skills is knowing when to use them. Thus, the processes of learning were focused on building an understanding of effective chemistry teaching and attempting to use appropriate strategic types of interactions to produce a better learning environment for UGs. The quality of judgment and decisions that GTAs made during the course depended on their understanding of the importance of their role as chemistry teachers, and the realization that UGs needed GTA guidance to integrate lecture concepts into lab.

Description of Program

At the end of the semester, how did the instructor– coaches assess the teaching of new graduate teaching assistants in comparison to assessments by undergraduate students whom those GTAs taught in laboratory sections?

New graduate students were assigned to teach one of either two general chemistry labs, two organic labs, or one analytical lab with preparative work. Instructors hoped to enhance GTAs’ abilities to perform specified procedural and conceptual teaching interactions in the laboratory. We designed a one-credit hour training for new GTAs in which they were coached while situated in their laboratory teaching role (3, 4). Training activities included coaching through formative assessments, and viewing audio-visual clips of exemplary interactions of GTAs while teaching. Weekly semi† Current address: Department of Chemistry and Biochemistry, Arizona State University, Tempe, AZ 85287-1604

www.JCE.DivCHED.org

•

Research Question

Subjects

Graduate Teaching Assistants The primary subjects were 14 new graduate students who were teaching labs in a chemistry department of a midwestern Research I university. Seven were assigned to general chemistry (GC), five to organic chemistry (OC), and two to analytical chemistry (AC). The GTAs’ teaching backgrounds were variable, although most had not taught a lab class before. The majority of GTAs planned careers in industry rather than teaching.

Vol. 83 No. 2 February 2006

•

Journal of Chemical Education

305

Research: Science and Education

Undergraduates Survey data were obtained from 245 UGs who enrolled in the observed lab sections. The UGs are similar in their interest in the procedural work and getting out of lab whether they are in introductory or advanced labs, as others have shown (5). In the general chemistry course we found that some of the lab section means on the California Chemistry Diagnostic Exam were significantly less than the class mean, indicating that enrollment in 44 GC lab sections is not totally random. We have no similar information for the OC sections.

Faculty The faculty member who developed the GTA course and a postdoctoral student experienced in working with GTAs in this course were the instructor–coaches and did the final assessment of all graduate students. Each Fac rating was compared with the other and negotiated until agreement was reached. Laboratory Sections

Time Factor The GC labs lasted three hours; the OC and AC labs were scheduled for five hours, which meant that we videotaped longer in organic and analytical labs to get the same proportion of lab time. Use of Lab Instruments The equipment and instrument demands were similar in GC and OC. Our general chemistry labs use Ocean Optics probeware and additional software programs, which were a struggle for GTAs and for UGs to learn. The organic labs used IR and polarimetry in the first semester. Analytical lab responsibilities were considerably different, for example, troubleshooting instruments on a continual basis and round-robin scheduling for experiments at individual instruments. In summary, we think that comparing performances of OC and GC GTAs is valid and reliable. Therefore, we illustrate the two AC GTAs only in the overall performance ratings in Figure 1. Conceptual Teaching We define conceptual teaching as teaching the more abstract chemical concepts on which the lab experiment is based. Abstract chemical conceptions, such as atom, electricity,

chemical bond, and specific heat comprise chemical explanations and are conceived (6) by chemists, rather than perceived. In contrast, procedural teaching involves procedural knowledge of general skills, measurement, techniques, and instrument operation at the perceptual level of observation and performance. Conceptual teachers link the lab to the lecture’s abstract concepts as often as possible. They explain abstract concepts in terms of students’ concrete perceptions in the lab. They expect students to reason through the lab experiment rather than just accomplish the lab procedurally. When UGs ask conceptual questions GTAs can guide their reasoning (7). The underlying concepts are equally important to discuss for GTAs in GC, OC, AC, and physical chemistry (as we found in a subsequent year). The OC labs build upon each other from week to week, whereas GC labs often do not as might be expected in a survey course. The OC GTAs have more instances to discuss the same concepts, such as characteristics of molecules that alter polarity and solubility. Abraham et al. found that conceptual teaching is not common in chemistry labs (8), which is unfortunate and avoidable.

Remote Audio-Visual Observation and Documentation To comply with university human subjects research stipulations, procedures were explained to GTAs and their UGs, and they were asked to sign written consent forms for the audio-video observations. A camera was mounted 10–12 ft above the laboratory floor near the center of the lab during the 3–4 hour observations. The camera connected to an audio-video transmitter linked through the local area network (LAN) for reception at the remote site. (The remote site is the Fac’s research lab.) During the scheduled observation the GTA wore a wireless microphone attached to a pocket-transmitter pack, which signaled an audio receiver similarly linked through the LAN for reception at the remote site. At the remote site the audio-video signals were sent to a monitor for observations and through another line for compression into MPEG format and subsequent recording on a computer hard drive. Thus, the system captured the GTA’s interactions with UGs in audio-visual documentation. Watching the audio-video data of each GTA, the Fac assessed performance on 12 codes, which are given on the Instructor coding of the TA Teaching (ITAT) instrument. The Fac assessed each GTA using the audio-video by assigning a priori codes on the ITAT, discussed below. Following the observation GTAs were given audio-video excerpts of their lab

4

Overall z Scores

3

Table 1. Scoring Rubric Used for UGATA and ITAT Assessments

2 1 0 -1 -2

Corresponding Quality or Frequency Descriptors

1

Very Poor / Never

2

Poor / Rarely

3

Mediocre /Sometimes

4

Good / Often

5

Very Good / Very Often

-3

undergraduates' rating faculty rating

-4 -5 1

2

3

4

5

6

7

8

9

10

11

12

13

14

GTA Code Figure 1. UG and Fac differences in overall z score ratings of new graduate students. Includes general, organic, and analytical GTAs.

306

Likert Scale Values


•


•

www.JCE.DivCHED.org


for review as well as a copy of the completed ITAT. A collage of exemplary clips was designed and used in the seminar class so that peers saw the good and very good examples of each other and previous year’s GTAs at work.

Instruments Two survey instruments were constructed: the ITAT and the UGATA (UG Assessment of TA Teaching). Table 1 illustrates the Likert scale used to assess GTAs’ teaching performance. The scale ranges on a continuum from very poor effectiveness (or never happened) given a rating of 1, up to very good effectiveness (or very often occurred) given a rating of 5. The scale is unusual because it allowed either a quality or a frequency descriptor. The Fac used the dual scale to increase momentum in skill building because a frequency increase of a desired interaction likely occurs before skill is maximized (as is true in any type of coached skill). Table 2 shows the ITAT instrument, which is organized into two columns. A factor analysis of previous survey data explained elsewhere (9) indicated two categories of interactions in laboratory teaching existed. 1. Chemical Manager interactions concerned teaching about and managing procedural knowledge of equipment, techniques, instrument operation, and basic lab skills, which occurred through talks and interaction with guidance and specific advice. 2. Chemical Teacher interactions linked the lab to the lecture’s concepts, explained abstract chemical concepts in terms of the concrete equipment, techniques, and procedural steps in the lab and expected UGs to reason through the lab experiment rather than just finish it.

Table 3 outlines the links between the ITAT and UGATA instruments. The ITAT and UGATA are analogues, that is, they were both organized within the framework of the same 12 interactions important to manage a chemical laboratory and teach chemical concepts that inspire the lab activity. The relevance of these interactions on effective teaching originated with the literature on best practices and is explained elsewhere with the validation of content and the measurability of each interaction on the ITAT and UGATA (10). The Cronbach α reliability coefficient for the UGATA was 0.953 for the 16 questions on GTAs’ teaching. This indicated that the data for undergraduates’ responses to GTAs’ teaching skills are reliable and internally consistent for undergraduates’ responses on GTAs’ teaching in the laboratory. The reliability of the ITAT was also high (Cronbach’s α ⫽ 0.863). The UGATA has 16 questions rather than 12 because we added items when discrepancies were discovered; for example, UGs’ meanings of respectful help (item 4) included one definition, “respect for me as a person” (item 5), and also “respect for my knowledge and capability” (item 6 on the UGATA). Similarly, a second item on troubleshooting was added to include the GTA facilitating a team (item 7) or an individual (item 8) to reason through their problem. Another concerning whether the explanations given (item 14) were at the level of UG knowledge (item 16) was added to chemical explanations. To facilitate a valid comparison between the UGATA and ITAT ratings, we averaged Troubleshooting 1 and 2 into one item and Respect 1, Respect 2, and Helpful into one item, and chemical explanation items into one, thus, grouping the UGATA’s 16 items into the same 12 items on the ITAT. Table 3 shows the links between the UG and Fac instruments.

Table 2. Rubric for Instructor Coding of Teaching Assistants’ Teaching (ITAT) Instrument by Role of GTA Managing a Chemical Work Environment In this role GTAs are evaluated relative to actions as a responsive mentor and procedural guide in a specific and timely manner throughout the lab, that is, a chemical manager.

Teaching Chemical Concepts In this role GTAs are evaluated relative to actions as a responsive chemistry teacher who initiates conceptual explanations and linkages to lecture and prompts UGs to reason using concepts.

A1

Interaction: GTA moves regularly throughout the lab space to interact with all UGs rather than waiting for students to come to them.

B1

Advice: GTA’s comments to individual UGs or student groups are timely and meaningful for the tasks to be completed.

A2

Safety: GTA models safety rules and enforces them (goggles worn, disposal directions followed, proper clothing and shoes, etc.)

B2

Links Concepts with Lab: GTA links the lab tasks to the underlying chemical concepts from the lecture portion of the course.

A3

Respect/Help: GTA’s attitude is respectful of individuals personally and of their ability to learn when responding to UGs asking for help.

B3

Chemical Explanations: GTA explains chemical concepts in concrete conversations based on UGs’ general understanding and common prior experiences.

A4

Opening Talk: Focused, concise, clear, and primarily procedural, but with a conceptual overview that illustrates why this experiment is important and relevant to UGs

B4

Prompts Students To Reason: GTA prompts UGs to think about chemical concepts underlying the lab when they ask questions.

A5

Awareness: GTA notices UGs having difficulty and moves to interact with them.

B5

Facilitates Reasoning among Students: GTA stimulates and facilitates discussion among UGs of concepts underlying the lab.

A6

Guidance: GTA’s explanations of the experiment are given at a level reflective of UGs’ general understanding and prior experiences when interacting with them.

B6

Urges Reasoning in Troubleshooting: GTA encourages UGs to think through and correct problems when mistakes are made rather than the GTA fixing problems.

www.JCE.DivCHED.org

•


•


307

Research: Science and Education Table 3. Linking the Undergraduate Assessment of GTAs with the ITAT ITAT Link

UGATAa

Construct b

A4

01

Directions

A2

02

Safety

A1

03

Interaction

A3

04

Helpful

A3

05

Respect 1

My TA is respectful of me as a person.

A3

06

Respect 2

My TA is respectful of my knowledge and ability to learn.

B6

07

Trouble 1

My TA encourages us to discuss our procedural problems together as a team.

B6

08

Trouble 2

My TA encourages us to think through problems or mistakes rather than telling us what to do or doing it for us.

A6

09

Guidance

My TA’s comments on troubleshooting are relevant and at the level of my knowledge.

A5

10

Awareness

My TA helps me when I am having difficulty with the experiment, even if I do not ask.

B1

11

Advice

B2

12

Links Concepts

My TA links underlying chemical concepts from the lecture to the lab experiment to help me understand how they connect.

B4

13

Prompts Ss

My TA prompts me about underlying concepts of the experiment when I ask questions.

B3

14

Chemical Explanations

B5

15

Discussions of Chemistry

My TA stimulates us to discuss the concepts underlying the experiment with each other.

B3

16

Chemical Explanations

When my TA explains chemical concepts, s/he does so at the level of my knowledge.

aItem

Operational Definition of Construct My TA’s short talks are helpful to my understanding. My TA models safety and other lab rules and enforces them. My TA interacts with me throughout the lab. My TA helps me when I ask for help.

My TA’s comments are helpful to my work as the lab progresses.

My TA explains the chemical concepts underlying the experiment with me.

number from the Undergraduate Assessment of Teaching Assistant Teaching (UGATA) Instrument.

bThese

are the variables showing interaction.

Table 4. Absolute Likert Scale Ratings: Comparison of Faculty and Undergraduate Assessment of 14 GTAs Teaching Skills Rated by Role of GTA

Undergraduates’ Means (n = 245)

Instructors’ Means (n = 2)

t-Test Values

p Values for Significance of Difference a

4.36

4.34

᎑0.28

0.781

1. Interactions

4.57

4.77

᎑1.05

0.295

2. Safety

4.56

4.40

᎑0.90

0.372

3. Respect/Helpful

4.60

4.80

᎑1.10

0.272

4. Directions

4.12

4.07

᎑0.25

0.803

5. Awareness

3.83

3.67

᎑0.61

0.540

6. Guidance

4.18

4.43

᎑1.12

0.264

4.21

3.22

᎑5.11

0.000

1. Advice

4.20

4.27

᎑0.28

0.781

2. Troubleshooting

4.24

3.63

᎑2.45

0.027

3. Link to Experiment

4.14

2.97

᎑6.21

0.000

4. Prompt Thinking

4.16

2.67

᎑6.09

0.000

5. Chemical Explanations

4.16

3.30

᎑3.31

0.005

6. Chemical Talk

3.98

2.00

᎑8.95

0.000

4.22

3.75

᎑3.51

0.003

AA. Chemical Managing

AB. Teaching Chemical Concepts

AAll Chemistry Teaching

A p value of ⱕ .05 indicates a significant difference.

a

308


•


•

www.JCE.DivCHED.org


Results and Discussion GTA performance by the end of the fall semester was quite variable. Figure 1 is a graph of individual GTA summation ratings of all 12 interactions by the Fac (light symbols) and the UGs (dark symbols). The z scores are relative ratings that describe the number of standard deviations above or below the mean for all GTAs on one or a set of interactions. The mean for all GTAs is subtracted from the individual GTA’s mean, and the positive or negative difference is divided by the standard deviation among all GTA scores. The z score method was a standard way for us to relate variable performance among individuals experiencing very similar opportunities. The total z rating was calculated as the sum of each GTA’s scores as a Chemical Manager and as a Teacher of Chemical Concepts (last column in Tables 5 and 6), and is graphed in Figure 1. Figure 1 shows similarities and differences in the ratings of GTAs by UGs and the Fac. Data show that the general pattern for rating GTAs indicated by both lines was similar. Several discrepancies are also shown, which will be discussed. When teaching assistants’ students and faculty ratings coincided, Brooks et al. (2) believed there was an implied “fairness” that might not survive the test of a more thorough evaluation. Would legitimate differences occur between faculty and UG ratings of GTAs? Prior to the implementation of the remote audio-visual observation (RAVO) system, the agreement between the Fac and UG assessments of our GTAs was generally used as a sign of Fac “fairness,” particularly in interactions that were most difficult to discern when an outsider observed (many of these interactions were removed from the survey instruments). Starting in Fall 2002, we were able to test the Brooks et al. hypothesis because of the consistency of the video data and carry out a rigorous analysis on Fac and UG assessments of GTAs’ teaching in three undergraduate laboratory courses. The two-tailed t-test values and significance of difference (p) values between the Fac and UG mean ratings are reported in Table 4 for each interaction and for the composite variables of Chemical Managing and Teaching Chemical Concepts. The rest of the paper compares and contrasts UG and Fac ratings of these new GTAs.

Likert Ratings by UGs and Fac Table 4 illustrates Likert mean ratings, which were based on perceived quality or frequency criteria of each interaction; thus, criterion rating. Two composite variables, chemical management and chemical teaching, were created. The specific components of each are also shown in Table 4.

Significant Results: Similarities and Differences Student’s t-tests were calculated for the mean differences between category ratings to determine whether the differences were statistically significant or due to chance variation. The UGs rated chemical managing very similarly to chemical teaching, yielding a value of 4.36 for management versus 4.21 for teaching. The ratings indicate good effectiveness of interactions and a frequency of often. The Fac rated the GTAs’

www.JCE.DivCHED.org

•

interactions of management and those of conceptual teaching significantly differently (4.34 versus 3.22). UGs and Fac rated all interactions involved in chemical management very similarly (4.34 versus 4.36). This overall rating meant that the quality and frequency of management interactions were done well and or often. Both UG and Fac rated highest the helpfulness and respect for UGs as individuals and as capable of learning, and the level of interaction with UGs throughout the lab period. Both Fac and UG rated awareness of the whole lab and the opening talk as the weakest parts of management; both rated the opening talk lowest, indicating mediocre effectiveness. UGs rated all interactions that taught chemical concepts as good or often done (4.21), whereas the Fac rated the observed interactions significantly lower in quality and frequency of concept teaching: 3.22, meaning mediocre in effectiveness or sometimes attempted. The Fc rated all five teaching interactions significantly lower: See interactions B2B6 rated at 3.63, 2.97, 2.67, 3.30, and 2.00, respectively. B1, advice, is different from the other teaching interactions because it contains components of both managing and conceptual teaching according to our factor analysis (9).

Performance Continua A continuum of individual GTA performances can be absolute (criterion-based) as on the Likert scale, or relative (normed to a population of peer GTAs) as with calculated z scores. Both show variation among GTAs. Tables 5 and 6 show mean Likert ratings; as in Figure 1 both tables use z scores. Table 5 shows the UGs’ ratings; here they are compiled as composite categories of Manager, Teacher, or Overall performance. Over 83% of the GTAs were rated at good effectiveness in managing and procedural guidance. Two of the GTAs were rated one standard deviation or more below the mean on management by their UGs, indicating that UGs note when management is mediocre. Very similarly to procedural chemical management, nearly all UGs rated their GTA as good in effectiveness for linking and explaining concepts from lecture. Table 6 shows the Fac ratings of the same GTAs. The data in Table 6 demonstrate that the Fac also rated chemical management as good for nearly all GTAs. The Fac likewise rated GTAs higher in management than in teaching concepts. The z score calculation was a way to use rankings, allowing us to compare UG (Table 5) and Fac ratings (Table 6), which had different means on many interactions. The rankings show that the Fac and UGs judge the criteria similarly for effective managing. The same cannot be said for conceptual teaching. Criteria for effectiveness between UGs and Fc do not appear to match in a linear fashion in concept teaching as they do for chemical management.

Disparities in Concept Teaching Our data indicates that legitimate differences can exist between UGs and Fac assessments of a GTA’s teaching. The disparity results from connotations of the meaning of types of teaching interactions as well as differences in the interpretation of quality.


•


309


Connotations of Procedural and Conceptual Teaching Disparities may be due to differences in the connotations ascribed to specific kinds of interactions; for example, the interactions in which GTAs linked lecture concepts with

labs, explained lecture concepts, and pushed their UGs to reason. We suspect that UGs might consider procedural talk equivalent to talking about chemical concepts that underlie the lab procedure. The procedural portion of the lab is very important for UGs because they want to finish, to have the

Table 5. Undergraduate Assessment Means and Calculated z Scores of Graduate Teaching Assistants, by Role Absolute Means a

Relative z Score

GTA as Chemical Manager

GTA as Teacher


GTA as Teacher

Total z Scores c

16

4.69

4.70

᎑0.51

᎑0.68

᎑1.20

10

4.72

4.61

᎑0.55

᎑0.56

᎑1.12

12

4.58

4.47

᎑0.33

᎑0.36

᎑0.70

11

4.57

4.38

᎑0.32

᎑0.25

᎑0.57

15

4.40

4.45

᎑0.15

᎑0.33

᎑0.49

17

4.52

4.36

᎑0.25

᎑0.21

᎑0.46

13

4.43

4.38

᎑0.11

᎑0.24

᎑0.35

19

4.49

4.05

᎑0.19

᎑0.21

᎑0.02

12

4.17

3.98

᎑0.28

᎑0.30

᎑0.60

11

4.10

3.84

᎑0.39

᎑0.50

᎑0.91

18

3.64

3.55

᎑1.11

᎑0.90

᎑2.02

14

3.28

2.89

᎑1.65

᎑1.83

᎑3.49

GTA ID

b

a

Overall mean values are 4.36 and 4.21 for Lab Manager and Teacher roles, respectively; the standard deviation values are 0.65 and 0.71, respectively. bOrganic chemistry and general chemistry GTAs are included in this population. cThe z values are presented in rank order.

Table 6. Faculty Assessment Means and Calculated z Scores of Graduate Teaching Assistants, by Role Absolute Means a

Relative z Score


GTA as Teacher


GTA as Teacher

Total z Scores c

11

4.86

4.42

᎑1.08

᎑1.82

᎑2.90

15

4.50

3.67

᎑0.33

᎑0.68

᎑1.07

16

4.64

3.42

᎑0.63

᎑0.30

᎑0.92

12

4.43

3.67

᎑0.19

᎑0.68

᎑0.87

19

4.50

3.50

᎑0.33

᎑0.42

᎑0.75

17

4.57

3.33

᎑0.48

᎑0.17

᎑0.64

13

4.43

3.33

᎑0.19

᎑0.17

᎑0.35

10

4.32

3.08

᎑0.04

᎑0.21

᎑0.25

11

4.14

3.17

᎑0.42

᎑0.08

᎑0.49

18

4.57

2.58

᎑0.48

᎑0.97

᎑0.49

12

3.86

3.17

᎑1.00

᎑0.08

᎑1.08

14

3.10

2.00

᎑2.00

᎑1.73

᎑4.22

GTA ID

b

aOverall mean values are 4.34 and 3.22 for Lab Manager and Teacher roles, respectively; the standard deviation values are 0.48 and 0.66, respectively. bOrganic chemistry and general chemistry GTAs are included in this population. cThe z values are presented in rank order.

310


•


•

www.JCE.DivCHED.org


correct data, and to be able to write a lab report. When UGs were given relevant procedural steps or specific help to solve a procedural problem, we observed that they were satisfied with this teaching. The rated interactions of GTA 11 in Table 6 provide an example of differing connotations. The Fac realized that this GTA was superlative in integrating concepts and getting students to talk about them. GTA 11 markedly and consistently worked at the level of teaching concepts with small groups of UGs. As the lab progressed, this GTA became more of a conceptual teacher since students had data or other results they needed to decipher. The Fac z score was ⫹1.80, whereas the UGs’ z score for GTA 11 was only ⫹0.25. Without seeing variability of several GTAs, the UGs may be unable to identify a high-performing conceptual teacher. The UGs might still have trouble recognizing exemplary concept teaching even after experiencing varying GTA performance, which is discussed next.

Meaning of Quality Indicators Differences in the meaning of quality indicators used on the survey scale may explain disparity between Fac and UG ratings, especially disparities in recognizing interactions linking lecture concepts with labs, explaining them at the level of UG knowledge, and expecting their UGs to reason. Although the Fac and UG used the same Likert scale (from poor to very good ) to assess GTAs, the scales are not equivalent when the Fac’s idea of good was not identical to the UG notion of good. Disparities in such scale meanings are not uncommon: A valid survey can show quite different Likert response patterns when tested with different populations of subjects. Reasons for different meanings in quality indicators are differing knowledge bases and differing values and expectations. Knowledge differences may account for interpretation of meaning of quality. For example, the Fac were more knowledgeable in GC, OC, and AC than were the UGs doing the lab. The Fac understood the procedural concepts involved in skills, measurement, techniques, and instrument operation. Further, the Fac knew the fundamental chemistry concepts that made the lab experiment work the way it did. Therefore, the Fac could discern missing elements or missing connections in GTAs’ teaching, whether they were procedural or conceptual. Expectation differences engender differing meanings of quality in teaching interactions. The Fac expected GTAs to look for opportunities to provide links and explanations in the experiments. The Fac had provided guidance so that GTAs had resources to develop good questioning and interactive skills that pushed UGs to inquire about their lab work. In contrast to the Fac, finishing the procedural steps and getting the right data to write a report is a high-priority UG expectation, regardless of whether the lab is connected with an introductory or advanced lab course. Generally, UG goals are met by procedural teaching, so this is their expectation. Differences in the meaning of quality depend on what is valued. What is considered to be a good practice by UGs may be different from the Fac idea of what a good practice is. The Fac had a very high set of procedural and conceptual standards for the GTA. These standards were the interactions coded on the ITAT form, the very standards for which GTAs had been trained in formative assessments, seminar discuswww.JCE.DivCHED.org

•

sion, and video collages of exemplary lab teaching. Pushing UGs to reason may not be considered as good to UGs as it does to the Fac. The UGs may value short-term help getting correct answers more than they value longer-term learning, in contrast to Fac values and expectations, for example. In summary, we suspect that the goals of UGs reinforce the procedural emphasis of most laboratory TAs. Many UG goals are short-term project completion rather than long-term chemical understanding. While many chemistry faculty members may realize that current lab teaching is primarily procedural, it is still noteworthy that concepts of chemistry were rarely discussed in many chemistry lab sessions at our institution and in the literature reviews of laboratory work (7, 10). Conclusions The Fac found that most GTAs were not strong in integrating chemistry knowledge from the lecture into the lab procedures in terms of linking concepts, making explanations, pushing students to reason using abstract classroom concepts underlying the lab, and getting UGs to talk chemistry—even with training and coaching to do so. (See ref 3 about successful increases in conceptual teaching by a majority of GTAs.) In this cohort of new graduate students only one GTA took the training and deliberately incorporated it: GTA 11. This GTA taught chemical concepts well and attempted to teach them often, most notably closer to the end of the lab session as groups were finishing. These results concerning chemistry teaching imply that linking abstract concepts that underlie the lab with the concrete work that UGs are doing in the lab is not internalized by most GTAs. Why? This is an important question to investigate further. This question clearly connects to others’ questions about whether the laboratory is worth the time and expense if it does not teach chemical concepts (11). A second question to investigate is the relationship between the conceptual teaching interactions of UGs with their GTAs and the learning gained by UGs. Many variables are associated with the achievement of UGs in chemistry; thus, measuring performance only by course exams is unlikely to have enough power to detect a difference due to conceptual teaching by the GTA once per week. A conceptual exam must be built with items that closely relate to the conceptual understanding that we believe should develop from experiences in GC, OC, or AC labs. Then the exam items’ specificity can provide enough power to find significant increases in achievement for UGs lucky enough to be in a high-performing GTA’s lab section. We hope to pursue this important link from GTA conceptual teaching to UG learning in the near future.

Implications: Superlative TAs May Not Get Recognition If science departments and the faculty utilizing specific GTAs have an expectation that these GTAs will be teaching concepts to UGs that support the lecture portion of the course, this goal must be explicit; otherwise GTAs focus on procedural management. When highly effective GTAs do teach conceptual concepts associated with the lab experiments, we have shown that most UG assessments cannot discriminate between procedural and conceptual teaching. Thus, these highly effective GTAs will not have higher ratings as Table 5 illustrates for GTA 11. In summary, the shorter-term


•


311


procedural teaching that UGs want to complete a chemistry lab may supersede looking for any other kind of knowledge from their GTAs. If student evaluations do not discriminate finely enough between short-term and long-term benefits, then student assessment cannot tell the whole story about teaching effectiveness. Ideally, teaching effectiveness reveals itself in longer-term effects, such as understanding how concepts in chemistry fit together and the kinds of applications these concepts reveal themselves in. Superlative TAs, those especially effective in terms of their ability to teach the underlying concepts of the lab activity, will likely not stand out from other GTAs based on UG assessments. Therefore, superlative GTAs may not get the recognition they deserve unless explicit Fac assessments of chemistry teaching frequently take place. Acknowledgements We extend our appreciation to the National Science Foundation (NSF Career Grant Number 0093319 to BondRobinson) for funding the most recent research on the development of pedagogical chemical knowledge in graduate teaching assistants.

312


•

Literature Cited 1. Levenson, H.; Brooks, D. W. Journal of College Science Teaching 1975, 5 (2), 85. 2. Brooks, D. W.; Kelter, P. B.; Tipton, T. J. J. Chem. Educ. 1980, 57, 294. 3. Bond-Robinson, J.; Rodriques, R. B. J. Chem. Educ. 2006, 83, 313–323. 4. Robinson, J. Bond. Journal of Graduate Teaching Assistant Development, 2000, 7 (3), 147. 5. Del Carlo, D. I.; Bodner, G. J. Res. Sci. Teach. 2004, 41 (1), 47. 6. Blumer, H. The American Journal of Sociology 1931, 36, 515. 7. Brooks, D. W.; Levenson, H. J. Chemistry Education 1974, 51, 161. 8. Abraham, M. R.; Cracolice, M. S.; Graves, A. P.; Aldhamash, A. H.; Kihega, J. G.; Palma Gil, J. G.; Varguese, V. J. Chem. Educ. 1997, 74, 591. 9. Bond-Robinson, J. Chem. Educ. Res. Pract. 2005, 6, 83–103. 10. Bond-Robinson, J.; Rodriques, R. B. Chem. Educator 2005, 10, 1–9. 11. Hilosky, A.; Sutman, F.; Schmuckler, J. J. Chem. Educ. 1998, 75, 100.


•

www.JCE.DivCHED.org

Comparing Faculty and Student Perspectives of Graduate Teaching

Recommend Documents