Article pubs.acs.org/jchemeduc
Target Inquiry: Helping Teachers Use a Research Experience To Transform Their Teaching Practices Deborah G. Herrington,*,† Karen Luxford,† and Ellen J. Yezierski‡ †
Department of Chemistry, Grand Valley State University, Allendale, Michigan 49401, United States Department of Chemistry and Biochemistry, Miami University, Oxford, Ohio 45056, United States
‡
S Supporting Information *
ABSTRACT: Research experiences for teachers (RET) programs report benefits to teachers and students. RET programs aim to give teachers authentic science research experiences based on the premise that these experiences will improve science instruction. Specifically, many programs require teacher development of lessons and units to help translate the teachers’ research experience into classroom practice. However, the design and implementation of activities incorporating proven, inquiry-based practices called for in national science education reform documents require that teachers also understand contemporary models of teaching and learning. Using a lesson plan analysis instrument producing scores found to be reliable and valid in measuring teachers’ instructional approaches, we compared teacher-developed lessons from traditional RET programs with those developed in Target Inquiry, a program that supplements research experiences for teachers with a materials development experience. Traditional RET lessons scored significantly lower, suggesting that supplementing RET programs with focused materials development is valuable for promoting instructional reform. KEYWORDS: Graduate Education/Research, High School/Introductory Chemistry, Chemical Education Research, Professional Development, Student-Centered Learning FEATURE: Chemical Education Research
■
INTRODUCTION AND RATIONALE In addressing the National Academy of Sciences, National Academy of Engineering, and Institute of Medicine, President Obama said, “Science is more essential for our prosperity, our security, our health, our environment, and our quality of life than it has ever been before”.1 It is imperative that we encourage and prepare students to pursue careers in science, and yet our education system does a tremendous job of turning students away from science.2 Across the country, science is primarily taught as a set of facts that students memorize and regurgitate on an exam or standardized test; it is not viewed as an ever-changing body of knowledge or a way to investigate the world.3 This disconnect between how science is done by practicing scientists and how it is taught in schools may stem from teachers’ lack of experience with authentic science research.4,5 Research experiences for teachers (RET) programs across the country are founded on the premise that “experience in the practice of science improves the quality and authenticity of science teaching and thereby increases student interest and achievement in science”.6 Numerous studies have shown that RET programs can have positive effects on teachers and their students.6−13 Most notable is the recent study published in Science, documenting that students of teachers who participated in RET performed significantly better on state content examinations than students of nonparticipating teachers.6 Most RET programs articulate the goal of helping teachers translate what they learn in research to classroom instruction.14 Thus, most RET opportunities include activities directed toward this goal, such as having teachers develop lessons or © 2012 American Chemical Society and Division of Chemical Education, Inc.
instructional units. Even the National Science Foundation (NSF), which funds RETs for individual principal investigators (PIs) and teachers, expects that RET programs will “lead to transfer of new knowledge to classroom activities”, and requires sustained follow-up activities to aid in this transfer.15 An examination of Web site descriptions of formalized and recurring RET programs in the United States and a report from a 2005 conference on teacher research experiences14 indicates that many programs require teacher development of lesson or unit plans; however, the level of support and amount of time devoted to this goal varies greatly among RET programs. RET program goals and supporting activities designed to connect teacher learning about science to classroom practice are encouraging. However, research tells us that developing and implementing lessons that more accurately model scientific inquiry for studentsallowing students to investigate phenomena, construct conceptual understanding, and develop an understanding of the tentative and experimental nature of sciencedemands that teachers understand contemporary models of teaching and learning.16 In addition to gaining a better grasp of how scientific knowledge is constructed, teachers also need knowledge of proven teaching practices and quality teaching materials to substantially reform their instruction.7,17−19 Given this, we predicted that supplementing an RET program with a rigorous materials development component would result in more substantial instructional changes. This led Published: January 13, 2012 442
dx.doi.org/10.1021/ed1006458 | J. Chem. Educ. 2012, 89, 442−448
Journal of Chemical Education
Article
Table 1. Program Features in RET Study Sample RET
Development of Inquiry Materials
1 (TI) 2
Teachers meet as a group weekly during RET 1 wk content and pedagogy workshop during RET
Yes Yes
2 1
3 4 5 6
During summer after RET Academic year following RET with some individual mentoring Focus of RET is development of inquiry activity During RET Last week of RET with some mentoring Last week of RET
None indicated None indicated Curriculum development workshop last week of RET None indicated
None indicated None indicated Yes None indicated
7
During RET, not sure when
Yes
8
During 2nd summer in collaboration with other teachers During summer RET During RET with some mentoring (at least one meeting)
One day per week participants meet for content/ pedagogy talks/discussions Teachers meet as group during RET
1 1 1 Second summer possible 1
9 10
Seminars/Workshops To Supplement RET
None indicated Weekly seminarsnot sure of focus
AY Follow-Up
3 group meetings/year None indicated None indicated
No. of Summers
2 required 1 2 required
Figure 1. Progression of courses in TI program. Burgundy = RET; brown = Materials Development; green = Action Research.
■
RET PROGRAM OVERVIEWS We are not suggesting that RET programs are not useful means of professional development. We do suspect, though, that the effects of such programs on teacher practices can be considerably enhanced by coupling carefully structured RET programs with rigorous and supported materials development experiences. As such, we have not identified individual RET programs outside of the TI program; rather this section describes the general structure of the remaining RET programs in this study ascertainable from their Web site descriptions.
to the development of the Target Inquiry (TI) professional development program, as well as the study described in this paper which aims to address the question: Can supplementing an RET experience with a rigorous materials development experience result in classroom chemistry activities more aligned with the proven practices called for by national science education reform documents? The inquiry focus of the TI program and results of this study should interest readers, particularly as numerous articles focused on inquiry instruction have been published in this Journal. An “inquiry” keyword search spanning two years, from May 2009 to May 2011, yielded 203 articles published in the Journal20 ranging from commentary,21 program description,22 and research,23 to high school,24 introductory college,25 and advanced college26 laboratory experiments. As observations of the hundreds of teachers across the country who have participated in RET programs were not possible, to answer our question we sought to systematically analyze instructional materials developed by teachers participating in RET programs and posted on RET Web sites. In this study, we analyzed only teacher-developed chemistry lessons because, to date, the TI program has enrolled only chemistry teachers. As these lessons were publically available, presumably for other teachers to use, we expected these materials to provide adequate detail for lesson implementation. Furthermore, if RET programs are transforming teaching practices, we would expect to see these lessons incorporate reformed, inquiry-based teaching practices called for in the National Science Education Standards (NSES).27
General RET Program Description
The chemistry lesson plans considered for this study were drawn from 10 RET programs that generally involved a summer research experience of six−eight weeks in which teachers were mentored by faculty, graduate students, or postdoctoral students. Noteworthy differences exist among the programs in terms of the level of support and time allocated for developing classroom materials related to the RET. These differences are described in Table 1. TI Program Description
The TI professional development model is built on three core experiences: (i) RET; (ii) materials development; and (iii) action research.28 This model was then translated into seven graduate courses taken in sequence as shown in Figure 1. The program begins with the RET component, which incorporates the features of effective RET programs. In addition to the actual research project, this component includes a pre-RET course (CHM 610) that prepares teachers to 443
dx.doi.org/10.1021/ed1006458 | J. Chem. Educ. 2012, 89, 442−448
Journal of Chemical Education
Article
Table 2. Summary of RET Programs in Databasea
conduct research and assist with teacher−research faculty matches, and a companion course (CHM 612), running concurrently with the research experience (CHM 611), that facilitates teacher reflection on research activities and connections to the high school classroom. Teachers also present their research results at a regional or national meeting at the conclusion of CHM 611.14 During the six-week CHM 611, teachers work closely with a chemistry faculty member on their research project. To facilitate lasting instructional reform and prepare for the materials development component of the program, during the following academic year, teachers take a course that immerses them in the chemistry education literature focusing on both teaching and learning of chemistry and chemistry education research methods (CHM 621). During the following summer, facilitated by chemistry education specialists, teachers then build upon their RET experience and their new understanding of chemistry teaching and learning from the chemistry education research literature to develop classroom materials that more accurately model how science is done (CHM 631). They pilot these materials with their peers, implement the materials in the classroom the subsequent academic year, and study how the materials affect student learning (CHM 632). Finally, teachers write up the results of their action research and submit the papers for publication (CHM 633). Student-centered teaching methods are modeled throughout the program because we believe, as Beisenherz and Dantonio stated,29 [T]eachers cannot be lectured at, demonstrated to, and asked to regurgitate facts in course after course, semester after semester, and then be expected to teach the processes of science without having experienced them. As TI is a cohort program, collaborations among teachers, chemistry faculty, and chemistry and science education faculty provide the necessary social structure for teachers to coconstruct meaning through negotiations with their peers and “experts”. Teachers also routinely reflect on the connections between their research experiences, their classroom practices, and education theory and research, providing the necessary foundation for changes in beliefs about teaching, learning, and science.
■
Salient Program Elements Number of RET programs requiring lesson/unit development Number of RET programs with lessons posted on Web site
Yes
No
May/Not Sure
Total
85
18
14
117
30
87
0
117
a
Note: At the time the database was established, some RET Web sites did not provide enough information to determine whether teachers were required to develop lesson or unit plans as part of the program. “May” means teachers have an option of materials development; “Not Sure” denotes some mention of curriculum/lesson materials but no indication of teacher development.
Table 3. Level Descriptors for Continuum of Science Inquiry Instrument Level
Descriptor
0
Problem area, methods of solution and “correct” interpretations are given or are immediately obvious from either statements or questions in the students’ laboratory manual or textbook. Includes activities in which students simply observe or “experience” some unfamiliar phenomena or learn to master a particular laboratory technique. Laboratory manual proposes problems and describes ways and means by which the student can discover relationships he or she does not already know from manuals and texts. Problems are provided, but methods as well as solutions are left open. Problems, as well as solutions and methods, are left open. The student is confronted with the “raw” phenomenon.
1
2 3
The CSI, based on the levels of openness framework,34−36 is a four-level continuum used to assess the extent to which students have the freedom to make choices with respect to the problem investigated, the methods used, and the construction of the solution. As this continuum was developed and used by Lederman to analyze high school science laboratories, it was chosen for this study to aid in lesson sampling and to help establish concurrent validity of the scores obtained using the second instrument, the Science Lesson Plan Analysis Instrument Revised (SLPAIR).37 Instrument 2: Science Lesson Plan Analysis Instrument Revised
Although the ideal measure of teaching practice is classroom observation, the resources required made this largely impossible for this study. Using lesson plans as a measure of teaching practices allows for the evaluation of larger portions of planned instruction for more teachers. The SLPAIR was adapted from the Science Lesson Plan Analysis Instrument (SLPAI).37 The SLPAI was designed to help researchers and evaluators of science education professional development programs measure changes in teachers’ practices. The SLPAI was chosen as it is aligned with the NSES science teaching standards,27 which focus on inquiry-based science instruction. Lesson plan scores obtained by instrument developers using the SLPAI were validated through triangulation with other measures of teaching practice: a teacher questionnaire, the Standards-Based Teaching Practice Questionnaire,38 and classroom observation analysis using the Reformed Teaching Observation Protocol.39 Each of the items on the SLPAI is weighted to add flexibility to the instrument to reflect the goals of a particular program. As the broad goals of the institute used in the development and piloting of the SLPAI were aligned with inquiry-based practices, the original items’ weights were retained for this study. However, some of the items in the SLPAI reflected the finergrained goals of the institute. Thus, the items Awareness of
METHODS
Selecting RET Programs for Study
A database of formalized RET programs30 in the United States was constructed using the Pathways to Science database,31 the RET network database,32 and an Internet search of RET programs to locate any programs not included in either of these two extensive listings. Table 2 summarizes the information from our RET program database (the complete database can be found in the online Supporting Information). Each RET program Web site was reviewed to determine whether: (i) teachers developed classroom activities; (ii) activities were available online; and (iii) activities included chemistry lessons. A total of 30 met the first two criteria (Table 2); 10 of the 30 RET Web sites met all three selection criteria and were selected for the study (Table 1). Instrument 1: Continuum of Science Inquiry
All chemistry lessons from each of the 10 identified RET Web sites were first analyzed using the Continuum of Science Inquiry (CSI) instrument33 (Table 3). 444
dx.doi.org/10.1021/ed1006458 | J. Chem. Educ. 2012, 89, 442−448
Journal of Chemical Education
Article
Table 4. CSI and SLPAIR Distributions for RET Sites with Teacher-Developed Chemistry Materials Inquiry Level, % (CSI)
Lesson RET Scores (SLPAIR).
RET Sites
Lessons Posted, N
Lessons Rated, N
0
1
2
3
Average
Std Dev
1 (TI) 2 3 4 5 6 7 8 9 10
33 4 5 1 1 26 8 5 20 87
24 4 5 1 1 6 8 5 6 18
0.0 0.0 40.0 0.0 0.0 44.5 62.5 60.0 55.0 80.0
44.0 75.0 60.0 0.0 100.0 50.0 25.0 20.0 45.0 16.7
56.0 25.0 0.0 100.0 0.0 5.5 12.5 20.0 0.0 3.3
0 0 0 0 0 0 0 0 0 0
81.7 61.8 54.9 48.3 42.1 34.8 30.3 29.9 29.1 27.6
3.3 4.6 6.3 n/a n/a 21.1 13.6 16.8 5.9 13.0
scores would result in higher scores on the SLPAIR. A positive correlation between the CSI and SLPAIR scores provides evidence of concurrent validity of the SLPAIR scores and supports the sampling methods (Figure 2).
Science Education Research, Equity, Students’ Attitudes about Science, and Nature of Science were omitted from the SLPAIR. Though these omitted items may seem relevant to evaluating the quality of lessons employing inquiry instruction (particularly Nature of Science), a closer look at the criteria for such items (see the online Supporting Information) calls for features not commonly found in high school chemistry lesson plans. The SLPAI was designed to evaluate longer portions of instruction (approximately five days in length) including copies of assignments, handouts, laboratory activities, and assessments, as well as graded student work if available. Because most of the lessons posted on RET Web sites were designed as one-day activities as opposed to weeklong units, the Variety and Student Engagement items were also omitted from the SLPAIR. The items that made up the SLPAIR and the omitted items from the original SLPAI along with the relative weights of the items are found in the online Supporting Information. Raw SLPAIR scores were converted to percentages. Lesson Sampling
For RET programs with six or fewer chemistry lessons posted, all lessons were scored using the SLPAIR. For those that had more than six chemistry lessons, 20% of the lessons posted with a minimum of six lessons per RET were selected. Lessons were chosen using stratified random sampling with strata defined by the CSI level (0, 1, 2, or 3) and the year in which the lesson was developed. For example, if an RET Web site had 20 lessons with CSI levels of 0 (20%), 1 (60%), 2 (20%), and 3 (0%), six lessons were selected to score: one at level 0, four at level 1, and one at level 2, ensuring that lessons from each year of the RET program were included in the sample. Table 4 summarizes the CSI distributions for each RET site in this study. For RET 1 (TI) more than 20% of the lessons were scored with the SPLAIR, as we began to use this instrument to evaluate the quality of the teacher-developed lessons in our program prior to this study. To ensure that the oversampling of these activities would not introduce bias, we compared a subset of the lessons, sampled using the sampling methods described above, to the entire set after the lessons were scored with the SPLAIR. Using two, one-tailed t-tests to test for equivalence according to Schuirmann’s method published in this Journal,40 we found the two samples to be equivalent.
Figure 2. Correlation of lesson plan CSI and SLPAIR scores. A correlation coefficient was computed to determine the relationship, if any, between CSI levels and SLPAIR scores. The Spearman’s rank correlation between CSI levels and SLPAIR scores for each of the lessons rated was found to be significant, ρ(74) = 0.788, p < 0.001. The results show that higher median SLPAIR scores are associated with higher levels of inquiry, as indicated by the CSI value.
Reliability of CSI and SLPAIR Scores
To establish inter-rater reliability for the CSI, the levels (0, 1, 2, or 3) for 56 lessons were determined independently by two of the authors. These ratings were in agreement for 53 of the lessons (95%). Furthermore, the level descriptors in Table 3 were provided to three external raters who rated 20% (12/56) of these lessons with a 73% agreement. As the external interrater agreement was above 70%,41 the CSI ratings were used as a sampling criterion. The SLPAI developers established reliability for the scores through independent double scoring of a sample of the lessons. Lessons were independently scored by two developers out of 100 points and the average difference in their scores was found to be 4/100, indicating an average agreement of 96%, based on 10 lesson plans. The agreement with an external trained rater was 89% based on 8 lessons. For our study, 25% of the lessons rated were independently scored by two of the authors with an average agreement of 96%. Three additional raters, trained in
Validity of the SLPAIR Scores
As the NSES teaching standards call for inquiry-based teaching methods in which teachers guide students in active scientific inquiry and focus on student understanding and use of scientific knowledge, ideas, and inquiry processes,27 it was reasonable to expect that, for lessons of similar quality, those with higher CSI 445
dx.doi.org/10.1021/ed1006458 | J. Chem. Educ. 2012, 89, 442−448
Journal of Chemical Education
Article
Follow-up Mann−Whitney U tests were conducted to evaluate pairwise differences among 8 of the 10 RET sites, controlling for Type I error across tests by using the Holm’s sequential Bonferroni procedure. Sites 4 and 5 were excluded because they each had only one SLPAIR score. The results indicated the typical SLPAIR score for RET site 1 (TI program) was significantly greater than those of each of the remaining sites, suggesting that supplementing an RET program with a materials development experience greatly improves the alignment of teacher-developed lessons with inquiry-based practices. Results also show that SLPAIR scores for RET sites 2 and 3, which included faculty-supported materials development separate from the RET summer (site 2) or focused on materials development as opposed to a research experience (site 3), were significantly greater than some of the other RET sites (Table 5).
chemistry education research, but not associated with the TI program, were trained to use the SLPAIR instrument and asked to rate 6−7 lessons each. For these “blind” ratings, lessons from each of the RET programs and representing the range of SLPAIR scores were chosen and stripped of formatting and identifying information. The average inter-rater agreement between the blind raters and the TI team was 87%. This interrater agreement among authors and between authors and external trained raters was consistent with the SLPAI developers’ prior work.
■
RESULTS A summary of the CSI and SLPAIR scores for each RET site studied are shown in Table 4. A box plot of SLPAIR scores for the 10 RET programs included in this study (Figure 3)
■
DISCUSSION AND CONCLUSIONS As the SLPAIR instrument was designed in alignment with the NSES science teaching standards’ description of inquiry instruction and classrooms, our data show that many of the chemistry lessons developed and posted by chemistry teachers as part of traditional RET programs do not meet these standards. However, some limitations of the study should be noted. First, many of the lessons provided little detail regarding the role of the teacher or sample student work, which limited our abilities to assess teacher enactment of the lessons. Although it is certainly true that the implementation of the same lesson plan by two different teachers can look completely different, the SLPAIR instrument places high value on activities in which students investigate, discuss, and reflect on phenomena, and construct conceptual understanding. Activities that simply introduce students to a new technology or new area of chemistry with no concept development requirement, provide cookbook instructions for students to work with new materials, or put a real-world context around a traditional verification activity do not address these criteria. An example of how two activities addressing the same topic scored differently on the SLPAIR is included in the online Supporting Information. Furthermore, as these activities were posted on RET program Web sites as examples of lessons created by the teachers based on their research experience, one should expect they would reflect how the teacher−designers would implement them in their own classrooms. This is in contrast to less formal lesson plans that teachers keep to themselves, often based on activities they find, annotate, and adapt.
Figure 3. Distribution of chemistry lesson SLPAIR scores by RET program. Boxes are bounded by first and third quartiles and banded at the median. Whiskers indicate the maximum and minimum for each RET site’s data set. Outliers are indicated by the open circles.
illustrates the wide distribution of scores and shows that the median lesson score for most RET programs is below 50%. A Kruskal−Wallis test, conducted to evaluate differences among the 10 RET sites based on median SLPAIR scores and corrected for tied ranks, was significant χ2 (9, N = 76) = 58.50, p < 0.001. The proportion of variability in the ranked SLPAIR score accounted for by the RET site was 0.78, indicating a very strong relationship between RET site and SLPAIR score. This suggests that teacher-developed chemistry lessons from some RET programs show better alignment with proven teaching practices.
Table 5. Comparison of Pairwise Differences among RET Sites with Two or More SLPAIR Scores p Values for Mann−Whitney U Follow-Up Testsa RET Sites 1 2 3 6 7 8 9
Site 2
