Effectiveness of a Workshop To Encourage Action: Evaluation from a

Research: Science and Education

W

Effectiveness of a Workshop To Encourage Action: Evaluation from a Post-Workshop Survey Scott E. Lewis and Jennifer E. Lewis* Department of Chemistry, University of South Florida, Tampa, FL 33620; *[email protected]

This article is an evaluation, based on a post-workshop survey, of the effectiveness of a set of workshops to promote curricular reform. Since workshops are a common vehicle for disseminating curricular innovations among college chemistry faculty and reports from post-workshop surveys of this population are rare (with the notable exception of ref 1), we hope that this article will be joined by subsequent reports from other workshop dissemination efforts, enabling us to continue to develop our understanding of workshop effectiveness. We believe that understanding workshop effectiveness as it relates to college chemistry faculty is a critical component of ongoing curricular reform. We begin by giving a brief description of the curricular reform project and of the workshops, and then discuss the effectiveness of the workshops. Curricular Reform: ChemConnections Anthony et al. (2) described the work of the ChemLinks Coalition and the Modular Chemistry Consortium, known collectively as ChemConnections, in developing and testing topical modules for the first two years of college chemistry. In short, each topical module revolves around an interesting socially relevant question such as “How can we make our water safe to drink?” (3). The interweaving of social context and chemistry content is generally accepted as one appropriate means to engage students to learn chemistry, as can be seen by the existence of the American Chemical Society-sponsored texts Chemistry in the Community (ChemCom, for high school) (4), Chemistry in Context (for nonscience majors) (5), and Chemistry (for general chemistry, in beta-testing during 2003) (6). Two important differences between the modules and these texts are that the modules feature embedded active and collaborative pedagogies and that the modules can be used with a companion (usually more traditional) text to produce a course that is tailored to match a particular instructor’s desire for balance between traditional and nontraditional content. One module takes 3–5 weeks to pursue fully, and students develop an understanding of chemistry by engaging in module activities and textbook readings that provide the chemistry content necessary for addressing the socially relevant question.1 During this topical module-based exploration of chemistry, students work in collaborative classroom activities and engage in inquiry-based laboratory projects as a means for collecting data, sharing information, exploring phenomena, and suggesting and evaluating explanations. Workshops: Dissemination of ChemConnections To disseminate information about the modules, a variety of workshops were offered from 1997–2000 at locations around the country. These workshops, planned and designed www.JCE.DivCHED.org

•

solely by the ChemConnections project, form the basis for the evaluation. Overall, the intent of the workshops was to provide faculty with enough information about and experience with the modules that they could consider using them in their own classes.2,3 Data Collection and Methods An email survey of the 216 attendees of the 16 workshops was undertaken after the workshops had all been delivered. The delayed survey was intended to allow time for instructor implementation of the modules and is in keeping with recommendations for workshop evaluation (8). A total of 106 workshop attendees responded to the survey;4 however 7 of respondents indicated that they either did not teach or were no longer teaching and were not considered in the analysis. Thus, 99 respondents served as the focus of the analysis. The survey was intended to determine whether the recipients had used modules (module uptake) or intended to use them in the future and what factors influenced this decision. For those using modules, the survey was designed to determine in what way the instructor had implemented the modules, whether the instructor had found the material effective for teaching chemistry, and if there had been any difficulties in implementation. To achieve these ends, the survey employed both open-ended and closed-ended questions. The survey responses were coded for analysis using an open coding scheme (9) and the resulting data placed into SPSS 11.0 (10) for statistical analysis. Initial analysis of the data consisted of frequency counts of the responses to present an overall picture, including an identification of the rate of uptake for modules. Since further analysis was required to determine which factors affected module uptake, cross-tabulations were frequently used to identify potentially important variables. Because “module uptake” is a nominal dependent variable (either respondents had implemented modules or they had not), all statistical significance was sought by evaluating the chi-square statistic on the generated cross-tabulations (11). Finally, to evaluate the impact of confounding variables, a forced-entry logistical regression model was used to determine the effect of each independent variable when the other variables were held constant. A logistic regression models a dichotomous outcome variable by using the following equation, p Module Uptake

1

= 1+ e

−(β0 + β1 X 1 + β2 X 2 + εi )

(1)

where p is the probability of module uptake, β0 is a constant, X1 and X2 are predictor variables, β1 and β2 are the weights attached to the predictor variables, and εi is a residual term for individual values (12). This allows a comparison of the effects of two independent variables on module uptake.

Vol. 83 No. 2 February 2006

•

Journal of Chemical Education

299


Results and Discussion

Module Uptake and Workshop Delivery Decisions

Evaluation of Workshop Effectiveness

An additional factor affecting workshop effectiveness in terms of module uptake is the particular workshop the respondent attended. No one workshop is exactly like any other workshop, despite identical overarching goals and activities. Workshop presenters adjust to meet the needs of each particular group of workshop attendees. Beyond this simple difference is the general atmosphere, which determines much of the character of the workshop and is extremely sensitive to the particular attitudes and experiences of the participants. These ineffable differences make it difficult to consider the effectiveness of any set of workshops. However, these ChemConnections workshops had a quantifiable difference— the duration of the workshops. Workshops that lasted two days or longer were designated as long-term workshops and those shorter than two days were considered short-term workshops. By this classification 10 of the 16 workshops in this sample were long-term, with 51 of the 99 respondents attending a long-term workshop. Long-term workshops featured a 59% module uptake, while short-term workshops had a 27% module uptake, (χ2 = 8.9, p < 0.05). If reports of planned use are added to reports of actual use, the disparity continues: 73% of the long-term workshop respondents reported that either they had used or were planning to use modules, compared to a 40% rate for those attending short-term workshops (χ2 = 10.9, p < 0.05). These two significant differences seem to indicate the benefits of featuring long-term workshops in terms of participant adoption, though it could also be argued that those attending long-term workshops were already more seriously considering adoption of modules. Complicating the issue is that long-term workshops disproportionately occurred in the earlier years of the workshop series. As has been discussed earlier, there may be a “lag time” related to teaching assignments before modules can be implemented. It is possible that the effects of workshop duration are a function of the length of time since the workshop, rather than being directly related to workshop duration. To investigate this possibility, a logistic regression was performed to determine the effects of workshop duration on module uptake while holding the effects of workshop year constant, and vice versa. Workshop duration was indeed found to be the significant factor in determining module uptake ( p < 0.05) with the year of workshop held constant. The results of the logistic regression are in eq 2 and Table 1, in which year ranges from 1 to 3 for 1998 to 2000, and duration is set to a 1 for short-term and a 0 for long-term workshops.

Module Uptake and Potential Adoption Delays Overall, of the 99 respondents who were currently in teaching positions, 43 had used modules.5 Since less than 50% of the respondents had used modules, it is important to inquire into factors that may have affected module uptake for the workshop attendees. To begin an examination of this question, the rate of module uptake for each year of workshops was explored. These rates of uptake can be seen in Figure 1.6 As the graph illustrates, the percentage of those using modules has decreased among respondents who attended workshops in the later years. This trend was found to be statistically significant (χ2 = 8.3, p < 0.05). The survey was administered in February 2002, ensuring that workshop attendees from the final workshops, in spring 2000, would have had more than one year to consider adopting modules.7 We were therefore looking for indications of possible lag time in the data, for example an increase in reports of planned use rather than actual use from later workshops. Looking at reports of both planned and actual use, the majority of the responders from 1998 either planned to use modules or had used them (77%), while 44% of the responders from 1999 were in this category, and 55% from 2000. Indeed, there is a marked increase of respondents who report planned future module use for workshops given in 2000 (9% of the 1998 responders reported planning to use modules, 9% for 1999, 23% for 2000) though the trend is not statistically significant, (χ2 = 3.2, p > 0.05). The question of how long to wait before looking for implementation remains open.

80

Module Use (%)

70 60 50 40 30 20 10 0

1998

1999

2000

Year of Workshop Figure 1. The percentage of respondents using modules for each year during the workshop period 1998–2000.

Table 1. Results of Logistical Regression Analysis

300

βi

Standard Error for βi

Significance

1.165

0.658

0.077

᎑0.494

0.316

0.117

᎑1.011

0.461

0.028


•

p Module Uptake = 1 + exp − 1.165 − 0.494 ( Year ) − 1.011 (Duration )

−1

(2)

It was also possible to consider whether the size of a workshop (i.e., number of participants) had any effect on the subsequent uptake of modules. Did smaller workshops, for example, provide more attention to each participant, enabling participants to get their questions answered and to adopt modules more readily following the workshop? For this set of workshops, with attendances varying between 5 and 33,


•

www.JCE.DivCHED.org


no significant relationship was found between workshop size and resulting module uptake.

Table 2. Module Uptake by Institution Type

Target Population Concerns Another pertinent issue to the effectiveness of any set of workshops in disseminating information is the variety of institution types reached. We addressed this issue by determining the range of institution types represented by those who reported module use in the survey. Institution type was not collected via survey but rather was determined based on the highest chemistry degree obtainable at the respondent’s institution, as per information given on departmental and institution Web pages. The results are presented in Table 2. The responses indicate that 79% of module users are at 2-year and 4-year institutions, which is representative of overall survey respondents, 80% of whom are at 2-year and 4-year institutions. This relative lack of representation from graduate institutions suggests that there is still much work to be done before this curricular change extends beyond undergraduate institutions. General Reports of Reform The survey also inquired as to any other ways, apart from module use, in which the ChemConnections workshop had influenced the workshop attendee’s teaching. Sixty-four respondents answered this question, some with multiple responses. The answers were categorized and the results are found in Table 3. By considering the categories representing changes made in current teaching practices, the impact the workshops had on overall teaching practices can be ascertained. To measure this, respondents were considered for either module use or changing teaching practices. The majority of the 28 respondents who reported changing teaching practices had not used modules. Thus, 57 respondents had either used modules or reported one of the teaching practice changes listed in Table 3. By combining the respondents who reported “other benefits” with this group, it was found that 72 respondents reported some benefit from attending the workshop. Thus 72 respondents out of 99 reported being affected in some positive way by their participation in a ChemConnections workshop.

Evaluation of Implementation Perceived Module Effectiveness For those who reported having used modules in their teaching, a series of questions were asked to determine how the modules had been used. One of the questions asked the respondents to rate the effectiveness of modules with various student groups, on a scale of 1 to 5, with 5 being most effec-

Number of Number using Respondents Modules

Institution Type

Modules Used (%)

High School

04

03

07

2-Year

33

14

33

4-Year

46

20

47

Masters

05

03

07

PhD

10

03

07

No Information

01

00

00

Total

99

43

100

Table 3. Other Workshop Influences Category

Frequency

Led to use of active learning techniques

Current practice changes

Other benefits

17

Led to use of module materials as examples

7

Affected teaching practice in a nongeneral chemistry course

4

Introduced new concepts about teaching and spurred reflection on current teaching practice

19

Led to introduction of real-world examples or use of a thematic approach

12

Promoted community-building

10

No effect or not applicable

8

tive. This also brought forth information concerning with which groups of students modules had been used. The responses are tallied in Table 4. Though the majority of responses for every student group indicated an effectiveness rating of 3 or above, there were fifteen responses (from eight people) with a ranking of 1 or 2, indicating that module use was thought by eight respondents to be ineffective for particular groups. Eleven of the fifteen low responses were generated from only four people, meaning that only four people believed modules were ineffective for more than one student group. Even though this represents only 9% of module users, it is important to look into their concerns, since they may shed some light on difficulties with module adoption. Of these four, two did not comment on why they believed the modules to be ineffective. One indicated several reasons for the perceived ineffectiveness of modules, including time constraints involved in fitting the modules into the material traditionally covered, not finding the modules user-friendly, and

Table 4. Effectiveness Rankings for Student Groups (Actual Responses) Effectiveness Ranking

Chemistry

Health

Non-Science

Other Science

Pre-Med

Engineering

Total

1

2

3

1

01

3

0

10

2

1

0

2

01

0

1

5

3

8

6

7

11

7

6

45

4

8

2

8

12

8

5

43

5

4

1

1

02

2

3

13

Average

3.48

2.83

3.32

3.48

3.3

3.67

---

23

12

19

27

20

15

116

Total Responses

www.JCE.DivCHED.org

•


•


301

Research: Science and Education Table 5. How Modules Have Been Used Number Who Used This Approach

Module Usage

Putting several modules together to form a whole course

12

Number Who Found This Approach Best

6

Number Who Found This Approach Best When Using the Method Below Several modules

6

Inserting one or two

2

Capstone

1

Other

1

Didn’t answer Total Several modules Inserting one or two Inserting one or two modules into a regular course

26

Capstone

12

Other

6

11

2

6

55

26

0 2

Didn’t answer

9 26

Several modules

1

Inserting one or two

2

Capstone

2

Other

0

Didn’t answer

1

Total

6

Several modules

1

Inseringt one or two

1

Capstone

0

Other

6

Didn’t answer Total

3 12

Other Total

Using single module as a capstone at end of course

2 12

3

Total

11

----------

55

teaching in a multi-section course in which not all the faculty members teaching the course endorsed the implementation of the modules. This respondent described student gains in terms of students wanting to “hear about” real-world applications of chemistry, indicating perhaps that the respondent focused on the content aspects of the modules rather than on the pedagogy. The last of the four believed modules were not effective for premed and pre-health students, since “Premed and pre-health students are normally only interested in ‘getting chemistry out of the way’ as quickly and as easily as possible.” However, this respondent viewed engineering and other science students as sometimes interested, depending on the particular module focus.8 Despite these concerns, overall, as Table 4 indicates, modules were seen to be effective throughout a diverse set of student groups.

quent use of the “other” category was to describe the use of portions of modules to teach certain aspects of chemistry or to show the relevance of certain aspects of chemistry. Also emerging from this question was the total number of institutions that had used fully modular courses. Does this present a desirable goal for module use? Responses were mixed. The twelve respondents in this category represented eleven different institutions. Of the six who believed this was the best use, all were at different institutions. Of the other six, two did not indicate a best method, but four did indicate an alternative method as better, citing reasons such as the time commitment for a wholly modular course, the preference to be able to explore other areas of general chemistry not covered by the modules, and a lack of support from the department for using a fully modular course.

Reported Variations in Module Use Modules were designed for use in several ways depending on the instructor’s preference, ranging from an entire course based on modules to the use of one or two modules to focus on a specific topic (2). To better understand instructor preferences for implementing modules, we asked respondents to select which method described the way they used modules and which method they believed to be the best use for modules. The survey responses are presented in Table 5. Of the 43 module users, 40 provided responses indicating how modules were used, and 26 of the 40 indicated a preference as to the best structure for module use. The most fre-

Perceived Student Responses to Module Use Instructor-perceived student response to the modules was investigated via open-ended questions, both in terms of student resistance to the use of modules and student learning gains. Of the 43 module users, 28 respondents gave their impressions of the student resistance they had encountered. While one respondent believed that students found the modules too slow, others indicated that students felt the modules required more work (number if respondents, n, is 4). There was also an indication that some students did not like group work (n = 3), which modules support. In addition, eight responses indicated that instructors perceived students as skep-

302


•


•

www.JCE.DivCHED.org


tical or apprehensive simply about departing from traditional teaching techniques, regardless of the form of the departure. There were also four responses revealing the instructor perception that students believed modular courses were not preparing them for future chemistry coursework or examinations. Considering these four, responses elsewhere in the survey showed that one instructor shared these doubts, while the others tended not to agree with their students on this issue. While the question specifically probed for student resistance, two instructors used this opportunity to stress that student response to modules was positive, with one respondent saying simply, “They liked them!” In another question, those who used modules were also asked to describe any observed gains for the students resulting from module use. Of the 43 module users, 32 respondents related their observations of student gains achieved. Eleven of the 32 reported what could be classified as learning gains found with the students. These gains included such specifics as better science reasoning skills (n = 6), better retention of information presented (n = 2) and a better understanding of the material presented (n = 6). More frequently reported were attitudinal gains found with the students, with 28 of the 32 answering the question reporting observing such gains. Attitudinal gains included increased student interest (n = 14) and interaction in the classroom (n = 7). Also present was an awareness of student appreciation for learning the relevance of chemistry to real-world situations (n = 12) and observations that modules increased student confidence in understanding chemistry (n = 5). These attitudinal gains are those that the developers of the modules had hoped to create via the modules’ combination of real-world context with active and collaborative learning.

Reported Barriers to Reform Respondents who had not used modules were asked to describe any factors that influenced their decision. Of the 56 respondents who had not used modules, 50 of them provided descriptions of barriers. Nine of those simply reported that they were not teaching a course for which a module had been developed, and another twelve cited the effort and time needed to implement change without going into much detail. Details that were provided included heavy teaching responsibilities and a perceived clash with research time. Descriptions of other structural barriers were provided by twelve respondents. Structural barriers, in general, can be defined as representing external hindrances over which the instructor feels he or she has no control or input. One structural barrier (n = 5) cited by the respondents was the need for all sections of a particular chemistry course to be taught in the same way or, at the very least, to have the same exams (thereby, it was felt, stifling an individual innovator). Some of the respondents (n = 3) reported that they were not permitted to change the curriculum (two of these in conjunction with the coordinated-course issues mentioned above). While these issues would hinder the implementation of any proposed innovation, other respondents (n = 4) cited difficulties arising from lab and lecture being run as separate courses. This structural barrier would present a larger problem for modules, which specifically integrate laboratory activities with course content, often using real-world laboratory activities to bolster contextual connections, than it would for www.JCE.DivCHED.org

•

innovations lacking that integration. An additional two respondents were concerned about variability in teaching assistant ability to run module activities with appropriate pedagogical strategies, which is an issue arising from the explicit inclusion of active-learning pedagogical strategies within the modules. Again, this would not be an issue for contentbased innovations lacking pedagogical imperatives, but the module developers, as discussed earlier, were in fact trying to create materials requiring both some nontraditional content and some nontraditional pedagogy. Most of the modules are designed to accompany a textbook, with assigned readings from pertinent textbook sections providing the students with the chemistry background. The modules provide the students with the impetus to learn the chemistry in the assigned textbook readings in order to come up with answers to the socially relevant questions posed by the modules. This structure ensures the students are applying the chemistry concepts as they learn them, so they can be integrated into the students’ growing body of chemistry knowledge and applied in a real-world context immediately. We were alert for instructor reports of not wanting to require students to purchase even an inexpensive module in addition to an expensive textbook; however, only one respondent cited the cost to students as a factor in the decision not to use modules. Given the nontraditional nature of the modules, we were also alert for concerns about content coverage. Six of the 50 respondents who decided not to use modules did in fact cite coverage concerns, with three explicitly stating they felt that using the modules would necessitate leaving out important topics. One of these three linked this concern with failing to prepare students adequately for future coursework, which was given by the other three respondents in this group as a factor in their decision not to adopt modules. The coverage issue also pops up in another guise from two additional respondents who believe that modules are inappropriate for use in the lecture portion of a course, but might do for the laboratory. Overall, concerns with content coverage were reported as a barrier to module use by only these eight of the 50 nonusers who responded to the question. As discussed previously, a portion of the instructors who reported using modules gave their impressions of negative student response. Anticipated negative student response may have played a role in the decision for some instructors who chose not to use modules. In fact, seven respondents were concerned enough about student response to the modules that they reported this concern as a factor in their decision. Interestingly, two respondents believed that the modules would be pitched too high for their students, while another two believed the modules were pitched too low. This split mirrors the split seen among module users, and reinforces the situation-specific nature of student preparation. The remaining three thought that students would not like the nontraditional aspects of the modules. Conclusions In the final analysis, ChemConnections workshops were effective. Although not everyone who attended a workshop went on to use modules, 43% of respondents did use modules by spring 2002 and an additional 13% were planning


•


303


to use modules in the future. The majority of these module users were teaching at 2-year and 4-year institutions. Overall, the modules were perceived to be effective for a wide range of student groups. Student responses to the modules were perceived, in general, to be positive, with many respondents reporting that they had observed both attitudinal and learning gains among their students. There are indications that workshops brought other benefits beyond simple uptake of modules, with 57% of respondents reporting some change in teaching practice and 72% expressing positive outcomes as a result of workshop attendance. For those instructors who had not used modules, influential factors were the required time and effort, inherent structural problems, and concerns regarding coverage. In terms of module uptake, workshops lasting two days or more appeared to provide better results than shorter workshops. It is hoped that those interested in curricular change will be interested in these findings. For our purposes, we look forward to others’ reports of workshop dissemination efforts and an increased understanding of effective workshop dissemination for college chemistry faculty. Acknowledgments The authors wish to thank Sandra Laursen, Eileen Lewis, Elaine Seymour, and Brock Spencer for helpful discussions both of the survey and of this report. We also wish to thank those who took the time to complete the survey. W

Supplemental Material

Literature Cited

The email survey used is available through this issue of JCE Online. Notes 1. The opening session in a module introduces the general question to the students, inviting them to contribute what they already know and to suggest some approaches for gathering the remaining information necessary to investigate the question fully. The module continues by breaking the general question into a number of essential sub-questions, which require learning key chemistry concepts to answer. These sub-questions not only contribute toward answering the general question, but also serve as a means to apply and illustrate the usefulness of the chemistry concepts. 2. Although the workshops varied in duration they had several commonalities: (i) Participants were introduced to the modular approach and participated in sample modular activities. (ii) Whenever possible, participants also worked with other faculty to plan for and teach a module activity to each other in a mock classroom setting to gain practice with the nontraditional pedagogical strategies prevalent in the modules. (iii) Opportunities were provided for participants to discuss barriers to change at their home institutions and to brainstorm about ways around the barriers. Workshop presenters also offered additional assistance upon request, as well as information about current module users who would be willing to talk about their experiences and help new users. 3. The ChemConnections project continued to offer a modified version of these workshops as part of the Multi-Initiative Dissemination project (7).

304


4. Attendance lists for the 16 workshops were obtained from the project directors. Of these 281 available participant names, 214 received the survey sent in February 2002. The majority of those who did not receive the survey could not be reached via the email address they had provided at the workshop, and no additional information could be found at their institutional Web page, leading to the belief that they were no longer at the institution. If an email address was valid, it was presumed that the intended recipient had received the survey. After two additional email solicitations, a total of 106 workshop attendees responded to the survey, for a 50% response rate from the 214 presumed survey recipients. 5. There were also 13 additional respondents who had not used modules in the past, but indicated that they planned to use modules in the future. 6. The 1997 responses were discarded for this comparison because the number of survey respondents from 1997 was only three, with only one workshop attendance list obtained from that year, making 1997 unsuitable for comparisons with other years. The response rate for each of the other years mirrored the overall response rate. 7. The practice of rotating faculty assignments in introductory courses, however, could lead to situations in which attendees would have had to wait two years or more before being in a position to make the change. 8. Also, there was an indication that this respondent had been working with draft versions of the modules, finding them to have many errors, both in typing and in the labs, and to be arranged poorly.

•

1. Burke, K. A.; Greenbowe, Thomas J.; Gelder, John I. J. Chem. Educ. 2004, 81, 897. 2. Anthony, S.; Mernitz, H.; Spencer, B.; Gutwill, J.; Kegley, S.; Molinaro, M. J. Chem. Educ. 1998, 75, 322–324. 3. Kegley, S. E.; Landfear, D.; Jenkins, D.; Gross, B.; Shomglin, K. Water Treatment: How Can We Purify Our Water?; John Wiley & Sons: New York, 1998. 4. Chemistry in the Community (ChemCom), 4th ed.; W. H. Freeman: New York, 2002. 5. Stanitski, C. L.; Eubanks, L. P.; Middlecamp, C. H.; Stratton, W. J. Chemistry in Context, 3rd ed.; McGraw-Hill: Boston, 2000. 6. Chemistry, 1st ed.; W. H. Freeman: New York, 2002. 7. Multi-Initiative Dissemination Project Home Page. http:// www.cchem.berkeley.edu/~midp/ (accessed Nov 2005). 8. Roessner, D.; Perakis, S. Evaluating the National Science Foundation’s Chemistry Initiative: A Workshop. SRI International: Menlo Park, CA, 2001. 9. Neuman, W. L. Social Research Methods: Qualitative and Quantitative Approaches, 5th ed.; Allyn and Bacon: Boston, 2003; pp 442–443. 10. SPSS, Version 11.0; SPSS Inc., 2003. http://www.spss.com/ (accessed Nov 2005). 11. Foster, J. J. Data Analysis Using SPSS for Windows Versions 8 to 10: A Beginner’s Guide; Sage Publications: Thousand Oaks, CA, 2001; pp 156–157. 12. Pedhazur, E. J. Multiple Regression in Behavioral Research: Explanation and Prediction, 3rd ed.; Harcourt: Fort Worth, TX, 1997; pp 714–717.


•

www.JCE.DivCHED.org

Effectiveness of a Workshop To Encourage Action: Evaluation from a

Recommend Documents