Using an ACS General Chemistry Exam to Compare Traditional and

Chapter 20

Using an ACS General Chemistry Exam to Compare Traditional and POGIL Instruction 1

2

Downloaded by EAST CAROLINA UNIV on June 5, 2014 | http://pubs.acs.org Publication Date: September 29, 2008 | doi: 10.1021/bk-2008-0994.ch020

Martin D. Perry and Randall D. Wight 1

Department of Chemistry, Patterson School of Natural Sciences, Ouachita Baptist University, 410 Ouachita Street, Arkadelphia, AR 71998 Department of Psychology, School of Social Sciences, Ouachita Baptist University, 410 Ouachita Street, Arkadelphia, AR 71998 2

This study uses a standard ACS exam for general chemistry to examine the effectiveness of Process-Oriented Guided Inquiry Learning (POGIL) instruction compared to traditional instruction. One cohort in this general chemistry study received a full year of POGIL instruction, while the second cohort received one semester of traditional instruction followed by a semester of POGIL instruction. The ACS General Chemistry Full-Year Exam was administered to both cohorts at the end of the second semester and statistical comparisons were made. The data indicated no statistical differences in overall exam performance; however, closer examination of individual questions did yield insights into the learning process.

The Process-Oriented Guided Inquiry Learning (POGIL) instructional method, described in detail elsewhere in this text and other sources (1-4), aims to improve students' process skills in addition to assisting them with mastering course content. The key process skills are information processing, critical thinking, problem solving, communication, teamwork, management, and assessment. Instructors often suggest a variety of reasons for not attempting this student-centered approach where learners work in self-managed teams. Chief 240

© 2008 American Chemical Society In Process Oriented Guided Inquiry Learning (POGIL); Moog, R., et al.; ACS Symposium Series; American Chemical Society: Washington, DC, 2008.

241


among them is the notion that course content or coverage of material will be lacking (5), leading thus to poor performance on standardized exams such as those published by the American Chemical Society (ACS) Examinations Institute or to inadequate preparation for future courses including graduate and professional training. Achievement in chemistry courses is often linked to performance on ACS standardized exams (6). Although these exams provide a benchmark for comparison and set the standards deemed appropriate by the profession, they are but one tool for assessing student learning. This study explores instructional method as a tool and uses a standard ACS exam for general chemistry to compare the effectiveness of POGIL and traditional instruction.

Methodology All students were enrolled in General Chemistry courses at a small liberal arts university located in the rural mid-South. During the 2001-02 academic year (2002 cohort), every student experienced POGIL instruction both semesters by the same instructor. During the 2002-03 academic year (2003 cohort), instruction ranged from traditional during the first semester and POGIL during the second. A different instructor taught each semester. Prior academic preparation of the two cohorts—measured by ACT scores (2002, N = 26, M= 27A, SD = 3.31; 2003, N = 34, M = 27.2, SD = 3.12) and high school grade point averages (2002, N = 26, M = 3.73, SD = .360; 2003, N= 34, M = 3.79, SD = .208)—exhibited no statistical differences, respectively, F ( l , 58) = .08,/? = .77, d = .06 and F(\, 58) = .65, p = .43, d = .24. Thus, the only discernable preparatory difference between the cohorts was the instruction used in the first semester. The 1999 General Chemistry Full-Year Exam (7) was administered to both cohorts at the end of the second semester as the final exam for the course sequence. Materials used for POGIL instruction were authored by Moog and Farrell (5). Each class period began with a brief (five min) question-and-answer period, followed by students working in self-managed teams of three to four students on the assigned activity. Exercises requiring students to apply concepts invented during class time were assigned and collected at the beginning of the next class meeting. The students were allowed to work with each other or seek assistance from the instructor between class sessions. Occasionally, quizzes were given in lieu of turning in the assigned exercises. In addition to the final exam, four exams were administered during the course of the semester. The exam format consisted of multiple choice, short answer, brief discussion, and algorithmic problem solving questions. While the class sessions were conducted using POGIL methods, laboratories for all cohorts were under the purview of a separate instructor who used a traditionally structured manual.

In Process Oriented Guided Inquiry Learning (POGIL); Moog, R., et al.; ACS Symposium Series; American Chemical Society: Washington, DC, 2008.

242


Results We began our analysis by comparing the average percent correct for local (M= 64.28, SD = 21.56) and national (M= 59.89, SD = 16.63) students for the two cohorts (N = 140 question pairs). A Pearson product-moment revealed a substantial positive relationship between the two percentage samples, r(209) = .745, p < .001 (two-tailed). Though the local sample contained greater variability, the local mean was significantly higher than the national mean, ¿(210) = 4.42, p < .001, d = .23. A comparison of the overall percent correct on the ACS questions (N= 70) revealed no statistical difference—F( 1,138) = 1.58,/? = .21, d = .21—for the two local cohorts: 2002 (M= 65.88, SD = 21.19) and 2003 ( M = 61.26, SD = 22.26). Figure 1 displays the difference for each item (2002 score minus 2003 score) in local cohort scores: positive differences indicate questions where the 2002 cohort scored higher with negative difference indicating the opposite. The 2002 cohort achieved greater percent correct values on 40 of the 70 items overall. To assess the efficacy of traditional instruction compared to POGIL instruction, we focused on Semester 1 where instruction type varied for the two local cohorts. Questions from the ACS General Chemistry Full-Year Exam were examined and assigned to Semester 1 (N = 35) or Semester 2 (N = 30) according to presentation of material. Five of the questions received no coverage in either semester. When comparing percent correct on the germane national questions, the 2002 (POGIL) cohort (M = 67.36, SD = 19.22, N = 35) and the 2003 (traditional) cohort (M= 60.59, SD = 23.52, N = 35) did not differ significantly, F ( l , 68) = 1.73, p = .19, d = .31. The 2002 cohort achieved greater percent correct values on 21 of the 35 Semester 1 items. Although statistically significant differences in ACS exam performance do not exist between the two cohorts overall or by individual semester, eight questions (see Table I) do differ significantly, with the 2002 cohort performing better on six of the eight. Additionally, sue of the eight questions were firstsemester topics where the cohorts received different methods of instruction. The 2002 cohort scored higher on topics such as molecular structure (bond order and bond angles), stoichiometry (limiting and excess reagents), equilibrium (graphical interpretation), and experimental (graphical interpretation and safety). Of the five exam questions (questions 42, 60, 62, 63, & 64) that did not receive any coverage during either semester for either cohort, the 2002 cohort scored higher on four of them (questions 42, 60, 62, & 64). Additionally, fifteen questions on the exam (questions 8,14,15,20, 31,44,47,49, 51, 55, 60, 61, 62, 66, & 68) yielded mean correct values (see Table I) that were 1.50 or higher for both cohorts signifying that both cohorts failed to answer above 50% correct. In most cases, these questions required generalizations or higher-order levels of analysis in order to determine the correct answer. Ten of these tougher questions



Figure 1. Percent correct difference for each of the 70 items on the ACS exam. Positive differences indicate questions where the 2002 cohort performed better. Negative differences indicate questions where the 2003 cohort performed better.


244 8


Table I. Mean Correct , Standard Deviations, and One-Way Analysis of Variance (ANOVA) Comparing Cohorts on ACS Standardized Questions

Question (Semester Coverage) KD 2(1) 3(1) 4(1) 5(1) 6(1) 7(1) 8(1) 9(1) 10(2) 11(1) 12(1) 13(1) 14(1) 15(1) 16(1) 17(1) 18(1) 19(1) 20(1) 21(1) 22(1) 23(2) 24(2) 25(1) 26(1) 27(1) 28(1) 29(1) 30(2) 31(2) 32(2) 33(2) 34(2) 35(2) 36(2) 37(2) 38(1) 39(1)

2002(N = 26)

2003(N = 34)

ANOVA

M

SD

M

SD

F(2,81)

1.58 1.12 1.38 1.42 1.19 1.08 1.35 1.54 1.19 1.19 1.27 1.46 1.08 1.62 1.62 1.19 1.08 1.27 1.42 1.50 1.08 1.12 1.04 1.31 1.42 1.38 1.46 1.19 1.12 1.27 1.54 1.23 1.00 1.38 1.23 1.12 1.50 1.12 1.50

.504 .326 .496 .504 .402 .272 .485 .508 .402 .402 .452 .508 .272 .496 .496 .402 .272 .452 .504 .510 .272 .326 .196 .471 .504 .496 .508 .402 .326 .452 .508 .430 .000 .496 .430 .326 .510 .326 .510

1.29 1.21 1.53 1.62 1.38 1.09 1.32 1.85 1.50 1.12 1.29 1.62 1.03 1.53 1.71 1.38 1.18 1.35 1.65 1.74 1.15 1.06 1.03 1.32 1.41 1.35 1.38 1.38 1.09 1.41 1.59 1.32 1.06 1.41 1.21 1.21 1.38 1.09 1.41

.462 .410 .507 .493 .493 .288 .475 .359 .508 .327 .462 .493 .171 .507 .462 .493 .387 .485 .485 .448 .359 .239 .171 .475 .500 .485 .493 .493 .288 .500 .500 .475 .239 .500 .410 .410 .493 .288 .500

4.48" 1.72 1.25 2.00 2.91 .18 .29 4.87" 3.37 .85 .27 .92 .37 2.10 1.83 2.16 2.78 .24 2.12 2.85 3.10*" 1.50 .47 .38 .06 2.49 .43 1.28 .09 .66 .47 .95 1.02 .18 1.06 .76 1.66 .19 .26


245 Table I. Continued.


2002 (N = 26) Question (Semester Coverage) 40(2) 41 (1) 42(0) 43(2) 44(2) 45(2) 46(2) 47(2) 48(2) 49(2) 50(2) 51(2) 52(2) 53(2) 54(2) 55(2) 56(2) 57(2) 58(1) 59(1) 60(0) 61 (1) 62(0) 63(0) 64(0) 65(1) 66(2) 67(2) 68(1) 69(1) 70(2) a

2003 (N= 34)

ANOVA

M

SD

M

SD

F(2,81)

1.31 1.08 1.23 1.15 1.65 1.35 1.15 1.50 1.27 1.65 1.42 1.73 1.04 1.12 1.38 1.81 1.50 1.12 1.38 1.08 1.50 1.62 1.69 1.27 1.35 1.54 1.92 1.12 1.62 1.38 1.42

.471 .272 .430 .368 .485 .485 .368 .510 .452 .485 .504 .452 .196 .326 .496 .402 .510 .326 .496 .272 .510 .496 .471 .452 .485 .508 .272 .326 .496 .496 .504

1.09 1.18 1.32 1.26 1.56 1.41 1.47 1.53 1.26 1.62 1.35 1.68 1.06 1.24 1.32 1.59 1.47 1.29 1.35 1.06 1.71 1.74 1.71 1.26 1.38 1.35 1.97 1.47 1.82 1.71 1.24

.288 .387 .475 .448 .504 .500 .507 .507 .448 .493 .485 .475. .239 .431 .475 .500 .507 .462 .485 .239 .462 .448 .462 .448 .493 .485 .171 .507 .387 .462 .431

2.79 .91 .36 .75 1.43 .80 3.63*" 1.79 2.77 2.50 .24 1.04 .70 .73 .25 1.97 .42 3.01 3.28 .88 1.78 1.70 .94 .01 .08 1.03 .97 4.85"* 2.27 4.16" 1.29

where 1 = Correct, 2 = Incorrect


246 (questions 8, 15, 20, 31, 47, 60, 61, 62, 66, & 68) were handled more aptly by the 2002 cohort with an average percent correct 13.1% higher than the 2003 cohort.


Discussion The statistical analysis reveals that both local cohorts scored significantly above the national average and that no differences existed between local cohorts. These data contradict suggestions that less content coverage or less instructor control would result in lower standardized exam scores. Even though the 2002 cohort covered approximately 85% of course material compared to the 2003 cohort, their overall performance was not significantly different. While small sample sizes and large standard deviations contributed to this finding, the 2002 cohort did average 4.5% correct higher exam scores. Because the only difference in instruction for the two cohorts occurred during the first semester of the course, a second analysis of the 35 first-semester questions on the ACS exam was conducted. Similarly, the results exhibited no statistical differences. However, when individual questions are examined, eight questions show significant differences including six first-semester questions where the 2002 cohort scored higher. Further examination of these questions revealed that these topics required higher-order reasoning, multiple steps, or data interpretation. These data suggest that key process skills are developed to a greater extent in the POGIL classroom. The final analyses continue to bear out this trend. The 2002 cohort's greater performance on no-coverage and tougher questions highlights their reliance on skills developed during class sessions. As the students work in self-managed teams, they are forced to grapple with new ideas, invent concepts, assess their learning, and communicate with their peers. Rarely do any of these practices occur within the framework of traditional classrooms. Without the practice needed to acquire these grappling skills, students may flounder when confronted with tough questions or novel ideas. Instructors bear the burden of equipping students with the tools needed for success, not just on exams, but in careers and life.

References 1. 2. 3. 4.

Farrell, J. J.; Moog, R. S.; Spencer, J. N. J. Chem. Educ. 1999, 76, 570-574. Hanson, D.; Wolfskill, T. J. Chem. Educ. 2000, 77, 120-130. Hinde, R. J.; Kovac, J. J. Chem. Educ. 2001, 78, 93-99. Lewis, J. E.; Lewis, S. E. J. Chem. Educ. 2005, 82, 135-139.


247 5. 6. 7.

rd


8.

Williams, G.; McTighe, J. Understanding by Design; ASCD: Alexandria, VA, 1998. Holme, T. J. Chem. Educ. 2003, 80, 594. General Chemistry Full-Year Exam, ACS Examinations Institute: Clemson, SC, 1999. Moog, R.S.; Farrell, J.J. Chemistry: A Guided Inquiry, 3 Edition; John Wiley & Sons: New York, 2006.


Using an ACS General Chemistry Exam to Compare Traditional and

Recommend Documents