Predicted versus Actual Performance in Undergraduate Organic

In the Classroom

Predicted versus Actual Performance in Undergraduate Organic Chemistry and Implications for Student Advising

W

David P. Pursell† Department of Chemistry and Life Science, United States Military Academy, West Point, NY 10996; [email protected]

Predicting college academic performance is a challenging and controversial proposition, yet most colleges use measures of pre-college academic merit and predicted academic performance as part of their admissions process. The high school academic record and exams such as the SAT or ACT are perhaps the most common measures used in this process. Once students are admitted and begin college academics, the usefulness of predictive measures typically is superceded by actual academic performance during the undergraduate experience. These measures are useful in advising students as they develop their academic and career goals and plans. The focus of this report is predictive measures for organic chemistry performance. It is important to note that performance for the purpose of this article is measured by the grade students earned in the organic chemistry course. Grades are not necessarily indicative of the student’s understanding of organic chemistry—rather, using grades is one of many possible ways to measure performance. As a related but secondary issue, this report examines the effectiveness of faculty without organic chemistry expertise who teach organic chemistry for the first time. Background The United States Military Academy (USMA) at West Point is a four-year undergraduate institution whose mission is to educate, train, develop, and inspire future Army officers (1). The Academy offers a diverse array of academic programs designed to educate future Army officers for a changing world (2). All cadets complete a 30-course core curriculum spanning the math, science (all cadets complete one year of general chemistry), engineering, humanities, national security, and public affairs disciplines and an 11-course academic major of their choosing. The Department of Chemistry and Life Science offers three traditional academic majors in chemistry, chemical engineering, and life science. Perhaps unique among colleges, our department faculty consists of a small permanent cadre of 11 military and civilian faculty and a rotating group of 22 military faculty. The permanent faculty, all Ph.D. educated, provide disciplinary expertise and academic leadership for the three academic majors. With only 11 permanent faculty and three academic majors, there are occasions when faculty must teach in areas out of their expertise. The rotating faculty arrive with recent Army operational experience (vital to developing and mentoring cadets as future officers) and having just completed a two-year masters degree program. They complete an eightweek teacher training and mentor program prior to teaching three years, typically in the core general chemistry program. † Current address: School of Science and Technology, Georgia Gwinnett College, Lawrenceville, GA 30043

1448

Journal of Chemical Education

•

At the end of the three years, they return to the operational Army (3). Organic chemistry is one of the first chemistry electives for students in the chemistry, chemical engineering, and life science programs and a required course for all cadets pursuing medical school admission. The course is a comprehensive, rigorous introduction to modern organic chemistry with laboratory (4). Ask any student on any campus in the nation about the organic chemistry course and you will likely hear what follows (excerpted from ref 5). “The infamous, dreaded ‘orgo’, a marathon of memorization.” Unfortunately, that’s how all too many college students view their first exposure to organic chemistry. Their trepidation is justified: one-quarter to one-half of beginning organic students don’t do well enough to continue on to the next course.

There has been much research on pedagogical approaches to teaching a demanding, rigorous course such as organic chemistry. Often the introductory organic chemistry course is a large lecture course with recitation sections and a separate laboratory course. Such pedagogy has limitations and educators have experimented with many alternative approaches. Alternatives related to teaching organic chemistry include active and cooperative learning (7–11), student-directed and team learning (10, 12, 13), grade-study contracts (14), problem-solving and collaborative learning (15–17), distance education (18–20), and meta tasks for organizing knowledge (21). These studies indicate that there is enhanced learning and greater student satisfaction when the traditional lecture course is supplemented with other instructional techniques. The USMA organic chemistry course is taught with two instructors, two sections each, 12–15 students per section, and incorporates many of the instructional techniques cited above. The small section size offers many opportunities for instructional flexibility (22). The instructors use the Thayer Method (23–26), named for Sylvanus Thayer, the USMA Superintendent from 1817–1833. Thayer increased academic rigor and made the focus of the academic program to train civil engineers for the growing nation. The Thayer Method views teaching as supporting student learning (27). The Method’s hallmark is that students prepare prior to class, so each lesson is published in advance with student lesson objectives, study assignment, terms, concepts, and homework problems. There is essentially a contract whereby students commit to preparing before class and instructors commit to flexibility in facilitating student learning during class (28). Each class period is 80 minutes, allowing sufficient time for discussion, exploration of more challenging topics in depth, and student recitation under the guiding and mentoring eye of the instructor (typically via student problem-solving chalkboard sessions) (26). The lab program is directly integrated with the class, using the same sections and instructors.

Vol. 84 No. 9 September 2007

•

www.JCE.DivCHED.org

In the Classroom

The Investigation For two consecutive years our department taught the course with one Ph.D. organic chemist (the same person both years) and two different Ph.D. chemists with specialties other than organic chemistry. Both non-organic instructors had previous experience teaching general chemistry, had never taught organic chemistry, and had their last organic experience as undergraduates. The course curriculum was virtually identical both years but graded events were different. The overall approach to the investigation was to examine the correlation between measures predicting student performance and actual performance (GPA) in organic chemistry. Predictive measures included measures based on pre-college information and predictions based on college academic performance. For the purpose of this article, performance is measured by the grades students had earned in college courses at West Point. The mathematics portion of the SAT (SAT–M) has been found to indicate mathematical skill that is an important factor in predicting grades in general chemistry (29). While the SAT–M score may not have a cause and effect relationship with grades, or more importantly understanding, students with high SAT–M scores tend to earn relatively high general chemistry grades while students with low averages tend to earn lower grades (30, 31). These studies also show that students who subsequently major in chemistry tend to have earned higher grades in general chemistry than do students who have not majored in chemistry. In addition, chemistry majors tend to earn appreciably higher general chemistry grades than those predicted by SAT–M scores. Organic chemistry is quite different from general chemistry as it is typically less mathematics oriented. Organic chemistry covers a more focused (although large) body of material in depth as compared to general chemistry, which tends to cover the broad field of chemistry in less depth. Recognizing that reading comprehension skill as measured by the SAT verbal (SAT–V) is important, the first predictive measure used in the investigation was the Cadet Entrance Evaluation Report (CEER) score. The CEER score is an internal admissions committee tool only and not released to students. The committee uses the CEER, along with many other factors, to gauge the likelihood of initial candidate success in the core academic program, which is a mix of mathematics, science, engineering, and humanities courses. The CEER has been a reasonably effective predictor of overall academic performance during initial semesters since its inception at the Academy in 1973. The CEER has a range of 200–800 and is computed by weighting SAT–M (48%), SAT–V (22%) and high school rank (30%). The high school rank is a scaled score that considers both academic class rank and size of the graduating class. The second pre-college predictive measure was Predicted GenChem, a predicted general chemistry percentage score developed by the Academy’s Office of Institutional Research in the 1980s that has been a reasonably good predictor of performance in general chemistry (26). The Predicted GenChem is a scaled score using the CEER (69%) in combination with a validation exam (31%) prepared by the department’s senior faculty supervising the general chemistry course. The senior faculty have chosen to use their own exam, developed and modified over many years, rather than www.JCE.DivCHED.org

•

a standardized exam (such as the ACS exam). The validation exam given to all students participating in this study was a 65 question, multiple-choice, calculator-free exam. Students never see the graded exam and only learn whether or not they have validated the year of general chemistry. Because the Predicted GenChem contains a chemistry-specific component, it is designed to be a more specific predictor of general chemistry performance than the CEER. While the CEER and the Predicted GenChem are measures that consider student “potential” prior to entering college, the investigation also considered predictive measures for organic chemistry based on actual student performance, as measured by grades in courses at West Point during the initial semesters of college. The first of the college academic performance-based predictive measures was GenChem GPA, the GPA average for the two-semester general chemistry course. The second performance-based predictive measure was the student’s overall, cumulative GPA up through the semester prior to beginning organic chemistry. This Pre-Organic GPA measure provides insight into how well the student has adapted overall to college academic life. At West Point students only complete core courses during their first four semesters, therefore, the Pre-Organic GPA provides a common comparison among students who have up to that point completed an identical set of courses. The final college academic performance-based predictive measure was Organic I GPA, the earned GPA for the first semester of organic chemistry. The organic chemistry sections were assigned by the Dean’s Operations and Registrar Division with no attempt to develop sections of specific composition. The instructors had no knowledge of student CEER, Predicted GenChem, GenChem GPA, or Pre-Organic GPA. All students included in the study had SAT scores, CEER scores, completed general chemistry, and completed both semesters of organic chemistry. Results and Discussion It is important to note again that this investigation examined performance as measured by grades. Many factors contribute to student grades, such as ease of exams, distribution of grades, instructors “coaching” the exams, exam similarity to previous years exams (and their solutions in student study files), to mention just a few. As with professional faculty at all institutions, the West Point chemistry faculty does its best to present a challenging, rigorous, relevant, and fair chemistry curriculum. That being said, the study nonetheless considers academic performance in terms of course grades and as previously stated, grades should be viewed as not necessarily reflecting student understanding. While there was no attempt to design section composition, descriptive statistics in Table 1 suggest that the sections were roughly equivalent. All of the sections are relatively small (N ⫽ 18–21), so one should view the results keeping in mind the small sample size. A noteworthy observation in Table 1, though, is that the section with the instructor who had nonorganic expertise (and for whom this was the first time teaching organic) in Year 1 of the study had the lowest average CEER (628), the lowest Predicted GenChem (83), but the highest Pre-Organic GPA (3.39). This apparent contradiction in the data reinforces the complex nature of predictions


•


1449

In the Classroom

versus performance and necessitates caution in drawing sharply defined, quantitative conclusions. Therefore, this report will make general observations. Correlations of predicted versus actual performance in first-semester organic chemistry are summarized in Table 2. The correlation of data sets was done by calculating the Pearson product-moment correlation coefficient, r. There are several notable observations. First, for the pre-college-based predictive measures, Predicted GenChem has a higher correlation with GenChem GPA than does CEER for each of the four sections. Both CEER and Predicted GenChem have a reasonably strong positive correlation with Pre-Organic GPA.

There are also data that appear uncorrelated, for example, 0.03 and ᎑0.06, for which there is no ready explanation. It is well known that predictions of academic success in college based on measures of “potential” (such as SAT scores) are fraught with contradictions (30, 31). There are many aspects of the college experience—especially during the initial year—that confound such predictions. Nonetheless, the first discussion point concerns the usefulness of CEER scores (SAT and high school rank) and Predicted GenChem grade (CEER and validation exam) as predictors of academic performance. The CEER measure is designed to predict performance in the overall core academic program spanning mathematics, science,

Table 1. Course Sections Composition and Data by Instructor Specialty Organic Chemistry I Course Data Year 1 Predictive Measures

Statistical Parameters

CEER Score Predicted GenChem Grade GenChem GPA Pre-Organic GPA Organic I GPA

Instructor Not an Organic Chemist (N ⫽ 20)

Year 2

Instructor an Organic Chemist (N ⫽ 21)



Mean

628

651

640

642

Std Dev

42

45

76

56

Mean

83

89

87

86

Std Dev

13

6

9

2

Mean

3.75

3.72

3.78

3.78

Std Dev

0.58

0.46

0.48

0.52

Mean

3.39

3.31

3.38

3.32

Std Dev

0.37

0.36

0.54

0.51

Mean

3.15

3.36

3.15

3.02

Std Dev

0.81

0.62

0.73

0.73

Table 2. Correlation of Prediction versus Performance for Students in Organic Chemistr y I Year 1 Predictive Measures Comparisons


Year 2




MCEER Score vs GenChem GPA

0.41

0.42

0.67

0.30

MCEER Score vs Pre-Organic GPA

0.49

0.62

0.85

0.47

MCEER Score vs Organic I GPA

0.03

0.41

0.77

0.28

MPredicted GenChem Grade vs GenChem GPA

0.49

0.58

0.73

0.65

MPredicted GenChem Grade vs Pre-Organic GPA

0.30

0.67

0.71

0.53

᎑0.06

0.56

0.59

0.30

MGenChem GPA vs Organic I GPA

0.23

0.52

0.68

0.27

MPre-Organic GPA vs Organic I GPA

0.42

0.82

0.75

0.52

MPredicted GenChem Grade vs Organic I GPA

1450


•


•

www.JCE.DivCHED.org

In the Classroom Table 3. Correlation of Prediction versus Performance for Students in Organic Chemistry II Predictive Measures Comparisons

Correlation V a l u e s

MCEER Score vs Organic II GPA

0.40

MPredicted GenChem Grade MMMvs Organic II GPA

0.29

MGenChem GPA vs Organic II GPA

0.45

MPre-Organic GPA vs Organic II GPA

0.60

MOrganic I GPA vs Organic II GPA

0.87

Table 4. Correlation of Organic Chemistry I GPA to Organic Chemistry II GPA for Organic Chemistry I to Organic Chemistry II by Instructor Specialty Year

Instructor Specialty

N Values

Correlation Values

Both

Overall Combined

79

0.87

First

Inorganic to Inorganic

07

0.93

Inorganic to Organic

13

0.94

Organic to Organic

12

0.67

Organic to Inorganic

09

0.87

Physical to Physical

11

0.90

Physical to Organic

09

0.94

Organic to Organic

09

0.96

Organic to Physical

09

0.92

Second

engineering, and humanities, while the Predicted GenChem measure is tailored more specifically to the two-semester general chemistry sequence. For each of the four Organic I sections, Predicted GenChem is the best predictor of general chemistry performance. This is not surprising since its prediction model includes a validation exam covering many of the same general chemistry topics included in the two-semester general chemistry sequence. Most USMA students begin organic chemistry in the fall semester of junior year. As a result, actual college performance rather than pre-college “potential” is intuitively a more reasonable predictor of performance in organic chemistry. Considering the correlation of demonstrated college academic performance with Organic I GPA for each of the four sections, Pre-Organic GPA is a much better predictor of organic chemistry performance than GenChem GPA. This is perhaps not surprising to those who have taught both general chemistry and organic chemistry. Organic chemistry tends to cover topics in depth and requires perhaps a more extensive set of student knowledge, skills, and abilities than does general chemistry; these are presumably reflected in the student’s overall GPA. Organic chemistry requires students to visualize and think in three dimensions more than does general chemistry www.JCE.DivCHED.org

•

and organic chemistry has a whole new nomenclature system in addition to that of general chemistry, to name just two examples. In addition, study skills and study discipline developed in a broad array of courses—in part reflected in Pre-Organic GPA—are perhaps more important to organic chemistry performance than what is predicted by a single general chemistry course. All things considered, it is reasonable that the overall GPA before organic chemistry is a better predictor than is general chemistry GPA. Indeed, Table 2 data for each of the sections show that Pre-Organic GPA has a substantially higher correlation with Organic I GPA than does GenChem GPA. Table 3 summarizes prediction versus performance in the second semester of organic chemistry. As shown in Table 3, all the predictive measures have a positive correlation with Organic II GPA. By far the strongest correlation (0.87) is Organic I GPA to Organic II GPA. The two-semester organic chemistry course is sequential. Knowledge, skills, and abilities developed in the first semester set the groundwork for success in the second semester. To some degree success in the first semester is also a result of affective issues such as motivation, study discipline, and so forth. In fact, most students find the second semester course more challenging because of the cumulative nature of the course material and the pace through which the course moves through a multitude of topics. It is not surprising, then, that the predictive measure of Organic I GPA correlates very well with Organic II performance. Based on the previous discussion, it is anticipated that Pre-Organic GPA is a better Organic II predictor that is GenChem GPA, which is indeed the case. Table 4 summarizes student performance in Organic II with respect to the chemistry expertise (organic, inorganic, or physical) of their Organic I and Organic II instructors. For all instructor expertise combinations, there is a very high positive correlation for Organic I GPA predicting Organic II GPA. This result is interesting as it seems to indicate that with regard to student performance, it does not matter whether the instructor is an organic expert. This is a topic needing further investigation as it only involved three instructors. Based on this limited sample, one should not draw any substantial conclusions. However, it is important to note that for both years of the study the organic chemistry expert ran the course. The organic expert mentored the non-organic instructors in both organic subject matter and organic teaching techniques. This mentoring included daily interaction outside of class, occasional visits to class, team grading of all exams, and support during each of the laboratory periods. In addition, the organic expert developed the syllabus, worked out the details of the laboratory program, and prepared virtually all of the course materials. The non-organic-expert instructor was thus able to focus on self-mastery of the material and teaching students, without handling administrative responsibilities for the course. Having non-organic-expert faculty teach organic chemistry—without negative effect on students’ performance—provides great flexibility to small chemistry departments. In fact, a recent survey of two-year colleges finds that 59% offer organic chemistry. Many of these two-year colleges have small chemistry faculty (median of 3, mean of 3.7) (32), so instructional flexibility in covering offered courses is an advantage for the department.


•


1451

In the Classroom

Summary

Literature Cited

Overall student college GPA prior to taking organic chemistry (Pre-Organic GPA) correlates quite well with academic performance in the first semester of organic chemistry (Organic I GPA) as measured by course GPA. The overall student college GPA correlates substantially better with first semester of organic chemistry performance than does performance in the two-semester general chemistry course (GenChem GPA), consistent with the notion that success in organic chemistry requires a more comprehensive set of knowledge, skills, and study habits than does success in general chemistry. Academic performance in the first semester of organic chemistry (Organic I) has a very high correlation with performance in the second semester of organic chemistry (Organic II). The high correlation is reasonable considering the cumulative and comprehensive nature of the organic chemistry two-semester course sequence. Given substantial mentoring by an instructor with organic chemistry expertise and organic chemistry teaching experience, an instructor with no organic chemistry expertise or organic chemistry teaching experience can enable student success in organic chemistry. Finally, and perhaps the most useful result of the investigation, predictions are helpful when advising students. It is useful to have a reasonable idea of how students might do in organic chemistry when faculty provide academic and career advising. While it is the role of faculty to be supportive, positive, and encouraging in helping students develop their academic and career plans and objectives, faculty must also enable students to develop reasonable plans and objectives. Although a high degree of success in organic chemistry may not be critical for chemistry, chemical engineering, and life science majors, organic chemistry success (as measured by grades and MCAT) is seen as a major component in the medical school admissions process. During the two-year study, 23 students were admitted to medical schools. Of this cohort, the average GPA for both Organic I and Organic II was 3.7. Of the 22, only two had combined Organic I and Organic II GPAs of less than 3.0. Perhaps this group of students seeking medical school admission benefit most from having reasonable expectations with regard to their success in organic chemistry. For those students whose predicted success is not at the top, but who aspire to attend medical school, faculty could fill a larger role in inspiring, coaching, and mentoring students to help them achieve their goal of outstanding performance in organic chemistry.

1. United States Military Academy. United States Military Academy Strategic Vision—2010; Office of Policy Planning and Analysis: West Point, NY, 2000; p 7. 2. United States Military Academy. Educating Future Army Officers for a Changing World; Office of the Dean: West Point, NY, 2002; pp 6–8. 3. Department of Chemistry and Life Science, United States Military Academy. http://www.dean.usma.edu/departments/ chem/ (accessed Jun 2007). 4. United States Military Academy, Academic Program Red Book. http://www.dean.usma.edu/sebpublic/curriccat/static/index.htm (accessed Jun 2007). 5. Zurer, P. A. Chem. Eng. News 2001, 79, 43. 6. Paulson, D. R. J. Chem. Educ. 1999, 76, 1136. 7. Oliver-Hoyo, M. T.; Allen, D. J. Chem. Educ. 2005, 82, 944– 949. 8. Oliver-Hoyo, M. T.; Allen, D.; Hunt, W. F.; Hutson, J.; Pitts, A. J. Chem. Educ. 2004, 81, 441. 9. Hinde, R. J.; Kovac, J. D. J. Chem. Educ. 2001, 78, 93. 10. Katz, M. J. Chem. Educ. 1996, 73, 440. 11. Landolt, R. G. J. Chem. Educ. 2006, 83, 334. 12. Frydrychowski, V. A. J. Chem. Educ. 1995, 72, 429. 13. Hass, M. A. J. Chem. Educ. 2000, 77, 1035. 14. Dougherty, R. C. J. Chem. Educ. 1997, 74, 722. 15. Browne, L. M.; Blackburn, E. V. J. Chem. Educ. 1999, 76, 1104. 16. Neeland, E. G. J. Chem. Educ. 1999, 76, 230. 17. Almeida, C. A.; Liotta, L. J. J. Chem. Educ. 2005, 82, 1794. 18. Kurtz, M. J.; Holden, B. E. J. Chem. Educ. 2001, 78, 1122. 19. Casanova, R. S.; Civelli, J. L.; Kimbrough, D. R.; Heath, B. P.; Reeves, J. H. J. Chem. Educ. 2006, 83, 501. 20. Kurtz, M. J.; Holden, B. E. J. Chem. Educ. 2001, 78, 1122. 21. Fountain, K. R. J. Chem. Educ. 1997, 74, 354. 22. Toth, L. S.; Montagna, L. G. College Student Journal 2002, 36, 253. 23. Staats, Elmer B. Academic and Military Programs of the Five Service Academies, Report to the Congress by the Comptroller General of the United States of America; United States General Accounting Office: Washington, DC, October 1975; pp 48–52. 24. Palladino, G. F. J. Chem. Educ. 1979, 56, 323. 25. Ertwine, D. R.; Palladino, G. F. J. Coll. Sci. Teach. 1987, 16, 524. 26. Pursell, D. P.; Jordano, F. A.; Ramsden, J. H. Academic Excellence Research, Department Head Memorandum for the Dean of the Academic Board, United States Military Academy: West Point, NY; September, 1989. 27. Samuelowicz, K.; Bain, J. D. Higher Education 1992, 24, 93. 28. Raymark, P. H.; Connor-Green, P. A. The Teaching Professor 2003, 17, 2. 29. Ozsogomonyan, A.; Loftus, D. J. Chem. Educ. 1979, 56, 173. 30. Spencer, H. E. J. Chem. Educ. 1996, 73, 1150. 31. Russell, A. A. J. Chem. Educ. 1994, 71, 314. 32. Ryan, M. A.; Neuschatz, M.; Wesemann, J.; Boese, J. J. Chem. Educ. 2003, 80, 129.

W

Supplemental Material

A number of documents cited in the article are available in this issue of JCE Online. These PDFs include: Strategic Vision for the United States Military Academy—2010; Educating Future Army Officers for a Changing World; Comptroller General’s Report to the Congress, Academic and Military Programs, 1975; and Executive Summary of Academic Excellence Research in Chemistry, 1989.

1452


•


•

www.JCE.DivCHED.org

Predicted versus Actual Performance in Undergraduate Organic

Recommend Documents