Symposium: lecture and learning: Rre They Compatible?
The Use of the GALT (Group Assessment of Logical Thinking) as a Predictor of Academic Success in College Chemistry Diane M. ~ u n c e 'and Kira D. Hutchinson The Catholic University of America, Washington DC 20064 I t is recognized that gaining scientific knowledge is especially difficult for students (1, 2). Formal lectures, which are the most common vehicle to disseminate information, are ineff~cient,and large class sizes usually serve to estrange the struggling student h m the instructor. Thus a student who is experiencing difficulty with the course material may remain anonymous to the instructor until very late in the semester when little can he done to aid himher. Math SAT scores are traditionally used to identify students with a n aptitude i n science and math-related courses (3).But this information is not always available to instructors, and self-reported scores (from students) are often not reliable. The GALT (Group Assessment of Logical Thinking) is a paper-and-pencil t e s t constructed by Roadrangka, Yeany, and Padilla (4, 5) to measure logical reasoning abilities. It is our belief that the administration of the GALT test will serve to quickly and painlessly identify students a t risk in college-level chemistry. These students can then be counseled to seek tutoring andlor directed to educational aids before they experience failure or withdraw from the class. The purpose of this study was to determine whether or not the GALT would enable an instructor to predict academic success in college-levelchemistry courses. The study was conducted on three different levels of freshman chemistry (vide infra). We believed that the GALT, a test of logical reasoning abilities, would successfully correlate with a student's ability in a n introductory chemistry course. Method Subjects There were three different populations under study. Population one included a large proportion of science and engineering majors electing Chem 104, General Chemistry I, spring semester 1991. This is the second of a two-semester course on introductory chemistry designed for science and ensineerine., students. The material covered traditional introducto~ychemistry topics such as atomic theory, stoichiometrv. kinetics, acid-baiie reactions, and thermodynamics. ail 50 students enrolled in Chem 104 in the spring of 1991 participated in the study. Most of the subjects were freshmen (&%), the rest sophomores (20%)and juniors and seniors (15%). Their majors included the following disciplines: liberal arts (40%).engineering and architecture (22%),biology (22%),chemistry (4%)and other (12%).The class was almost equally distributed between males (46%)and females (54%).
-
Population two comprised the students electing Chem 125 or 126 (Chemistry in the World Around Us and Chemistry in Modern Times, respectively) during the fall or spring semesters of the three academic years 1988-89, 1989-90, and 1990-91. This is a chemistry course designed 'Author to whom correspondence should be addressed.
for people who are not concentrating their studies in science, and it covers topics in chemistry that effect daily lives, such as food additives and water and air pollution. There were 181 subjects involved in this study. They were enrolled in the School of Arts and Sciences and their majors included politics (21%), education (20%), history (IS%), undecided (14%), English (8%), and other (19%). The class was divided between freshmen (13%), sophomores (36%).iuniors (24%). and seniors (24%). The class conswted of co"nsiderablymore female students 167';) than male t230I1. We comblnpd the s~udcntsIn Chem 125 and 126 into one group because the courses are taught at the same level of difficulty and by the same professor. The courses differed only in the specific topics under study but any variation in methodof administrationhetween the two courses was insignificant. Population three comprised nursing students enrolled in Chem 101 and 102 (Chemistry for the Health Sciences GeueraliInorganic Chemistry and OrganicBiochemistry) during the fall and spring semesters of the three academic years 198849, 1989-90, and 1990-91. This course sequence is designed primarily for the students in the School of Nursing. Chem 101 covers one semester of inorganic chemistry and Chem 102 covers one semester of organic/biochemistry. The 62 students in this population were predominantly female (99%). Most of the students were college freshmen (75%);the rest were sophomores (19%) and juniors (6%). Chem 101 and Chem 102 are significantly different in terms of the course material covered. However, the students enrolled in Chem 101 are the same students enrolled in Chem 102. Thus we can analyze the relationship between the GALT and achievement scores in both organic and inorganic chemistry for the same population. This enables us to observe whether the GALT will predict success more effectivelyfor one division of chemistry versus another. Research Design The relationship between a student's total GALT score and a student's achievement score in a particular chemistry class (see populations 1-3 above) was measured by linear regression using the SPSS-X Release 3.1 for VAXNMS (6).We then determined the correlation between both the student's math and verbal SAT scores, independently with achievement. \Vc also measured the correlation of the SAT (Scholastic Aptitude Test) scores (both verbal and math, independently) with the total GALT score The relative predictive power of achievement (dependent variable) by math SAT, verbal SAT, and GALT score (independent variables) was measured using stepwise multiple regression by forced entry of the independent variables into the multiple regression equation. We examined the respective P weights of all independent variables to determine how much variance in achievement was explained by each variable. For example, inspection of the P weights in the multiple regression equation determines whether the GALT Volume 70 Number 3 March 1993
183
explains a significant amount of additional variance over and above the math SAT and subsequently whether the math SAT and GALT are measuring the same variable. The 12-itemGALT was administered a t the beginning of the semester and compared to the student's achievement, measured as a percentage of the points obtained to those possible over the semester. The SAT scores used in the study were those on record in the respective schools within the University. Although some students reported their SAT scores at the beeinnine of the semester. the self-reported scores tended Lo be very different from'those on record and were deemed unreliable. In one case (nonscience majors) the GALT test was administered a second time a t the end of the semester to detect anv im~rovementin GALT score over a period of one semestkr. Instrument GALT Test (Group Assessment of Logical Thinking)
students must provide their 0% logical combinatorial patterns. Procedure The GALT was administered to every student in the class; the students were given 30 minutes to complete the test. Math and verbal SAT scores for the populations were obtained from the schools within the University where the students were enrolled. Achievement scores were entered into the data file as percentages of points achieved to the points possible over the entire semester. Results and Discussion Table 2 displays the means for the students'achievement scores, verbal and math SAT scores, and their GALT scores Table 2. Mean Population Scores on Achievement, Verbal and Math SAT and GALT
n Achieve- Verbal Math G A L T ~ Population Science courses attempt to foster the development of reamenia SAT^ SAP soning abilities among students. These reasoning ahilities are d~videdintocategories defined by Piagetians 17, as corScience Majors 50 relational. combinational. ~robabilistic.and ~ m ~ o r t i o n a l mean swre 80.98 498.00 581.50 8.50 logic. ~lso'includedare thLabilities to (dent& &d to constandard deviation 8.87 101.56 110.81 2.20 trol variables. The abbreviated GALT is al2-item DauerNonscience Majors 181 and-pencil Piagetian test of logical thinking that &&enmean swre 83.49 483.11 508.35 7.13 trates on the above six modes of reasonine. one of which is standard deviation 6.95 86.49 89.30 2.37 concrete operational (conservation of matter) and five of which are formal operational (proportionalreasoning, conNursing-Inorganic 62 mean score 72.78 480.34 491.70 7.14 trolling variables, probabilistic reasoning, correlational standard deviation 13.26 77.75 88.38 2.52 reasoning, and combinatorial logic). The GALT has been demonstrated to vredict critical Nursingarganic 62 thinking abilities and grades assigned by teachers in scimean swre 71.86 480.34 491.70 7.14 standard deviation 12.52 ence and mathematics for students in grades 9-12 (8).Bit77.75 88.38 2.52 measure of logical ner (9)has shown that the GALT is ; 'Highest possible smre for achievement = 100. thinking ability of eighth grade students and a predictor of b~ighest possible smre forverbal SAT = 8W. math and science achievement. The validity of the GALT 'Highest possible smre for math SAT = 800. d~ighestpossible smre far GALT = 12. was established by Roadrangka ( 4 ) on a student sample that varied in ages from sixth grade through college. Bitner (10) has also used the GALT as a predictor of academic by population. achievement for middle and sewndary students in 6 1 2 The nursing and nonscience majors demonstrate compagrade. rable ability as seen by comparison of their mean GALT Bitner (8, 9, 10) reports a 0.85 reliability coefficient for and math and verbal SAT scores. The nursing students did the internal wnsistenw value of the GALT bv calculatine not score as high in achievement as the nonscience majors Cronbach's alpha fo; the 21-item test. we obtainez but this may be attributed to different standards between Cronbach's alpha values for the 12-item GALT. and. exthe courses. The science majors demonstrated slightly tmpolatingtoihe 21-item test using the ~ ~ e a r m a n - ~ & v n higher GALT and math and verbal SAT scores than the formula, we obtained the values listed in Table 1. All are other groups. But because the students typically electing nearly comparable to those obtained by Bitner. this course have already demonstrated ability in the use of The abbreviated GALT test consists of 12 illustrated math and logic, it is not unexpected that the GALT and problems. For 10 of the 12 questions, studentsmust choose math SAT scores would be elevated in this oooulation. ~*~~~-~~~ -.--. the appropriate multiple-choice response for the problem Williams (11)rewrts a ~ositivecorrelation between the ----.-.. and the rationale, which is also in multiple-choice format, number of science courses a student has taken and the for the selected response. The question is considered to be student's score on the GALT test. Manv science educators correct only if both parts of the question, answer and ratiobelieve that actinty-hased sciencccuursescan enhance the nale, are correct. The remaining two questions, which condevelopment of logical thinking skills as measured bv the cern combinatorial logic, are not multiple choice and the GALT: We attempted to test this by measuring the students' GALT scores before and after a one-semester Table 1. Cronbach's Alphas for the 12-Item and 21-Item nonscience major's course. We found no significant correla(derived) GALT tion but believe this to be the result of inadequate length of exposure to science-based logical-thinking skills. Population Calculated a Cronbach's a Change takes time and a 13-week wurse may not be lone 12-ItemTest 21-Item Test enough to bring about measurable improvem&t. Science majors 0.6164 0.7377 Nonsdence majors 0.6267 0.7461 Com arison of Achievement Scores with GALT and Math and eehal SAT scores Nursinginorganic 0.6970 0.8010 The relationship between the students' achievement -oroanic scores, for all three populations, and their math SAT, verA
~
~~~
-
184
Journal of Chemical Education
bal SAT, and total GALT scores were analyzed by linear remession. The results are resented in Table 3. From Table 3 one can see that the GAL'I' correlates significantlv with achievement for all three oooulations but the degree of correlation depends upon population. The highest correlation between the GALT score and achievement exists for the nonscience majors @ = 0.0000), followed closely by the nursing-inorganic students @ = 0.0001), and nursing-organic students @ = 0.0002). The science majors correlation ofp = 0.0093 is also highly significant. The math SAT and verbal SAT also correlate with achievement and again, this varies with population. The three independent variables, math and verbal SAT and GALT, display a trend in terms of correlation to academic achievement in chemistry. The math SAT explains a good deal of the variance of achievement for the science majors (R2= 0.39961, less for the nursing populations (R2= 0.2667 and 0.2980), but little for the nonscience majors (R2 = 0.0739). A similar trend is observed for the correlation of verbal SAT scores with achievement-adequate explanation of variance for the nursing-inorganic population (R2= 0.2010), the nursing-organic population, (R2 = 0.20241, and the science majors (RZ= 0.1527), but low for the nonscience majors (R2= 0.0866). The GALT displays a completely different trend in that the variance in achievement scores is adequately explained for the nursing-inorganic (R2= 0.2406) and nursing-organic students (R2= 0.2186) and less well for both the science (R2 = 0.1411) and nonscience majors (R2 = 0.1268). In the case of the science majors the math SAT explains the variance in achievement better than does the GALT (R2= 0.3996 vs R2 = 0.1411). However, for the nonscience majors the GALT does better than the math SAT (R2 = 0.1268 vs R2 = 0.0739). For the nursing-inorganic students, the GALT and math SAT both explain comparable levels of variance in achievement (R2= 0.2406 vs R2 = 0.2667). The nursing-organic students follow the same trend as the science majors, that is, the math SAT explains
..
Table 3. Comprehensive Linear Regression Statistics of Achievement on Math SAT. Verbal SAT and GALT. Independently for the Three chemistry ~ o ~ u l a t i o n s Population
Correlation coefficient
2-tailed significance
Science Majors math SAT verbal SAT GALT
0.6322
0.0000
0.3908
0.0127
0.3757
0.0093
Nonscience Majors math SAT verbal SAT GALT
0.2718
0.0006
0.2943
0.0002
0.3560
0.0000
Nursing-Inorganic math SAT verbal SAT
0.5164
0.0000
0.4483
0.0004
GALT
0.4905
0.0001
NursineOrganic math SAT verbal SAT GALT
more variance in achievement than the GALT (RZ= 0.2980 vs R2 = 0.2186). Though it is obvious that the GALT is as good or better at ex~laininp ..variance in achievement than the math SAT scorei fur the nonscience and nursing-inorganic populations, the GALT was not air effective as the math SAT with the skience majors or nursing-organic groups. The GAL.^ was as good or better a t explaining variance in achievement than the verbal SAT with all populations. Comparison of GALTand Math SAT Scores
The next question to be addressed is whether or not the GALT and math SAT were both predicting the same eritical thinking abilities. We explored the correlation of math SAT and total GALT score by linear regression. The results are presented in Table 4. Table 4. Linear Regression Statistics of Total GALT Score on Math SAT Score Population
Correlation coefficient
2-tailed significance
Science majors
0.6813
0.0000
Nonscience majors
0.4930
0.0000
Nursinginorganic
0.5791
0.0000
A significant correlation exists between the math SAT score and the total GALT score for all populations. The highest correlation occurred for the science major population ( r = 0.6813) and lowest for the nonscience majors ( r = 0.4930). The nursing population, including both inorganic and organic sections, had identical correlation coefficients ( r = 0.5791) as their only difference stems fmm different achievement scores in Chem 101 and 102. All correlations in all populations were significant (2-tailed significance = 0.0000 for all). The f a d that the correlation is moderate to high for all populations in this study indicates that there is overlap in the variables measured by both the GALT and the math SAT tests. Comparison of GALT and Verbal SAT Scores
We next investigated the overlap between the GALT and verbal SAT. From Table 5 we see that the verbal SAT score is positively correlated to the total GALT score for all populations. We can see that the highest correlationoccurs for the science majors ( r = 0.6450) and then for the nursing majors ( r = 0.5905) but that the correlation is low for the nonscience majors (r = 0.2943). This is similar to the pattern that describes the relationship between the GALT and math SAT except that there is a lower correlation between the Table 5. Linear Regression Statistics of Total GALT Score on Verbal SAT Score Population
Correlation coefficient
2-tailed significance
Science majors
0.6450
0.0000
0.5459
0.0000
Nonscience majors
0.2943
0.0002
0.4499
0.0004
Nursing-inorganic
0.5905
0.0000
0.4676
0.0002
Nursing-arganic
0.5905
0.0000
Volume 70 Number 3 March 1993
185
GAET and verbal SAT for the nonscience majors than that between the GALT and math SAT. Does the GALT Measure the Same Variable as Verbal and Math SAT? Hence we embarked w o n a multide-reeression analvsis to determine if the GALT was indeed measuring the &me variables as the math and verbal SAT scores. Analvsis of B weights as calculated in a multiple-regression &alys& with forced entry of variables in the following order: math SAT, verbal SAT, and then the total GALT score shows the relative importance of all three variables to the prediction of success in chemistry for three populations. From Table 6 we see that all three inde~endentvariables make significant contributions rp = 0.000, to the prediction Bv analvzine the B weiehts of these of achievement ~R'A variables, we can compare the relative impo&nce of each independent variable (math SAT, verbal SAT, GALT) to achievement within a given population (science majors, nonscience majors, nursing-inorganic, nursing-organic). Beta weights are a standardized measure of the relative contribution of each independent variable in a regression equation. Because they are a standardized measure, we can compare the p weights of a particular variable both within and across populations. Within the science major population, it is obvious that the addition of GALT to the regression equation boosts the amount of variance explained (R2)from 0.4165 (math and verbal SAT) to 0.4255. Atotal R2 of 0.4255 for a given population is an important indication that mathematical aptitude and logical thinking ability make up a major part of the success in chemistry for this population. Here the P weights show that the contibution of math SAT (0.7195)is more than five times as important as that of the GAJiT (0.1351) in the regression equation. For nonscience majors, the picture is very different. The GALT score increases the amount of variance explained Table 6. Multiple Regression Summary for Prediction of Academic Achievement bv the Variables Math SAT. ,~ ~
Verbal SAT and GALT
Population
p
$
WeigM
F significance
Science majors math SAT verbal SAT
0.6322
0.3996
0.7195
25.30
0.0000
0.6453
0.4165 -0.2382
13.20
0.0001
GALT
0.6523
0.4255
8.88
0.0002
0.1351
Nonscience majors math SAT verbal SAT
0.2718
0.0739
0.0738
12.44
0.0006
0.3270
0.1069
0.1313
9.28
O.MM2
GALT
0.3944
0.1555
0.2658
9.45
0,0000
Nursing-inorganic math SAT verbal SAT GALT
0.5164
0.2667
0.3061
20.73
0.0000
0.5337
0.2849
0.0868
11.15
0.0001
0.5705
0.3255
0.2620
8.85
0.0001
Nursingarganic math SAT verbal SAT
0.5459
0.2980
0.3665
23.35
0.0000
0.5635
0.3176
0.1085
12.56
0.0000
GALT
0.5833
0.3402
0.1955
9.11
0.0001
' f ? recalculated after addition of
lion.
186
each predictor variable for a given papula-
Journal of Chemical Education
(R2) from 0.1069 (math and verbal SAT) to 0.1555. In this population, mathematical aptitude and logical thinking ability, combined, account for only about one-third as much variance as they did in the science major population. Analysis of the p weights shows that in this population, the coutribution of the GALT score (0.2658) is more than three and a half times as important as that of the math SAT score in the regression equation. The results for the nursing inorganic and organic population are in-between those of the science and nonscience populations. Here the GALT boosts the amount ofvariance explained (R2)from 0.2667 to 0.3255for nursing-inorganic and similarly from 0.2980 to 0.3402 for nursing-organic. In both groups, the math SAT makes a larger contribution to the regression equation than GALT (0.3061 vs. 0.2620 for nursing-inorganic) and (0.3665 vs. 0.1955 for nursingorganic). This analysis leads to the conclusion that the math SAT is the most important contributor to success in chemistry of the three independent variables measured for the science majors, but the least important for the nonscience majors. The GALT, which adds little to the prediction of success for science major,s is the most important variable of those measured for nonscience majors. Nursing majors fit in-between science and nonscience majors with the math SAT as the single most important predictor, but followed closely in importance by the GALT. In answer to the question of whether GALT and math and verbal SAT scores measure the same variable, the answer is they are highly correlated with each other (Tables 4 and 5) and seem to be measuring similar variables for the science and nursing populations, but, in the nonscience population, GALT is a much more important contributor to the prediction of success than either the math or verbal SAT scores. Summary of Results In summary, for science majors, the GALT appears to be measuring a similar variable to that measured by the math SAT score although it does not da so as well a s the math SAT. For nursing students, GALT appears to be closer to the math SAT score in predicting success, but for nonscience majors the GALT is definitely better than the math SAT score in predicting success in chemistry. Taking into account the correlations between GALT scores and achievement in chemistry for all three populations (Table 3), together with the high correlations between GALT and math SAT scores (Table 41, we conclude that scores on the GAET test can be used as an alternative to the math SAT score to predict success in chemistry. For special populations such as nonscience majors, the GALT score may even serve as a better predictor of success than the math SAT. This may be due to the fact that nonscience major chemistry courses typically include less mathematical manipulation than a chemistry course for science majors. It is the ability to do math manipulations that is measured by the math SAT. The same logic can be used to explain why the GAIT is at least as good, if not better, as a predictor for success in a nursing chemistry course. Nursing chemistry courses typically include mathematical manipulations similar in kind but not in level of difficulty to those in a science majors course. Nursing chemistry courses also place a good deal of emphasis on the understanding of chemistry concepts without relying on math. Since a nursing course may be considered intermediate in its use of math between that of a science major and nonscience major's course, it is logical that the GALT would be nearly as good a predictor of success as the math SAT score here.
Conclusion It seems reasonable then that the GALT test, which is quick and easy to administer within a chemistry course, can be used to identify students at risk of failure regardless of the level of chemistry taught. Math SAT scores, which might serve as a better predictor of success in a t least the science majors course, are often difficult for a professor to obtain. The GALT scores, on the other hand, are relatively easy. If both scores measure similar variables for the success of ~ciencemajors then the CA1.T could be the tcst of chuice for college professors. The GALT is even more effective as a predictor-in nursing and nonscience major courses. Of course there is always an element of fear that test scores will be used as a barrier for admission to a course or as a "weed out" instrument. No single predictor is foolproof and this is especially true here. GALT scores should be interpreted as an early warning device to alert both parties to the need for early intervention designed to
help assure student success in the course. Further studv is needed to ascertain what combination of helptutorial, study aids, computer or videodisk interaction, or the use of models-will be the most effective for a particular student. Literature Cited 1. Chsmpagne, A . B.;Ho-8,
L. E. Studmtr and S c k e L e o r n i n g ; AAAS: Washington,D.C., 1987. 2. Resnick. L.B.Edumti~nondhingfoThink;NatianalAeademyPress:WaahingUrn, D.C.. 1987. 3. 0zsognmanyan.A:LoRus.D.J. Chsm. Educ 1979.56,173-175. 4. Raadranga, *; Yeany, R. H.:Padilla, M. J., University of G ~ o r g avnpvbliahed , results, 1982. 5. i7osdrangka.Y Y e w R. H.; Padilla. M. J.. NARST Annual Conference, unpub. h h e d results. 1983. 6. SPSSI Users'Gutdp, 3rd ed.:SPSS Ine: Chicago, 1988. 7. Good. R.: Mellon. E. K.: h m h o u t . R. A. J. Chem. Educ.1978 55.. 6 8 8 4 3 8. Bimer, B.L J. R ~ Z sci. . n o c h 19h,z8,2 ~ 1 4 . 9. Biker. B. L., NARSTAnnual Conference, unpublished results. 1986. 10. Biker, B.L.,NARSTAnnual Conference, unpublished results, 1988. 11. Wiliiams.R. L..NARST Annual Conference. unoublished results. 19R9
.
Volume 70 Number 3 March 1993
187
