The Effect of Language on Student Performance on Multiple Choice Tests in Chemistry J. R. T. C a s s e l s and A. H. Johnstone University of Glasgow, University Avenue, Glasgow 612 8 0 0 Many chemistry students at all levels are assessed by means of multiple choice questions. I t is easy to assume that differences in performance are due solely to differences in chemical ability. This study has shown that the language in which the question is couched may be a very significant factor in the student performance. Fuller details of this study are to he found elsewhere (1-6). Matched questions testing the same chemistry in different words were designed and organized into two tests and administered t*, two groups of pupils in each oithree st~ccessiw years. The total sample was in rxcess of3600 pupils at ahnut age 16. '1'0 rheck whether this method of samnline did test similar populations, both tests contained questink wiich were identical (control auestions). The results for the control questions hdicated ;hat the samples of pupils sitting the tests did not differ sienificantlv. and conseauentlv. it was assumed that differences exhihitkd between'any pair of matched questions were due to factors operating within the questions rather than factors associated with differences in the sample uouulation. The similaritv in the results for those control qukstions even between successive years with different pupils indicated the reuroducibilitv of the experimental conditions. Results Key Words Results indicated that the substitution of words (in key oositions) hv simoler words hroueht about an imurovement in performake. fn the examplesin Tahle 1nnly'the part of the question which has been changed is shown along with the percentage of the group choosing the correct answer. (In all of the examples in this paper the improvement in performance was statistically significant a t better than the 5%level.) Replacement of expressions such as "distinguish visually," "tendency t o predominate," "following procedures . . . expected to produce" brings about an improvement in performance. In multiple choice items there is a tendency for the style of the question to become formal in order to achieve precision. If, however, this formality becomes pompous and alien to the candidate, then it mav make the questions more difficult. Terms of Quantity Terms of quantity have within them an implicit direction which seems to affect performance (see Table 2). "Most ahundant" seems to be easier to understand than either "least concentrated" or "most dilute." This may he a reflection of the fact that the language change is influencing the thinking oattern necessarv to answer the auestion. Consider the comparison between "most abundant" and "least abundant." "Most" is an exoression indicatine" a . oositive measure of magnitude and so is "abundant." "Least" is an expression indicating a necatiue ., measure of magnitude and. conseauentlv by this argument, is operating in a direction opposite to "ahundant." Therefore "most ahundant" mav he easier to undt:r>tnnd berausr it does not invulw the unntvelling ofth*: ~louhlethink o f ' l m i t ahundnnt." In t,thrr words. there may
-
Table 1. Effect of Changlng Key Words %
%
Oriainai Question
correct
Which one of the following is a pungent gas? Which is the least stable sulfideamong the following?
56
Which One of the following requires a non-aqueous solvent to di~soiveit?
Simtliified Question
correct
the foiiowing is a choking gas? Which one of me fallowing sulfides is easiest to break down to its dements? Which one of the following requires a liquid other than water to dissolve
63
Which one of
40
34
49
49
it?
Table 2. Effect of Changlng Terms of Ouantlty %
%
Orioinai Question
correct
brass contains 75 pet cent copper. 22 per cent zinc. 2 per cent aluminum, and 1 per cent lead. Which is the leas abundant element in this brass? Which one of the foiiowing soiutions of a salt in water is the least concenhated?
75
A
59
Sim~lified Question
correct
brass contains 75 per Cent copper. 22 per cent zinc. 2 per cant aluminum, and 1 per cent lead. Which is the most abundant element in this brass? Whlch one of the foiiowing solutions of a salt in water is the most concentrated?
84
A
85
be fewer thinking stages necessary to solve the questions containing "most concentrated" and "most ahundant." Negative Forms It is well established that negative forms of auestions create problems for students. The rekoval of negat&es, in general, seems to improve the performance (Table 3). and again this may indicate that the language change influences theihinking necessary to answer the question. In a question without a negative there are a number of thinking stages in order to get to the correct answer, but in a question with anegative it may hc poisihlr t h ~ there t ii at leasimeadditional thtnking stage Ifdoul~lenrgatives occur (as in the second exam plea hove^ the complexity may rise alarmingly. Large Numbers of Words and Arrangement of Clauses Wordy questions with embedded clauses tended to he more difficult than short auestions written in short sentences (Table 4). In multiple choick questions information has to he read and considered bv the candidate before a resnonse is made. The organization of this information in the question has an influence on performance. The removal of extraneous information by reducing the number of apparently unnecessary words is a simplification. Presenting the information in a clear, Volume 61
Number 7
July 1984
613
Table 3.
Enect ot Changing from Negative to Posllive Forms %
Sb correct
Oriainal Question
correct
Sim~lifiedQuestion
Which one of the following particles does not have the same number of electron^ as a calcium ion?
24
Which one of the following particles has the same number of elecbons as a calcium ion?
80
Which statement is not true? a) The proton does not havethesame mass as an electron. b i The nydrogen malecu e conlams two atoms C ) sotopes of chlor ne do not have different numbers of protons in their atoms. d) The element with Atomic Number 13 is a nonmetal.
80
Which statemem is true?
80
a) The proton has the same mass as an electron. 0 ) The nyorogen m l e c u e ~ontilm tnrce atoms. C ) is010pe~01 chinme have dilferent numbers of protons in their atoms. d) The elemem wim Atomic Number 13 is a metal.
organized fashion or in simde sentences rather than lareer. - . complex sentences with embedded clauses can bring about an im~rovementin ~erformance.This mav again reflect the language change influencing the thinking hec&ary to answer the question. If the external organizational changes in the presentation of the information in the question reduces the internal intellectual organization necessary in order to answer the question then the finguistic change brings about a change in performance. Minor Changes in Parts of Speech Minor changes in parts of speech do not seem to influence performance, and framing the question in the passive or active voice seems to have little effect (Table 5). A possible reason may he that these language changes do not obscure the underlvine - meanine of the auestion as a chanee in vocahularv might do, nor do they hring about any major organizational changes in the oresentation of information. Since these changes have little influence on the thinking necessary to answer the auestion thev do not alter the measured ~ e r f o r mance.
-
Ened ot Rewording Long w Complex Ouestlons %
Original Question
When a metal Z was
Simplified Question
47
Three metals 2 , T, and Y were added to separate solutions containing metal X ions. Z precipitatesX; Thad no effect: and with Y a aas was glven oll from the ~oluttonWhich one show* me metals m ihe wrrect order of anivity (the most reactive first)?
67
38
Atoms of X have two slecbons in the outer shell. Atoms of Y have seven electrons in me outer shell. Atoms of X and Y combine. Which one of the following statements is true?
48
added lo me sulfate of a metal X, the metal X was precipitatedand there was no effervescence. When the test was repeated using the metal T in place of Z no reaction occurred. When a metal Y was added to a solution of the sulfated X. a brisk effervescence o c ~ u r r e dWhich one of the following is the Comct order of decreasing activity (i.e.. the most reactive first) of me four metals? Atoms of element X. which have two electrons in me Outer shell, combine with atoms of element Y. Which have seven electrons in the outer shell. Which one of the following statements is
% consct
mect
,.,,a?
Table 5.
Enect ot Changing from Passtve to Active Voice
OTlginal (luestion In which one of me following compounds does covalency predominate?
% correct
69
% Simplified Question
conect
Which one of the following compounds is predominantly covalent?
71
memory and long term memory) a widely supported idea (11,
Discussion
.1.5 - , -1-6,.) ~
T o think of reading as primarily a visual process would he erroneous (7). Reading involves the interaction of the information that the reader receives through his visual system and. the information that reader already has available in his long-term memory. This enables the reader to recognize and organize the incoming information and to make sense of it. Unfamiliar or confusing words and construction come into conflict with the organizational process. Smith (8)quotes the example, przyjezdzac, which a Polish speaker would recognize a t a glance hut a non-Polish speaker would not understand. The cognitive processes may he considered to involve the interaction of the comnonents of memorv: workine memorv and long-term memo&(9,10). The working memo6 in which the interaction is thought to take lace seems to have a finite capacity. Miller (11) observed inkariences in memory-span experiments indicating a working memory capacity of about "seven, plus or minus two" familiar units or chunks of any kind. The existence of chunks, i.e., coherent memorv or oerceptual units, in language processing and recalling has deen established (12, 13). However, the size and nature of each chunk that can he stored in the working memory is determined not so much by external considerations as by what the brain is prepared to do with the information (14). This brings us hack to the processing (the interaction between working 614
Table 4.
Journal of Chemical Education
The suggestion in the previous section was that the language in the method multiple choice questions was influencing the thinking orocesses necessarv to answer the auestion. A numhe;oi sources would tend to add support to this statement. For examole, investinations into neeative exoressions (17-21) seem to ;uigest that they are encoded in the memory as a separate chunk aoart from the rest of the sentence. The claim is that sentences with a negative require more of the working memorv caoacity . . than do otherwise identical sentences &king the negative. In addition, several investigations (22-25) seem t o suggest that pupils have more difficulty understanding negative comparative adjectives (e.g., shorter, less) than their opposites (taller, more). Their interpretation is again based on the presence of an additional chunk encoded in the limited space of the working memory. Similarlv there are sueeestions: (1)that the memorv snan is not determined by thenumber of words hut by the grammatical structures (e.e.. embedded clauses) that mav themselves load the memo6 (26) and; (2) that the importa& factor in the sentence is its meanine and the svntax is foreotten when e are consistent the memory is unravelled (i7).~ h e s kndings with a theory of comprehension which contends that the meaning of a sentence is derived from the original string of words by an active interpretative process. d
.
Conclusion
Our investigation with matched multiple choice questions may he able to he placed in this wider theoretical framework. If the change in the question either reduces the number of thinking stages or makes the thinking easier, then the performance is improved. If, on the other hand, the change does not alter the number of chunks in the short term memory, does not make the meaning more accessible, and does not influence the thinking necessary to solve the question, then there is no influence on performance. If the influence of laneuaee on oerformance in multiole choice questions can he demkstrated then what effect m&t laneuaee .. have on the teachine and learnine that takes l lace prior to any assessment? If ;I student makes sm5e of what the teachrr snvs hv orocessinr! it in thr light of what he already knows (or'thi"ks he knows) i t is littl'wonder that students can lay hold of completely wrong ideas. It is hoped that this research makes some contribution toward the real need identified in the Bullock Report (28).
..
(3) Cameis. J. R. T., and Johnstonc.A. H.. Bull. Scot. Centre Malh., Sci. Tech. E d u . , 11, 19-24 (1977). (4) Caaael8, J. R. T., and J0hnatona.A. H. in "Reportof Education DivisionConfnfnfnee: Research for theClassrwm and Beyond," The Chemical Saeiety,London, 1971,pp. d."e F-. d
Caxrir..l R.T..md .inhnrtonr.A. H..Nru i r r m r , a l , l R 2 t18Msy 19181 16, C s c v 1 R T . m d .ll+n.mnr.A tI..'ThcIlndmtnndingnf Non-lrrhuirnl Wordr ~n 5&nm-a Ropnn of s Rcsmwh El;rr&r." Thr Chcmtral Swtrry, I.nndon. 1980. (7) Kalem, P. A,, "Psycholinguistics and the Teaching of Reading, (Editon: Gmdman, K. S., end Fleming, J. T.) lntpmational Reading Assoeistian, Newark, 1969. (8) Smifh.F.,in"FoundatianaofLanguageDwelopment:AMultidiaeipiinaryAppmaeh," (Editor Imncberg, E. H.) Academic Press. New York, 1915. Vol. 2 , p p 317360. (9) White, R. T., Re$. Sci. Edue.. 7.25 (1977). (10) Feigenbaum. E. A , i n "MdelsofHuman Memory,"(Edilor Norrnan,D. A),Academie Press. New York. 1970. (11) Miller,G.A.,Psyehol. Rou.,63.81 (19561. (121 F d a r , J. A , and Beuer, T. G., J Verbal Learning Verbal Behouiour, 1, 414 IS
~.""-,. (lM5,
(13) McLean. R. S., and Grsgg, L. W., J. Erp. P~ych.,74,455(IWI). (14) Smith, F. and Halmes. D. L.. Reading Re8 Quart., 6,394 (1971). (15) Miller. G. A..andChamsky, N.,in "Hsndbwkof MathematiealPsycholw,"(Edifors: Luce, R. D. et ai.) John Wiley & Sons, London, 1963. Vol. 2. (16) Simon,H. A.,andChase,W.G.,Amrr. Sci.,61,394 (1973). (17) Wason, P. C., QuolifyJ Erp. Psych., 11.92 (1969). (18) Wason, P. C., Brit. J. Psych., 52,133 (1961). (19) Wason, P. C.."The Contentp of Plauaibla Denial,'. J Verbal Looming VerbolBeholiinlrr . .. .. ,A.,7. ,1969 ,. ...,.
(20) Eihrmann, R. R., AetaPsychol., 18, 258 (1961). (21) Ssvin, H.B..andPerchonaeh,E., J. of V e v
\.""",.
b
b
l
L
P
o
3
4
8
,*WE,
them; and that such an understanding involves his payingparticular attention to the part language plays in learning.
Literature Cited (11 Casela, J. R.T.,"Langusae in Chemistry: TheEffectofSomeAspeetrofI.anguageon WGrade Chemiatw Candidates,'' M.Sc. thesis. University of Giasgow, 1976. (2) b e l s . J. RT.."Languageand Thinking in kionee: Somelnve.tigsiionsvlth Multiple ChoiceQuestions,"Ph.D. thesis,Univerrity of Gbw,ow. 1980.
(221 Donaldm, M., and Wa1ea.R J.. "Cognition and theDwelopment of L8n-e" John Wiley & Sona. New Vork. 1910. (23) Donaldsan, M., and Balfour, G., Brit. J. Pqvch., 69,461 (1968). (24) Palerrno. D. S.. "More about L e d . J. Verbol Lmrning Verbal Behouiov. l2.211
,,"-"\ \Lz,.,,.
(25) Clerk, H.H.,Sebeok,T. A.,ii "CcccctTr~rnddiiLi"guiitititi,"(Editor: T h e H w e ) , Mouton, 1912,Vol. 12, pp. 1291-1428. (26) Miller, G.A.,and Iaard, S.,Informotion and Control. 1,292 (1961). (27) Sachs. J. S.,Perception ondPsyehophysics, 2.437 (1967). (28) Bulinck,A.,"ALsnguagefffLife," Reportof the CcmmittetetefInquiryapp~iifedby thcSecretaryof'tz:*hiRd.nt,iou :mii L:iexevnder the Chairma"ship0fSir.A. Rullock, F. B. A,. HMSU. 19'75.
Volume 61
Number 7 July 1984
615
