Difficulty and Discrimination of Multiple-choice Questions:A Counterintuitive Result John P. Sevenair Xavier University of Louisiana, New Orleans, LA 70125 Allan R. Burkett Dillard University, New Orleans, L A 70122 Over the nast several vears. , . we have develooed test hanks ur multiple choice questions in organic chemistry1 and genhave beenevalerulchemistrv .'Chcureanicrh(~mi.itr\~itcms uated using ieveral methods. In pakicular, the difficulty factor (the nercentaee of students who eive an incorrect answer) and the dis&mination factor measure of the correlation between student performance on an individual item and performance on the exam as a wholeI3 were calculated for 230 test items. One of the results of these calculations is strongly counterintuitive. Questions can be more difficult, and can discriminate more strongly, than is predicted hv one ~lausihlemodel of student behavior. The d"iscrin;ination factor of an item is calculated using the equation
(a
H-L
discrimination = 0.275 N
H a n d L are the number of correct responses from the highest and lowest scoring 27.5% of the students, respectively, and N is the total numher of students. The discrimination factor must have a value of zero when the difficultv factor is either 0% or 100%. The maximum possible value is attained when all of the stronger students get the question right and all of the weaker students get the question wrong; this is 1.00 for difficulty factors between 27.5% and 72.5%. This theoretical maximum is shown by the solid line A in the figure. A plausible intuitive model predicts a lower maximum. The top students are expected to get the correct answer, as before. This model states that weaker students guess, gettine" the correct answer 25% of the time for four-foil items. The maximum possible discrimination factor from this model is 0.75, reached as difficulty increases when the top 72.5% of the students get the item correct and the bottom 27.5% euess. The difficultv factor is (0.275)(0.75). or 20.6%. a t this point. The maximum will remain at 0.75 until on]" the top 27.3'"~r r t the correct nnswrr nnd the rcst mess ~diiiiculls= (0.725)(0.75) = 54.5%), and should sink'to zero when t h e difficulty reaches 75%. This model states that all students are guessing if the difficulty factor of an item is 75%, and predicts that the difficulty factor should never exceed 75%. This prediction of the maximum is shown by the dotted line B in the figure. Approximately 80 students took each version of the organic chemistry tests used in this study; in many cases two versions with scrambled question order were given, each to 80 students. In these cases statistics were determined for the two versions se~arately.The difficulty factors were almost always reprodukble td within 5%. he discrimination factors for all items whose difficulty factors fell within each 5% interval were averaged. These vdues are indicated by open circles in the figure. It is notable that the average experimentally determined discrimination factor exceeds the maximum value predicted by the intuitive model for all difficulty factors greater than
Theoretical maxima and observed values far the discrimination factor vs. the difficultyfactor. A, top students get correct answer and bottom get incorrect: 8, top students get correct answer and bottom students guess: open circles. averages of observed values.
65%. The individual discrimination factors exceeded the theoretical maximum for 23 such items, were equal to the intuitive model maximum in eieht cases. and were less than the maximum for only six. he difficulty factor exceeded 75% for 14 items. These unusual results can he interpreted in terms of the overload of working memory hypothesis of A. H. Johnstone4. Stronger students, who can answer the item in less than five cognitive steps, will get the correct answer. Weaker students appear to follow a series of steps that stops short of the correct response a t an incorrect one. We have apparently learned to oredict such resoonses and include them amone" the foils. In some cases. a t least. the reason for hizh difficultv and significant discrimination is apparent when the question is examined. One such question is:
-
What are the major products of the reaction shown? CH3C=C:-Nat (CH,),CBr ?
+
'
Sevenair. J. P.; Burkett, A. R. Multiple Choice Duestions To Accompany Organic Chemisty; Allyn and Bacon: Boston, in press. Burken. A. R.; Sevenair, J. P. Multiple Choice Questions To Accompany Chemistry; Allyn and Bacon: Boston. 1986. Bodner, F. M. J Chem. Educ. 1980,57, 188-190. Johnstone, A. H. J. Chem. Educ. 1984, 61,847-849. Volume 65
Number 5
May 1968
441
This question had a difficulty factor of 0.79 and a discrimination factor of 0.2; the nucleophilic substitution products were the most freouentlv- eiven answer amona- the lower " scoring students. A similar question about the reaction of (CH&CHCH=CH2 with HI gave a difficulty factor of 0.80, a discrimination factor of 0.3, and the unrearranged product (CHhCHCHICHs as the most popular answer. One of the provided striking example from an refer& of this introductory chemistry course. The question was "What is the molecular weight of copper(I1) iodide?"This item gave a difficulty factor of 0.69 and a discrimination factor of 0.8; the
a
442
Journal of Chemical Education
answer most often given was the molecular weight of "Cu21". The results are a striking statistical example of the adage "A little learning is a dang'rous thing."bThose students who have learned nothing guess, and get the correct answer 25% of the time. Those students who have learned a little (apparently a much larger group than those who have learned nothing) often follow a thought process that leads to the wrong answer, with a chance of getting the item right considerably less than 25%. Pope. A. An Essay on Criticism; 1711; Part II. line 15.
