Analysis of Student Performance on Multiple-Choice Questions in

ARTICLE pubs.acs.org/jchemeduc

Analysis of Student Performance on Multiple-Choice Questions in General Chemistry JudithAnn R. Hartman* and Shirley Lin* Department of Chemistry, United States Naval Academy, Annapolis, Maryland 21402 United States

bS Supporting Information ABSTRACT: The percentage of students choosing the correct answer (PSCA) on 17 multiple-choice algorithmic questions taken from general chemistry exams is analyzed. PSCAs for these questions varied from 47 to 93%, and a decrease of 4.5% in PSCA was observed with each additional step in the algorithm required for solving the problem (R2 = 0.80). Variants of a subset of these questions were also examined and reveal the effect of making changes to a particular algorithmic question on the ability of students to choose the correct answer. KEYWORDS: First-Year Undergraduate/General, Chemical Education Research, Problem Solving/Decision Making, Testing/ Assessment, Learning Theories, Nonmajor Courses FEATURE: Chemical Education Research

A

continuing challenge in chemical education is the development of instruments to accurately assess the learning and knowledge of our students. These measures may include written quizzes or examinations,1
3 homework,4
9 laboratory reports,10
15 oral examinations,16 students presentations,17,18 and laboratory practicals.19
22 In the category of written examinations, a commonly used format is the multiple-choice question. This format enables questions answered by large classes of students to be graded as correct or incorrect with minimal error in a timely manner. However, multiple-choice questions typically do not lend as much insight into the students’ fundamental understanding of the material as do hand-graded, worked-out questions. Furthermore, in general chemistry, the majority of multiplechoice questions found in test banks accompanying textbooks tend to be focused on calculations that may be solved by algorithmic means without an understanding of the underlying concepts. The lack of equivalence between the ability to solve problems and understanding of molecular concepts is already well-known in the chemistry pedagogical literature.23
29 Multiple-choice questions have been a component of general chemistry examinations at our institution for many years. A standard practice in this course has been the administration of “common exams”, exams taken by all students registered in the course, regardless of instructor (see below). In order to facilitate writing these exams and to compare student performance over time, a database was constructed from the multiple-choice questions given on these exams over the last 20 years. The questions were organized by topic, and the percentage of students who gave the correct answer (PSCA) each time the question appeared on an exam was recorded. To date, the database contains >1800 multiple-choice questions and >2400 PSCAs; the majority of PSCAs (∼60%) were collected from exams administered recently (Spring 2003
Fall 2009). We realized that analysis of the PSCA data in the exam database could be of interest to a larger audience outside of This article not subject to U.S. Copyright. Published 2011 by the American Chemical Society

our own department for several reasons. The first is that these data were gathered in a rigorous way. Every one of these questions was answered by at least 900 students each time it appeared on an exam. This large sample size would presumably mitigate many of the statistical fluctuations one might expect from small sample sizes. Second, all students at USNA are enrolled in general chemistry regardless of major unless they achieve a sufficiently high score on an in-house exam. The data presented here therefore are a measure of the abilities of students destined for both technical and nontechnical majors. Furthermore, unlike many other institutions, all our students are required to attend class unless they have a valid excuse. Although instructors cannot guarantee the engagement of each and every student while they are in the classroom, the PSCA data is largely free from the complication of a variable number of frequently absent students. We also note that these questions were answered by a geographically diverse (representing all 50 states and some foreign countries) and high-achieving student population. For the students who provided the data for the PSCAs on exam questions administered from 2003 to 2009, the average SATM score was 663 and SATV score was 639. The ratio between technical and nontechnical majors varied between 1.3 and 1.8 during this time. The most notable difference between the student population at USNA and at most other colleges is that the average percentage of women in these graduating classes was 19.5%. However, we believe the PSCAs presented here are a reliable representation of how a large, diverse, student population performs on certain general chemistry exam questions.

Published: July 14, 2011 1223

dx.doi.org/10.1021/ed100133v | J. Chem. Educ. 2011, 88, 1223–1230

Journal of Chemical Education

ARTICLE

Table 1. General Chemistry Algorithmic Multiple-Choice Exam Questions and the Percentage of Students Choosing the Correct Answer (PSCA) PSCAs for this question algorithmic PSCA av, item

question subject

steps

A1

calculate % composition

3

type (no. of different

% (SD)a questions in database)b 91 (5)

6 (4)

What is the percent by weight of sodium in Na2CO3?

from molecular formula A2

determine molecular

example questions (bold indicates the correct answer)

a. 11.3%; b. 21.7%; c. 30.2%; d. 43.4% 3

77 (2)

6 (3)

A compound contains 40.0% C, 6.71% H, and 53.29% O by mass.

formulas from % mass

The molar mass of the compound is 60.05 g/mol. The molecular formula for the compound is a. CH2O; b. C2H4O2; c. C3H8O; d. C2H2O4

A3

limiting reactant

2
5

75 (8)

13 (9)

The following reaction is used as a test for limestone: 2HCl(g) þ CaCO3(s) f CaCl2(s) þ H2O(l) þ CO2(g). When 4.50 g of HCl are reacted with 15.00 g of CaCO3, which is the limiting reagent and how many grams of CO2 are produced? [Molar masses: HCl (36.46 g/mol); CaCO3 (100.09 g/mol); CO2 (44.01 g/mol)] a. CaCO3, 3.30 g; b. CaCO3, 6.60 g; c. HCl, 2.72 g; d. HCl, 5.42 g

A4

use NA to find no. of

3

75 (9)

The number of H atoms in 22.2 g of glycine, C2H5NO2 (75.07 g/mol) is

6 (2)

a. 1.48 atoms; b. 1.78  1023 atoms; c. 8.90  1023 atoms;

atoms given molecular

d. 3.01  1024 atoms

formula A5

stoichiometry in acid
base reactions

3

89 (2)

3 (2)

A6

Hess’s law

3

85 (8)

6 (13)

What volume of 0.336 M nitric acid (HNO3) solution is required to neutralize 32.8 mL of 0.135 M sodium hydroxide (NaOH) solution? a. 13.2 mL; b. 81.6 mL; c. 2.49 mL; d. 25.0 mL Given the following information, calculate the ΔHrxn for the reaction Na(s) þ HCl(aq) f 1/2H2(g) þ NaCl(aq) 2Na(s) þ 2H2O(l) f H2(g) þ 2NaOH(aq); ΔH =
367.6 kJ H2O(l) þ NaCl(aq) f HCl(aq) þ NaOH(aq); ΔH = þ55.86 kJ a.
431.4 kJ; b.
239.7 kJ; c.
4.2 kJ; d.
10.0 kJ

A7

relationship between

2
3

85 (13)

8 (6)

When Cs metal is irradiated with light, it takes a photon with an energy of 2.65  10
19 J to eject an electron.

E, h, n, l, and c

What is the wavelength of this light? a. 900 nm; b. 750 nm; c. 400 nm; d. 285 nm A8

determine no. of valence electrons

A9

determine molecular

2
3

76 (3)

4 (3)

6

77 (11)

6 (3)

The total number of valence electrons in the perchlorate anion, ClO4
, is a. 28; b. 30; c. 31; d. 32 For the oxygen atom of H3Oþ, the electron domain geometry is ___ and the molecular geometry is ___.

shape using VSEPR

a. tetrahedral, bent; b. tetrahedral, trigonal pyramidal; c. trigonal bipyramidal, linear; d. trigonal pyramidal, bent A10

determine molecular

A11

ideal gas law

7

69 (3)

2 (2)

2
3

86 (9)

21 (16)

Which one of the following compounds is nonpolar? a. CO; b. HCl; c. NCl3; d. CCl4

dipoles

What pressure is exerted by a 0.922 mol sample of a gas at 110 °C in a volume of 0.696 L? a. 1.00 atm; b. 12.0 atm; c. 22.4 atm; d. 41.6 atm

A12

rate law from

4

73 (11)

9 (4)

Use these data below to determine the rate law for the following reaction: 2A(aq) þ 3B(aq) f 2C(aq)

initial rates

Trial 1: [A] = 0.15; [B] = 0.12; Initial Rate (M/s) = 0.10 Trial 2: [A] = 0.15; [B] = 0.24; Initial Rate (M/s) = 0.20 Trial 3: [A] = 0.45; [B] = 0.24; Initial Rate (M/s) = 1.80 a. rate = k[A]3[B]; b. rate = k[A]2[B]2; c. rate = k[A]2[B]; d. rate = k[A]2 A13

calculate Keq

2

93 (5)

6 (5)

An equilibrium mixture at a constant temperature contains 0.0574 M NO, 0.126 M Cl2, and 0.248 M NOCl. Calculate Kc for the reaction: 2NO(g) þ Cl2(g) f 2NOCl(g) a. 2.92  10
3; b. 6.75  10
3; c. 34.3; d. 148 1224

dx.doi.org/10.1021/ed100133v |J. Chem. Educ. 2011, 88, 1223–1230

Journal of Chemical Education

ARTICLE

Table 1. Continued PSCAs for this question algorithmic PSCA av, item

question subject

A14

calculate Kb from Ka

steps 1

type (no. of different

% (SD)a questions in database)b 92 (3)

example questions (bold indicates the correct answer) The Ka of hydrocyanic acid, HCN, is 4.0  10
10 at 25 °C.

5

What is the Kb for cyanide ion, CN
, at 25 °C? a. 5.0  10
4; b. 2.5  10
5; c. 3.0  10
6; d. 3.3  10
7 A15

þ

calculate [H ] from pH

1

90 (5)

At 25 °C, the pH of a vinegar solution is 2.60. What is the [H3Oþ]

5 (4)

in the solution? a. 7.4  10
2 M; b. 2.5  10
3 M; c. 4.0  10
12 M; d. 1.3  10
13 M A16

calculate pH during a titration

A17

balance redox equations

10

47 (1)

4 (4)

10.0 mL of 0.200 M NaOH is added to 50.0 mL of 0.100 M acetic acid solution (Ka = 1.8  10
5). What is the pH of the resulting solution?

63 (11)

7 (6)

The unbalanced equation for the oxidation of Fe2þ by permanganate ion

a. 2.52; b. 4.57; c. 4.74; d. 5.24 7

(MnO4
) in an acidic solution is:

(5 to solve)

Fe2þ(aq) þ MnO4
(aq) f Fe3þ(aq) þ Mn2þ(aq) In the simplest balanced equation, the coefficient for Fe2þ is a. 2; b. 4; c. 5; d. 8

Percentage of students choosing the correct answer; ∼1000 students responded to the question each time it appeared on an exam. b PSCAs for specific question types and number of this type of question in the database. a

’ ORIGINS OF PSCA VALUES AND METHODS OF DATA COLLECTION The United States Naval Academy (USNA) is a four-year undergraduate college that graduates and commissions officers for service in the United States Navy and Marine Corps. Currently 22 technical and nontechnical majors are offered to students, with a core technical curriculum required for all graduates. A twosemester general chemistry sequence is one of the core classes taken by first-year students with typical annual enrollment of approximately 1000. During the 16-week semester, students receive three hours of lecture per week of classroom instruction accompanied by two hours of laboratory exercises. Lecture sessions are conducted in groups of 20
30 students with 20 students in each laboratory section. These small section sizes necessitate enlisting approximately 30 faculty members per semester to teach general chemistry with each instructor responsible for no more than two lecture sections and three lab sections a week. A variety of teaching methods are employed during classroom time, ranging from traditional lecture style to active group learning, such as POGIL.30 From 1988 to 2009, all registered students except for those with valid excuses (illness, official travel, etc.) received “common exams” in general chemistry. For the majority of these years, three exams of this kind were given per semester, one during the sixth week of the semester, one during the 12th week of the semester, and a final exam. Students were given 50 min to complete 6-week and 12-week exams and 3
4 h to complete final exams. The format for the 6-week and 12-week exams varied over this period from all multiple-choice (25 questions or less) to some combination of fewer than 25 multiple-choice and some handgraded work-out questions (Fall 2000
Fall 2003). The majority of exams were given to all students simultaneously during an early morning examination period (6-week and 12-week exams administered from Spring 1988 to Spring 2005, Fall 2008 to present, and all final exams) or over the course of 36 h during each student’s regularly scheduled lecture section (6-week and 12-week exams in Fall 2005 to Spring 2008). A subset of questions for each of these exams was chosen from the exam

database so that the average of the exam would be 70
75%, based on PSCAs obtained from previous uses of the same question. New questions written for each exam and their PSCA values were added to the database after the conclusion of each common exam. Student answers to multiple-choice questions were collected and read by an optical scanner to provide a distribution of the number of students choosing each answer. Questions that were left blank or registered as having stray marks were not included in the calculation of PSCA.

’ RELATIONSHIP BETWEEN PSCA AND NUMBER OF STEPS IN A PROBLEM-SOLVING ALGORITHM Table 1 lists the average PSCAs, derived from the PSCA of each question each time it was used, with standard deviations for 17 different types of algorithmic questions (A1
A17) on a variety of first-semester (A1
A11) and second-semester (A12
A17) topics; a representative example of each question type is shown in column 6. These questions were chosen for analysis because the database contained multiple questions of this type and, in the majority of cases, at least one of the questions had been used multiple times. Column 3 lists the number of algorithmic steps (Ns) for each type of question, and those steps are listed in parentheses in Table 1 of the Supporting Information. The steps represent the shortest path to a correct answer and are commonly modeled for each type of question in mainstream general chemistry textbooks. Repetitions of steps, such as calculating moles of each reactant, are counted as single steps as they do not represent a unique skill required to solve the problem. In the case where the number of algorithmic steps varied depending on the exact form of the question, a range is given. Ns values varied from 1 (using a single equation) to 10 with the majority of questions requiring 2
4 steps to solve. We assumed that although some questions, such as the example for A17, do not require all the steps in an algorithm to arrive at a correct answer, novice learners would generally complete the entire algorithm rather than risk making an error by skipping steps. While PSCAs for types of questions with Ns > 7 1225

dx.doi.org/10.1021/ed100133v |J. Chem. Educ. 2011, 88, 1223–1230

Journal of Chemical Education

Figure 1. Relationship between PSCA for an algorithmic multiplechoice question and the number of steps required to solve the problem in the question.

were desirable, there were not many examples of such questions contained in the database. This is in part owing to the time constraint that students face during 6-week and 12-week exams, spending only an average of two minutes to solve each question. Column 4 reports the average PSCA and the standard deviations for each type of question; average PSCA varied from 47 to 91%, with standard deviations ranging from 2 to 13%. In some cases, the standard deviations on the PSCA for the same question were large (>15%). These differences may be attributable to factors not recorded in the database, such as the abilities of the different student populations answering the questions, the presence of a supplementary equation sheet accompanying the exam (given some years but not others), which textbook students were using, and whether or not students were provided with a practice exam containing similar questions. Only question types for which the standard deviation of PSCAs was