What We Don't Test: What an Analysis of Unreleased ACS Exam Items

Article pubs.acs.org/jchemeduc

What We Don’t Test: What an Analysis of Unreleased ACS Exam Items Reveals about Content Coverage in General Chemistry Assessments Jessica J. Reed,† Sachel M. Villafañe,‡,◊ Jeffrey R. Raker,‡,§ Thomas A. Holme,⊥ and Kristen L. Murphy*,† †

Department of Chemistry and Biochemistry, University of WisconsinMilwaukee, Milwaukee, Wisconsin 53211, United States Department of Chemistry, University of South Florida, Tampa, Florida 33620, United States § Center for Improvement of Teaching and Research in Undergraduate STEM Education, University of South Florida, Tampa, Florida 33620, United States ⊥ Department of Chemistry, Iowa State University, Ames, Iowa 50011, United States ‡

S Supporting Information *

ABSTRACT: General chemistry courses are often the foundation for the study of other science disciplines and upper-level chemistry concepts. Students who take introductory chemistry courses are more often from health and sciencerelated fields than chemistry. As such, the content taught and assessed in general chemistry courses is envisioned as building blocks for concepts in other science courses and across many science disciplines. American Chemical Society (ACS) exams are developed by committees of expert chemists and serve as representative artifacts of content valued by the chemistry community. Before an exam is released, items developed by an examination committee undergo trial testing, and items deemed too easy, too hard, or not discriminating based on item statistics are removed. Analysis of content coverage from items on multiple released general chemistry ACS exams from the past decade and their associated unreleased trial tests revealed content areas where few exam items were written. Content coverage was analyzed by aligning exam items to the ACS Anchoring Concepts Content Map for General Chemistry. By comparing released and unreleased trial items, we found that many of the content coverage gaps on released exams are the result of a lack of items developed about these concepts. These conceptual holes warrant a discussion of whether what is being assessed is what is desired to be assessed and how curricular improvement and assessment reform efforts address content coverage. KEYWORDS: First-Year Undergraduate/General, Chemical Education Research, Testing/Assessment, Curriculum FEATURE: Chemical Education Research

■

INTRODUCTION General chemistry courses serve a broad population of undergraduate majors and are often considered gatekeeper courses for other science, technology, engineering, and math (STEM) disciplines.1−3 It is conceivable that a student’s only exposure to chemistry concepts may come solely from what is taught and assessed in general chemistry courses. As such, there is an expectation from the faculty in other STEM programs that their students are learning fundamental chemistry concepts that can be transferred to other disciplines.2 With this in mind, it is worth considering whether what is assessed in general chemistry matches the expectations of what students ought to know after taking general chemistry.4 It is not practical to test every nuance of chemistry content that is taught in general chemistry; thus, only a subset of concepts and knowledge can pragmatically be assessed. It is therefore valuable to understand how the content taught and content assessed intersect. In this regard, work of the American © XXXX American Chemical Society and Division of Chemical Education, Inc.

Chemical Society Examinations Institute (ACS-EI) provides insight about the intersection of learning objectives and corresponding assessments. The primary function of the ACS-EI is to develop standardized chemistry content exams for use across the undergraduate chemistry curriculum and for use in secondary chemistry courses.5 In general chemistry, there are five types of ACS exams: first-term, second-term, full-year, paired-questions, and conceptual.6 The manner in which ACS exams are developed, via committees of chemistry educators from a wide array of institutions, results in exams that are likely to broadly capture the content being taught in chemistry courses across the United States. These exams can be considered artifacts which reflect current trends in topics taught in chemistry courses at the time the exam was written.5,7 Received: November 8, 2016 Revised: January 31, 2017

A

DOI: 10.1021/acs.jchemed.6b00863 J. Chem. Educ. XXXX, XXX, XXX−XXX

Journal of Chemical Education

Article

Level 3 (Subdisciplinary Articulation) and Level 4 (Content Details) statements are specific to a subdiscipline. This structure provides a means for departments to make comparisons of course content and assessment across their undergraduate curriculum.10 Due to the interconnected nature of chemistry concepts, there is content overlap across the ten Big Ideas. Readers are encouraged to view the ACCM-GC15,16 in its entirety to understand better the map’s structure.

By using these exams as artifacts, it is possible to identify trends in assessment of various chemistry topics, including the lack of assessment.

■

TRIAL EXAM ITEMS AS ARTIFACTS ACS exams are developed by committees of chemistry faculty and practitioners who come together to write exam items reflective of the expectations found within their own classrooms.5 The Examinations Institute does not impose content coverage guidelines beyond the directive that content must be appropriate for the targeted course and student population to be assessed. The committee process ensures that content coverage is representative of what is being taught in a variety of institution types and course demographics. During this process, potential exam items are trial-tested, and performance metrics are analyzed when determining what items are used in the released version of the exam. ACS exam items, therefore, can serve as artifacts to understand assessment coverage over time.7 An analysis of 20 years of released ACS general chemistry exams was conducted by Luxford and colleagues.8 Additional work by Luxford and Holme identified content areas where few released ACS exam items existed.9 The process of exam development results in items that are not included in the final released form of the exam. In most cases of exam development, approximately 50% of the items trial tested by the committee are not included in the released exam. Because there are known gaps in content coverage, it is important to consider both released and unreleased items and determine if unreleased items mimic or close gaps in content coverage found among released items. The question becomes whether (1) items are not being written and trial-tested for certain topics or (2) items do not perform well enough during trial-testing to be included on released exams. The analysis presented herein focuses on comparing the content covered on released and unreleased general chemistry items over the past 10 years.

■

■

ALIGNING EXAM ITEMS TO THE ACCM

■

UNRELEASED TRIAL EXAM ITEM CONTENT

The multiple-choice trial exam items used in this analysis were aligned to content locations on the ACCM to evaluate the specific content assessed by the items. Alignment involves determining where each exam item fits within the ten Big Ideas and subsequent Enduring Understanding, Subdisciplinary Articulation, and Content Details statements. Each exam item results in four coordinates indicating its location on the ACCM. A more detailed description of the alignment process is provided elsewhere.19,20 While every effort was made to align items to Content Details (Level 4) statements, not all items were able to be aligned to this level of content specificity. For example, an item related to phase changes as an example of dynamic equilibrium may be aligned to the Subdisciplinary Articulations level (VIII.A.1) and not to the Content Details level (VIII.A.1.a) unless the item requires the interpretation of phase diagrams. Additionally, on occasion, more than one location on the ACCM was suitable for an item. When this occurred, the item was designated with both alignments with no priority given to one over another. Author J.J.R. aligned all of the unreleased trial test items in the data set to the ACCM. A subset of 10% of those items was aligned by authors S.M.V., J.R.R., and K.L.M. In cases where there was disagreement among the raters, discussion was had until agreement was reached between at minimum three of the four raters. An item was then assigned to the agreed upon ACCM mapping. Similar items to those aligned by the rating team that had previously been aligned by author J.J.R. were realigned, if necessary, to reflect the alignment mappings determined by the four raters.

ACS ANCHORING CONCEPTS CONTENT MAP

Anchoring Concepts Content Map (ACCM) Development

Development of the ACS ACCM10 was motivated by calls for assessment reform and accountability in undergraduate chemistry education11−13 and the need to provide resources for program assessment.14 The primary purpose of the ACCM is to provide a broad representation of the content associated with a subdiscipline of chemistry (e.g., organic chemistry or biochemistry) and to provide an organizational scheme for chemistry educators and departments to analyze the content assessed across their undergraduate curriculum. The ACS-EI has held workshops since the inception of the ACCM to build and vet the maps. At present, ACCMs have been published for general chemistry (ACCM-GC)15,16 and organic chemistry courses.17 The ACS-EI is also nearing completion of the maps for biochemistry, analytical chemistry, inorganic chemistry, and physical chemistry.

Unreleased ACS exam trial items from the last 10 years of general chemistry exams were aligned to the ACCM. Exam types included in this analysis assess content covering a full-year course (GC), General Chemistry Conceptual (GCC) exams, First Term General Chemistry (GCF) exams, and Second Term General Chemistry (GCS) exams. The full-year GC exam series is the most abundantly produced ACS exam and is most frequently released; the majority of items in the analysis came from GC exams. In total, items from seven GC exams (six released exams and the unreleased trial tests for a GC exam currently in development), three GCF exams, two GCS exams, and two GCC exams were analyzed. The released and unreleased items associated with an exam were considered as one unit. ACS exams, even the trial exams, carry secure copyrights; therefore, specific items are not presented herein. In total, 1018 unreleased trial items were aligned to the ACCM. To make judgments about the presence of content coverage gaps in the unreleased trial items, it was necessary to make comparisons to the released items from the same exam types. Many of the 925 released exam items analyzed had been aligned to the ACCM previously.8 Table 1 details the

ACCM Structure

The ACCM was designed in a manner commensurate with backward design18 and has a hierarchical structure with the top tier (Level 1) comprised of ten subdisciplinary-independent Anchoring Concepts (i.e., Big Ideas) and the bottom tier (Level 4) comprised of subdisciplinary concepts (i.e., lecture-level learning objectives). The ACCM was designed such that the Level 1 (Big Idea) and Level 2 (Enduring Understanding) statements are the same across all subdisciplines of chemistry; B



Article

distribution of unreleased trial items and released items by exam type.

Table 3. Enduring Understanding Statements from Each Big Idea that are Commonly Found on ACS Exams and Others That Are Less Frequently Represented

Table 1. Items Analyzed by Exam Type Exam Type

Unreleased Trial Items Analyzed

Released Exam Items Analyzed

Total

GC GCC GCF GCS Total

547 128 216 127 1018

420 160 210 135 925

967 288 426 262 1943

Big Idea I. Atoms II. Bonding III. Structure and Functiona IV. Intermolecular Forces V. Reactionsa

It is possible for an individual item to be aligned to more than one location on the ACCM; therefore, data are reported in number of alignments rather than number of items. There are 1104 total alignments made for the 1018 unreleased trial items and 969 total alignments for the 925 released exam items. Before item content is distilled in detail, overall results of the alignments at Levels 1 and 2 of the ACCM can be concisely enumerated in two tables. A distribution of the number of item alignments across the ten Big Ideas for released and unreleased items by exam type is presented in Table 2. It is apparent that

Most Frequently Tested (No. of Released Exam Items)b

Least Represented (No. of Released + Unreleased Exam Items)b

Frequently Trial Tested But Not Released (No. of Unreleased Exam Items/ No. of Released Exam Items)b

I.C (39) II.C (26) III.B (43)

I.G (16) II.G (2) III.F (5)

I.F (20/11) II.E (10/2) III.A (24/10)

IV.A (53)

IV.B (1)

IV.A (62/53)

V.A (44) V.D (44)

V.B (2)

V.F (29/11)

VI. Energy and Thermodynamics

VI.C (55)

VI.B (1)

VII. Kineticsa VIII. Equilibrium IX. Experimentsa X. Visualizationa

VII.B (30) VIII.G (53) IX.B (20) X.D (20)

VII.E (14) VIII.A (2) IX.D (2) X.C (1)

VI.D (12/6) VI.E (28/22) VII.B (47/30) VIII.D (28/12) IX. E (13/2) N/A

a

Denotes presence of a Level 2 statement with no items aligned to it within the Big Idea. bDetailed description of each Enduring Understanding statement and the number of items aligned to it by exam type can be found in Tables B−K in the Supporting Information.

Table 2. Item Alignments by Big Idea and Exam Type Exam Type Big Idea I. Atoms II. Bonding III. Structure and Function IV. Intermolecular Forces V. Reactions VI. Energy and Thermodynamics VII. Kinetics VIII. Equilibrium IX. Experiments X. Visualization

Item Status

GC

GCC

Unreleased Released Unreleased Released Unreleased Released Unreleased Released Unreleased Released Unreleased Released Unreleased Released Unreleased Released Unreleased Released Unreleased Released

89 80 37 24 52 24 63 53 80 57 84 70 56 36 88 67 24 18 19 10

19 25 4 5 10 8 22 36 18 15 22 30 9 11 24 31 5 5 4 13

GCF GCS 55 54 27 22 25 24 35 32 42 39 21 21 0 0 3 0 24 16 4 7

4 1 1 1 2 2 23 16 9 14 25 26 21 17 52 54 1 0 1 5

Total

statements did not have any items aligned to them, including both released and unreleased items. To focus on areas where differences in testing of Enduring Understanding statements might be most informative, statements without item alignments are not represented in Table 3 but can be found in Table A in the Supporting Information. Additionally, the text of each Enduring Understanding statement and the number of item alignments associated with the statement by exam type can be found in Tables B−K in the Supporting Information. A more in-depth analysis of specific content coverage within each Big Idea follows herein.

167 160 69 52 89 58 143 137 149 125 152 147 86 64 167 152 54 39 28 35

Big Idea I: Atoms

The Big Idea of Atoms was one of the most frequently tested content areas. The number of items about atoms was relatively equal for unreleased and released exam items. All seven Enduring Understanding statements that fall under the Big Idea of Atoms had unreleased and released items aligned to them. Enduring Understanding statements “Most information about atoms is inferred from studies on collections of atoms often involving an interaction with electromagnetic radiation” (I.D) and “Atoms maintain their identity, except in nuclear reactions” (I.F) were represented by items that were only on trial tests more often than they were represented on released exams (Figure 1). To the contrary, the statement “Atoms display a periodicity in their structures and observable phenomena that depend on that structure” (I.C) had more alignments from released exam items than from unreleased trial items. Even though Enduring Understanding I.C was well represented on released exams, it is interesting to consider more specifically what was being assessed. In this case, 39 (58.2%) of the total alignments made to Enduring Understanding I.C were aligned to I.C.1, which relates to periodic trends such as atomic radius and ionization energy. In comparison, only approximately 12% of the alignments made to I.C were mapped to I.C.2, relating to effective nuclear charge, and about 28% aligned to I.C.3, which describes the relation between periodic and reactivity trends.

some content Big Ideas are tested more frequently than others, but within a Big Idea, the number of items written but not released is similar to the number of items making it onto released exams. More variation is noted when looking at each exam type individually. Table 3 highlights the Enduring Understanding (Level 2) statements from each Big Idea that (1) are most commonly found on released exams, (2) least frequently have items written about them, and (3) frequently do to not make it past trial testing. For Enduring Understanding statements that had items that were frequently trial tested but not included in a released exam, the statements listed represent the largest difference between number of trial items and number of released items. Seven Enduring Understanding (Level 2) C



Article

Figure 1. Distribution of released and unreleased item alignments from the Full Year General Chemistry (GC), the General Chemistry Conceptual (GCC), the First Term General Chemistry (GCF), and the Second Term General Chemistry (GCS) ACS exams across Enduring Understanding (Level 2) statements for ACCM Big Idea I: Atoms.

Figure 2. Distribution of released and unreleased item alignments from four types of general chemistry ACS exams across Enduring Understanding (Level 2) statements for ACCM Big Idea II: Bonding.

exams as well, comprising approximately 5% of alignments, which was consistent with the previous analysis.9 The small total count of items both released and unreleased related to this content suggests that items about bonding are simply not being written. For example, as shown in Figure 2, fewer than 10 items total were written (unreleased + released items) in the past 10 years about sharing electrons (II.B), energy input required to break bonds (II.D), and metallic bonding (II.G). While it is possible that the lack of items about energy consequences of breaking bonds (II.D) can be explained by content overlap with the Energy and Thermodynamics Big Idea (VI), the lack of items about the other two Enduring Understanding statements (II.B and II.G) is not readily explained. Perhaps, due to constraints in the number of items allowed on an ACS exam, the exam committees deemed these topics of lesser importance to be assessed. In other cases, such as II.A and II.E, the number

While this unequal division of content coverage is not necessarily problematic, it could be important to consider why basic periodic trends are being assessed more frequently than the other concepts within Enduring Understanding I.C. Consideration of whether a topic is being frequently assessed because it is of great importance for students to know or because it is of relative ease to assess with a multiple-choice question may be of merit during future item development. On the surface, it may appear that a content area from the ACCMGC is well represented on ACS exams, but a more targeted analysis may reveal holes in content coverage due to over assessment of specific concepts. Big Idea II: Bonding

Items related to the topic of bonding accounted for approximately 6% of the unreleased trial items analyzed. Bonding was one of the least tested topics on released ACS D



Article

Figure 3. Distribution of released and unreleased item alignments from four types of general chemistry ACS exams across Enduring Understanding (Level 2) statements for ACCM Big Idea III: Structure and Function.

Figure 4. Distribution of released and unreleased item alignments from four types of general chemistry ACS exams across Enduring Understanding (Level 2) statements for ACCM Big Idea IV: Intermolecular Forces.

of unreleased trial items is greater than the number of items that make it to the released test, suggesting that exam committees wrote items to test the content of these Enduring Understanding statements but, for some reason, likely unsatisfactory item statistics during trial testing or content decisions by the committee, the items did not make it to the released exam. Regardless, considering the evidence that students struggle with misconceptions related to bonding such as students not understanding that energy is required to break bonds or differences in covalent, ionic, and metallic bonding,21−25 it is somewhat troubling that bonding continues to be one of the least assessed topics in ACS general chemistry exams. Additionally, within the most assessed Enduring Understanding (II.C), 85% of the total alignments were made to II.C.3, which is related to valence bond theory, while less than 5% of the items aligned had content related to total energy

being lowered during bond formation (II.C.1). Once again, it could be important to consider whether over assessment of a particular concept is leading to conceptual holes in the content coverage of general chemistry ACS exams. Big Idea III: Structure and Function

The Big Idea of Structure and Function was another relatively untested content area for general chemistry. Approximately 8% of unreleased item alignments and 6% of released exam item alignments were made to this Big Idea. The most common alignment for the Big Idea of Structure and Function, as shown in Figure 3, was related to the prediction of molecular shapes (III.B). Items that were aligned to this Enduring Understanding statement frequently tested understanding of the VSEPR model by asking students to predict the shape around the central atom of a small molecule and were most commonly found on GC and GCF exams. While it may initially appear ominous that there are no alignments for Enduring Understanding stateE



Article

Figure 5. Distribution of released and unreleased item alignments from four types of general chemistry ACS exams across Enduring Understanding (Level 2) statements for ACCM Big Idea V: Reactions.

law, and only 17% of the alignments to IV.A related to intermolecular forces being overcome during physical change (IV.A.3). Additionally, statements IV.C and IV.D relating to the relationship between IMF and dipoles and physical properties influenced by IMF, respectively, were also commonly assessed. Even though there appears to be a large assessment effort directed at concepts relating to IMF, some concepts such as noncovalent forces within large molecules (IV.B) and energy consequences of chemical reactions in solution include IMF (IV.E) were seemingly overlooked in the past 10 years of exam development. After the analysis conducted by Luxford and Holme revealed a conceptual hole for Enduring Understanding IV.B,9 one item assessing this concept has since made it onto a released GC exam. While one exam item is certainly not enough to close a gap in assessment coverage, it suggests that creating awareness of holes in concept coverage can lead to creation of items to address conceptual holes. Additionally, two items were written about Enduring Understanding IV.E by the GC15 exam committee, which used the ACCM in exam construction, but ultimately, the items did not make it onto the released exam.

ments III.C, III.D, or III.E, it is not entirely problematic because these statements are examples of cases where the ACCM is designed for use across the undergraduate chemistry curriculum. There are no Subdisciplinary Articulation or Content Details statements for these Enduring Understanding statements on the ACCM-GC. Thus, it is not surprising that these Enduring Understanding statements, which are important in upper-level courses, are not assessed on general chemistry exams. It is somewhat surprising, however, that very few items tested students’ understanding of functional groups (III.F) because this is a concept that may be introduced at the end of general chemistry as students prepare to transition to an organic chemistry course. While statements related to periodic trends and their implications for structure and function (III.G) and the influence of geometric structures on the properties of solid state and extended systems (III.H) were relatively untested on released exams, it is interesting to note the number of unreleased items written and aligned to these statements. For example, while only one item from a released exam aligned to III.H in the past 10 years, 10 unreleased items had been written about this statement, typically assessing content related to unit cells. This suggests that exam committees deemed this content important even though it ultimately did not make it onto the released exams likely due to performance during trial testing.

Big Idea V: Reactions

Concepts that are often easily assessed such as balancing equations (V.A) and classifying reactions into categories (V.D) constituted the majority of item alignments for this Big Idea (Figure 5). Few, if any, item alignments were made for concepts relating to chemical change being the result of bonds breaking and forming (V.B), chemical change at the particulate and macroscopic levels (V.C), and control of chemical reactions for the synthesis of new materials (V.G). Interestingly, the number of unreleased trial item alignments for the concept of controlling chemical change by choice of reactants, conditions, or catalysts (V.F) was nearly three times the number of released item alignments. Without further analysis, it is unclear why these items did not make a released exam, but the fact that so many items were written about this concept suggests that it was deemed important to assess by exam committees.

Big Idea IV: Intermolecular Forces

Intermolecular forces (IMF) as a Big Idea was a very commonly assessed topic (Figure 4). Approximately 13% of unreleased trial item alignments and 14% of released exam item alignments were found in this Big Idea. Alignment trends were consistent with observations made previously by Luxford and Holme.9 Enduring Understanding statement IV.A relating to intermolecular forces being generally weaker than chemical bonds was assessed most frequently in both released and unreleased items. This is expected because common general chemistry concepts such as the physical properties of gases, the ideal gas law, phasediagrams, and phase changes are all included under this Enduring Understanding statement. A more in-depth look at Enduring Understanding IV.A revealed that roughly 50% of the alignments were made to IV.A.1, which pertains to the ideal gas F



Article

Figure 6. Distribution of released and unreleased item alignments from four types of general chemistry ACS exams across Enduring Understanding (Level 2) statements for ACCM Big Idea VI: Energy and Thermodynamics.

Figure 7. Distribution of released and unreleased item alignments from four types of general chemistry ACS exams across Enduring Understanding (Level 2) statements for ACCM Big Idea VII: Kinetics.

Big Idea VI: Energy and Thermodynamics

the system” (VI.A) had four alignments from unreleased trial items and only two alignments from released exam items. Additionally, statement VI.B about energy input to initiate a chemical reaction had no alignments from unreleased trial items and only one alignment from released exam items. The concept that energy can be harnessed by devices (VI.F) such as a battery had only three alignments from released exam items and no unreleased trial item alignments. Finally, the concept of energy changes associated with nuclear chemistry processes (VI.I) appeared in one released GCS exam item and three unreleased trial items (one item each from GC, GCC, and GCS trial exams). It is important to consider that some concepts appear in multiple locations on the ACCM. For example, even though the concept of reactions requiring an energy input (VI.B) was essentially not tested, another facet of this concept appears in the context of activation energy found in the Big Idea related to kinetics.

Approximately 13.7% (151 alignments) of unreleased trial test item alignments and 15.2% (147 alignments) of released exam item alignments were assigned to this Big Idea. As seen in Figure 6, Enduring Understanding VI.C, relating to the types of energy associated with chemical change, was the most assessed topic from this Big Idea with nearly double the number of alignments compared to other readily assessed Enduring Understanding statements (VI.E, VI.G, and VI.H). This is likely due to the fact that basic ideas related to the types of energy associated with chemical change (VI.C) appear in the first semester and advanced ideas related to this concept appear in the second semester of the course, whereas content related to VI.E, VI.G, and VI.H typically appears in one of the two semesters but not both. Consistent with previous analyses of released exam items, there are four Enduring Understanding statements that have rarely been tested.9 The statement “Most chemical changes are accompanied by a net change of energy of G



Article

Figure 8. Distribution of released and unreleased item alignments from four types of general chemistry ACS exams across Enduring Understanding (Level 2) statements for ACCM Big Idea VIII: Equilibrium.

Figure 9. Distribution of released and unreleased item alignments from four types of general chemistry ACS exams across Enduring Understanding (Level 2) statements for ACCM Big Idea IX: Experiments.

Big Idea VII: Kinetics

the Kinetics Big Idea were aligned to Enduring Understanding statement VII.B as compared to 47% (30 alignments) of released exam item alignments. More specifically, approximately 83% of the alignments made to Enduring Understanding statement VII.B were mapped to VII.B.1, which relates to the order of a reaction and has Content Details (Level 4) statements about methods for determining and use of rate laws. This suggests that items related to rate laws are the primary focus of exam writing committees when assessing kinetics concepts. No items relating to the products of a reaction being influenced by controlling the reaction rate or reaction energy (VII.F) were found in this analysis, likely because this is a topic traditionally taught and assessed in organic chemistry.

The Big Idea of Kinetics was one of the least tested topics across ACS exams for general chemistry with 7.8% of unreleased trial item alignments and 6.6% of released item alignments falling under this heading. It should be noted that kinetics concepts are typically taught in the second semester of a full-year course, which explains why they are not observed on GCF exams. The GCS exam is released less frequently than GC or GCF exams, so topics that are predominantly taught in the second semester are likely to be represented less frequently in the sample analyzed because the GCS exam series comprised only 13.5% of the items in the sample. As seen in Figure 7, the total number of alignments for released and unreleased items was approximately equal for each Enduring Understanding statement except VII.B. Questions related to Enduring Understanding VII.B typically asked students about rate laws and the dependence of reaction rates on concentrations of reactants and temperature. Approximately 55% (47 alignments) of the unreleased trial item alignments for

Big Idea VIII: Equilibrium

With 15.1% (167 alignments) of unreleased trial item alignments and 15.6% (152 alignments) of released exam item alignments falling under this Big Idea, it is easy to see that H



Article

Figure 10. Distribution of released and unreleased item alignments from four types of general chemistry ACS exams across Enduring Understanding (Level 2) statements for ACCM Big Idea X: Visualization.

were made to IX.G, and they all came from the GC exam series. It should be noted that the ACS-EI offers a laboratory exam which was not included in this analysis and may include more assessment of safety concepts, but the number of users of the laboratory exam pales in comparison to the number of users of exams in the general chemistry series. Therefore, test developers may consider the merits of including more items to assess chemical safety in future ACS general chemistry exams. Additionally, quantitative observations of matter (IX.A) and consideration of the representativeness of samples (IX.E) appeared predominantly in unreleased trial exam items. Likely, these items did not test well (too easy or too hard) during trial testing and were therefore omitted from released versions of the exam, but it is encouraging that assessment of concepts related to experiments, measurement, and observation beyond nomenclature and error analysis has at least been attempted by exam developers.

exam writing committees consider equilibrium an important concept to assess. Yet, several concepts within the realm of equilibrium are assessed far more frequently than others (Figure 8). Interestingly, the application of equilibrium concepts (VIII.G) such as buffers, pH, and titration is the most assessed concept within Big Idea VIII, yet foundational concepts such as reversibility of reactions and phase changes (VIII.A) and the idea that at equilibrium the net change of reactant and product amounts is zero (VIII.B) were essentially not tested. While the ability to apply equilibrium concepts is certainly important to understanding concepts found in subdisciplines of chemistry, biology, and nursing courses, it may be worth considering whether explicit testing of foundational concepts of equilibrium is warranted as students may be able to apply a concept without fully understanding it.26−28 Because equilibrium is primarily taught in the second semester of a year-long course, only three alignments came from GCF exam items, and all three were aligned to the same location (VIII.D.2.a) relating to strength of acids and bases.

Big Idea X: Visualization

The Big Idea of visualization is the least tested topic across ACS general chemistry exams. Only 2.5% of unreleased trial item alignments and 3.6% of released exam item alignments were found within this anchoring concept. Moreover, most of the items that were aligned to this Big Idea were also aligned to another location on the ACCM-GC because the visualization component of the item was often in relation to a specific content topic. As seen in Figure 10, the predominant locations for alignment related to making connections between the particulate and macroscopic levels of representation (X.A) and interpreting graphical representations (X.D). One unreleased trial test item was aligned to Enduring Understanding X.C about statistical methods providing a link between particulate and macroscopic levels of representation. This Enduring Understanding is likely more readily assessed in upper-level chemistry courses rather than at the general chemistry level. The Enduring Understanding statement “The mole represents a key factor for translating between the macroscopic and particulate levels”(X.B) has not been assessed by any ACS general chemistry items in the past 10 years. This is not to suggest that there are not questions pertaining to the

Big Idea IX: Experiments

Fewer than 5% of unreleased trial item alignments and released exam item alignments were aligned to Big Idea IX: Experiments. The majority of alignments (88%, 82 alignments) came from the GC and GCF exam series, while only 1 unreleased trial item from the GCS series was aligned to this content. Alignments for this Big Idea were made most frequently to Enduring Understanding IX.B, which is essentially chemical nomenclature, and were predominantly from GCF items (Figure 9). Concepts related to the experimental control of reactions playing a role in the synthesis of new materials (IX.C) and measurements being based on mass, charge, or interaction with electrons or photons (IX.D) were relatively unassessed with zero alignments to IX.C and two alignments to IX.D from released exam items. This is not entirely unexpected as these concepts are likely more commonly associated with upper-level chemistry courses. It is surprising, however, that very few items addressing chemical safety (IX.G) were aligned because it is considered an important concept for students to learn early on.29,30 Only four alignments total (3 unreleased, 1 released) I



Article

Also, due to the fixed length of ACS exams, it is impossible to test all the content taught in a general chemistry course. Therefore, some topics are inherently not assessed because they are deemed to be of lesser importance by the committees of practitioners who write the exams. This is not to suggest that instructors should stop teaching this content or that it is not of value to the learner.

importance and calculation of moles (see I.E); rather, these traditional calculation questions did not require the student to make a conceptual connection between macroscopic and particulate levels, which could be difficult to assess with a multiple-choice exam item. Consideration of expectations about students’ development of representational competence in chemistry31,32 and studies relating to students’ understanding, or lack thereof, of concepts related to scale33,34 suggest that assessing concepts found within this Big Idea is important.

Limitations of Research

■

There are several limitations to this study. First, the current research is only able to acknowledge the existence of holes in content coverage on ACS exams but cannot make claims beyond speculation as to why specific gaps exist. Data providing historical insight into exam committees’ item creation and selection processes was not available but may have provided more concrete evidence as to why specific holes in content coverage exist. Additionally, this study did not examine individual item statistics to explore how item difficulty and discrimination measures for trial test items may have led to their exclusion from released exams. Finally, the study was limited to the past 10 years of ACS general chemistry exam items, so exploration of trends or changes in exam content coverage beyond that timespan is not possible with the current data set.

DISCUSSION, IMPLICATIONS, AND CONCLUSIONS Overall, because general chemistry courses serve a variety of student majors, consideration of what is being taught and assessed in the course is important. Addressing the questions of “What do we assess?” versus “What do we want to assess?” may provide useful insight into how to serve these student populations better. The ACS-EI has an obligation to deliver assessments that are not only fair but also meet the needs of the community by ensuring appropriate content is assessed. To do this, it is important to understand what ACS exams for general chemistry are currently assessing and compare that to what the community deems most important to assess. Prior research by Luxford and Holme identified conceptual holes in content coverage in ACS exams for general chemistry.9 They posited that these gaps in content coverage on exams could be due to items that cover such topics not making it past the trial-testing phase of exam development.9 The research herein examined the content of ACS exam items that never made it past the trial-testing phase of development to a released exam. Comparison of item alignments for unreleased trial items and released exam items revealed that, unfortunately, many of the conceptual holes exist because items are not being written to assess that content. While this may not be problematic in some cases because the content is likely to be taught and assessed in upper-level chemistry courses, other conceptual holes in content assessment could suggest worrisome problems, especially for the many students who are not required to take higher level chemistry courses. For example, lack of assessment items about intermolecular forces in large molecules suggests that the topic is not of priority in general chemistry. Yet, many of the students in introductory chemistry courses are life science majors, not chemistry majors, so focusing only on intermolecular forces in small molecules limits the ability for students to transfer and make meaningful connections about these concepts between chemistry and biological science courses.35 Additionally, assessment coverage gaps for topics such as metallic bonding, solid state systems, and harnessing energy via devices could be problematic for other student populations such as engineering majors. Indeed, making larger connections either within chemistry or between disciplines is being considered with the format of the ACCM. A possible 11th Big Idea related to systems, which could address these concerns in a small part, is currently in discussion. It is important to consider that very few of the items analyzed were constructed with the aid of the ACCM-GC or alignment data. Therefore, propagation of similar patterns of exam content coverage by exam development committees is not surprising because committees have often used prior exams as guides for how content is considered. Additionally, because ACS exams are multiple-choice tests, there is a possibility that conceptual holes arise because it is not as easy to test certain topics by using this item format.

Using the ACCM to Write Exam Content

There are some practical implications of this research. First, as ACS exam writing committees become more aware of the content coverage gaps on ACS general chemistry exams, they are more cognizant of what the items they develop are assessing and may choose to address conceptual holes. Currently, two exams (GC15F and GC17) have been developed with the aid of the ACCM-GC and have attempted to address some of the gaps in assessment coverage.20 Additionally, departments and instructors concerned about how their course content and assessments intersect may consider use of the subdisciplinary ACCMs. By aligning course content and assessment materials, instructors can get a better picture of how the content they are teaching and the content they are testing intersect. Instructors may also find it useful to use the ACCM-GC as they write their own course exams, as the Content Details (Level 4) statements of the map are designed to represent content specificity that can be assessed in an exam item.

■

ASSOCIATED CONTENT

S Supporting Information *

The Supporting Information is available on the ACS Publications website at DOI: 10.1021/acs.jchemed.6b00863.

■

Tables representing Enduring Understanding statements and item alignments (PDF; DOCX)

AUTHOR INFORMATION

Corresponding Author

*E-mail: [email protected]. ORCID

Jessica J. Reed: 0000-0003-4791-6094 Jeffrey R. Raker: 0000-0003-3715-6095 Thomas A. Holme: 0000-0003-0590-5848 Kristen L. Murphy: 0000-0002-7211-300X J



Article

Present Address

(17) Raker, J.; Holme, T.; Murphy, K. The ACS Exams Institute Undergraduate Chemistry Anchoring Concepts Content Map II: Organic Chemistry. J. Chem. Educ. 2013, 90, 1443−1445. (18) Wiggins, G. P.; McTighe, J. Understanding by Design; ACSD: Alexandria, VA, 2005. (19) Zenisky, A. L.; Murphy, K. L. Developing a Content Map and Alignment Process for the Undergraduate Curriculum in Chemistry. ACS Symp. Ser. 2013, 1145, 79−91. (20) Reed, J. J.; Luxford, C. J.; Holme, T. A.; Raker, J. R.; Murphy, K. L. Using the ACS Anchoring Concepts Content Map (ACCM) to aid development of ACS general chemistry exam items. Chapter 10. In Technology and Assessment Strategies for Improving Student Learning in Chemistry; American Chemical Society: Washington, D.C., 2016; pp 179−194. (21) Galley, W. C. Exothermic Bond Breaking: A Persistent Misconception. J. Chem. Educ. 2004, 81, 523−525. (22) Nicoll, G. A report of undergraduates’ bonding misconceptions. Int. J. Sci. Educ. 2001, 23, 707−730. (23) Luxford, C. J.; Bretz, S. L. Moving beyond definitions: what student generated models reveal about their understanding of covalent bonding and ionic bonding. Chem. Educ. Res. Pract. 2013, 14, 214− 222. (24) Taber, K. S. An alternative conceptual framework from chemistry education. Int. J. Sci. Educ. 1998, 20, 597−608. (25) Ö zmen, H. Some Student Misconceptions in Chemistry: A Literature Review of Chemical Bonding. J. Sci. Educ. Technol. 2004, 13, 147−159. (26) Orgill, M. K.; Sutherland, A. Undergraduate chemistry students’ perceptions of and misconceptions about buffers and buffer problems. Chem. Educ. Res. Pract. 2008, 9, 131−143. (27) Nakhleh, M. B. Why Some Students Don’t Learn Chemistry: Chemical Misconceptions. J. Chem. Educ. 1992, 69, 191−196. (28) Treagust, D. F. Development and Use of Diagnostic Tests to Evaluate Students’ Misconceptions in Science. Int. J. Sci. Educ. 1988, 10, 159−169. (29) Hill, R. H.; Nelson, D. A. Strengthening safety education of chemistry undergraduates. Chem. Health Saf. 2005, 12, 19−23. (30) Miller, G. J.; Heideman, S. A.; Greenbowe, T. J. Introducing proper chemical hygiene and safety in the general chemistry curriculum. J. Chem. Educ. 2000, 77, 1185−1187. (31) Kozma, R. B.; Russell, J. Multimedia and understanding: Expert and novice response to different representations of chemical phenomena. J. Res. Sci. Teach. 1997, 34, 949−968. (32) Kozma, R.; Russell, J. Students becoming chemists: Developing representational competence. In Visualization in science education; Gilbert, J. K., Ed.; Springer: Netherlands, 2005; pp 121−145. (33) Gerlach, K.; Trate, J.; Blecking, A.; Geissinger, P.; Murphy, K. Investigation of absolute and relative scaling conceptions of students in introductory college chemistry courses. J. Chem. Educ. 2014, 91, 1526−1537. (34) Gerlach, K.; Trate, J.; Blecking, A.; Geissinger, P.; Murphy, K. Valid and reliable assessments to measure scale literacy of students in introductory college chemistry courses. J. Chem. Educ. 2014, 91, 1538−1545. (35) Villafañe, S.M.; Loertscher, J.; Minderhout, V.; Lewis, J.E. Uncovering students’ incorrect ideas about foundational concepts for biochemistry. Chem. Educ. Res. Pract. 2011, 12, 210−218.

◊

S.M.V.: Department of Chemistry and Biochemistry, California State UniversityFullerton, Fullerton, CA 92834, United States.

Notes

The authors declare no competing financial interest.

■

ACKNOWLEDGMENTS The authors would like to acknowledge Cynthia Luxford for her work related to updating the ACCM-GC and aligning ACS exam items to it. Additionally, partial funding for the development of the ACCM was provided by the National Science Foundation (NSF-DUE Grants 071779, 0943783, and 1323288). Any opinions, findings, and conclusions communicated in this material are those of the authors and do not necessarily reflect the views of the NSF.

■

REFERENCES

(1) Gasiewski, J. A.; Eagan, M. K.; Garcia, G. A.; Hurtado, S.; Chang, M. J. From Gatekeeping to Engagement: A Multicontextual, Mixed Method Study of Student Academic Engagement in Introductory STEM Courses. Res. High. Educ. 2012, 53, 229−261. (2) Wright, A.; Provost, J.; Roecklein-Canfield, J. A.; Bell, E. Essential Concepts and Underlying Theories from Physics, Chemistry, and Mathematics for “Biochemistry and Molecular Biology” Majors. Biochem. Mol. Biol. Educ. 2013, 41, 302−308. (3) Tai, R. H.; Sadler, P. M.; Loehr, J. F. Factors influencing success in introductory college chemistry. J. Res. Sci. Teach. 2005, 42, 987− 1012. (4) Bretz, S. L. Navigating the Landscape of Assessment. J. Chem. Educ. 2012, 89, 689−691. (5) Holme, T. A. Assessment and Quality Control in Chemistry Education. J. Chem. Educ. 2003, 80, 594−596. (6) American Chemical Society Division of Chemical Education Examinations Institute. http://uwm.edu/acs-exams/ (accessed Dec 2016). (7) Raker, J. R.; Holme, T. A. A Historical Analysis of the Curriculum of Organic Chemistry Using ACS Exams as Artifacts. J. Chem. Educ. 2013, 90, 1437−1442. (8) Luxford, C. J.; Linenberger, K. J.; Raker, J. R.; Baluyut, J. Y.; Reed, J. J.; De Silva, C.; Holme, T. A. Building a Database for the Historical Analysis of the General Chemistry Curriculum Using ACS General Chemistry Exams as Artifacts. J. Chem. Educ. 2015, 92 (2), 230−236. (9) Luxford, C. J.; Holme, T. A. What do Conceptual Holes in Assessment Say about the Topics We Teach in General Chemistry? J. Chem. Educ. 2015, 92 (6), 993−1002. (10) Murphy, K.; Holme, T.; Zenisky, A.; Caruthers, H.; Knaus, K. Building the ACS Exams Anchoring Concept Content Map for Undergraduate Chemistry. J. Chem. Educ. 2012, 89, 715−720. (11) Liu, O. L. Value-added assessment in higher education: A comparison of two methods. High. Educ. 2011, 61, 445−461. (12) Liu, O. L. Outcomes Assessment in Higher Education: Challenges and Future Research in the Context of Voluntary System of Accountability. Educ. Meas. Issues Pract. 2011, 30, 2−9. (13) Spellings, M. A Test of Leadership: Charting the Future of U.S. Higher Education; U.S. Department of EducationEducation Publication Center: Jessup, MD, 2006. (14) Towns, M. H. Developing Learning Objectives and Assessment Plans at a Variety of Institutions: Examples and Case Studies. J. Chem. Educ. 2010, 87, 91−96. (15) Holme, T.; Murphy, K. The ACS Exams Institute Undergraduate Chemistry Anchoring Concepts Content Map I: General Chemistry. J. Chem. Educ. 2012, 89, 721−723. (16) Holme, T.; Luxford, C.; Murphy, K. Updating the General Chemistry Anchoring Concepts Content Map. J. Chem. Educ. 2015, 92, 1115−1116. K


What We Don't Test: What an Analysis of Unreleased ACS Exam Items

Recommend Documents