Using the ACS Anchoring Concepts Content Map (ACCM) To Aid in

Downloaded by PURDUE UNIV on November 29, 2016 | http://pubs.acs.org Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch010

Chapter 10

Using the ACS Anchoring Concepts Content Map (ACCM) To Aid in the Evaluation and Development of ACS General Chemistry Exam Items Jessica J. Reed,1 Cynthia J. Luxford,2 Thomas A. Holme,3 Jeffrey R. Raker,4 and Kristen L. Murphy*,1 1Department

of Chemistry & Biochemistry, University of Wisconsin-Milwaukee, Milwaukee, Wisconsin 53211, United States 2Department of Chemistry & Biochemistry, Texas State UniversitySan Marcos, San Marcos, Texas 78666, United States 3Department of Chemistry, Iowa State University, Ames, Iowa 50011, United States 4Department of Chemistry, University of South Florida, Tampa, Florida 33620, United States *E-mail: [email protected]

The ACS Anchoring Concepts Content Map (ACCM) for general chemistry provides an increasingly detailed framework for examining specific components of chemistry content across ten big ideas of chemistry. The ACCM can serve as a valuable tool for evaluating the content coverage of an existing assessment, and can aid in the creation of new assessment materials by allowing assessment creators to ensure items covering a broad range of chemistry topics are included. Previous work aligned ACS general chemistry exam items to the ACCM and identified locations on the map lacking exam item coverage. This chapter presents insights of how committees of practitioners used the ACCM as a framework to create new ACS general chemistry exams that aimed to eliminate some of the gaps in assessment coverage found in previous ACS exams. Gaps related to topics within the big ideas of bonding and energy and thermodynamics will be highlighted. The results indicate a modest shift that begins to eliminate coverage gaps © 2016 American Chemical Society Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

in ACS general chemistry exams when the ACCM is used as a guide for writing exam items. These results also provide implications for chemistry educators about the usefulness of the ACCM when creating assessment materials.


Introduction Introductory level chemistry courses are often gateway courses for many other science, technology, engineering, and math (STEM) courses (1, 2). In many cases, a student’s only exposure to chemistry concepts comes from what is taught and assessed in these foundational general chemistry courses. In this regard, it may be worth considering whether what is being assessed is in line with the expectations of what students ought to know after taking these courses. While it would be counterproductive to assess every nuance of chemistry content taught, it could be valuable to understand how the content taught and content assessed intersect. In this regard, the questions of “What do we assess?” versus “What do we want to assess?” become important to address. In order to bridge the gap between these two questions, it is necessary to have some means to determine what we are currently assessing in our chemistry courses and make a comparison as to whether that is what we actually want to assess. In the K-12 sector this is often accomplished through the use of standards, for example the Next Generation Science Standards (NGSS) (3). While the use of standards at the collegiate level is not anticipated, there are a growing number of calls for assessment accountability in higher education (4–6). In this regard, the use of a framework to bridge course content with assessment endeavors may be beneficial to many voluntary mechanisms institutions and departments are using to address issues of teaching and learning accountability (7). In chemistry, the development of such a framework, the Anchoring Concepts Content Map (ACCM) (8), provides a mechanism for chemistry faculty and departments to align their assessment materials to an externally generated content framework for measurement of learning outcomes. Here we explore the use of the ACCM framework to aid in the development of chemistry exam content when a mismatch between what we assess and what we want to assess occurs.

Development and Structure of the Anchoring Concepts Content Map (ACCM) The ACCM project was developed by the American Chemical Society Examinations Institute (ACS-EI) as a means to provide instructors enhanced assessment abilities across the undergraduate chemistry curriculum (8, 9). In this regard, the goal was to create a series of interlinking content maps that span the undergraduate chemistry curriculum to which ACS Exam items and instructors’ personally generated assessment items could be aligned for the purpose of measuring learning outcomes. It is important to note that the ACCM is not meant to serve as a set of standards for any chemistry curriculum. Rather, the ACCM should be construed as a tool available to faculty and administrators for the 180 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.


purpose of understanding and evaluating the content taught and assessed within their courses and departments. Content maps for all subdisciplines of the undergraduate curriculum (general, organic, analytical, inorganic, physical, and biochemistry) are currently in various stages of development and publication (10–12), however, the work in this chapter will focus on the ACCM for general chemistry (10, 11). The ACCM for general chemistry is readily available for use by chemistry practitioners and researchers (10, 11). The development of the various subdiscipline content maps has been conducted through numerous workshops with chemistry faculty and instructors. Murphy, et. al. (8, 9), provides a timeline and a more detailed description of how the structure of the ACCM was developed. The ACCM was developed using backward design (13) and is constructed in a hierarchical structure with ten broad, subdiscipline independent Big Ideas or Anchoring Concepts comprising the top-tier (Level 1) and spanning to fine grained subdiscipline specific Content Details (Level 4).

Figure 1. Hierarchical Structure of the ACCM. Figure 1 displays the hierarchical structure of the map and Table 1 displays the titles of the ten Big Ideas and the Anchoring Concepts that support them. The Big Ideas and Enduring Understanding statements, Levels 1 and 2, respectively, are the same across all subdisciplines, while the subdiscipline articulations and content details, Levels 3 and 4, respectively, are specific to the subdiscipline. Therefore, there are a fixed number of Big Idea and Enduring Understanding statements, but the number of subdiscipline articulations and content details vary depending upon the subdiscipline. An example of how the ACCM varies at the levels of 181 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.


subdiscipline articulation and content details for general and organic chemistry is shown in Table 2. An important feature of the ACCM is the fact that the Level 1 and Level 2 statements remain the same across all subdisciplines because it provides a means for departments to set learning objectives that span the entire undergraduate chemistry curriculum and then have an external reference to align course content and assessments. Additionally, the ACS-EI has aligned general chemistry ACS Exam items (14) to the ACCM so ACS Exams users may examine student performance on specific content details or make comparisons of content coverage between courses. As more subdiscipline content maps are completed and ACS Exam items are aligned to the maps, the utility of the ACCM for longitudinal comparison of curriculum and assessment materials will increase.

Table 1. Big Ideas and Anchoring Concepts of ACCM (level 1: subdiscipline independent) Big Idea

Anchoring Concept

I. Atoms

Matter consists of atoms that have internal structures that dictate their chemical and physical behavior.

II. Bonding

Atoms interact via electrostatic forces to form chemical bonds.

III. Structure and Function

Chemical compounds have geometric structures that influence their chemical and physical behaviors.

IV. Intermolecular Interactions

Intermolecular forces—electrostatic forces between molecules—dictate the physical behavior of matter.

V. Chemical Reactions

Matter changes, forming products that have new chemical and physical properties.

VI. Energy and Thermodynamics

Energy is the key currency of chemical reactions in molecular scale systems as well as macroscopic systems.

VII. Kinetics

Chemical changes have a time scale over which they occur.

VIII. Equilibrium

All chemical changes are, in principle, reversible; chemical processes often reach a state of dynamic equilibrium.

IX. Experiments, Measurement, and Data

Chemistry is generally advanced via experimental observations.

X. Visualization

Chemistry constructs meaning interchangeably at the particulate and macroscopic levels.

182 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Table 2. Example of all four levels within General and Organic Chemistry (showing the difference beginning at level 3)


Level

General Chemistry

Organic Chemistry

1

Matter consists of atoms that have internal structures that dictate their chemical and physical behavior.

2

Electrons play the key role for atoms to bond with other atoms

3

For a neutral atom there are as many electrons as there are protons, but the electrons can be categorized as core (inner) and valence (outer) electrons

Electrons play a role in understanding the relative stability of resonance structures.

4

Valence electrons, which determine the properties of elements, are correlated with the groups in the periodic table

Stabilization of anions helps to explain pKa values and relative acidities of protons.

Development of ACS Exams ACS Exams are developed in a grassroots fashion, meaning that the ACS-EI does not specify what content must be included in an examination other than it must be appropriate for the level of chemistry to be tested (15). Committees of chemistry faculty and practitioners work together to generate exam items which are then trial tested and validated by student performance metrics. It takes approximately two years for an ACS Exam to be developed and released to the community. A timeline of this process is shown in Figure 2.

Figure 2. Exam development timeline. Because exam committees are comprised of practitioners, ACS Exams reflect the content deemed important to assess by the chemistry community as a whole. While exam committees are not given item specifications to follow when constructing the exam, there is often a tendency for exam content coverage to mimic recently released ACS Exams from the same domain. This can lead to gaps in assessment of certain concepts as shown by Luxford and Holme (16). In order to address these gaps in assessment and avoid perpetuation of conceptual holes, exam committees ought to be made aware of the existence of such gaps and provided the opportunity to remedy these gaps by writing items that assess frequently overlooked content details. Use of the ACCM as a tool to guide these efforts may prove beneficial. 183 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Methods


Exam Item Alignment ACS Exams provide a unique snapshot of what the chemistry community values in assessment because the exams are created by committees and measure the content the committee deems important. However, released exams may not show the full picture of the content the exam writing committee wanted to assess because items that do not perform well are removed following trial testing. These items that never make it to a released exam are useful artifacts to analyze to understand the relation between what was actually tested versus what was intended to be tested. In the current research project, it is postulated that exam writing committees that were made aware of the conceptual holes found in prior ACS Exams (16), and used the ACCM to guide the creation of new exam content, were able to generate assessment items that tested content that had previously been overlooked. Previous work aligned released ACS general chemistry exams to the ACCM (14) and identified areas, particularly in Big Idea II (Bonding) and VI (Energy and Thermodynamics), where few ACS Exam items were being aligned (16). The current project examined how content coverage from released and unreleased exam items from the ACS First-Term General Chemistry Exams from 2012 (GC12F) and 2015 (GC15F) compare because the GC15F exam was developed with aid of the ACCM and the GC12F was not. The project also examined content coverage from the trial tests developed for the 2017 full-year General Chemistry Exam (GC17) because the exam committee used the ACCM during the exam development process. This analysis was conducted by a research team from the ACS EI in order to understand more thoroughly how use of the ACCM during the exam development process may influence exam content coverage and aid in the elimination of conceptual holes in ACS Exams. Assessment alignment can occur in a variety of ways and can be useful for identifying how assessment content and curriculum interect (17–19). In order to evaluate the content coverage of these exams for research purposes, the exam items were first aligned to the ACCM by an experienced rater. The alignment process involved looking at individual items and determining, first, which Big Idea matched the item, then reading through the Enduring Understanding, Subdisciplinary Articulation, and Content Details statements of that Big Idea to determine where the item best aligned. So, for example, an item asking for the number of neutrons in an atom of fluorine would first be aligned to Big Idea I (Atoms) and then aligned to Enduring Understanding “A” about the number of protons in the nucleus giving rise to an atom’s unique identity. From here, Subdisciplinary Articulation “2” is selected and then Content Detail “a” regarding protons and neutrons summing to contribute the mass of an atom is selected as the final coordinate for where the item belongs. Thus, the item now has four coordinates indicating its location on the ACCM. In this example, the item has been aligned to the location “I A 2 a” on the map. A visual representation of the alignment process described in this example is illustrated in Figure 3. Every effort was made to align items to statements at the Content Details (most specific) level, which provided each item with a set of four “coordinates” indicating its location on the ACCM, but on occasion some items were unable to be aligned 184 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.


to the content specificity of Level 4 statements. Also, sometimes an item fit in multiple locations on the ACCM, in which case it was aligned to multiple coordinates. When this occurred, there was no priority assigned to multiple alignment locations for one item, treating the multiple locations of equivalent value. While the alignment process may appear tedious at first, with practice it can be completed with ease.

Figure 3. Example of the alignment process for a general chemistry assessment item.

The item alignments for the current project were completed by an experienced rater and then a subset of the items was aligned by a team of three additional raters in order to ensure consistency and accurate alignment. In situations where the rating team did not agree on the alignment location for an item, the team discussed possible alignment locations until 100 percent agreement was reached. In total, 424 released and unreleased items were aligned in this analysis and a distribution of items by exam is shown in Table 3. 185 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

Table 3. Distribution of Analyzed Items by Exam Type


ACS Exam Items Aligned to ACCM Exam

Released Items Analyzed

Unreleased Trial Items Analyzed

GC12F

70

72

GC15F

70

72

GC17

n/a

140

Total

140

284

Using the ACCM To Create ACS Exam Content As of this writing, two general chemistry exam committees have used the ACCM to aid in the development of ACS Exams. The committee chairs for the GC17 and the GC15F exams shared their experiences using the ACCM to assist in exam development. These experiences were shared in the form of written responses to questions posed by the authors and each committee chair provided written consent to use her name and quotes in this manuscript. When asked about how their respective committees used the ACCM, GC15F exam committee chair Sharmistha Basu-Dutt described a process of first becoming familiar with the ACCM and then “We looked at a couple of old [ACS] exams and aligned them to the ACCM […]. After we agreed on how each question fit the ACCM topics, we generated a list to see if there were topics that were over-represented or under-represented in previous [ACS] exams.” She went on to explain that the committee members were given the opportunity to develop their own questions, but were “charged to fit their question to specific topics and subtopics on the ACCM.” The committee members’ questions were then compiled and all of the items relating to a particular Big Idea and Enduring Understanding were considered at the same time making the process “very streamlined.” In regard to using the ACCM as opposed to following the content distributions previous exam committees had used, GC17 exam committee chair Yasmin Patell stated: “The original plan was to explore the general chemistry topic grid that had been used by previous exam committees as a blue-print […]. However, once the committee members became familiar with the ACCM, it became clear that the ACCM provided a far superior template that offered genuinely comprehensive coverage of the general chemistry curriculum.” Basu-Dutt also commented on this by saying “[…] the ACCM helped us have a good understanding of the general chemistry curriculum and we were able to use our time efficiently to develop a test that had a good balance of topics included.” Patell also pointed out that using the ACCM “brought about some helpful self-realization for many committee members such as, what topics do I cover in my own day-to-day teaching, and should I consider expanding/changing my topic 186 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.


selections, or should I reassess how much/little time I spend on different areas?” In this regard, the ACCM can be a valuable tool for instructors to use in their own classrooms. Use of the ACCM to create assessment materials is certainly not limited to ACS Exams, and in fact, instructors may find it beneficial to align their own course exam items to the ACCM to identify what they test and how that compares to what they want to test. Additionally, the Content Details (Level 4) statements of the ACCM may help instructors generate content for individual exam items. It is not expected that the ACCM would serve as a mandated set of test blueprints, but rather as an external framework for instructors to use to create, analyze, and compare assessment materials within and across courses. Overall, both committee chairs felt that the ACCM helped them to see where previous assessments may have fallen short in terms of breadth of content coverage and allowed them to remedy these shortcomings by creating assessment items to fill the gaps. It appears that lack of awareness of the ACCM and its potential uses for test development are key barriers to its implementation. Both committee chairs described that their exam committees agreed unanimously to use the ACCM once they became familiar with its design and usefulness. It is important to note that the ACS-EI does not prescribe the use of the ACCM for the creation of new exams, rather it is a tool available to assist exam development committees as they write assessment items. Content coverage decisions are always made by the exam development committees based on the collective experience of the committee.

Results and Analysis The GC15F and GC17 exam committees used the ACCM to guide development of their respective exams. Currently, the GC15F has been released for purchase and use by the chemistry community, but the GC17 exam is still in stages of trial testing and development. Therefore, the results presented for the GC17 exam will represent all unreleased trial items and are considered independently from the GC15F results because the exams have different content coverage. A closer look at alignments for the GC15F and GC17 exams reveals that use of the ACCM for test item creation showed a modest shift toward elimination of gaps in assessment coverage. Because ACS Exams are secure, copyrighted exams, the specific items created to address these conceptual holes cannot be shown. It is important to note that the y-axis represents the number of alignments rather than the number of items because some items were aligned to more than one location in the ACCM. The figures presented here will show all written items for GC12F, GC15F and GC17 exams. For the first-term exams (GC12F and GC15F), the total number of exam items written (prepared for trial testing) is the sum of the released and unreleased items. The released items are on the exam, and the unreleased items were trial items that were not selected for the released test. Therefore, the number of released items in one content category can exceed the number of unreleased items as more items were selected for the released test compared to those that were not. In general, approximately 50 percent of the items written by an exam committee make it onto the released version of the test. 187 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.


Assessing Big Idea II: Bonding Luxford and Holme (16) identified that the Big Idea of Bonding was one of the least tested across all ACS general chemistry exams analyzed, and pointed to conceptual holes in assessment related to Enduring Understanding statements 2D and 2G about bond breaking requiring energy input and metallic bonding, respectively. Comparison of released and unreleased items associated with GC12F and GC15F exams across the Enduring Understanding statements of Big Idea II is shown in Figure 4. The GC12F exam was not developed with the aid of the ACCM while the GC15F exam was. This comparison revealed that there were still conceptual holes in the assessment, particularly for statement 2D, but growth in the number of items created addressing Enduring Understanding statements 2A (7 items total for GC15F compared to 2 for GC12F) and 2F (3 items total for GC15F compared to 1 for GC12F). The lack of items on the GC15F exam assessing Enduring Understanding 2D could be due to the fact that a similar statement is found in Big Idea VI related to energy and thermodynamics.

Figure 4. Distribution of released and unreleased items from GC12F and GC15F across Enduring Understanding statements for Big Idea II: Bonding. The distribution of unreleased trial items for the GC17 exam across the Enduring Understanding statements of Big Idea II is shown in Figure 5. Because it is yet to be determined how many of these items will make it to the released exam, a comparison between the distribution of GC17 unreleased trial items and the distribution of items for a released full-year general chemistry exam would not be very meaningful. What can be observed, however, from the GC17 item distribution across the Big Idea of Bonding is that while the greatest number of items are still found within Enduring Understanding 2C, as also observed in the first-term exam distributions, relatively untested topics are beginning to appear. For example, statement 2D about bond breaking requiring an energy input has one trial item and 2E regarding molecular orbital theory has three trial items. How these items will perform during trial testing and whether they will make it to the final released version of the exam is still unknown, but the fact that items 188 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.


are at least being constructed about these topics suggests that there has been acknowledgment of at least some of the conceptual holes present in previous assessments.

Figure 5. Distribution of unreleased trial items from GC17 across Enduring Understanding statements for Big Idea II: Bonding. Assessing Big Idea VI: Energy and Thermodynamics An additional area where conceptual holes were noted was in Big Idea VI: Energy and Thermodynamics (16). There were several Level 2 statements with few assessment items aligned to them, however, it could be argued that lack of items assessing net change in energy of a system (6A) and energy input for bond breakage (6D) are the most disconcerting considering their importance and relevance to the general chemistry curriculum. Figures 6 and 7 show various distributions of items across the Enduring Understanding statements of Big Idea VI. The idea that bond breaking requires an energy input was not readily addressed in Big Idea II, however, items aligning with this content are found in Enduring Understanding 6D. In Figure 6, it is observed that only one item was written about this topic for the GC12F exam, and that item did not make it to the final released version of the exam, however, for the GC15F exam a total of four items were written to address this topic and two made it to the released exam. Figure 7 shows that two trial items were written for the GC17 exam regarding the energy input required when bonds break (6D). While there are still some Enduring Understanding statements without assessment items, the distribution of GC17 unreleased trial items shows fairly comprehensive coverage of Big Idea VI. Additionally, the GC17 trial items contain two items related to the net change in energy of a system (6A) which has historically been tested very infrequently (16). Concepts related to energy and thermodynamics at the macroscopic scale such as harnessing energy via devices (6F) and concepts related to implications of nuclear chemistry (6I) have historically not been assessed by ACS Exams (16), likely because of limitations due to the number of items on these exams. While 189 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.


it is disappointing that items related to these applications of chemistry did not appear in trial items, it is not unexpected.

Figure 6. Distribution of released and unreleased items from GC12F and GC15F across Enduring Understanding statements for Big Idea VI: Energy and Thermodynamics.

Figure 7. Distribution of unreleased trial items from GC17 across Enduring Understanding statements for Big Idea VI: Energy and Thermodynamics. Distribution of Exam Items across the ACCM Because there is a finite number of items on ACS Exams, there may be concern about incorporating assessment items to fill in content coverage gaps at the detriment to assessment of another topic. Comparison of the distributions of item alignments across all ten Big Ideas of the ACCM for GC12F and GC15F 190 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.


(Figure 8) reveals minor differences congruent with shifting some items from one content area to another in order to address conceptual holes, but not so much as to warrant concern. Additionally, Figure 9 shows all ten Big Ideas as represented in the GC17 unreleased trial items. Still, it is important to consider, at least in the realm of ACS Exams, the notion of whether what we want to assess is worth the sacrifice of what we are currently assessing. It is speculated that in some instances, for example metallic bonding (2G), while an exam committee may want to include a topic on an exam, the content coverage that would need to be sacrificed does not make it worth doing.

Figure 8. Distribution of released GC12F and GC15F exam items across the ten Big Ideas of the ACCM. The GCF exam series is designed for a first-semester general chemistry course and therefore intentionally does not include the topics of kinetics or equilibrium.

It is important to note that just because a topic is not assessed does not mean that it is not taught or valued in the chemistry curriculum. Additionally, some instructors may choose to test these topics more frequently in their own course exams compared to how frequently they appear on ACS Exams. As described earlier, the limited number of items included on an ACS Exam means that not all topics can be given equal representation on exams. Overall, modest shifts in the number of items developed to test topics previously overlooked in ACS Exams was observed when exam development committees used the ACCM to guide exam construction. Dramatic changes to exam content coverage were not expected due to the limited number of items on ACS Exams and the nature of content to be assessed. These modest shifts in exam content coverage suggest a growing awareness of potential gaps in assessment coverage and a need to address them.



Figure 9. Distribution of GC17 unreleased trial items across the ten Big Ideas of the ACCM.

Summary and Implications In general, the use of the ACCM to aid in exam construction proved to be a valuable experience for two ACS general chemistry exam committees. The researchers’analysis of items created by these committees suggests that as the committees became aware of important gaps in content coverage on previous ACS general chemistry exams they sought to create new exams that provide a more comprehensive assessment of the curriculum. The results of these endeavors showed modest increases in the number of items being constructed about important topics related to bonding and energy and thermodynamics that had rarely been assessed on previous ACS general chemistry exams. The implications of this work are twofold. First, there are implications for ACS Exams developers and second there are implications for instructors generating their own assessment materials. It is important to revisit the question of “What are we testing, and how does that align with what we want to test?” From the perspective of ACS Exams development, the use of the ACCM by exam writing committees creates greater awareness of what is being tested and how it fits within the curriculum. This creates the potential for more comprehensive content coverage on exams. It also allows committees to better judge the cost of including some topics on an exam while excluding others. From the perspective of instructors, the use of the ACCM when creating assessment materials has a variety of implications. First, instructors are able to align their own previously created exam items to the ACCM to better understand their own patterns of assessment. Perhaps they may notice content areas that receive much of the focus of their assessments while other topics they value for their students to know are assessed very little; or perhaps they will notice that their assessments are well distributed across the topics that they value for student mastery. This ability for self-evaluation may provide opportunity to 192 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.


enhance alignment between course content and assessment content. Secondly, the ACCM allows instructors to have an external framework to support creation of assessment items to measure content learning objectives. When writing an exam, the instructor can look to the Content Details (Level 4) statements on the ACCM for aid in determining the specific content to include in test items. Additionally, large courses with multiple sections and instructors may find it beneficial to use the ACCM when creating exams to ensure consistent content assessment between multiple exam forms. Finally, as more subdiscipline ACCMs are completed, a department may choose to use align assessment materials to the various subdiscipline ACCMs to evaluate programmatic assessment endeavors across the undergraduate chemistry curriculum. In summary, there is a lot of flexibility in how individual instructors and chemistry departments may choose to use the ACCM, but it is anticipated that by using the ACCM their teaching and assessment efforts will be aided. Overall, this statement by Yasmin Patell, GC17 exam committee chair, sums up the relevance of the ACCM to the chemistry community: “Simply put, the ACCM provides an invaluable pedagogical tool for chemistry teaching, learning, and assessment.”

References 1.

2.

3. 4. 5.

6.

7. 8.

Gasiewski, J. A.; Eagan, M. K.; Garcia, G. A.; Hurtado, S.; Chang, M. J. From gatekeeping to engagement: A multicontextual, mixed method study of student academic engagement in introductory STEM courses. Res. High. Educ. 2012, 53, 229–261, DOI: 10.1007/s11162-011-9247-y. Wright, A.; Provost, J.; Roecklein-Canfield, J. A.; Bell, E. Essessential concepts and underlying theories from physics, chemistry, and mathematics for “Biochemistry and Molecular Biology” majors. Biochem. Mol. Biol. Educ. 2013, 41, 302–308, DOI: 10.1002/bmb.20728. NGSS Lead States. Next Generation Science Standards:For States, By States. The National Academies Press: Washington, DC, 2013. Liu, O. L. Value-added assessment in higher education: A comparison of two methods. High. Educ. 2011, 61, 445–461. Liu, O. L. Outcomes assessment in higher education: challenges and future research in the context of voluntary system accountability. Educ. Meas. Issues Pract. 2011, 30, 2–9, DOI: 10.1111/j.1745-3992.2011.00206.x. Spellings, M. A Test of Leadership: Charting the Future of U.S. Higher Education; U.S. Department of Education, Education Publication Center: Jessup, MD, 2006. https://www2.ed.gov/about/bdscomm/list/hiedfuture/ reports/pre-pub-report.pdf (accessed August, 2016). Bretz, S. L. Navigating the landscape of assessment. J. Chem. Educ. 2012, 89, 689–691, DOI: 10.1021/ed3001045. Murphy, K.; Holme, T.; Zenisky, A.; Caruthers, H.; Knaus, K. Building the ACS Exams Anchoring Concept Content Map for undergraduate chemistry. J. Chem. Educ. 2012, 89, 715–720, DOI: 10.1021/ed300049w. 193

Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.

9.

10.

11.


12.

13. 14.

15. 16.

17.

18.

19.

Zenisky, A. L.; Murphy, K. L Developing a Content Map and Alignment Process for the Undergraduate Curriculum in Chemistry. In Trajectories of Chemistry Education Innovation and Reform, 2013; Chapter 6, pp 79−91. Holme, T.; Murphy, K. The ACS Exams Institute undergraduate chemistry Anchoring Concepts Content Map I: General Chemistry. J. Chem. Educ. 2012, 89, 721–723, DOI: 10.1021/ed300050q. Holme, T.; Luxford, C.; Murphy, K. Updating the General Chemistry Anchoring Concepts Content Map. J. Chem. Educ. 2015, 92, 1115–1116, DOI: 10.1021/ed500712k. Raker, J.; Holme, T.; Murphy, K. The ACS Exams Institute undergraduate chemistry Anchoring Concepts Content Map II: Organic Chemistry. J. Chem. Educ. 2013, 90, 1443–1445, DOI: 10.1021/ed400175w. Wiggins, G. P.; McTighe, J. Understanding by Design. ACSD: Alexandria, VA, 2005. Luxford, C. J.; Linenberger, K. J.; Raker, J. R.; Baluyut, J. Y.; Reed, J. J.; De Silva, C.; Holme, T. A. Building a database for the historical analysis of the general chemistry curriculum using ACS General Chemistry exams as artifacts. J. Chem. Educ. 2015, 92, 230–236, DOI: 10.1021/ed500732q. Holme, T. A. Assessment and quality control in chemistry education. J. Chem. Educ. 2003, 80, 594–596, DOI: 10.1021/ed080p594. Luxford, C. J.; Holme, T. A. What do conceptual holes in assessment say about the topics we teach in general chemistry? J. Chem. Educ. 2015, 92, 993–1002, DOI: 10.1021/ed500889j. Webb, N. In Handbook of Test Development; Downing, S. M., Haladyna, T. M., Eds.; Lawrence Erlbaum Associates, Inc.: Mahwah, NJ, 2006; pp 155−180. Pinto, G. The Bologna Process and its impact on university-level chemical education in Europe. J. Chem. Educ. 2010, 87, 1176–1182, DOI: 10.1021/ ed1004257. Plaza, C. M.; Draugalis, J. R.; Slack, M. K.; Skrepnek, G. H.; Sauer, K. A. Curriculum mapping in program assessment and evaluation. Am. J. Pharm. Educ. 2007, 71, Article 20.


Using the ACS Anchoring Concepts Content Map (ACCM) To Aid in

Recommend Documents