An Introduction to the Tools of Chemistry Education Research - ACS

Chapter 1

Downloaded by UNIV OF TASMANIA on September 1, 2014 | http://pubs.acs.org Publication Date (Web): July 31, 2014 | doi: 10.1021/bk-2014-1166.ch001

An Introduction to the Tools of Chemistry Education Research Renée S. Cole*,1 and Diane M. Bunce2 Department of Chemistry, University of Iowa, Iowa City, Iowa 52242, United States 2Department of Chemistry, The Catholic University of America, Washington, DC 20064, United States *E-mail: [email protected] 1

This chapter provides an overview of the issues and tools associated with chemistry education research projects and introduces the different chapters. The intent is to highlight the information available that may prove useful to individuals engaged in chemistry education research projects. To facilitate use by readers, the material has been organized into four sections: strategies for qualitative research; analyzing quantitative research data; cognitive-based tools for chemistry education research; and practical issues for planning, conducting, and publishing chemistry education research.

Introduction As the field of chemistry education research matures, more researchers are using more tools to answer a greater range of questions. In the Nuts and Bolts of Chemical Education Research (1), we provided an overview of the field and discussed how chemistry education research questions could be addressed. The intended audience for that book was quite diverse, including those who wanted to learn about aspects of chemistry education research (CER) from many perspectives: novice researchers, scholarly teachers who wanted to improve assessment of practice, grant writers, and chemists who want to be more informed about chemistry education research. In this volume, the audience has been more narrowly defined as those who wish to learn more about specific techniques used

© 2014 American Chemical Society In Tools of Chemistry Education Research; Bunce, D., et al.; ACS Symposium Series; American Chemical Society: Washington, DC, 2014.


in chemistry education research, although many aspects may still be useful for a broader audience. Many active areas of research in chemistry education were described in the 2013 National Research Council report on Discipline-Based Education Research (2), hereafter referred to as the DBER report. These areas included student conceptual understanding, the use of technology to support student learning, analysis of student discourse and argumentation patterns, the use of heuristics in student reasoning, and the development of assessment tools to measure student thinking about chemistry. More detail into many of these studies is provided in a review of the peer-reviewed literature conducted by Towns and Kraft (3). The review includes research with many different areas of focus and that use a variety of research designs (qualitative, quantitative, and mixed methods). The review also summarizes several instruments that have been used by the chemistry education research community. The DBER report (2) also includes a series of recommendations to advance DBER, which includes CER, as a field of inquiry. These recommendations include a research agenda that emphasizes the following: exploring the similarities and differences among different groups of students, research in a wide variety of course settings, research that measures a wider range of outcomes and that explores the relationships among those outcomes, research that includes more nuanced aspects of instructional strategies and their implementation, and longitudinal studies. The areas of research and specific tools selected for this book represent a range of approaches (including qualitative and quantitative). The selection of topics to be included in this volume was based on interactions with members of the audience for this book. We received several requests from readers of the Nuts and Bolts book for more information on some topics, while other topics were chosen based on new directions and opportunities for growth in chemistry education. For example, we selected R for particular attention due to its growing use in many disciplines. It is an open-source program that is more easily available to many researchers and has capabilities to address data analysis for some areas of research (such as eye-tracking) in ways that are not yet easily available in other programs. The book is not intended to present an exhaustive list of tools and strategies that can be used in investigating chemistry education research questions, but should present a starting point and encourage broader perspectives of what can be done.

Strategies for Qualitative Research Qualitative research methods are necessary to address “how” and “why” questions. Bretz (4) broadly described qualitative research methods in the Nuts and Bolts of Chemical Education Research, including issues related to data collection and quality, theoretical perspectives, data analysis, and other practical considerations. Towns (5) also addressed qualitative research methods in her chapter on mixed methods research designs. In this volume, the emphasis is on highlighting particular qualitative research methods that can be used to answer chemistry education research questions. 2 In Tools of Chemistry Education Research; Bunce, D., et al.; ACS Symposium Series; American Chemical Society: Washington, DC, 2014.


The qualitative analysis section starts with a chapter on using classroom observations as a tool for investigating chemistry teaching and learning. Yezierski (Chapter 2) describes reasons to use classroom observation protocols as a component of research projects as well as guidelines for collecting and analyzing data. The table of research-based observation protocols that can be found in the literature provides a starting point for any researcher that would like to take advantage of existing instruments. This is extended into a more complete discussion of observation protocols that have particular promise for use in chemistry education research. This is followed by a discussion of using student interviews by Herrington and Daubenmire (Chapter 3). They focus on using open-ended and think-aloud interviews, including several examples from the chemistry education research literature. They provide guidance on developing interview protocols, constructing questions/tasks, selecting participants, conducting the interview, and analyzing the data. The chapter by Cole, Becker, and Stanford (Chapter 4) introduces the area of discourse analysis as a tool for research in chemistry education. They begin by defining discourse and discourse analysis and then describe the types of questions that can be addressed through discourse analysis. They continue by providing an overview of methodological considerations, including data collection and analysis. The chapter concludes with a summary of some CER studies that have used discourse analysis. The section concludes with a chapter by Talanquer (Chapter 5) describing how computer assisted qualitative data analysis (CAQDAS) programs can be used to facilitate organization and analysis of data in qualitative research. He describes a variety of programs that are available, but focuses on how they can be used to support research activities. This ranges from handling and organizing data to assisting in the coding annotation of data to visualizing data.

Analyzing Quantitative Research Data Quantitative research starts with the collection of data, but statistical methods are required as part of the analysis. There are a number of articles that describe problems with how statistical analysis are conducted and reported in educational research (6–9). Sanger (10) provided an overview of inferential statistics in the Nuts and Bolts book, including the steps in conducting a quantitative research study and common misconceptions. In this volume, we add to the previous discussion of statistics and extend it by including chapters on additional statistical techniques and on the software package R. Pentacost (Chapter 6) builds on the work of Sanger by describing how analysis of variance (ANOVA) techniques can be used to support claims in chemistry education research. He begins with a discussion of what it means to determine a difference in data sets and the assumptions about the data that must be met to use these techniques. He then helps readers decide which type of ANOVA is most appropriate for their study and provides examples of how each approach would work using examples from the CER literature. 3 In Tools of Chemistry Education Research; Bunce, D., et al.; ACS Symposium Series; American Chemical Society: Washington, DC, 2014.


The majority of the discussion of statistics in the Nuts and Bolts of Chemistry Education Research focused on parametric statistics, which make certain assumptions about the normality of the data. Many of the studies in chemistry education research do not meet these assumptions and require the use of non-parametric statistics. In the second chapter of this section, S. Lewis (Chapter 7) presents an overview of nonparametric statistics. He begins by discussing the different data scales and comparing nonparametric and parametric statistics. He then summarizes a number of nonparametric statistical tests that are useful in chemistry education studies. The section closes with a chapter by Tang and Ji (Chapter 8) on the statistical program R. They begin with reasons that researchers may want to learn how to use R, including a description of some of the advantages while acknowledging the disadvantages of this particular environment. This is followed by a discussion of the program itself and it’s capabilities. They also describe some areas of research where R has some functionality that makes it better suited to completing the data analysis as compared to other programs such as SPSS.

Cognitive-Based Tools for Chemistry Education Research As chemistry education research has developed as a field, more sophisticated tools and methods have also been identified and developed by researchers to address research questions. There are a number of tools that are emerging as being particularly useful for chemistry education research, many of which have foundations in cognitive psychology. The examples included here highlight some methods that are showing increased use in chemistry education research. The first chapter in this section is a discussion of concept inventories by Bretz (Chapter 9). She begins with a discussion of a variety of design and development methods. This is followed by an extensive discussion of the validity and reliability of the data generated by these instruments as well as their limitations. The chapter ends with a discussion of how concept inventories can be used to measure what and how much chemistry is learned and provides recommendations for chemistry education researchers. In the next chapter, Neiles (Chapter 10) explores the use of tools that can be employed to measure students’ structural knowledge of chemistry. After defining what is meant by structural knowledge, she describes two approaches, concept mapping and proximity data techniques, that have been used to create representations of the connections. She then presents a detailed discussion of how to analyze the resulting structural knowledge networks, particularly with the aid of computer programs such as Pathfinder and GEPHI. The third chapter in this section focuses on eye-tracking technology and its use in chemistry education research. Havanki and VandenPlas (Chapter 11) begin with a discussion of how vision works and how eye movements relate to cognitive processes. They describe different types of eye tracking instruments before describing the types of research that are amenable to eye tracking studies. The heart of the chapter is a discussion of considerations that guide experimental design, data collection, and data analysis. 4 In Tools of Chemistry Education Research; Bunce, D., et al.; ACS Symposium Series; American Chemical Society: Washington, DC, 2014.


In the last chapter in this section, Cooper, Underwood, Bryfczynski, and Klymkowsky (Chapter 12) present a short history of how they have used technology to model and analyze student data. Of particular emphasis is how to use tools to both support student learning and capture data that can be analyzed for research purposes. They describe the features of IMMEX, Organic Pad, and beSocratic from the perspectives of how they were designed to support student learning and as research tools to provide insights into how students develop knowledge and science practice skills.

Practical Issues for Planning, Conducting, and Publishing Chemistry Education Research Important areas that are rarely addressed in other forums are the practical issues of how to plan, conduct, and publish chemistry education research. While the importance of careful design and planning are emphasized for data collection and analysis, this is generally done in the context of ensuring the quality of the data. Several chapters are included in this volume that provide valuable advice on how to ensure that a project goes as smoothly as possible and culminates with results that are publishable. In the first chapter in this section, Bunce (Chapter 13) describes a two-pronged approach to dealing with nonsignificant results. She begins by describing the specific issues in chemistry education research that make this topic important. She then explores two ways of ensuring that studies with statistically nonsignificant results are still valid and contribute to the body of knowledge about teaching and learning in chemistry—planning and post-hoc analysis. She uses examples from her own work to demonstrate how these two approaches can result in quality, publishable studies even if the results indicate there are nonsignificant effects. In the chapter on doing chemistry education research in the “real world,” J. Lewis (Chapter 14) describes the challenges of conducting research projects that involve collecting student data in real classrooms at multiple institutions. This chapter takes a more conversational tone and provides valuable guidance to researchers whose research or evaluation activities involve this type of project design. She focuses on two phases of projects that involve data collection in multiple classrooms, involving multiple instructors, and often times multiple institutions. The first phase addressed is that of planning. After exploring many aspects of planning that make it more likely that the project will result in usable data, she moves on to discussing the monitoring and controlling phase of a project. She ends with discussing how a well-designed and executed project can result in publishable results, even if events do not unfold as planned. The topics of the ethical treatment of participants and Institutional Review Boards (IRBs) should be of concern to all chemistry education researchers (as well as to any one sharing data collected from students or other people). Bauer (Chapter 15) presents an overview of the fundamental principles, purposes, and process of obtaining appropriate IRB oversight of studies involving human subjects. He describes how to get started and some of the expectations for completing applications for IRB approval. He provides several examples to help 5 In Tools of Chemistry Education Research; Bunce, D., et al.; ACS Symposium Series; American Chemical Society: Washington, DC, 2014.


researchers understand the nature and scale of risks of participating in chemistry education research as well as strategies for minimizing these risks. He then describes the levels of review and what these imply for the researcher. The final chapter in this section addresses an issue of concern to most working on chemistry education research projects – how to publish the results. Taber, Towns, and Treagust (Chapter 16) present an overview of the process of publishing chemistry education research, including what makes a manuscript suitable for publication as well as the practical issues of how the process works. Many aspects of what make research publishable have been addressed in previous chapters, but the authors reinforce the importance of thinking about what makes research publishable during the design phase by describing a variety of factors, including following ethical guidelines for research with human participants. The majority of the chapter focuses on the practical aspects of publication including preparing the manuscript for submission, the submission and review process, and what happens after approval or acceptance.

Application The final chapter of the book (Bunce and Cole, Chapter 17) illustrates how the resources provided in the book can be used to assist a researcher in planning, conducting, and publishing their research. Two different research questions are explored to provide comparisons of how the nature of the question drives further decisions.

Final Thoughts The book has been designed so that readers can either focus in to learn about a particular topic or read through the entire book for a broader view. While the information in each chapter should provide enough information for a reader to get started in an area, additional reading is likely to be needed to gain further expertise in a specific area. Our hopes are that the material here will facilitate conversations with colleagues or other researchers to continue to develop and expand the field of chemistry education research.

References 1.

2.

3.

Bunce, D. M., Cole, R. S., Eds.; Nuts and Bolts of Chemical Education Research; ACS Symposium Series 976; American Chemical Society: Washington, DC, 2008. Discipline-Based Education Research: Understanding and Improving Learning in Undergraduate Science and Engineering; National Research Council: Washington, DC, 2012. Towns, M.; Kraft, A. Review and Synthesis of Research in Chemical Education from 2000-2010. Paper presented at the Second Committee Meeting on the Status, Contributions, and Future Directions of DisciplineBased Education Research, 2011. 6 In Tools of Chemistry Education Research; Bunce, D., et al.; ACS Symposium Series; American Chemical Society: Washington, DC, 2014.


4.

Bretz, S. L. Qualitative Research Designs in Chemistry Education Research. In Nuts and Bolts of Chemical Education Research; Bunce, D. M., Cole, R. S., Eds.; ACS Symposium Series 976; American Chemical Society: Washington, DC, 2008; Chapter 7. 5. Towns, M. H. Mixed Methods Designs in Chemical Education Research. In Nuts and Bolts of Chemical Education Research; Bunce, D. M., Cole, R. S., Eds.; ACS Symposium Series 976; American Chemical Society: Washington, DC, 2008; Chapter 9. 6. Keselman, H. J.; Huberty, C. J.; Lix, L. M.; Olejnik, S.; Cribbie, R. A.; Donahue, B.; Kowalchuk, R. K.; Lowman, L. L.; Petoskey, M. D.; Keselman, J. C.; Levin, J. R. Statistical practices of educational researchers: An analysis of their ANOVA, MANOVA, and ANCOVA analyses. Rev. Educ. Res. 1998, 68 (3), 350–386. 7. Lewis, S. E.; Lewis, J. E. The same or not the same: Equivalence as an issue in educational research. J. Chem. Educ. 2005, 82 (9), 1408. 8. Henson, R. K.; Hull, D. M.; Williams, C. S. Methodology in our education research culture: Toward a stronger collective quantitative proficiency. Educ. Res. 2010, 39 (3), 229–240. 9. Kirk, R. E. Promoting good statistical practices: Some suggestions. Educ. Psychol. Meas. 2001, 61 (2), 213–218. 10. Sanger, M. J., Using Inferential Statistics to Answer Quantitative Chemical Education Research Questions. In Nuts and Bolts of Chemical Education Research; Bunce, D. M., Cole, R. S., Eds.; ACS Symposium Series 976; American Chemical Society: Washington, DC, 2008; Chapter 8.

7 In Tools of Chemistry Education Research; Bunce, D., et al.; ACS Symposium Series; American Chemical Society: Washington, DC, 2014.

An Introduction to the Tools of Chemistry Education Research - ACS

Recommend Documents