Article pubs.acs.org/jchemeduc
The Citation Index of Chemistry Education Research in the Journal of Chemical Education from 2008 to 2016: A Closer Look at the Impact Factor Jon-Marc G. Rodriguez,† Kinsey Bain,† Alena Moon,‡ Michael R. Mack,§ Brittland K. DeKorver,∥ and Marcy H. Towns*,† †
Department of Chemistry, Purdue University, West Lafayette, Indiana 47907, United States Department of Chemistry, University of Michigan, Ann Arbor, Michigan 48109, United States § Te Waka Urungi, UNITEC Institute of Technology, Mt. Albert, Auckland 1025, New Zealand ∥ Department of Chemistry, Lyman Briggs College, Michigan State University, East Lansing, Michigan 48825, United States ‡
S Supporting Information *
ABSTRACT: Journal impact factors are a metric often used to evaluate journals; they are calculated by considering a journal’s citation and publication rates during a specified time period. In some cases, impact factors can be misleading because they do not take into account the publication of different types of papers. In the Journal of Chemical Education, many papers provide useful laboratory exercises and classroom activities for teaching faculty and instructors, but they are not formally considered chemistry education research. Laboratory and classroom activity papers may positively affect classroom practices and curriculum development through adaptation and implementation, yet are cited less frequently. This study calculates a citation index for chemistry education research articles published between 2006 and 2015 and compares that index to the published impact factor. We find that the calculated citation index at the twoyear and five-year intervals is larger than the corresponding impact factor for the same time period, demonstrating that chemistry education research articles are cited with greater frequency than previously suggested by impact factor calculations. KEYWORDS: Chemical Education Research, Graduate Education/Research, Communication/Writing, Administrative Issues, Professional Development FEATURE: Chemical Education Research
■
INTRODUCTION
an indication that the work has been useful. One method of measuring a particular paper’s influence on the field is to examine how often it has been cited and by whom. Similarly, the citation indexes can be calculated to indicate the rate at which papers within a journal are cited. These metrics may be used by researchers when determining where to submit a manuscript or by tenure and promotion committees to evaluate the journals in which researchers publish. Thus, it is important to examine citation data.
The Nature of Citations
The practice in scientific communities is to publish discoveries in order to share them with the community and recognize the work of reseachers.1 Publication allows knowledge to become available, cumulative, and transferable within and across fields. Furthermore, citations within publications help disseminate work by allowing readers to find related research. This allows scholars to use research designs, theories, and findings to inform studies, expand knowledge, or contrast new findings to previous work.2 In the absence of publication, knowledge can only be transmitted proximally through scientists who are closely associated with the researcher’s network of colleagues. Moving from the network model to the broadcast model allows researchers to share their findings and discourage duplication of research efforts, which increases the pace of innovation and promotes the expansion of our collective knowledge. Citations also serve the function of evaluating the utility and quality of previous work. When a researcher cites a paper, it is © XXXX American Chemical Society and Division of Chemical Education, Inc.
Rationale
According to the 2016 Journal Citation Report from Thomson Reuters (Clarivate Analytics), the 2015 two-year journal impact factor (IF) for the Journal of Chemical Education (JCE) is 1.225, which is calculated by dividing the number of citations in 2014 and 2013 by the number of citable items (articles, Received: January 24, 2017 Revised: April 11, 2017
A
DOI: 10.1021/acs.jchemed.7b00062 J. Chem. Educ. XXXX, XXX, XXX−XXX
Journal of Chemical Education
Article
commentaries, and reviews) published in 2014 and 2013.3−5 Chemistry education research (CER) articles published in this Journal contain the findings of research for the purpose of informing CER, sharing knowledge about classroom practice, and presenting theories regarding how students reason and learn about chemistry.6 Discipline-based education researchers, especially CER scholars, cite these articles to support their ideas about student learning, influence considerations of research design, and interpret findings (see, for example, refs 7−89). Furthermore, chemistry educators cite evidence from CER literature published in this Journal to support their pedagogical innovations in classroom and laboratory courses, indicating CER articles are used and cited differently than other manuscripts published in this Journal (see for example refs 10−12). However, the Journal also publishes a variety of other types of papers, such as classroom activities, laboratory experiments, technology reports, and commentaries. These papers make educational resources available to faculty and impact instruction and curriculum development, but are infrequently cited in subsequent work. For example, laboratory experiments are published with Supporting Information containing notes regarding practical implementation for faculty and materials for students. If a faculty member wanted to use or adapt an experiment, it is likely she would simply download the paper and consider how to incorporate it into her teaching. Thus, laboratory experiment papers have the potential to influence classroom practices without leading to citations. The journal impact factor is intended to measure citations of the published research. For this Journal the calculation includes a significant number of papers that are used differently by the community and cited infrequently, suggesting there is a need to determine a measure of the frequency of citation based solely on the citations of CER articles. This report seeks to respond to that need and provide a service to the chemistry education research community.
■
and discussed articles in the sample that did not have a CER keyword. An article remained in the sample if all three researchers agreed it met the criteria. Building on the initial guidelines used to assemble the sample, this review provided an additional layer of evaluation of the articles across three researchers. The citations of articles in the sample were manually gathered during November and December of 2016, and then the articles were rechecked in March of 2017 during article revision, which completed data collection. Due to current publication practice, this included articles published ASAP using DOIs, as well as formally published articles with citations that have volume, issue, and page numbers. The process of determining the number of citations for each CER article in the sample involved using Google Scholar to identify and evaluate citations. The evaluation used two criteria for inclusion in this study: the citation must come from the peer-reviewed literature and be written in English (which is necessary in order for the authors to be able to evaluate the paper based on the established criteria). This included articles, reviews, and proceedings, but excluded theses, dissertations, abstracts (such as ACS conference abstracts), and editorials. Self-citations were included in the analysis because researchers may have a research trajectory in which the findings of subsequent studies build on one another. ACS Symposium book chapters (which are peer-reviewed) were identified and counted, but book chapters from other publishers that could not be identified as peer-reviewed were excluded from this study. These criteria may have led to a more conservative citation index calculation relative to the Thomson Reuters (Clarivate Analytics) impact factor, as any citation is considered valid for that calculation.4,5 A similar procedure was followed using the Web of Science database, although we noted that the number of citations in Web of Science was smaller than the number of citations obtained from Google Scholar, with similar findings reported by Ye et al.14 The Supporting Information contains a table with citation counts for each individual CER article that was published between 2006 and 2015, as well as a column with compiled citation counts for each year.
METHODS
Sampling
Calculation of Citation Index
This study focused on generating a citation index for CER articles published in this Journal from 2006 to 2015. In 2013, guidelines for CER manuscripts were published in the Journal.6 Thus, the samples in 2014 and 2015 were the articles identified as CER articles by keyword. Between 2006 and 2013 a different strategy was used in the absence of the published guidelines. CER articles were defined as having one of the following: (i) a research question with collection and analysis of data or a declared intention of collecting data; (ii) a review of the research with implications for research and the chemistry classroom; (iii) a theoretical perspective or approach which provided framing for the field. We note that perhaps these articles would not meet all of the criteria currently in place for CER with regard to inclusion of theoretical frameworks and data analysis (e.g., the reporting of effect sizes or coding methods in qualitative studies). However, as other authors have noted, the field has grown, and overall publication rates across the CER field have increased.13 With that increase has come a desire to codify what is meant by a CER publication. Thus, the sampling methods encompass those evolving practices across 2006−2015. To ensure the identified articles met the criteria for inclusion, three of the authors (J.G.R., K.B., and M.H.T.) jointly reviewed
Using our data set, a modified impact factor was calculated; however, given that Thomson Reuters (Clarivate Analytics) has proprietary right to the calculation of a journal impact factor, we refer to our calculation as a CER citation index (CI). As shown below, the two-year citation index for 2016 was calculated by taking the total number of citations during 2016 for CER articles published in 2014 and 2015 and dividing it by the number of CER articles published in this Journal in 2014 and 2015. This general formula was used to calculate the two-year citation indexes for 2008−2016, the five-year citation indexes for 2011−2016, and the ten-year citation index for 2016. Each citation index was calculated twice, corresponding to calculating the citation index with (+) and without (−) peerreviewed book chapters. 2 yr Citation Index of 2016 #citations in 2016 of CER articles published 2014 and 2015 = #CER articles published in JCE 2014 and 2015
5 yr Citation Index of 2016 #citations in 2016 of CER articles published 2011−2015 = #CER articles published in JCE 2011−2015 B
DOI: 10.1021/acs.jchemed.7b00062 J. Chem. Educ. XXXX, XXX, XXX−XXX
Journal of Chemical Education
C
0 2 1 0 0 2 1 1 7 1 26 44 55 32 57 54 61 330 0 2 2 1 1 0 0 6
a
3 3
0 0
1 12 13
0 0 0
0 11 26 37
0 0 0 0
1 6 27 42 76
0 0 0 1 1
0 4 18 37 45 104
0 0 0 0 2 2
2 5 19 43 47 55 171
0 3 1 0 0 0 4
1 5 39 49 63 56 53 266
Due to the ability to make papers immediately available through online and ASAP publications, some articles published in an issue of a specific year may be cited the previous year when they were first published online.
0 0 2 1 0 0 0 0 1 2 6 25 18 32 27 9 14 4 13 15 13 170 15 9 22 22 5 11 3 2 3 3 95 95 68 132 137 73 89 45 70 62 75 846 1 3 9 8 6 4 3 4 0 1 39 40 69 159 127 78 75 43 59 48 72 770 0 3 53 25 13 9 5 11 6 7 131 0 19 116 139 84 95 42 90 50 86 721 0 5 15 6 2 8 15 8 8 67
A B 2015 A B 2014 A B 2013 A B 2012
A B
2011
A B
2010
A B
2009
A B
2008
A B
2007
A B
2006
A
2015 (33) 2014 (34) 2013 (55) 2012 (51) 2011 (26) 2010 (25) 2009 (19) 2008 (17) 2007 (26) 2006 (30) Total
It is important to note that direct comparisons between our citation indexes and Thomson Reuters (Clarivate Analytics) impact factor values may be limited. This is largely a result of the proprietary nature of the impact factor, which is likely calculated using methods and a database different from those used by the authors. In addition, our citation counts included self-citations, because researchers tend to cite their previous work within a research trajectory. We did not separately calculate a self-cite and non-self-cite citation index. Finally, we note that in designing the study a key decision was to establish the criteria that defined which articles would be
0 15 90 79 72 66 82 79 77 560
B A
2017
Limitations
B
Table 1. Citations of CER Articles in the Journal of Chemical Education by Year
CONCLUSION Impact factors are calculated and disseminated yearly for a wide variety of journals.4 We have demonstrated that it is possible to calculate a citation index, a CI, for CER articles published in this Journal. We discovered that the 2-year and 5-year CIs are larger than the reported IFs, which includes a wider variety of papers that the chemistry education community values and uses, but cites less frequently. Although impact factors are often used as an indication of the relative prestige of a journal, this study supports the notion that there are a variety of factors that can influence the magnitude of the citation index, demonstrating the importance of utilizing other means to evaluate the quality of academic work and acknowledging that citations and the CIs provide only a partial picture of the visibility and impact of a particular scholar or journal.15−19 The CIs in this report may prove helpful for faculty comparing journals in the field of chemical education and may be helpful to faculty preparing promotion and tenure dossiers.
2005
■
A
Table 1 displays the number of citations for CER articles in each year based on our review of published articles in the Journal and their citations. Each column shows counts of the citations with the peer-reviewed books tabulated separately in Column B for those who wish to exclude books. Each row contains a year-by-year count of citations from the year the articles were published to 2017. The resulting citation indexes calculated in this study are presented in Table 2 and graphically represented in Figure 1, along with the impact factor values for the Journal reported by Thomson Reuters (Clarivate Analytics).4 From a comparison of each CI with its corresponding IF in Figure 1, it is readily observed that the CI is 2−3 times larger than the IF. This is especially impressive because, as discussed above, our CI calculation may be more conservative than that of the IF.4,5 For the Journal, the general trend is that both the CI and IF increase over time, and the IF appears less susceptible to yearly fluctuations. When including the peer-reviewed book chapters in the calculation of the CI, in most cases the increase in the CI was small (although there was one year where there was no change), with the smallest change being a 0.018 increase and the largest change being a 0.726 increase in the citation index.
Citations: Column A Only Counts Articles, Reviews, and Proceedings; Column B Only Counts Peer-Reviewed Book Chapters
Citation Indexes and Trends
2016
B
RESULTS AND DISCUSSION
Year (N, JCE CER Articles)a
■
Total
10 yr Citation Index of 2016 #citations in 2016 of CER articles published 2006−2015 = #CER articles published in JCE 2006−2015
176 189 546 620 407 475 326 536 506 645 4426
Article
DOI: 10.1021/acs.jchemed.7b00062 J. Chem. Educ. XXXX, XXX, XXX−XXX
Journal of Chemical Education
Article
Table 2. Comparison of JCE Multiyear Citation Indexes and Impact Factors for CER Articles Analysis Results by Year Parameter
2008
2009
2010
2011
2012
2013
2014
2015
2016
2-year IF 2-year CIa (−) 2-year CIb (+) 5-year IFc 5-year CI (−) 5-year CI (+) 10-year CI (−) 10-year CI (+)
0.538 1.232 1.250 0.647 d d d d
0.586 1.279 1.279 0.677 d d d d
0.571 1.722 1.750 0.673 d d d d
0.739 2.000 2.068 0.763 2.222 2.256 d d
0.817 1.941 1.961 0.831 2.142 2.171 d d
1.001 2.195 2.468 1.046 2.819 3.152 d d
1.106 2.406 3.132 1.089 0.2.705 3.295 d d
1.225 2.562 2.697 1.240 2.660 2.817 d d
e 2.433 2.791 e 2.538 2.905 2.677 2.978
a (−) indicates the citation index (CI) was calculated in this work without including peer-reviewed book chapters. b(+) indicates the citation index was calculated in this work by including peer-reviewed book chapters. cIFs (impact factors) come from Thomson Reuters (Clarivate Analytics). dCI calculation requires data beyond the scope of this study. eNot available at time of publication.
■
(1) Sorenson, O.; Fleming, L. Science and the Diffusion of Knowledge. Research Policy 2004, 33 (10), 1615−1634. (2) Buntrock, R. E. Using Citation Indexes, Citation Searching, and Bibliometrics To Improve Chemistry Scholarship, Research, and Administration. J. Chem. Educ. 2015, 93 (3), 560−566. (3) Journal of Chemical Education. http://pubs.acs.org/journal/jceda8 (accessed Apr 2017). (4) Journal Citation Reports. https://jcr.incites.thomsonreuters.com/ JCRJournalProfileAction.action?pg= JRNLPROF&journalImpactFactor=1.225&year=2015&journalTitle= JOURNAL%20OF%20CHEMICAL%20EDUCATION&edition= SCIE&journal=J%20CHEM%20EDUC (accessed Apr 2017). (5) Cameron, B. D. Trends in the Usage of ISI Bibliometric Data: Uses, Abuses, and Implications. portal: Libraries and the Academy 2005, 5 (1), 105−125. (6) Content Requirements for Chemical Education Manuscripts. http:// pubs.acs.org/paragonplus/submission/jceda8/jceda8_CER_Guide.pdf (accessed Apr 2017). (7) Cooper, M. M.; Grove, N.; Underwood, S. M.; Klymkowsky, M. W. Lost in Lewis Structures: An Investigation of Student Difficulties in Developing Representational Competence. J. Chem. Educ. 2010, 87 (8), 869−874. (8) Kelly, R. M.; Jones, L. L. Investigating Students’ Ability To Transfer Ideas Learned from Molecular Animations of the Dissolution Process. J. Chem. Educ. 2008, 85 (2), 303−309. (9) Bunce, D. M.; VandenPlas, J. R.; Havanki, K. L. Comparing the Effectiveness on Student Achievement of a Student Response System versus Online WebCT Quizzes. J. Chem. Educ. 2006, 83 (3), 488−493. (10) Collison, C. G.; Cody, J.; Stanford, C. An SN1−SN2 Lesson in an Organic Chemistry Lab Using a Studio-Based Approach. J. Chem. Educ. 2012, 89 (6), 750−754. (11) Shea, K. M.; Gorin, D. J.; Buck, M. E. Literature-Based Problems for Introductory Organic Chemistry Quizzes and Exams. J. Chem. Educ. 2016, 93 (5), 886−890. (12) Vázquez, A. V.; McLoughlin, K.; Sabbagh, M.; Runkle, A. C.; Simon, J.; Coppola, B. P.; Pazicni, S. Writing-To-Teach: A New Pedagogical Approach To Elicit Explanative Writing from Undergraduate Chemistry Students. J. Chem. Educ. 2012, 89 (8), 1025− 1031. (13) Craig, A. F.; Koch, D. L.; Buffington, A.; Grove, N. Narrowing the Gap? Revisiting publication rates in chemistry education. J. Chem. Educ. 2012, 89 (12), 1606−1608. (14) Ye, L.; Lewis, S. E.; Raker, J. R.; Oueini, R. Examining the Impact of Chemistry Education Research Articles from 2007 through 2013 by Citation Counts. J. Chem. Educ. 2015, 92 (8), 1299−1305. (15) Hendee, W.; Bernstein, M. A.; Levine, D. Scientific Journals and Impact Factors. Med. Phys. 2011, 38 (12), i. (16) Shanta, A.; Pradhan, A. S.; Sharma, S. D. Impact Factor of a Scientific Journal: Is It a Measure of Quality of Research? J. Med. Phys. 2013, 38 (4), 155−157.
Figure 1. JCE multiyear CER citation indexes (CI) and journal impact factors (IF) by year. The impact factors come from Thomson Reuters (Clarivate Analytics), and the citation indexes are the results of the analysis carried out in this article. In the graph above, (−) indicates the citation index was calculated without including peer-reviewed book chapters, and (+) indicates the citation index was calculated by including peer-reviewed book chapters.
included. Future researchers may use different criteria that would alter the sample and subsequent CI calculations.
■
ASSOCIATED CONTENT
S Supporting Information *
The Supporting Information is available on the ACS Publications website at DOI: 10.1021/acs.jchemed.7b00062. Table of individual CER articles with citation counts (XLSX)
■
REFERENCES
AUTHOR INFORMATION
Corresponding Author
*E-mail:
[email protected]. ORCID
Marcy H. Towns: 0000-0002-8422-4874 Notes
The authors declare no competing financial interest. D
DOI: 10.1021/acs.jchemed.7b00062 J. Chem. Educ. XXXX, XXX, XXX−XXX
Journal of Chemical Education
Article
(17) Alberts, B. Impact Factor Distortions. Science 2013, 340 (6134), 787. (18) San Francisco Declaration on Research Assessment. https://en. wikipedia.org/wiki/San_Francisco_Declaration_on_Research_ Assessment (accessed Apr 2017). (19) Towns, M. H.; Kraft, A. The 2010 Rankings of Chemical Education and Science Education Journals by Faculty Engaged in Chemical Education Research. J. Chem. Educ. 2012, 89 (1), 16−20.
E
DOI: 10.1021/acs.jchemed.7b00062 J. Chem. Educ. XXXX, XXX, XXX−XXX
