SC I E N C E & T EC H N O LO GY
THE IMPORT OF IMPACT
COU RT ESY O F J O RG E H I RSC H / U O F CA L I FO R N I A , SA N D I EG O
plains, “some papers are very highly cited; other papers are not cited at all.” The same goes for an individual author’s collection of papers. New types of JOURNAL METRICS grow more One way to address this issue is to use influential in the scientific community other types of measures, such as the h inSOPHIE L. ROVNER, C&EN WASHINGTON dex, Pringle says. “It’s a statistical way of showing how many papers within the individual’s work rank at a certain level.” “The main advantage of the h index is lists of researchers whose papers have reAT ONE POINT in his career, Nobel Laurethat it is less susceptible to distortions and ceived the highest number of citations over ate Sir Harold W. Kroto was the second fluctuations,” says Jorge E. Hirsch, a physthe past 10 or 20 years. most highly cited chemist in Britain— ics professor at the University of California, The impact factor of a journal is caltopped only by the University of SouthSan Diego, who introduced the h index in culated by first counting the number of ampton’s Martin Fleischmann, one of the 2005 (Proc. Natl. Acad. Sci. USA 2005, 102, citations received in a given year by items proponents of cold fusion. 16569). He defines the h index as the maxithe journal published in the previous two Kroto, who codiscovered C60 and is mum number of an author’s papers that years. That quantity is then divided by the currently a chemistry professor at Florida have been cited at least h times. State University, declines to Other methods continue to draw any conclusions from emerge. Last year, for instance, that experience. But given the University of Washington ultimate fate of cold fusion, the biologist Carl T. Bergstrom anecdote suggests that citation and colleagues proposed the statistics aren’t always a good inEigenfactor. This metric, which dicator of scientific excellence. is calculated by using network Still, citation counts are used theory, ranks journals according to rank the performance of into influence. A journal is considdividual academic researchers, ered influential if its articles are their departments, and even heavily cited within five years of their universities, and thus they publication by other influential help influence promotion and journals. The Eigenfactor team funding decisions. says the metric measures a jourCitation metrics and other nal’s importance to the scienstatistics related to usage are tific community. also used to evaluate the signifiThe Eigenfactor, h index, cance and impact of individual and impact factor are all calcunumber of articles published in journals. The statistics help librarians METRIC EQUALIZER Hirsch lated from citations, meaning the journal in those two years. select which journals to subscribe to and introduced the h that an author has to have read Although the impact factor help authors decide where to submit their index as a “more an article and then written is widely viewed as a general manuscripts. democratic another article citing the first measure of a journal’s quality, The journal impact factor “is probably assessment of importance, and influence, it has people’s research.” one. However, readers based in the most well-established metric that’s out industry “tend to read but not some drawbacks. “If you look at there,” according to Susan King, senior vice cite because they are publishchemistry journals, quite often president in the journals publishing group ing patents,” King says. “If you’ve got an articles will be cited long after that twoat the American Chemical Society, which applied journal, you may be distressed year period,” King says. “Just looking at a publishes C&EN. “Librarians, publishers, because it’s got a low impact factor. When relatively narrow window—two years— authors, and grant-funding bodies all pay you look at the usage, however, you may may not truly capture those journals that attention to the impact factor.” see that it’s used very heavily.” In that case, have a longer citation half-life.” Eugene Garfield first introduced the even though the journal has a low impact Furthermore, the impact factor meaconcept of an impact factor in 1955, when factor, it’s actually a very important journal sures the average number of citations per he was director of the Institute for Scienbecause it’s widely used, she says. article in a given journal. But by itself it tific Information (ISI). Since then, ISI, Such usage is tracked by counting downdoes not give a full picture of the underlywhich is now part of Thomson Reuters, has loads of online documents. A raw usage ing data because “citation data tend to developed dozens of other metrics, includnumber isn’t all that informative, however. be very skewed,” says James Pringle, vice ing the immediacy index (which indicates “All other things being equal, a journal that president of product development for how soon after publication a journal’s arpublishes 2,000 articles a year is going to Thomson Reuters’ scientific business. For ticles are cited on average), the percentage generate significantly more requests for a given journal over a period of time, he exof self-citations by individual authors, and downloads than one that publishes merely 50,” says Richard Gedye, research director Get tips on coping with cheating, errors, and ambiguities in MORE ONLINE for the journals division at Oxford Univercitations at C&EN Online, www.cen-online.org. W W W.C E N - O N L I N E .O RG
8621sci1_live.indd 39
39
M AY 2 6, 20 0 8
5/21/08 3:19:55 PM
SC I E N C E & T EC H N O LO GY
sity Press in the U.K. A “journal usage factor” that compensates for such differences is under development. He adds that usage factors offer an advantage over citation data in that “usage data are available pretty well from day one of publication.” Gedye chairs the Counting Online Usage of Networked Electronic Resources group (COUNTER), which was established in 2002 to make sure that publishers, librarians, and others measure usage in a consistent way. COUNTER is working with the U.K. Serials Group, which promotes collaboration on matters related to publishing, and other organizations to develop the usage factor.
“A lot of work might be flashy and get cited right away, but the true value of lots of very high-quality work is only known over time.”
In addition, the usage factor team is “aiming to compensate, in a way that the impact factor doesn’t, for the point in a calendar year when an article is published,” Gedye says. “If you are looking at average amounts of downloads per article over a given period of time for articles published in a given year, clearly there’s going to be more usage of an article published in January than there is of one published in NoCHEMISTRY CITATIONS vember,” he explains. Scripps and Harvard garnered “So we’re looking at the most citations per paper average downloads of articles in the first 12 CITATIONS months of their life or RANK INSTITUTION PAPERS PER PAPER the second 12 months 1 2,178 Scripps Research Institute 32.71 of their life, irrespec2 2,773 Harvard University 32.49 tive of when in the 3 2,217 California Institute of Technology 31.91 year they were pub4 1,766 Yale University 30.35 lished. We feel that 5 3,734 Massachusetts Institute of Technology 26.69 will create a more 6 2,888 Northwestern University 25.75 7 5,271 University of California, Berkeley 24.25 level playing field.” 8 2,225 University of California, Santa Barbara 22.90 The group also 9 2,729 Stanford University 22.72 wants to avoid “the 10 University of Utrecht, the Netherlands 2,469 20.35 issue that you get with the impact NOTE: Based on institutions with 50,000 or more citations. SOURCE: Thomson Reuters’ Essential Science Indicators database covering factor, where some journal articles published from January 1997 through December 2007 items appear in the numerator but not in the denominator,” Gedye says. “For exACS is among the publishers that are ample, the count of citable articles doesn’t helping scope out parameters for this metinclude letters to the editor, but if a letter ric, such as how it should be calculated and to the editor is cited, it counts in the nuwhat time frame should be used. Gedye says merator. So if you publish a lot of letters to the usage factor will have some similarity to the editor that are very controversial and the impact factor. For instance, it might be interesting, then your impact factor is gocalculated by taking total usage during peing to get a bit of a boost.” riod x of articles published during period y, The numerator-denominator discrepand dividing that number by the total numancy isn’t the only criticism that’s lobbed at ber of articles published during period y. Thomson Reuters’ impact factor. Indeed, The next stage of the project will test the company periodically comes under fire proposals for calculating and auditing the over the accuracy and import of its metmetric with real data provided by publishrics. For instance, Mike Rossner, executive ers, but there are several nuances to condirector of Rockefeller University Press, sider. “If we want to look at the usage in a and colleagues recently took the company given year of articles published in specific to task for what the authors perceive as erother years, then we have to address how rors and inconsistencies in its data (J. Cell we define when an article was published,” Biol. 2008, 180, 254, and 2007, 179, 1091). Gedye says. “Very often an article goes up Thomson Reuters posted a rebuttal on its online months before it appears in a desigwebsite (scientific.thomsonreuters.com/ nated issue in print. We have to make sure that we are looking at statistics from when citationimpactforum/8427045). articles originally came online.” As a general rule, Thomson Reuters’ W W W.C E N - O N L I N E .O RG
8621sci1_live.indd 40
40
Pringle says, “there’s going to be some small percentage of things that need to be corrected,” given the tremendous volume of articles the firm analyzes. He adds that the company fixes errors brought to its attention by authors and publishers. Despite the controversies, impact factors and other journal metrics have a wide following in academia. Librarians use impact factors to guide decisions about journals, such as whether to renew a subscription to a particular journal or to subscribe to a new one, ACS’s King says. “Librarians are also increasingly using usage, and particularly cost per use, as a means of helping them develop their collections.” PUBLISHERS USE several metrics to as-
sess how their journals measure up against competing publications. For example, King evaluates ACS journal performance by impact factors and numbers of manuscripts submitted and articles published. She is also examining the utility of the h index and Eigenfactor for journal evaluation. “I think you want as many tools as possible when you’re looking at journals,” she explains. King tracks changes over time in the impact factors for ACS journals and their competitors. “If the impact factor of an ACS journal is declining while the other journals in that area are going up,” she notes, “I’d want to try and find out why. Has there been some change in the number or types of articles published in a given journal?” For instance, an increase in the number of review articles can boost a journal’s impact factor because reviews tend to be cited more heavily than original research articles, she says. On the other hand, if the impact factors of all of the journals in a particular field are dropping, it could indicate that scientific interest in that field is waning, she says. Thomson Reuters notes that the impact factor must be used judiciously. With this metric, “the object under study is a journal. It’s not an individual or a department or a university or a country,” Pringle says. Despite the cautions voiced by Thomson Reuters, the impact factor is “used in ways that were never anticipated,” Pringle says.
M AY 2 6, 20 0 8
5/21/08 3:19:56 PM
for an author’s work in a particular field doesn’t necessarily mean the work is lowcaliber. For example, Kroto says the papers describing his most original and intellectually important work garnered few citations, “partly because I left the field and didn’t continue to tell people that we started it off.” The work, performed at the University of Sussex with John F. Nixon, showed that multiple bonds could be formed between carbon and phosphorus and launched the field of phospha-alkyne and phospha-alkene chemistry, Kroto says. “I doubt whether I would get a citation at all today, although it’s a major field of chemistry.” “Evaluation, particularly where individuals are involved, has to be done extremely carefully, because one is dealing with people’s careers,” Pringle says. Thomson Reuters has developed a set of guidelines for using citation analysis to
t t t t t t
Power Supplies Quartz Mercury Vapor Immersion Lamps Safety Cabinets Borosilicate Glass Reaction Vessels Quartz & Borosilicate Glass Immersion Wells Custom Configurations Available Toll-free:
P.O. Box 688 Vineland, NJ 08362-0688
Toll-free fax:
800-223-4524 800-543-6752 856-692-3333 t Fax: 856-692-8919
www.aceglass.com
[email protected] Microscale or Benchscale Apparatus!
W W W.C E N - O N L I N E .O RG
8621sci1_live.indd 41
41
COU RT ESY O F R I C H A R D G E DY E /OX FO R D U N I V E RS I T Y P R ESS
HOWEVER, a low number of citations
the true value of lots of very high-quality work is only known over time.” She says that makes the h index a “questionable parameter to use for people who are at a very early stage in their career or if you are trying to analyze the impact of work that’s just been done in the last couple of years, which is typically the case for someone coming up for advancement to tenure. The work may be superb, but the true impact has yet to manifest itself.” As a result, Garrell adds, “many people have concluded that the h index is more useful as a retrospective metric, where you’re looking at someone at a more advanced stage in their career. You could say, ‘This person’s work has been cited a lot more than someone else’s work, or the individual has a higher h index.’ Then the metric has value to people who perhaps aren’t in that field,” Garrell says. “The fact is, at that point, people in the field really know the substance of the work, so the metric doesn’t add much to what they think.” When making hiring and promotion decisions, Garrell’s department also discusses whether individuals are publishing in the super-high-impact journals. “There are very different feelings among different people about the value of that metric,” she says. “Some say that it is abso-
R EQU EST M O R E AT A D I N FO N OW.O RG
evaluate research. For instance, when analyzing metrics about a particular author, university, or country, it’s important to establish appropriate benchmarks against which the scientist or institution can be compared, Pringle says. “If I’m analyzing papers produced by a university within a field, I need to take into account the expected citation rate of papers in that field, in the journals underlying it, in the years in which the articles were produced.” Pringle adds, “If I’m a biologist, I would expect to see different levels of citation counts than if I were a chemist or a physicist or an economist” because citation practices vary tremendously across fields. Just how could citation Gedye counts be used in chemistry departments? Robin L. Garrell, a chemistry and biochemistry professor at the University of California, Los Angeles, says her department has discussed the value of h indexes in decisions involving promotion to tenure and the hiring of senior faculty. As with other metrics, the h index must be used with care. “It’s widely understood that assessing the true impact of work can only be done over a fairly long timeline,” Garrell says. “A lot of work might be flashy and get cited right away, but
“For example, people will take the impact factor of each journal in which someone has published and then come up with a ranking of an individual author on that basis, and the metric was never really intended for that purpose.” Just because that author published a paper in a journal with a high impact factor, it would be incorrect to assume that that specific paper would be highly cited, Pringle explains. An individual author’s work could be better assessed by summing up the number of citations to each of that scientist’s papers.
M AY 2 6, 20 0 8
5/21/08 3:19:56 PM
SC I E N C E & T EC H N O LO GY
R EQU EST M O R E AT A D I N FO N OW.O RG
lutely essential for candidates for tenure to have published in Science and Nature. This has been challenged by many people who raise the concern that not all topics are suitable for those magazines,” Garrell says.
Visit us at:
www.btemsoftware.com
R EQU EST M O R E AT A D I N FO N OW.O RG
An advanced method for recovering pure component spectra
Self-modeling curve resolution for multi-component infrared and Raman spectra
FOR INSTANCE, it would be tough to write up a total synthesis or the development of a theoretical model for the broad readership of Science and Nature, she explains. In addition, neither journal “allows for a full presentation of the results,” which can make replication of a reported experiment difficult for other scientists. “So the more traditional journals—many of the high-impact ACS journals, for example—are often more useful to the scholarly community” for advancing science, she says. Furthermore, “I feel that the expectation to publish in Science and Nature potentially disadvantages outstanding scholars whose personal style is just less aggressive,” Garrell says. “To get your work in these journals, you may have to communicate with the editorial board and hammer on them.” The likelihood that a manuscript will be accepted by Science or Nature can be swayed by factors apart from quality, such as how hot the subject matter is, Garrell says. In addition, “once you start getting into Science and Nature, it’s easier to get into Science and Nature. Certain people have published a lot of papers in Science and Nature. Jan Hendrik Schön is one,” she says, referring to the Bell Labs physicist disgraced for publishing articles based on falsified experimental data (C&EN, Sept. 30, 2002, page 9). His papers were later retracted. Hirsch’s own troubles with breaking into the ranks of the top-notch journals helped drive his determination to create the h index. “I have had difficulty publishing in very high-impact journals because of the somewhat controversial nature of my research,” which questions the conventional theory of superconductivity, he says. “I have had several papers that got published in not-as-high-impact journals and then got lots of citations because after some time people realized that they were good papers. So I partly was motivated by the desire to have more of a democratic assessment of people’s research that doesn’t depend on whether they got their paper published in a highimpact journal.” Hirsch adds that “it’s good to have some quantitative parameters” to suppleW W W.C E N - O N L I N E .O RG
8621sci1_live.indd 42
42
ment and validate subjective criteria “when you’re thinking of hiring or promoting people.” At the same time, he emphasizes that the h index and other quantitative parameters should be “evaluated together with more detailed evidence on the research, on the content of the papers, and on recommendations.” “If you’ve actually gone through that detailed reading of the scholarly work, you might conclude that the metrics don’t add much, and so the people who are using them more are the people who can’t or won’t go through that effort,” Garrell says. That might be the case at a university’s administrative level. For instance, a dean or other administrator might reject a new hire recommended by a faculty committee because the candidate has a low h index, Hirsch says. Reliance on metrics can arise even at the department level if the department includes professors from a lot of different subdisciplines, “not all of whom would feel competent or interested in reading all the papers by someone who’s coming up for promotion,” Garrell says. That’s why departments solicit the opinions of outside reviewers who can “assess quality and creativity and also whether the work has had or is beginning to have an impact.” RUMORS ABOUND that citation metrics play a part in grant allocation. Garrell says she’s never observed peer review panels in the U.S. using such metrics when making decisions about awarding grants. But she notes that reviewers do examine the quality and quantity of the investigator’s work. Likewise, the U.K.’s national researchfunding apparatus currently relies on peer review rather than citation analysis when making grant decisions for its universities and other educational institutions. But that situation is changing. After this year, the next national evaluation of academic research departments in the U.K., known as the Research Excellence Framework, will utilize citation metrics. The Higher Education Funding Council for England, which distributes public money to universities and colleges for teaching and research, is implementing the shift. The new assessment method will be phased in beginning in 2011. Observations drawn from the U.K. experience should help clarify exactly how large a role citation metrics should play in the research sector. ■
M AY 2 6, 20 0 8
5/21/08 3:19:56 PM
