government &
policy
BENCHMARKING U.S. RESEARCH
To set parameters for the three benchmarking experiments, COSEPUP looked to its own 1995 report, "Science, Technology, and the Federal Government: National Goals for a New Era." That study suggested two specific goals: first, that the U.S. should be among the world leaders in all major scientific areas, and, second, that it should maintain clear leadership in some areas of science. The benchmarking review panels that the COSEPUP benchmarking group assembled concluded that the U.S. is at least among world leaders in all three fields studied, but there were some important caveats that would likely affect the future leadership status of U.S. research in thesefields.In mathematics, for instance, "there are storm clouds on the horizon" because of thefield'sreliance on foreign mathematicians who have recently immigrated to the U.S., said David R. Challoner, also a member of the benchmarking guidance group. Challoner is director of the Institute for Science & Health Policy at the University of Florida, Gainesville, and foreign secretary of the academies' Institute of Medicine. In materials science and engineering, Challoner continued, difficulty lies in attracting students to the field, "and the U.S. is only among the leaders—but barely." Furthermore, the benchmarking group found that "a general area of U.S. weakness for most subfields [in materials science and engineering] was in materials synthesis and processing. Of particular concern in thisfieldwas the lack of ad-
Experiments to compare research leadership among nations show U.S. may fall behind in math, materials science and engineering William Schulz C&EN Washington nation's or a region's leadership status in research can be compared with world standards of excellence, a National Academies committee has concluded. Using a technique known as international benchmarking, policymakers and federal agencies can reliably determine the leadership status of the U.S. in particularfields,subfields, and, possibly, sub-subfields, says the academies' Committee on Science, Engineering & Public Policy (COSEPUP). A set of recent experiments in benchmarking arranged by the COSEPUP Benchmarking Guidance Group, while validating the technique, also provided a disquieting view of the U.S. ranking for selectedfieldsofresearch: The U.S. is definitely at risk of falling behind other countries in two critical areas—namely, mathematics and materials science and engineering. Asked which was more notable—the utility of international benchmarking or the state of the U.S. basic research enterprise revealed in the COSEPUP group's experiments—one group member replied, T o the extent that this exercise gives us a look at the state [of U.S. research] byfield,it may force consideration of who's in charge of funding decisions and how that responsibility is split up between mission agencies as well as the White House and Congress." International benchmarking is a somewhat loosely defined method or methods that rely primarily on review panels (C&EN, June 28, 1999, page 21). According to COSEPUP, a review panel is composed of researchers qualified to examine a particularfield,and it should be independent, "consisting of researchers who work in afield,individuals who work in closely relatedfields,and research 'users' who follow the field closely. Some of these individuals [approximately onefourth of the total number of panel mem-
A
bers] should be outstanding foreign scientists in thefieldbeing examined." The technique itself "is a way of comparing the quality and impact of research conducted within countries or regions," said Marye Anne Fox, chancellor of North Carolina State University, Raleigh, and chair of the COSEPUP guidance group. She made her remarks at a public briefing on the issue held earlier this month at the academies in Washington, D.C. At the briefing, Fox said a technique such as international benchmarking is necessary "because science and technology are among the most powerful tools in moving nations toward economic and quality-of-life goals." The COSEPUP group, she said, based its conclusions on three experiments in international benchmarking. Three benchmarking panels evaluated U.S. basic research leadership in mathematics, materials science and engineering, and immunology. "For these experiments, the committee deliberately chose tough subjects," Fox said.
Fox and Challoner chat before benchmarking briefing. MARCH 20,2000 C&EN
33
government &
policy
equate funding to modernize major research facilities in the U.S., many of which are much older than those in other countries, and to build the new facilities needed to maintain research leadership." In fact, the materials science and engineering panel sounded a strong note about the state and importance of materials science and engineering in its written report: aTo be leaders in industrial growth and to maintain a vibrant economy, it is critical that the U.S. lead the world in materials science and engineering innovations. Materials have been central to economic growth and societal advancement since the dawn of history." The panelists continued, however, that "U.S. leadership is likely to be eroded in composites, catalysts, polymers, and biomaterials because of the high priorities given to these subfields by other countries. Of particular concern is the catalysts subfield, where there are not enough university multidisciplinary centers to conduct cutting-edge research and reduce the development cycle time for commercialization.,, In sum, the panelists who reviewed materials science and engineering concluded: "The ability of the U.S. to capitalize on its leadership opportunities could be curtailed because of shifting federal and industry priorities, a potential reduction in access to foreign talent, and deteriorating facilities for natural materials characterization." The U.S. is the world leader in immunology, according to the results of that benchmarking experiment Still, the panel wrote, "although U.S. dominance was evident in the major subfields, the U.S. was only among the world leaders in some parts of subfields." The international benchmarking panel for immunology noted two overarching concerns: the increasing cost of maintaining facilities for housing and feeding laboratory mice that are fundamental to research in immunology, and the increasing difficulty in recruiting patients for clinical trials. The immunology panel said the major obstacle to recruiting such patients is health maintenance organizations, which strictly control health care for many U.S. citizens. "The tool works," Challoner said of international benchmarking. Despite initial skepticism noted by the COSEPUP benchmarking group in its report, he described international benchmarking as a rapid and inexpensive way to take a snapshot of a givenfieldof research within a matter of months. About $50,000 was spent on each of the COSEPUP experimental studies. 3 4 MARCH 20,2000 C&EN
An example of international benchmarking International benchmarking as a tool for comparing nations' or regions' status in fields of scientific research seeks to answer three questions about that research. For example, if the field of chemistty were being reviewed by an international benchmarking panel, panelists would be charged with answering the following questions: • What is the position of U.S. research in chemistry relative to that in other regions or countries? • On the basis of current trends in the U.S. and worldwide, what will be the relative position of the U.S. in the near and longer term future? • What are the key factors influencing relative U.S. performance in chemistry? The answers to these questions would be found through a variety of methods. Panelists would set out to identify the world's best researchers in the field or subfield and note what country or region they work in. The panelists would also use citation and journal publications analysis, and would compare the numbers of graduate students, their degrees, and employment status. Also considered would be scientific and technical prizes awarded to researchers in a country or region. A benchmarking study can be completed in a few months, says the National Academies' Committee on Science, Engineering & Public Policy.
Furthermore, Challoner said, results of the group's experiment for mathematics were replicated by a National Science Foundation (NSF) benchmarking study of mathematics. That study was part of NSFs effort to comply with the 1993 Government Performance & Results Act (GPRA) (C&EN, March 22,1999, page 23). In fact, in its report, "Experiments in International Benchmarking of U.S. Research Fields," the COSEPUP guidance group writes that benchmarking "might also help federal agencies to comply with [GPRA] by evaluating the quality of their own performance.,, And the group noted some particular strengths of research benchmarking that emerged during the experiments. Among them were the following: • " Panels were able to identify institutional and human-resource factors crucial to maintaining leadership status in a field that is unlikely to have been identified by other methods."
• "Benchmarking allows a panel to determine the best measures for a particular field while providing corroboration through the use of different methods, as opposed to the One-size-fits-alT approach of some common evaluation methods." • "Benchmarking can produce a timely but broadly accurate 'snapshot* of a field." In addition, the group noted that the benchmarking experiments provide guidance for future research evaluations. For example, it stated that the choice of panelists for international benchmarking panels is key to the benchmarking process. "In particular, it is critical to include non-U.S. participants in the selection of panelists and as panel members because they provide perspective and objectivity." Because major fields of research change slowly, the group suggested that benchmarking can probably only detect important changes in quality, relevance, and leadership in fields when conducted at significant intervals, say of three tofiveyears. Annual benchmarking is not likely to detect changes. Choosing researchfieldsto be evaluated is both challenging and critical, the group writes. It suggests that a "field" is an "array of related domains between which investigators can move without leaving their primary area of expertise." Benchmarking, the group says, "produces information that administrators, policymakers, and funding agencies find useful as they make decisions as to what activities a federal research program should undertake and respond to demands for accountability," such as the demands set forth in GPRA. Twofinalnotes on guidelines by the group suggest that federal agencies, because of their wide variance in missions, must tailor the benchmarking technique to suit their own needs. Along with that, the group suggests that information on degrees of uncertainty and reliability in reporting benchmarking results "might enhance the presentation of panel assessments of leadership status." Nonetheless, at the public briefing and in the group's written report, the benchmarking experiments were characterized "as an encouragingfirststep" toward an effective and reliable evaluation tool. More such experiments, the group continued, "will lead to more effective methods, better understanding of the factors that promote research excellence, and better decision-making by funders of science and technology."^
