GOVER NMENT & POL ICY
BROADER IMPACTS, TAKE TWO Confusion about NSF’S GRANT REQUIREMENT leads to revised review criterion ANDREA WIDENER, C&EN WASHINGTON
THE NATIONAL SCIENCE Foundation’s
controversial broader impacts criterion is getting an overhaul. The action comes after years of confusion about what is required and complaints that the criterion is inconsistently applied. The revision was prompted by an analysis of grant review criteria that was released earlier this year by the National Science Board (NSB), an oversight body that sets policy for NSF. The board affirmed its support for the two fundamental NSF review criteria—broader impacts and intellectual merit—and made recommendations for how to interpret them. NSF has incorporated the NSB findings into its annual update of its grantee handbook. This year’s changes—the most significant of which is the reworking of broader impacts evaluations—go into effect for grants due on or after Jan. 14, 2013. The biggest change will be for grant reviewers, who will be asked to use one set of five aspirational questions to evaluate both broader impacts and intellectual merit. Grant seekers will also have to change how they write proposals to more clearly delineate their projects’ broader impacts, in response to both the review changes and additional reporting requirements laid out by NSF. Under the new rules, proposals will include a specific section on how a grant meets the broader impacts criterion instead of including this information in the body of the grant as is now done. And successful grantees will have to gauge their progress on their broader impacts goals in annual reports to the agency, something that hasn’t previously been required. “We care that proposals are of the highest quality, that they have a broader impact on societal goals, and that we can assess how effective they are,” says Joanne S.
Tornow, deputy assistant director in NSF’s Social, Behavioral & Economic Sciences Directorate and executive secretary for the NSB task force. NSF is also asking universities to provide more support for the broader impacts criterion. In the past few weeks, NSF has been holding a series of meetings to get the word out to the scientific community about how the changes will impact grantees. “By and large, there has been a positive response,” Tornow says.
awarding grants in the 1950s. It has had few fundamental changes over time. Broader impacts became an explicit part of the merit review picture in 1997. And almost immediately, scientists began complaining that the criterion wasn’t clear. The agency had set out examples of broader impacts, which could be met through the research project itself or by working toward several societal goals defined by the agency, such as improving retention of women and minorities in science. But many scientists weren’t sure whether or how their projects had to meet any or all of those goals. AFTER SEVERAL YEARS, it also became
obvious that different NSF divisions and peer reviewers were applying the broader impacts criterion differently. In 2010, NSB decided to reevaluate its merit review criteria, in large part because of the ongoing confusion about broader impacts. At the time, NSF was also working on a strategic plan for the agency, and
NSB BROADER IMPACTS REVIEW BY THE NUMBERS
NSF stakeholders who provided feedback:
Division evaluations examined:
Past grant proposals evaluated for broader impacts goals (approximate):
4,516 195 150,000 Review criteria confirmed:
2
Changes take effect for grants due on or after:
Last NSB merit review:
Jan. 14, 2013 1997
SOURCE: “Merit Review Criteria: Review & Revisions,” NSB
Seth M. Cohen, a professor and chair of the department of chemistry and biochemistry at the University of California, San Diego, says most of his colleagues understand that broader impacts goals are part of NSF grants and have learned how to work within the system. “I’m not sure if the change is going to make things better or just make people more nervous about it,” he says. Intellectual merit has been at the heart of NSF’s mission—and its grants review process—since the agency first started
“It was time to take a whole-cloth look at the review criteria.” WWW.CEN-ONLIN E .ORG
38
D EC E M BE R 10, 20 1 2
NSB wanted to make sure its review criteria for grants matched the agency’s larger goals. “It was time to take a whole-cloth look at the review criteria and rationalize their descriptions and their use,” explains Alan I. Leshner, chief executive officer of the American Association for the Advancement of Science and cochair of the NSB Task Force on Merit Review. To do that, the task force undertook an extensive evaluation of the intellectual merit and broader impacts criteria and how they were applied throughout NSF. The task force gathered opinions from NSF employees, grantees, and other stakeholders, including the public. It examined
NSF SETS FIVE QUESTIONS FOR ASSESSING GRANTS For the first time, National Science Foundation grant reviewers will be required to ask five fundamental questions when assessing both intellectual merit and broader impacts of proposals. 1. What is the potential for the proposed activity to (a) Advance knowledge and understanding within its own field or across different fields (for intellectual merit)? (b) Benefit society or advance desired societal outcomes (for broader impacts)?
reasoned, well organized, and based on a sound rationale? Does the plan incorporate a mechanism to assess success?
2. To what extent do the proposed activities suggest and explore creative, original, or potentially transformative concepts?
5. Are there adequate resources available to the primary investigator—either at the home institution or through collaborations—to carry out the proposed activities?
3. Is the plan for carrying out the proposed activities well
4. How well qualified is the individual, team, or institution to conduct the proposed activities?
regularly scheduled divisional reviews done by outside evaluators. It hired evaluators to examine thousands of grant applications to see how the broader impacts criterion was being used by scientists. In the middle of the NSB review, Congress passed the America Competes Reauthorization Act, which includes language that requires, among other things, NSF to keep broader impacts and lays out several societal goals that the agency should be striving to achieve through this criterion. Although the NSB task force had considered eliminating the broader impacts criterion altogether, ultimately it decided to keep both criteria—intellectual merit and broader impacts—because they exemplify the goals for the agency, Leshner says. But the task force did aim to clear up confusion about broader impacts by outlining specific questions reviewers should ask—the same questions that they use to evaluate intellectual merit. “Most of the surrounding principles are the same,” Leshner says. “The report speaks to the importance of quality and a commitment to broader impacts.” ALTHOUGH NSF IS CLARIFYING the broader impacts criterion,
Tornow acknowledges that scientists may still be unsure how they should evaluate their work’s broader impacts, which is one of the questions reviewers will use to examine their grants. Tornow suggests that grantees think of evaluating broader impacts in the same way they think of evaluating their own research. For example, NSF is not asking a researcher to know whether the curriculum he or she is developing leads students to win Nobel Prizes 20 years from now, she explained. But there should be some way to conduct a basic evaluation of how the students did. UCSD’s Cohen is an advocate of broader impacts in general, and he has worked extensively on the broader impacts portion of his own NSF grants. But he wonders if the additional evaluation burden will make it too difficult to apply for NSF grants. “If you are WWW.CEN-ONLIN E .ORG
39
D EC E M BE R 10, 20 1 2
GOVER NMENT & POL ICY
NSF C HEM ISTRY
Gervay-Hague Of UC Davis Named New Division Head
going to add a much higher standard for broader impacts and make it a more onerous project, you need to increase the size of the awards,” he says. One solution to make it simpler for scientists to meet the new broader impacts criterion, Cohen suggests, might be to al-
perienced researcher, administrator, and She was particularly intrigued when a professor who has a deep understandfellow UC Davis professor from the mediing of the chemical sciences and their cal school asked for her help creating a impacts on other fields,” says Celeste protein that could serve as a cancer drug. Rohlfing, acting assistant director for “That was an eye-opening experience NSF’s Directorate for Mathfor me. It allowed me to see the ematical & Physical Sciences. power of chemistry in other sci“Her research has crossed ences,” she recalls. “I think that many disciplines, including is already being fostered at NSF.” biology and virology, preparing Although Gervay-Hague has her for the challenges posed been interested in science since by the increasingly multidiscibeing captivated by the first plinary nature of chemistry,” manned moon landing in 1969, Rohlfing notes. she wound her way through CalIn fact, Gervay-Hague seeks fornia community colleges and Gervay-Hague out interdisciplinary work, state universities before landboth in her lab and in academic ing in a sophomore-year organposts. “That’s what excites me,” she says. ic chemistry class at UCLA. She thought “I know my chemistry well, but I get to apshe might want to be a doctor, but that ply it in new settings.” class—taught by chemist and future menOne theme of her research is applying tor Michael E. Jung—changed her mind. synthetic organic chemistry to biological Gervay-Hague went on to get her bachapplications like immunology, virology, elor’s and doctoral degrees in chemistry and nanomaterials. at UCLA. She did postdoctoral research PETER CUT TS P HOTO GRAP HY
Jacquelyn Gervay-Hague, a professor and chair of the department of chemistry at the University of California, Davis, has been chosen as the next director of the National Science Foundation’s Chemistry Division. A synthetic organic chemist, GervayHague comes to NSF after 20 years in academia, at both UC Davis and the University of Arizona. She has received a number of NSF grants throughout her career and served as a grant reviewer for the agency. “I’m excited about the opportunity to work with the national chemistry community to assess the needs and trends in research and education that lead to important scientific discoveries,” GervayHague tells C&EN. Gervay-Hague succeeds Matthew S. Platz, who is leaving this month to become vice chancellor for academic affairs at the University of Hawaii, Hilo. She will begin her appointment in July 2013. “Gervay-Hague stands out as an ex-
low principal investigators (PIs) to tie this piece of their grants to effective, already proven outreach programs. NSF is implementing another change that may also help scientists more easily meet the revised criterion. The change, which is the result of a requirement in the
WWW.CEN-ONLIN E .ORG
40
D EC E M BE R 10, 20 1 2
America Competes Act, directs universities to certify that for any NSF grant their faculty members are applying for, the school will provide adequate support to help the scientists meet stated broader impacts objectives. Universities already have to affirm their commitment to providing laboratory
affairs for the Federation of American Societies for Experimental Biology. In the past, his organization has been ambivalent about the broader impacts criterion. The criterion is important for society, but it is difficult for individual scientists to meet such lofty goals in individual projects, given their short duration and limited amount of funding, he says.
at Yale University, where she first worked at the interface of chemistry and biology. In 1992, Gervay-Hague took a position at the University of Arizona, and she moved to UC Davis in 2001. She is a Fellow of the American Chemical Society and an associate editor of ACS’s Journal of Organic Chemistry. It was actually a stint working as the associate vice provost for outreach and engagement at UC Davis that showed Gervay-Hague the opportunities outside the lab. There, she worked on a health initiative that crossed the main campus as well as medical, veterinary, and law schools and the community. “I really believe in the power of problem solving brought by considering different perspectives and expertise,” she says. Gervay-Hague’s goal now is to start a similar conversation with the chemistry community. “First, listen,” she says, “then understand the context, then work with the leadership to see how we can have the broadest impact.”
support for scientists, such as animal care facilities, electron microscopes, or housekeeping services. AS A RESULT OF THE CHANGE, universi-
ties may need to provide more resources to support broader-impacts-related activities, Tornow says. Those services could include hiring outreach or education staff who can assist in sculpting proposals, making connections in the K–12 community, or even designing projects that scientists could participate in. “I think this clarifies what PIs should be thinking about: Do they have the resources on campus that they need to do their project, or do they need to ask for money to support it?” Tornow says. NSB’s description of broader impacts sets lofty societal goals for NSF, ranging from increasing the public’s science literacy to developing a globally competitive science workforce. But Tornow emphasizes that those goals are for NSF as a whole. No one project is expected to meet all of them. This clear explanation that broader impacts is an agency-wide portfolio objective is particularly good for scientists, says Howard H. Garrison, director of public WWW.CEN-ONLIN E .ORG
41
D EC E M BE R 10, 20 1 2
“The change to measure broader impacts on a portfolio level makes a lot of sense,” he says. “It recognizes the value of those goals without tying it to each individual application.” For now, Tornow is continuing to help scientists understand the changes. “We just have to watch how it plays out once people start submitting proposals.” ◾
