GOVERNMENT & POLICY
ASSESSING DATA FOR QUALITY Workshop probes factors that EPA will consider when using published studies CHERYL H0GUE, C&EN WASHINGTON
T
HE ENVIRONMENTAL PROTEC-
tion Agency depends on scientific research from a variety of sources as it assesses the risks posed by chemicals and sets limits on pollutants. EPA scientists conduct studies. The agency receives data from researchers working under EPA grants. But most of the scientific information EPA uses to support risk assessments and policy choices that often result in regulations comes from studies published in peer-reviewed scientific journals. "The agency does not rely solely on its own informationforits decisionmaking," says Paul Gilman, EPA assistant administrator for research and development and science adviser to agencyAdministrator Christine Todd Whitman. "We rely heavily on the peer-reviewed literature," he tells C&EN. Building on the new U.S. law concerning information quality, EPA is the first federal agency attempting to lay out criteria by which it will assess the quality of so-called thirdparty data, Gilman says. The agency in 2002 drafted a guidance document describing the factors it uses to scrutinize third-party data for measures of quality. Among EPAs draft assessment factors are soundness of the data; the extent to which the information is applicable and appropriate for the agency's use; and how clear and complete scientific studies are on documentation of data, assumptions, methods, quality controls, and analysis. According to EPA, the assessment feetors document is "a way to communicate to parties who publish {in} the open liter-
ature and/or wish to submit information voluntarily to the agency in hopes of influencing an agency action." At the agency's request, the National Academy of Sciences held a workshop on the draft document onJan. 21. The workshop, held by the NAS Committee on EPA
view at the White House Office of Management & Budget (OMB). In contrast, the draft assessment factors paper is an internal document that will apply to data that EPA gleans from peerreviewed, published studies and other research submissionsforuse in risk assessments or to support pollution control standards. The agency says the document is "intended to encourage third parties to consider quality issues while the information is btmggmemted.wThe document will not need OMB approval and will be updated regularly
THE ASSESSMENT factors will describe how EPA scientists evaluate published studies. However, Gilman said they will not be prescriptive and will allowforprofessional judgment. NAS committee member Ellen Paul, executive director ofthe Chevy Chase, Md.-based Ornithological Council, an association of 10 scientific societies, said it is important for EPA to communicate how it evaluates the studies it uses in risk and regulatory analyses. Halûk Ôzkaynak is cochair of the EPA working group that is crafting the assessment factors document He told the workshop that the draft guidance describes how the agency determines that the procedures, measures, methods, or models used in a study are scientifically reasonable and consistent with EPAs intended application of the resulting data. The document would not apply to the results ofresearch that EPA conducts through its own contracts, grants, cooperative agreements, and interagency agreements. EPA exercises a great deal of control over the Assessment FactorsforData Quality, was quality of this research as it is generated, designed to give general feedback to the Ozkaynak said, and thus these data do not agency and will not result in formal rec- have to go through a formal assessment. In addition, the assessment factors will not ommendations or a report. Gilman told the workshop that the as- apply to information submitted to EPAas sessment factors document is separate required by statute, regulation, pollution from EPAs new information quality guide- control permit, or court order, Ozkaynak lines (C&EN, Oct. 7, 2002, page 22). said. Those guidelines are directed only at inTerry F. Tfosie, American Chemistry formation the agency disseminates to the Council vice president for Responsible public. The guidelines had to undergo re- Care, said EPA should use the same fee-
The assessment factors document will apply to data that EPA gleans from peer-reviewed, published studies for use in risk assessments or to support pollution control standards. HTTP://WWW.CEN-ONLINE.ORG
C & E N / FEBRUARY 10, 2003
21
GOVERNMENT & POLICY tors in assessing information regardless of the data's source. However, the agency should not necessarily conduct the same level of scrutiny of data for all types of decisions, said Yosie, a member of the NAS committee. EPA should determine
chemical engineering at the University of Texas, Austin, said, "As a scientist, I want to produce good-quality data." "That's music to our ears," Gilman tells C&EN. The workshop touched on situations where available data are not ideal for EPAs purposes. Scientists and policymakers often have to deal with data that are not perfectly suited for their needs, said committee member David Korn, senior vice president of the Association of American Medical Colleges' Division of Biomedical & Health Sciences Research. He expressed concern that ifthe federal government estabGilman Paul lishes "absolute" standards which kinds of policy choices the data for information quality, public health proassessment factors should apply to, he tection agencies such as EPA would be stopped from regulating because they said. Workshop panelist Virginia Ashby would be severely limited in the data they Sharpe, who directs the Integrity in Sci- could use. ence project at the Washington, D.C.Panelist Robert M. O'Keefe, vice presbased advocacy group Center for Science ident of the Health Effects Institute, an in the Public Interest, said the assessment independent research orfactors could limit the influence of those ganization in Boston fundwho voluntarily submit data to EPA in ed jointly by EPA and hopes of swaying a policy decision. Paul industry, said the environnoted that lobbying groups often want mental agency must idenEPA to include certain scientific informa- tify the limits and flaws of tion in its analyses—or to exclude some the data it uses for regulastudies. The assessment factors document tion and risk assessment. should assure industry, environmentalists, Knowledgeable scientists and others with a stake in a regulatory out- will have to determine come that the agency is following a fair and whether data are usable for open process that is used consistently for EPAs purposes, said comanalyzing scientific data, Paul said. mittee Chairman John C. Bailar III, a statistician SEVERAL PANELISTS at the workshop who is professor emeritus Yosie expressed doubt that EPAs assessment fac- in health studies at the tors document would have any effect on University of Chicago. researchers. Edmund A. C. Crouch, senGlenJ. Barrett, a senior health scientist ior scientist at Cambridge Environmen- with the American Petroleum Institute, tal, a Cambridge, Mass., consulting firm, provided comments to the NAS worksaid EPAs assessment factors guidance will shop. He suggested that EPA use a classinot change the behavior of scientists. The fication system that would rank studies document lays out procedures that "every- from excellent to poor or set up a matrix body in science tries to do," Crouch said. to grade the quality of data against the inNu-May Ruby Reed of the California En- tended use of the information. vironmental Protection Agency agreed Lawyers would use such a quantitative that most scientists are already familiar ranking of studies to challenge EPAs sciwith the criteria that the federal agency is entific assessments asflawed,warned pandrafting into a formal document. Dennis elist 'Wendy E. Wagner, a professor at the J. Paustenbach, a toxicologist and corpo- University of Texas, Austin, School of Law rate vice president for consultingfirmEx- The more prescriptive EPAs assessment ponent, said EPAs assessment factors "are factors are, the greater the chances they a reiteration ofwhat we do to publish pa- will be used in litigation, she said. Wagner pers." And David T. Allen, professor of recommended that EPA scrap the draft 22
C & E N / FEBRUARY 1 0 . 2 0 0 3
assessment factors document. The agency should conduct its assessments of data quality on a case-by-case basis, she said. Paul said even if the assessment factors were used as a basis for litigation against an EPA scientific assessment, federal courts would defer to the agency's judgment as long as it did not act arbitrarily or capriciously. Committee members and panelists offered their own suggestions for changes to the draft assessment factors document. Sharpe recommended that assessment factors include disclosure of funding sources for research. Dorothy E. Patton, a former EPA scientist who is an adjunct professor at Georgetown University said EPAshould describe who at the agency evaluates scientific data—not by name but byfieldof expertise and experience. Bailar suggested the agency develop assessment factors it will consider when evaluating a whole body of evidence and not just individual studies. In addition, EPA needs to explain its analyses ofquality clearly and concisely he added, warning against descriptions that are too long. He also recommended that the assessment factors document apply to studies used to delay
Sharpe
or reject the need for environmental regulations and not merely to those used to support new rules. Ozkaynak tells C&EN that EPA will revise the draft assessment factors in light of comments made at the NAS workshop and those submitted by the public. The revision is expected to be completed by midyear, Ozkaynak says. EPAs draft "Assessment Factors for Evaluating the Quality of Information from External Sources" is available at http:/Avww.epa.gov/oei/qualityguidelines/ afLassessdraft.pdf, and the agency's response to public comments on the document are at http://www.epa.gpv/oei/quali tyguidelines/AssessmentFactors-Clarifi cationOfIssues01-10-03a.pd£ • HTTP://WWW.CEN-ONLINE.ORG
