Safety assessments and multiplicity adjustment: comments on a recent

which is a compulsory part of the safety assessment according to current EU legislation4. 20 ... 24. Another currently ongoing EU research project is ...
0 downloads 3 Views 144KB Size
Subscriber access provided by UNIV OF DURHAM

Correspondence/Rebuttal

Safety assessments and multiplicity adjustment: comments on a recent paper Hilko Van der Voet J. Agric. Food Chem., Just Accepted Manuscript • DOI: 10.1021/acs.jafc.7b03686 • Publication Date (Web): 18 Feb 2018 Downloaded from http://pubs.acs.org on February 20, 2018

Just Accepted “Just Accepted” manuscripts have been peer-reviewed and accepted for publication. They are posted online prior to technical editing, formatting for publication and author proofing. The American Chemical Society provides “Just Accepted” as a service to the research community to expedite the dissemination of scientific material as soon as possible after acceptance. “Just Accepted” manuscripts appear in full in PDF format accompanied by an HTML abstract. “Just Accepted” manuscripts have been fully peer reviewed, but should not be considered the official version of record. They are citable by the Digital Object Identifier (DOI®). “Just Accepted” is an optional service offered to authors. Therefore, the “Just Accepted” Web site may not include all articles that will be published in the journal. After a manuscript is technically edited and formatted, it will be removed from the “Just Accepted” Web site and published as an ASAP article. Note that technical editing may introduce minor changes to the manuscript text and/or graphics which could affect content, and all legal disclaimers and ethical guidelines that apply to the journal pertain. ACS cannot be held responsible for errors or consequences arising from the use of information contained in these “Just Accepted” manuscripts.

Journal of Agricultural and Food Chemistry is published by the American Chemical Society. 1155 Sixteenth Street N.W., Washington, DC 20036 Published by American Chemical Society. Copyright © American Chemical Society. However, no copyright claim is made to original U.S. Government works, or works produced by employees of any Commonwealth realm Crown government in the course of their duties.

Page 1 of 4

Journal of Agricultural and Food Chemistry

1

Correspondence

2

Safety assessments and multiplicity adjustment: comments on a recent

3

paper

4 5

Hilko van der Voet, Wageningen University & Research

6 7

Many endpoints representing possible non-target effects have to be evaluated in safety

8

assessments that compare new and accepted products. This is known as the multiple-

9

comparison or multiplicity problem. One popular method to adjust statistical testing

10

procedures for multiplicity is the False Discovery Rate (FDR) method1, often implemented

11

via adjustment of p values, as for example provided in SAS procedure MULTTEST. FDR-

12

adjusted p values are obtained by multiplication of the raw p values with factors between

13

1 and , where  is the number of tested hypotheses. Let  ≤ ⋯ ≤  be the ordered p

14

values for  endpoints. Then the FDR-adjusted p values according to a linear step-up

15

algorithm are sequentially calculated as  =  ;  = min   ,    for  =  −

16

1, … ,1.



17 18

Recently, Hong et al.2 published an evaluation of the European Food Safety Authority

19

(EFSA)’s framework for safety assessment of GM crops using a rat 90-day feeding study3

20

which is a compulsory part of the safety assessment according to current EU legislation4.

21

The appropriateness of these animal studies and of the EFSA framework on how to

22

conduct such studies are both under discussion. For example, the EU research project

23

GRACE (http://www.grace-fp7.eu) has performed and evaluated four 90-day and one 1-

24

year study contributing to this discussion (see Schmidt et al.5 and references therein).

25

Another currently ongoing EU research project is G-TwYST (https://www.g-twyst.eu)

26

which is evaluating two 90-day studies and one combined chronic/carcinogenicity (2-

27

year) study. Hong et al.2 also assessed the appropriateness and applicability of the EFSA

28

recommendations using a 90-day study and a battery of statistical approaches including

29

retrospective and prospective power analyses. This short Correspondence is not the place

30

to give a full appraisal of all aspects of this discussion. The discussion here is restricted

31

to just one element of the statistical approach used, which is the treatment of the

32

multiplicity due to many endpoints. Hong et al. evaluated a very large number of

33

endpoints, and adjusted the p values of their tests according to the FDR method. The

34

maximum number of endpoints for each of the sexes was  =146, so FDR-adjusted p

1

ACS Paragon Plus Environment

Journal of Agricultural and Food Chemistry

1

values are between 1 and up to 146 times as large as the raw p valuesa. The main result

2

of Hong et al. regarding the comparisons between test and control groups is that ‘no

3

treatment-related differences were observed’. This can be contrasted with the detailed

4

comparisons in Appendix D of the paper, where 32 out of 816 of the 95% confidence

5

intervals for observed differences do not contain the value 0, and therefore indicate

6

significant differences in an unadjusted test. Note that this rate of significant results

7

(3.9%) is close to the expected rate of 5% false positives that is expected under a null

8

hypothesis of equality for all endpoints, and therefore in itself is not a reason for concern

9

about safety. However, the reported absence of any statistically significant difference

10

should be seen as the direct consequence of using the FDR adjustment. Clearly, with no

11

‘discovery’ at all in this set of results, the false discovery rate is zero by definition, and in

12

this respect the methodology can be said to have operated very effectively. In summary,

13

FDR-adjustment is not a minor detail, but is a main factor that determines the test

14

results.

15 16

I have two serious concerns about the methodology in this paper.

17 18

First, the use of standard FDR-correction or any other multiple-testing scheme makes no

19

sense in food safety testing. It controls false discoveries, and is therefore connected to

20

difference testing, where the null hypothesis is equality of means and false positives are

21

considered the error of the first kind: you want to have a small probability of erroneously

22

reporting a difference. This is useful in studies that set out to find differences between

23

groups, perhaps to find new explanations for biological phenomena or effective

24

treatments. However, in the context of safety or equivalence testing, the purpose is to

25

demonstrate safety with a chosen confidence level. Therefore the statistical hypotheses

26

are reversed: the null hypothesis is that some difference exists and we want to show

27

equivalence by rejecting such a null hypothesis (some possible approaches allowing for

28

endpoints with widely different variation have been described6,7,8,9). In equivalence

29

testing, the errors of the first kind are the false negatives rather than the false positives,

30

to guarantee a small probability of erroneously reporting equivalence. Consequently, the

31

commonly used methods for multiplicity correction including FDR are addressing the

32

wrong type of error, and should not be used in safety assessments.

33 34

Secondly, contrary to the tests used to report results in Hong et al., the statistical power

35

analyses in the same paper (both prospective and retrospective) do not use FDR

36

adjustments. Therefore, the results of these power analyses cannot be interpreted as

a

FDR adjustment was performed for the set of all endpoints that were reported across-sex, and separately for the male or female-specific comparisons, so  may have been lower in practice, but exact values are not given.

2

ACS Paragon Plus Environment

Page 2 of 4

Page 3 of 4

Journal of Agricultural and Food Chemistry

1

statements about the statistical power obtained using the FDR-adjusted tests. Clearly,

2

the statistical power for any endpoint separately is much lower than stated (because the

3

p values are adjusted upward). The potential danger of the paper is the message that its

4

approach would be an appropriate procedure, because 1) a high power of the difference

5

tests for the proposed effect sizes seems to be attained, and at the same time: 2) not a

6

single statistically significant difference is obtained. But the statistical approaches

7

followed for 1 and 2 (without and with FDR correction, respectively) are inconsistent. It is

8

misleading to present FDR-adjusted test results together with power analyses which do

9

not incorporate these adjustments.

10 11

As an additional point, Hong et al. also claim that FDR adjustments would be endorsed by

12

EFSA. Whereas EFSA in its guidance3 did acknowledge the multiplicity problem (‘the issue

13

of multiple testing […] should be addressed’), they have however not given an

14

endorsement of FDR adjustment. Instead, EFSA3 leaves the matter to the statistical

15

analyst (‘Any methods used to adjust for multiplicity should also be clearly documented

16

and referenced’). EFSA10 already concluded on this: ‘FDR as usually applied (i.e. in a

17

context of difference testing) is a property of the subset of endpoints for which a

18

significant difference has been found. It does not address the endpoints for which no

19

significance has been found and therefore FDR applied to difference testing does not

20

seem sufficient as a measure in GMO risk assessment. It could be of interest to adapt the

21

FDR concept for equivalence testing, i.e. for a situation where hypotheses are reversed,

22

but the GMO Panel is not aware that this has yet been done.’ By now alternative

23

methods for multiple or multivariate equivalence testing for safety evaluations have been

24

proposed11,12,13,14, which are currently under debate.

25 26 27 28 29

References 1. Benjamini, Y.; Hochberg, Y. Controlling the false discovery rate: a practical and powerful approach to multiple testing. J. Roy. Statist. Soc. B 1995, 57, 289–300. 2. Hong, B.; Du, Y.Z.; Mukerji, P.; Roper, J.M.; Appenzeller, L.M. Safety assessment of food

30

and feed from GM crops in Europe: Evaluating EFSA’s alternative framework for the rat 90-

31

day feeding study. J. Agric. Food Chem. 2017, 65, 5545-5560.

32 33 34

3. EFSA. Guidance on conducting repeated-dose 90-day oral toxicity study in rodents on whole food/feed. EFSA Journal 2011, 9, 2438. 4. EC Commission. Implementing Regulation (EU) No 503/2013 of 3 April 2013 on

35

applications for authorisation of GM food and feed in accordance with Regulation (EC) No

36

1829/2003 of the European Parliament and of the Council and amending Commission

37

Regulations (EC) No 641/2004 and (EC) No 1981/2006. Off. J. Eur. Communities 2013,

38

L157, 1−52.

39 40

5. Schmidt, K.; Schmidtke, J.; Schmidt, P.; Kohl, C.; Wilhelm, R.; Schiemann, J.; van der Voet, H.; Steinberg, P. Variability of control data and relevance of observed group 3

ACS Paragon Plus Environment

Journal of Agricultural and Food Chemistry

1

differences in five oral toxicity studies with genetically modified maize MON810 in rats.

2

Arch. Toxicol. 2017, 91: 1977-2006. https://dx.doi.org/10.1007/s00204-016-1857-x

3

6. van der Voet, H.; Perry, J.N.; Amzal, B.; Paoletti, C. A statistical assessment of differences

4

and equivalences between genetically modified and reference plant varieties. BMC

5

Biotechnol. 2011, 11: 15.

6

7. Meyners, M. Equivalence tests – a review. Food Qual. Prefer. 2012, 26, 231-245.

7

8. Kang, Q.; Vahl, C.I. Statistical analysis in the safety evaluation of genetically-modified

8 9

crops: equivalence tests. Crop Sci. 2014, 54, 2183-2200. 9. van der Voet, H.; Goedhart, P.W.; Schmidt, K. Equivalence testing using existing reference

10

data: an example with genetically modified and conventional crops in animal feeding

11

studies. Food and Chemical Toxicology 2017, 109: 472-485.

12

https://doi.org/10.1016/j.fct.2017.09.044

13 14 15 16 17

10. EFSA. Statistical considerations for the safety evaluation of GMOs. EFSA Journal 2010, 8, 1250. 11. Qiu, J.; Cui, X.Q. Evaluation of a statistical equivalence test applied to microarray data. J Biopharm. Statist. 2010, 20: 240-266. 12. van Dijk, J.P.; Souza de Mello, C.; Voorhuijzen, M.M.; Hutten, R.C.B.; Maisonnave Arisi,

18

A.C.; Jansen, J.J.; Buydens, L.M.C.; van der Voet, H.; Kok, E.J. Safety assessment of plant

19

varieties using transcriptomics profiling and a one-class classifier. Regulatory Toxicology

20 21 22 23 24

and Pharmacology 2014, 70: 297-303. http://dx.doi.org/10.1016/j.yrtph.2014.07.013. 13. Pallmann, P.; Jaki, T. Simultaneous confidence regions for multivariate bioequivalence. Statistics in Medicine 2017, 36: 4585-4603. 14. Vahl, C.I.; Kang, Q. Statistical strategies for multiple testing in the safety evaluation of a genetically modified crop. Journal of Agricultural Science. 2017, 155, 812-831.

4

ACS Paragon Plus Environment

Page 4 of 4