Ind. Eng. Chem. Res. 2010, 49, 9561–9562
9561
Some Observations on the Paper “Optimal Experimental Design for Discriminating Numerous Model Candidates: The AWDC Criterion” Guido Buzzi-Ferraris* CMIC Department “Giulio Natta”, Politecnico di Milano, Piazza Leonardo da Vinci 32, I-20133 Milano, Italy The paper “Optimal Experimental Design for Discriminating Numerous Model Candidates: The AWDC Criterion” by Claas Michalik, Maxim Stuckert, and Wolfgang Marquardt1 proposes a novel criterion to generate experimental points for rival model discrimination. Since this novel criterion is compared to the one I proposed with my co-workers (which I’ll refer to as our criterion in the following), I would have the opportunity to point out some observations. 1. The paper affirms that our criterion is unable to discriminate among more than two competing models. This is not true. Actually, both the original criterion, T, and the modified one, B, that was subsequently proposed are specifically tailored to discriminate among many rival models. It is demonstrated by research papers published to describe our criterion as well as by the subsequent applications. To discriminate among many models, Tij or Bij: Tij ) Bij )
(gi(x) - gj(x))2 2s2 + si2(x) + sj2(x) 2s2 + (gi(x) - gj(x))2 2s2 + si2(x) + sj2(x)
are evaluated for each couple of models, and hence the j or B j are calculated. arithmetic means T 2. The formula adopted in the paper to adapt our criterion for discriminating many models misrepresents the philosophy behind our criterion. • The objective function of our criterion has a relevant statistical meaning that is unavoidably lost in the proposed criterion of this paper: If the function is not sufficiently larger than 1, there is no reasonable evidence to discriminate models for the time being. It may happen either when all proposed models are poor or when the best models are practically equivalent. In this case it is important that new experiments are carried out to improve the parameter estimation or to better explore the experimental domain, otherwise new models should be considered. • Good or bad models are differently weighted as the prediction variance is included in the denominator; since the variance depends on the mean square error of the model, bad models are thus penalized. • Conversely, in the form adopted in the paper, each model is associated to some probabilities. Personally, I was always contrary to attribute models a probability since I think it conceptually wrong and practically dangerous. • It is wrong because there are ever infinite models that evenly simulate reasonably well a finite number of experimental points: a null probability should be therefore attributed to each one of them. • It is dangerous because such a philosophy may lead to the following two issues: if all the selected models are poor, * To whom correspondence should be addressed. Tel.: +39 02 2399 3257. Fax: +39 02 7063 8173. E-mail:
[email protected].
there is the risk to assign a probability 1 to be the true model to the less worse model; if all models are good and not easy to discriminate, there is the risk to select randomly one of them by thinking it the true one and discard all the other models. • Another drawback arising when using probabilities is that the possibility to stop the search of experiment for the model discrimination is lost: the search goes on until one model is better than the rival ones. These problems arise in the case of probability updating by means of both the Bayes theorem and the Akaike weights. • Beyond the problems related to probabilities listed here above, the novel criterion proposed in the paper requires that a good model is included in the set of rival alternative models. What happens when such a model is not within the set of collected models? New experiments are however proposed, but their relevance is doubtful. 3. The example discussed in the paper to highlight the features of the novel criterion and to show its superior performances against our criterion is inadequate. In fact, note the following: • Rival models are proposed without any relation to the physical phenomenon they are considering: 4 of 10 proposed models account for the catalyst concentration dependence, whereas it should be evident to people who carried out the experiments that such a dependence does not exist and it is therefore pointless to take this into consideration. Which is the criterion behind the selection of catalyst? • Some models can be removed without proposing any experiment since they are clearly illogical: actually, these models propose that the reaction A + B f C proceeds even in the absence of reactant A or B: r 1 ) k1 r2 ) k2cA r3 ) k3cB
r4 ) k4ccat r6 ) k6cAccat r7 ) k7cBccat
The remaining models r5 ) k5cAcB r8 ) k8cAcBccat
r9 ) k9cA2cB r10 ) k10cAcB2
are so different from each other that it is not surprising that a single experimental point randomly selected among cA * cB * 5 and with ccat < cA · cB can discriminate them. For example, cA ) 2; cB ) 10; ccat ) 1. • The initial set of experiments consists of a single observation. Even though all models had a single parameter, it would be illogical to analyze them by using a single starting point. It is illogical for different reasons and only the most obvious ones are reported hereinafter. First of all, we are forced to use an exact interpolation, whereas it would be avoided in the presence of experimental error; second, any possibility to check the presence of outliers is anyhow
10.1021/ie100373t 2010 American Chemical Society Published on Web 09/02/2010
9562
Ind. Eng. Chem. Res., Vol. 49, No. 19, 2010
blocked. Such a check is crucial before starting using models to generate additional experiments. Since three independent variables are involved, it should be more appropriate to provide a set of initial experiments consisting at least of four points (more points is preferable). • The range proposed to select the additional experiments: 0 e cA e 10, 0 e cB e 10, and 0 e ccat e 100 is illogical. The choice of an appropriate range is important to select new reasonable experimental points, and it must be consistent with the physical problem. It is pointless to carry out an experiment without A or B. What is the meaning of proposing an experiment when cA ) 0 or cB ) 0 when the selected reaction is A + B f C? It is not a coincidence if the selected experiments pointed out by our criterion appear not so reasonable. • All models contain one single parameter. It should be more appropriate to propose an example involving more parameters, for example involving Hougen-Watson form.2 Such models are harder to discriminate, whereas the proposed example does not show the need to make use of model discrimination because of the relevant differences in the collected models. 4. In addition, the example proposed to highlight the possibility to have the lumping is not realistic since the four models are considered as equivalent for both their mean square
error and the error on their previsions. Nevertheless, the question behind the example is important in order to discriminate among many models: is it suitable to use an experiment that partially discriminates many couples of models, or is it suitable to use an experiment that significantly discriminates a few couples of j or models? It is evident that considering an average value of T j , the latter experiment is preferred for the features of the mean B to be strongly sensitive against large values of its terms. By knowing the value of Tij or Bij for each couple of models and each experimental point, it is easy to obtain the experiment, or better still the experiments, that maximize the amount of couples of models having Tij or Bij larger than an assigned value. On this subject, I inserted in my regression program included in BzzMath3 using even this alternative to support the other already j and B j ), to discriminate among models, considered (included T and to simultaneously improve parameter estimation. Literature Cited (1) Michalik, C.; Stuckert, M.; Marquardt, W. Optimal Experimental Design for Discriminating Numerous Model Candidates.The AWDC Criterion. Ind. Eng. Chem. Res. 2010, 49 (2), 913–919. (2) Buzzi-Ferraris, G.; Manenti, F. Kinetic models analysis. Chem. Eng. Sci. 2009, 64 (5), 1061–1074. (3) Buzzi-Ferraris, G. BzzMath: Numerical library in C++. Politecnico di Milano, www.chem.polimi.it/homes/gbuzzi, 2010.
IE100373T
